{"thread":{"id":"66136","subject":"[PATCH 0/5] odb: make packfile generation pluggable","startedAt":"2026-08-07T10:45:42Z","lastAt":"2026-08-21T12:38:15Z","messageCount":58,"participants":["Patrick Steinhardt","Junio C Hamano","Taylor Blau","Elijah Newren","Justin Tobler","Karthik Nayak"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"549971","messageId":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","threadId":"66136","inReplyTo":null,"subject":"[PATCH 0/5] odb: make packfile generation pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-07T10:45:06Z","receivedAt":"2026-08-07T10:45:42Z","isPatch":true,"body":"Hi,\n\nthis patch series makes packfile generation pluggable.\n\nNote that this series only makes those parts pluggable that are required\nfor the transport layer. The other parts that relate to packfile\ngeneration as required by our repository maintenance is kept as-is, as\nthere is a bunch of options there that are way too specific to the\n\"files\" backend to be portable. This should ultimately not be much of a\nproblem though, as maintenance itself is already pluggable in the first\nplace.\n\nIt's a bit of a shame though for git-pack-objects(1), which still isn't\nusable with alternate backends. I tried several times to find good\nsolutions for making it fully pluggable, but due to the backend-specific\noptions it's an utter mess. I want to eventually address this though:\nsame as with git-refs(1), I want to introduce git-objects(1) to care\nabout all things ODB. And as part of that command we can also introduce\na command that generates packfiles in a generic fashion, without all the\ncruft that git-pack-objects(1) has. This is part of a future patch\nseries though.\n\nThe series is built on top of 2c78326f81 (The 11th batch, 2026-08-05).\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (5):\n odb: introduce interface to generate packfiles\n upload-pack: generate packfiles via the object database\n send-pack: generate packfiles via the object database\n builtin/bundle: refactor option handling for progress meter\n bundle: generate packfiles via the object database\n\n builtin/bundle.c | 31 ++++------\n bundle.c | 68 +++++++++++-----------\n bundle.h | 3 +-\n odb.c | 21 +++++++\n odb.h | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-files.c | 144 +++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h | 33 +++++++++++\n send-pack.c | 101 +++++++++++----------------------\n t/t5516-fetch-push.sh | 12 ++--\n upload-pack.c | 125 +++++++++++++++--------------------------\n 10 files changed, 482 insertions(+), 208 deletions(-)\n\n\n---\nbase-commit: 2c78326f810173a4f3aefd8021f1e07575412481\nchange-id: 20260807-b4-pks-odb-generate-pack-f30fbcdef3fc\n\n"},{"id":"549972","messageId":"20260807-b4-pks-odb-generate-pack-v1-1-7dec431ae7cd@pks.im","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","subject":"[PATCH 1/5] odb: introduce interface to generate packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-07T10:45:07Z","receivedAt":"2026-08-07T10:45:43Z","isPatch":true,"body":"Packfiles have two primary use cases:\n\n - They are used to store objects at rest in a Git repository.\n\n - They are used on the transport layer to transfer objects between two\n repositories.\n\nThe first class is closely tied to a given object database backend, and\nas such this use is highly specific to how such a backend decides to\nstore its data. This shows in git-pack-objects(1), which is used by\ngit-repack(1) et al to optimize the object database, which supports lots\nof options that are closely coupled with how data is stored.\n\nBut the second class is quite a lot more generic: we don't care about\nspecifics of how the object database stores its objects, but to generate\nthe packfiles we only care about the object graph itself. Still, this\nuse case is also coupled with git-pack-objects(1).\n\nUnfortunately, because git-pack-objects(1) covers both classes, the\nresult is that it is very hard to port the whole command to properly\nsupport pluggable object databases. There are simply way too many\noptions that an alternative implementation will have a very hard time to\nsupport in the first place.\n\nAnd despite being hard to implement, it's also quite unnecessary to\nimplement those backend-specific options. Optimizing the object database\nhas already been made pluggable, and an alternative implementation is\nunlikely to care about cruft packs, unpacked objects, keep packs and the\nlike. But we still need to make at least _parts_ of the packfile\ngeneration pluggable so that backends can generate packfiles for the\ntransport layer itself.\n\nIntroduce a new interface that lets backends generate a new packfile and\nimplement that interface for the \"files\" backend. The options supported\nby the callback are exactly the set of options that are required for the\ntransport layer, but nothing more.\n\nThis means that git-pack-objects(1) itself cannot be ported over to this\nnew interface, but as explained above that's a hard feat to pull off due\nto the backend-specific features. Ideally though, we should expose the\nability to generate arbitrary packfiles using this interface. The intent\nof this is to eventually introduce a git-objects(1) subcommand (similar\nto git-refs(1)) that exposes generic interfaces for accessing everything\nrelated to the object database. In that case, we are able to expose only\nthose options that are generic.\n\nSubsequent commits will convert git-upload-pack(1), git-send-pack(1) and\ngit-bundle(1) to use this interface.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 21 ++++++++\n odb.h | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-files.c | 144 ++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h | 33 ++++++++++++\n 4 files changed, 350 insertions(+)\n\ndiff --git a/odb.c b/odb.c\nindex caf1d0f542..cd9d5b48bc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1046,6 +1046,27 @@ bool odb_optimize_required(struct object_database *odb,\n \treturn odb_source_optimize_required(odb->sources, opts);\n }\n \n+void odb_generate_pack_options_release(struct odb_generate_pack_options *opts)\n+{\n+\toid_array_clear(&opts->wants);\n+\toid_array_clear(&opts->haves);\n+\toid_array_clear(&opts->shallows);\n+}\n+\n+int odb_generate_pack(struct object_database *odb,\n+\t\t struct odb_pack_generator **out,\n+\t\t const struct odb_generate_pack_options *opts)\n+{\n+\tif (!odb->sources->generate_pack)\n+\t\treturn error(_(\"primary object source does not support generating packfiles\"));\n+\treturn odb_source_generate_pack(odb->sources, out, opts);\n+}\n+\n+int odb_pack_generator_finish(struct odb_pack_generator *generator)\n+{\n+\treturn generator->finish(generator);\n+}\n+\n struct object_database *odb_new(struct repository *repo,\n \t\t\t\tconst char *primary_source,\n \t\t\t\tconst char *secondary_sources)\ndiff --git a/odb.h b/odb.h\nindex fca67e8253..fc1442f243 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -2,6 +2,7 @@\n #define ODB_H\n \n #include \"object.h\"\n+#include \"oid-array.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -677,6 +678,157 @@ int odb_write_object_stream(struct object_database *odb,\n \t\t\t struct odb_write_stream *stream, size_t len,\n \t\t\t struct object_id *oid);\n \n+/*\n+ * Options for generating a packfile via `odb_generate_pack()`.\n+ */\n+struct odb_generate_pack_options {\n+\t/* Tips of the object graph that shall be packed. */\n+\tstruct oid_array wants;\n+\n+\t/*\n+\t * Boundary of the object graph. Objects reachable from any of these\n+\t * tips are expected to already be available to whoever consumes the\n+\t * pack and shall thus not be packed.\n+\t */\n+\tstruct oid_array haves;\n+\n+\t/*\n+\t * The shallow boundary that shall be used when computing object\n+\t * reachability. When set, any shallow information of the repository\n+\t * itself shall be ignored in favor of these objects.\n+\t */\n+\tstruct oid_array shallows;\n+\n+\t/*\n+\t * Pre-expanded object filter specification that limits the set of\n+\t * objects that shall be packed. May be `NULL` in case no filter shall\n+\t * be applied.\n+\t */\n+\tconst char *filter_spec;\n+\n+\t/*\n+\t * Protocols that may be used to offload objects via packfile URIs.\n+\t * May be `NULL` in case packfile URIs shall not be used.\n+\t */\n+\tconst struct string_list *uri_protocols;\n+\n+\t/*\n+\t * Hook command that shall be executed instead of the internal\n+\t * machinery to generate the pack. It is up to the specific backend\n+\t * whether or not this hook is supported. May be `NULL` in case no\n+\t * hook shall be executed.\n+\t */\n+\tconst char *pack_objects_hook;\n+\n+\t/*\n+\t * File descriptor that the generated pack shall be written to. If set\n+\t * to `-1`, a pipe will be created and exposed via the pack generator's\n+\t * `out` field. If set to `0`, the pack will be written to the standard\n+\t * output stream. Otherwise, the provided descriptor will be written to\n+\t * and is consumed by the generator.\n+\t */\n+\tint pack_fd;\n+\n+\t/*\n+\t * File descriptor that progress output shall be written to. The same\n+\t * semantics as for `pack_fd` apply, except that `0` will cause the\n+\t * generator to write to stderr instead of stdout.\n+\t */\n+\tint progress_fd;\n+\n+\t/* Whether to print progress or not. */\n+\tenum {\n+\t\t/* Don't print progress output. */\n+\t\tODB_GENERATE_PACK_PROGRESS_NONE,\n+\n+\t\t/*\n+\t\t * Print progress while computing the packfile, but stop\n+\t\t * printing progress once starting to write it.\n+\t\t */\n+\t\tODB_GENERATE_PACK_PROGRESS_STANDARD,\n+\n+\t\t/*\n+\t\t * Similar to STANDARD, but also print progress when writing\n+\t\t * the packfile.\n+\t\t */\n+\t\tODB_GENERATE_PACK_PROGRESS_VERBOSE,\n+\t} progress;\n+\n+\t/* Allow the pack to contain deltas against unpacked objects. */\n+\tunsigned thin:1;\n+\n+\t/* Use offset deltas instead of reference deltas. */\n+\tunsigned ofs_delta:1;\n+\n+\t/* Include unasked-for annotated tags of packed objects. */\n+\tunsigned include_tag:1;\n+\n+\t/* The generated pack is destined for a shallow consumer. */\n+\tunsigned shallow:1;\n+\n+\t/* Allow objects that may be missing due to a promisor remote. */\n+\tunsigned missing_allow_promisor:1;\n+\n+\t/* Do not use bitmap indices when computing reachability. */\n+\tunsigned disable_bitmaps:1;\n+};\n+\n+#define ODB_GENERATE_PACK_OPTIONS_INIT { \\\n+\t.wants = OID_ARRAY_INIT, \\\n+\t.haves = OID_ARRAY_INIT, \\\n+\t.shallows = OID_ARRAY_INIT, \\\n+\t.pack_fd = -1, \\\n+}\n+\n+/* Release resources associated with the options. */\n+void odb_generate_pack_options_release(struct odb_generate_pack_options *opts);\n+\n+/*\n+ * A handle for an ongoing packfile generation as started via\n+ * `odb_generate_pack()`.\n+ */\n+struct odb_pack_generator {\n+\t/*\n+\t * File descriptor from which the generated pack can be read. Only set\n+\t * when the pack generation was started with `pack_fd == -1`. The\n+\t * caller is responsible for closing the descriptor.\n+\t */\n+\tint out;\n+\n+\t/*\n+\t * File descriptor from which progress output can be read. Only set\n+\t * when the pack generation was started with `progress_fd == -1`. The\n+\t * caller is responsible for closing the descriptor.\n+\t */\n+\tint err;\n+\n+\t/*\n+\t * Callback function to finish this generator. This callback is\n+\t * expected to wait for the packfile generation to complete and to then\n+\t * free the generator itself.\n+\t */\n+\tint (*finish)(struct odb_pack_generator *);\n+};\n+\n+/*\n+ * Start generating a packfile from the object database with the given\n+ * options. The pack is generated asynchronously; the caller is expected to\n+ * consume the file descriptors exposed via the pack generator and to then\n+ * wait for completion via `odb_pack_generator_finish()`.\n+ *\n+ * Returns 0 on success and populates the `out` pointer with the pack\n+ * generator. Returns a negative error code otherwise.\n+ */\n+int odb_generate_pack(struct object_database *odb,\n+\t\t struct odb_pack_generator **out,\n+\t\t const struct odb_generate_pack_options *opts);\n+\n+/*\n+ * Wait for the packfile generation to complete and free the pack generator.\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int odb_pack_generator_finish(struct odb_pack_generator *generator);\n+\n void parse_alternates(const char *string,\n \t\t int sep,\n \t\t const char *relative_base,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5a68af7d84..64a0417be7 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -4,6 +4,7 @@\n #include \"chdir-notify.h\"\n #include \"config.h\"\n #include \"gettext.h\"\n+#include \"hex.h\"\n #include \"lockfile.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n@@ -729,6 +730,148 @@ int odb_source_files_optimize(struct odb_source *source,\n \treturn ret;\n }\n \n+struct odb_pack_generator_files {\n+\tstruct odb_pack_generator base;\n+\tstruct child_process cp;\n+};\n+\n+static int odb_pack_generator_files_finish(struct odb_pack_generator *_generator)\n+{\n+\tstruct odb_pack_generator_files *generator =\n+\t\t(struct odb_pack_generator_files *)_generator;\n+\tint ret;\n+\n+\tret = finish_command(&generator->cp);\n+\tfree(generator);\n+\n+\tif (ret) {\n+\t\t/*\n+\t\t * On failure, pack-objects is expected to have written a\n+\t\t * useful error message to its standard error stream already.\n+\t\t * Death by signal is worth mentioning, though, with the\n+\t\t * exception of SIGPIPE: that is a normal occurrence when the\n+\t\t * consumer of the pack hangs up.\n+\t\t */\n+\t\tif (ret > 128 && ret - 128 == SIGPIPE)\n+\t\t\treturn -1;\n+\t\tif (ret > 128)\n+\t\t\terror(_(\"pack-objects died of signal %d\"), ret - 128);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_source_files_generate_pack(struct odb_source *source UNUSED,\n+\t\t\t\t\t struct odb_pack_generator **out,\n+\t\t\t\t\t const struct odb_generate_pack_options *opts)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tstruct odb_pack_generator_files *generator;\n+\tFILE *in;\n+\n+\t/*\n+\t * The hook is expected to spawn \"$hook git pack-objects \"\n+\t * and to behave like git-pack-objects(1) would have. This can for\n+\t * example be used to serve precomputed packfiles.\n+\t */\n+\tif (opts->pack_objects_hook) {\n+\t\tstrvec_push(&cp.args, opts->pack_objects_hook);\n+\t\tstrvec_push(&cp.args, \"git\");\n+\t\tcp.use_shell = 1;\n+\t} else {\n+\t\tcp.git_cmd = 1;\n+\t}\n+\n+\t/*\n+\t * The caller-provided shallow boundary overrides any shallow state\n+\t * that the repository itself may have, so the shallow file needs to\n+\t * be neutralized.\n+\t */\n+\tif (opts->shallows.nr) {\n+\t\tstrvec_push(&cp.args, \"--shallow-file\");\n+\t\tstrvec_push(&cp.args, \"\");\n+\t}\n+\tstrvec_push(&cp.args, \"pack-objects\");\n+\tstrvec_push(&cp.args, \"--revs\");\n+\tstrvec_push(&cp.args, \"--stdout\");\n+\tif (opts->thin)\n+\t\tstrvec_push(&cp.args, \"--thin\");\n+\tif (opts->shallow)\n+\t\tstrvec_push(&cp.args, \"--shallow\");\n+\tif (opts->ofs_delta)\n+\t\tstrvec_push(&cp.args, \"--delta-base-offset\");\n+\tif (opts->include_tag)\n+\t\tstrvec_push(&cp.args, \"--include-tag\");\n+\tif (opts->missing_allow_promisor)\n+\t\tstrvec_push(&cp.args, \"--missing=allow-promisor\");\n+\tif (opts->disable_bitmaps)\n+\t\tstrvec_push(&cp.args, \"--no-use-bitmap-index\");\n+\tswitch (opts->progress) {\n+\tcase ODB_GENERATE_PACK_PROGRESS_NONE:\n+\t\tstrvec_push(&cp.args, \"--quiet\");\n+\t\tbreak;\n+\tcase ODB_GENERATE_PACK_PROGRESS_STANDARD:\n+\t\tstrvec_push(&cp.args, \"--progress\");\n+\t\tbreak;\n+\tcase ODB_GENERATE_PACK_PROGRESS_VERBOSE:\n+\t\tstrvec_push(&cp.args, \"--all-progress\");\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown progress option %d\", opts->progress);\n+\t}\n+\tif (opts->filter_spec)\n+\t\tstrvec_pushf(&cp.args, \"--filter=%s\", opts->filter_spec);\n+\tif (opts->uri_protocols)\n+\t\tfor (size_t i = 0; i < opts->uri_protocols->nr; i++)\n+\t\t\tstrvec_pushf(&cp.args, \"--uri-protocol=%s\",\n+\t\t\t\t opts->uri_protocols->items[i].string);\n+\n+\tcp.in = -1;\n+\tcp.out = opts->pack_fd;\n+\tcp.err = opts->progress_fd;\n+\tcp.clean_on_exit = 1;\n+\n+\tif (start_command(&cp))\n+\t\treturn error(_(\"could not spawn pack-objects\"));\n+\n+\t/*\n+\t * Feed the objects to pack-objects. This is safe to do synchronously\n+\t * because pack-objects consumes all of its standard input before it\n+\t * starts to generate the pack.\n+\t */\n+\tin = xfdopen(cp.in, \"w\");\n+\tfor (size_t i = 0; i < opts->shallows.nr; i++)\n+\t\tfprintf(in, \"--shallow %s\\n\", oid_to_hex(&opts->shallows.oid[i]));\n+\tfor (size_t i = 0; i < opts->wants.nr; i++)\n+\t\tfprintf(in, \"%s\\n\", oid_to_hex(&opts->wants.oid[i]));\n+\tfprintf(in, \"--not\\n\");\n+\tfor (size_t i = 0; i < opts->haves.nr; i++)\n+\t\tfprintf(in, \"%s\\n\", oid_to_hex(&opts->haves.oid[i]));\n+\tfprintf(in, \"\\n\");\n+\tfflush(in);\n+\tif (ferror(in)) {\n+\t\terror(_(\"error writing to pack-objects\"));\n+\t\tfclose(in);\n+\t\tif (opts->pack_fd < 0)\n+\t\t\tclose(cp.out);\n+\t\tif (opts->progress_fd < 0)\n+\t\t\tclose(cp.err);\n+\t\tfinish_command(&cp);\n+\t\treturn -1;\n+\t}\n+\tfclose(in);\n+\n+\tCALLOC_ARRAY(generator, 1);\n+\tgenerator->base.out = opts->pack_fd < 0 ? cp.out : -1;\n+\tgenerator->base.err = opts->progress_fd < 0 ? cp.err : -1;\n+\tgenerator->base.finish = odb_pack_generator_files_finish;\n+\tgenerator->cp = cp;\n+\n+\t*out = &generator->base;\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t bool local)\n@@ -756,6 +899,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.write_alternate = odb_source_files_write_alternate;\n \tfiles->base.optimize = odb_source_files_optimize;\n \tfiles->base.optimize_required = odb_source_files_optimize_required;\n+\tfiles->base.generate_pack = odb_source_files_generate_pack;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex d69f8e2d1c..e2129766fc 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -278,6 +278,23 @@ struct odb_source {\n \t */\n \tbool (*optimize_required)(struct odb_source *source,\n \t\t\t\t const struct odb_optimize_options *opts);\n+\n+\t/*\n+\t * This callback is expected to start generating a packfile with the\n+\t * given options. The pack shall be generated asynchronously so that\n+\t * the caller can consume the pack data and progress output while the\n+\t * pack is being generated.\n+\t *\n+\t * This callback is optional. Sources that cannot generate packfiles\n+\t * shall leave it unset.\n+\t *\n+\t * The callback is expected to return 0 on success and populate the\n+\t * `out` pointer with the pack generator, a negative error code\n+\t * otherwise.\n+\t */\n+\tint (*generate_pack)(struct odb_source *source,\n+\t\t\t struct odb_pack_generator **out,\n+\t\t\t const struct odb_generate_pack_options *opts);\n };\n \n /*\n@@ -520,4 +537,20 @@ static inline bool odb_source_optimize_required(struct odb_source *source,\n \treturn source->optimize_required(source, opts);\n }\n \n+/*\n+ * Start generating a packfile from the given source with the given options.\n+ * The pack is generated asynchronously; the caller is expected to consume the\n+ * file descriptors exposed via the pack generator and to then wait for\n+ * completion via `odb_pack_generator_finish()`.\n+ *\n+ * Returns 0 on success and populates the `out` pointer with the pack\n+ * generator, a negative error code otherwise.\n+ */\n+static inline int odb_source_generate_pack(struct odb_source *source,\n+\t\t\t\t\t struct odb_pack_generator **out,\n+\t\t\t\t\t const struct odb_generate_pack_options *opts)\n+{\n+\treturn source->generate_pack(source, out, opts);\n+}\n+\n #endif\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549973","messageId":"20260807-b4-pks-odb-generate-pack-v1-2-7dec431ae7cd@pks.im","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","subject":"[PATCH 2/5] upload-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-07T10:45:08Z","receivedAt":"2026-08-07T10:45:45Z","isPatch":true,"body":"When serving a fetch, git-upload-pack(1) spawns git-pack-objects(1)\ndirectly to generate the packfile that gets sent to the client. This\nhard-codes the assumption that the object database is able to serve\npackfiles via git-pack-objects(1), which is specific to the \"files\"\nbackend.\n\nConvert git-upload-pack(1) to instead use the pack generation interface\nof the object database.\n\nSigned-off-by: Patrick Steinhardt \n---\n upload-pack.c | 125 +++++++++++++++++++++-------------------------------------\n 1 file changed, 45 insertions(+), 80 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex a52856d869..75a857eaa8 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -197,11 +197,11 @@ static void send_client_data(int fd, const char *data, ssize_t sz,\n \twrite_or_die(fd, data, sz);\n }\n \n-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n+static int append_one_shallow(const struct commit_graft *graft, void *cb_data)\n {\n-\tFILE *fp = cb_data;\n+\tstruct oid_array *shallows = cb_data;\n \tif (graft->nr_parent == -1)\n-\t\tfprintf(fp, \"--shallow %s\\n\", oid_to_hex(&graft->oid));\n+\t\toid_array_append(shallows, &graft->oid);\n \treturn 0;\n }\n \n@@ -299,7 +299,8 @@ static int relay_pack_data(int pack_objects_out, struct output_state *os,\n static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t\t const struct string_list *uri_protocols)\n {\n-\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n \tstruct output_state *output_state = xcalloc(1, sizeof(struct output_state));\n \tchar progress[128];\n \tchar abort_msg[] = \"aborting due to possible repository \"\n@@ -307,78 +308,42 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \tuint64_t last_sent_ms = 0;\n \tssize_t sz;\n \tint i;\n-\tFILE *pipe_fd;\n-\n-\tif (!pack_data->pack_objects_hook)\n-\t\tpack_objects.git_cmd = 1;\n-\telse {\n-\t\tstrvec_push(&pack_objects.args, pack_data->pack_objects_hook);\n-\t\tstrvec_push(&pack_objects.args, \"git\");\n-\t\tpack_objects.use_shell = 1;\n-\t}\n \n \tif (pack_data->shallow_nr) {\n-\t\tstrvec_push(&pack_objects.args, \"--shallow-file\");\n-\t\tstrvec_push(&pack_objects.args, \"\");\n-\t}\n-\tstrvec_push(&pack_objects.args, \"pack-objects\");\n-\tstrvec_push(&pack_objects.args, \"--revs\");\n-\tif (pack_data->use_thin_pack)\n-\t\tstrvec_push(&pack_objects.args, \"--thin\");\n-\n-\tstrvec_push(&pack_objects.args, \"--stdout\");\n-\tif (pack_data->shallow_nr)\n-\t\tstrvec_push(&pack_objects.args, \"--shallow\");\n-\tif (!pack_data->no_progress)\n-\t\tstrvec_push(&pack_objects.args, \"--progress\");\n-\tif (pack_data->use_ofs_delta)\n-\t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n-\tif (pack_data->use_include_tag)\n-\t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n-\tif (repo_has_accepted_promisor_remote(the_repository))\n-\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n-\tif (pack_data->filter_options.choice) {\n-\t\tconst char *spec =\n-\t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\", spec);\n-\t}\n-\tif (uri_protocols) {\n-\t\tfor (i = 0; i < uri_protocols->nr; i++)\n-\t\t\tstrvec_pushf(&pack_objects.args, \"--uri-protocol=%s\",\n-\t\t\t\t\t uri_protocols->items[i].string);\n+\t\tfor_each_commit_graft(append_one_shallow, &opts.shallows);\n+\t\topts.shallow = 1;\n \t}\n-\n-\tpack_objects.in = -1;\n-\tpack_objects.out = -1;\n-\tpack_objects.err = -1;\n-\tpack_objects.clean_on_exit = 1;\n-\n-\tif (start_command(&pack_objects))\n-\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n-\n-\tpipe_fd = xfdopen(pack_objects.in, \"w\");\n-\n-\tif (pack_data->shallow_nr)\n-\t\tfor_each_commit_graft(write_one_shallow, pipe_fd);\n-\n \tfor (i = 0; i < pack_data->want_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->want_obj.objects[i].item->oid));\n-\tfprintf(pipe_fd, \"--not\\n\");\n+\t\toid_array_append(&opts.wants,\n+\t\t\t\t &pack_data->want_obj.objects[i].item->oid);\n \tfor (i = 0; i < pack_data->have_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->have_obj.objects[i].item->oid));\n+\t\toid_array_append(&opts.haves,\n+\t\t\t\t &pack_data->have_obj.objects[i].item->oid);\n \tfor (i = 0; i < pack_data->extra_edge_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->extra_edge_obj.objects[i].item->oid));\n-\tfprintf(pipe_fd, \"\\n\");\n-\tfflush(pipe_fd);\n-\tfclose(pipe_fd);\n-\n-\t/* We read from pack_objects.err to capture stderr output for\n-\t * progress bar, and pack_objects.out to capture the pack data.\n-\t */\n+\t\toid_array_append(&opts.haves,\n+\t\t\t\t &pack_data->extra_edge_obj.objects[i].item->oid);\n+\n+\topts.thin = pack_data->use_thin_pack;\n+\tif (!pack_data->no_progress)\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_STANDARD;\n+\topts.ofs_delta = pack_data->use_ofs_delta;\n+\topts.include_tag = pack_data->use_include_tag;\n+\topts.missing_allow_promisor = repo_has_accepted_promisor_remote(the_repository);\n+\tif (pack_data->filter_options.choice)\n+\t\topts.filter_spec = expand_list_objects_filter_spec(&pack_data->filter_options);\n+\topts.uri_protocols = uri_protocols;\n+\topts.pack_objects_hook = pack_data->pack_objects_hook;\n+\topts.pack_fd = -1;\n+\topts.progress_fd = -1;\n+\n+\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n+\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n+\todb_generate_pack_options_release(&opts);\n \n+\t/*\n+\t * We read from generator->err to capture stderr output for the\n+\t * progress bar, and generator->out to capture the pack data.\n+\t */\n \twhile (1) {\n \t\tuint64_t now_ms = getnanotime() / 1000000;\n \t\tstruct pollfd pfd[2];\n@@ -393,14 +358,14 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tpollsize = 0;\n \t\tpe = pu = -1;\n \n-\t\tif (0 <= pack_objects.out) {\n-\t\t\tpfd[pollsize].fd = pack_objects.out;\n+\t\tif (0 <= generator->out) {\n+\t\t\tpfd[pollsize].fd = generator->out;\n \t\t\tpfd[pollsize].events = POLLIN;\n \t\t\tpu = pollsize;\n \t\t\tpollsize++;\n \t\t}\n-\t\tif (0 <= pack_objects.err) {\n-\t\t\tpfd[pollsize].fd = pack_objects.err;\n+\t\tif (0 <= generator->err) {\n+\t\t\tpfd[pollsize].fd = generator->err;\n \t\t\tpfd[pollsize].events = POLLIN;\n \t\t\tpe = pollsize;\n \t\t\tpollsize++;\n@@ -437,15 +402,15 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t\t/* Status ready; we ship that in the side-band\n \t\t\t * or dump to the standard error.\n \t\t\t */\n-\t\t\tsz = xread(pack_objects.err, progress,\n+\t\t\tsz = xread(generator->err, progress,\n \t\t\t\t sizeof(progress));\n \t\t\tif (0 < sz) {\n \t\t\t\tsend_client_data(2, progress, sz,\n \t\t\t\t\t\t pack_data->use_sideband);\n \t\t\t\tlast_sent_ms = now_ms;\n \t\t\t} else if (sz == 0) {\n-\t\t\t\tclose(pack_objects.err);\n-\t\t\t\tpack_objects.err = -1;\n+\t\t\t\tclose(generator->err);\n+\t\t\t\tgenerator->err = -1;\n \t\t\t}\n \t\t\telse\n \t\t\t\tgoto fail;\n@@ -455,15 +420,15 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \n \t\tif (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) {\n \t\t\tbool did_send_data;\n-\t\t\tint result = relay_pack_data(pack_objects.out,\n+\t\t\tint result = relay_pack_data(generator->out,\n \t\t\t\t\t\t output_state,\n \t\t\t\t\t\t pack_data->use_sideband,\n \t\t\t\t\t\t !!uri_protocols,\n \t\t\t\t\t\t &did_send_data);\n \n \t\t\tif (result == 0) {\n-\t\t\t\tclose(pack_objects.out);\n-\t\t\t\tpack_objects.out = -1;\n+\t\t\t\tclose(generator->out);\n+\t\t\t\tgenerator->out = -1;\n \t\t\t} else if (result < 0) {\n \t\t\t\tgoto fail;\n \t\t\t}\n@@ -498,7 +463,7 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t}\n \t}\n \n-\tif (finish_command(&pack_objects)) {\n+\tif (odb_pack_generator_finish(generator)) {\n \t\terror(\"git upload-pack: git-pack-objects died with error.\");\n \t\tgoto fail;\n \t}\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549974","messageId":"20260807-b4-pks-odb-generate-pack-v1-3-7dec431ae7cd@pks.im","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","subject":"[PATCH 3/5] send-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-07T10:45:09Z","receivedAt":"2026-08-07T10:45:49Z","isPatch":true,"body":"When pushing, git-send-pack(1) spawns git-pack-objects(1) directly to\ngenerate the packfile that gets sent to the remote. Same as with\ngit-upload-pack(1), which has been adapted in the preceding commit,\nthis hard-codes the assumption that objects can be packed via\ngit-pack-objects(1), which is specific to the \"files\" backend.\n\nConvert git-send-pack(1) to use the pack generation interface of the\nobject database instead.\n\nNote that this requires us to adapt t5516 because the parameters passed\nto git-pack-objects(1) are changing:\n\n - The order of arguments changes.\n\n - We pass \"--quiet\" instead of \"-q\".\n\n - We don't pass \"--all-progress-implied\" anymore when not generating\n output.\n\nAll of these changes are benign though and should not result in a change\nin behaviour.\n\nSigned-off-by: Patrick Steinhardt \n---\n send-pack.c | 101 +++++++++++++++++---------------------------------\n t/t5516-fetch-push.sh | 12 +++---\n 2 files changed, 40 insertions(+), 73 deletions(-)\n\ndiff --git a/send-pack.c b/send-pack.c\nindex 3bb5afc687..f20460fbf4 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -42,16 +42,17 @@ int option_parse_push_signed(const struct option *opt,\n \tdie(\"bad %s argument: %s\", opt->long_name, arg);\n }\n \n-static void feed_object(struct repository *r,\n-\t\t\tconst struct object_id *oid, FILE *fh, int negative)\n+static void append_negative_object(struct repository *r,\n+\t\t\t\t struct oid_array *haves,\n+\t\t\t\t const struct object_id *oid)\n {\n-\tif (negative && !odb_has_object(r->objects, oid, 0))\n+\t/*\n+\t * The remote end may have advertised objects that we do not have in\n+\t * our object database. Skip those, as we cannot use them as boundary.\n+\t */\n+\tif (!odb_has_object(r->objects, oid, 0))\n \t\treturn;\n-\n-\tif (negative)\n-\t\tputc('^', fh);\n-\tfputs(oid_to_hex(oid), fh);\n-\tputc('\\n', fh);\n+\toid_array_append(haves, oid);\n }\n \n /*\n@@ -62,92 +63,58 @@ static int pack_objects(struct repository *r,\n \t\t\tstruct oid_array *negotiated,\n \t\t\tstruct send_pack_args *args)\n {\n-\t/*\n-\t * The child becomes pack-objects --revs; we feed\n-\t * the revision parameters to it via its stdin and\n-\t * let its stdout go back to the other end.\n-\t */\n-\tstruct child_process po = CHILD_PROCESS_INIT;\n-\tFILE *po_in;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n \tint rc;\n \n \ttrace2_region_enter(\"send_pack\", \"pack_objects\", r);\n-\tstrvec_push(&po.args, \"pack-objects\");\n-\tstrvec_push(&po.args, \"--all-progress-implied\");\n-\tstrvec_push(&po.args, \"--revs\");\n-\tstrvec_push(&po.args, \"--stdout\");\n-\tif (args->use_thin_pack)\n-\t\tstrvec_push(&po.args, \"--thin\");\n-\tif (args->use_ofs_delta)\n-\t\tstrvec_push(&po.args, \"--delta-base-offset\");\n-\tif (args->quiet || !args->progress)\n-\t\tstrvec_push(&po.args, \"-q\");\n+\n+\topts.thin = args->use_thin_pack;\n+\topts.ofs_delta = args->use_ofs_delta;\n \tif (args->progress)\n-\t\tstrvec_push(&po.args, \"--progress\");\n-\tif (is_repository_shallow(r))\n-\t\tstrvec_push(&po.args, \"--shallow\");\n-\tif (args->disable_bitmaps)\n-\t\tstrvec_push(&po.args, \"--no-use-bitmap-index\");\n-\tpo.in = -1;\n-\tpo.out = args->stateless_rpc ? -1 : fd;\n-\tpo.git_cmd = 1;\n-\tpo.clean_on_exit = 1;\n-\tif (start_command(&po))\n-\t\tdie_errno(\"git pack-objects failed\");\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n+\topts.shallow = is_repository_shallow(r);\n+\topts.disable_bitmaps = args->disable_bitmaps;\n \n \t/*\n-\t * We feed the pack-objects we just spawned with revision\n-\t * parameters by writing to the pipe.\n+\t * The pack is either written directly to the remote's descriptor, or,\n+\t * in the case of a stateless RPC, read back from a pipe so that we\n+\t * can wrap the pack data into pkt-lines.\n \t */\n-\tpo_in = xfdopen(po.in, \"w\");\n+\topts.pack_fd = args->stateless_rpc ? -1 : fd;\n+\n \tfor (size_t i = 0; i < advertised->nr; i++)\n-\t\tfeed_object(r, &advertised->oid[i], po_in, 1);\n+\t\tappend_negative_object(r, &opts.haves, &advertised->oid[i]);\n \tfor (size_t i = 0; i < negotiated->nr; i++)\n-\t\tfeed_object(r, &negotiated->oid[i], po_in, 1);\n+\t\tappend_negative_object(r, &opts.haves, &negotiated->oid[i]);\n \n \twhile (refs) {\n \t\tif (!is_null_oid(&refs->old_oid))\n-\t\t\tfeed_object(r, &refs->old_oid, po_in, 1);\n+\t\t\tappend_negative_object(r, &opts.haves, &refs->old_oid);\n \t\tif (!is_null_oid(&refs->new_oid))\n-\t\t\tfeed_object(r, &refs->new_oid, po_in, 0);\n+\t\t\toid_array_append(&opts.wants, &refs->new_oid);\n \t\trefs = refs->next;\n \t}\n \n-\tfflush(po_in);\n-\tif (ferror(po_in))\n-\t\tdie_errno(\"error writing to pack-objects\");\n-\tfclose(po_in);\n+\tif (odb_generate_pack(r->objects, &generator, &opts))\n+\t\tdie(\"git pack-objects failed\");\n+\todb_generate_pack_options_release(&opts);\n \n \tif (args->stateless_rpc) {\n \t\tchar *buf = xmalloc(LARGE_PACKET_MAX);\n \t\twhile (1) {\n-\t\t\tssize_t n = xread(po.out, buf, LARGE_PACKET_MAX);\n+\t\t\tssize_t n = xread(generator->out, buf, LARGE_PACKET_MAX);\n \t\t\tif (n <= 0)\n \t\t\t\tbreak;\n \t\t\tsend_sideband(fd, -1, buf, n, LARGE_PACKET_MAX);\n \t\t}\n \t\tfree(buf);\n-\t\tclose(po.out);\n-\t\tpo.out = -1;\n+\t\tclose(generator->out);\n \t}\n \n-\trc = finish_command(&po);\n-\tif (rc) {\n-\t\t/*\n-\t\t * For a normal non-zero exit, we assume pack-objects wrote\n-\t\t * something useful to stderr. For death by signal, though,\n-\t\t * we should mention it to the user. The exception is SIGPIPE\n-\t\t * (141), because that's a normal occurrence if the remote end\n-\t\t * hangs up (and we'll report that by trying to read the unpack\n-\t\t * status).\n-\t\t */\n-\t\tif (rc > 128 && rc != 141)\n-\t\t\terror(\"pack-objects died of signal %d\", rc - 128);\n-\t\ttrace2_region_leave(\"send_pack\", \"pack_objects\", r);\n-\t\treturn -1;\n-\t}\n+\trc = odb_pack_generator_finish(generator);\n \ttrace2_region_leave(\"send_pack\", \"pack_objects\", r);\n-\treturn 0;\n+\treturn rc;\n }\n \n static int receive_unpack_status(struct packet_reader *reader)\n@@ -768,7 +735,7 @@ int send_pack(struct repository *r,\n \t\t\tgoto out;\n \t\t}\n \t\tif (!args->stateless_rpc)\n-\t\t\t/* Closed by pack_objects() via start_command() */\n+\t\t\t/* Consumed by the pack generator in pack_objects() */\n \t\t\tfd[1] = -1;\n \t}\n \tif (args->stateless_rpc && cmds_sent)\ndiff --git a/t/t5516-fetch-push.sh b/t/t5516-fetch-push.sh\nindex f3b3efc47f..b982b209bf 100755\n--- a/t/t5516-fetch-push.sh\n+++ b/t/t5516-fetch-push.sh\n@@ -1903,20 +1903,20 @@ test_expect_success 'push with config push.useBitmaps' '\n \ttest_unconfig push.useBitmaps &&\n \tGIT_TRACE2_EVENT=\"$PWD/default\" \\\n \tgit push --quiet testrepo main:test &&\n-\ttest_subcommand git pack-objects --all-progress-implied --revs --stdout \\\n-\t\t--thin --delta-base-offset -q \n---\n builtin/bundle.c | 33 ++++++++++++++++-----------------\n 1 file changed, 16 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 1e170e9278..bfafadc984 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -70,35 +70,34 @@ static int parse_options_cmd_bundle(int argc,\n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\t struct repository *repo UNUSED) {\n \tstruct strvec pack_opts = STRVEC_INIT;\n+\tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n-\tint ret;\n \tstruct option options[] = {\n-\t\tOPT_PASSTHRU_ARGV('q', \"quiet\", &pack_opts, NULL,\n-\t\t\t\t N_(\"do not show progress meter\"),\n-\t\t\t\t PARSE_OPT_NOARG),\n-\t\tOPT_PASSTHRU_ARGV(0, \"progress\", &pack_opts, NULL,\n-\t\t\t\t N_(\"show progress meter\"),\n-\t\t\t\t PARSE_OPT_NOARG),\n-\t\tOPT_PASSTHRU_ARGV(0, \"all-progress\", &pack_opts, NULL,\n-\t\t\t\t N_(\"historical; same as --progress\"),\n-\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n-\t\tOPT_PASSTHRU_ARGV(0, \"all-progress-implied\", &pack_opts, NULL,\n-\t\t\t\t N_(\"historical; does nothing\"),\n-\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n+\t\tOPT_NEGBIT('q', \"quiet\", &progress,\n+\t\t\t N_(\"do not show progress meter\"), 1),\n+\t\tOPT_BIT(0, \"progress\", &progress,\n+\t\t\tN_(\"show progress meter\"), 1),\n+\t\tOPT_BIT_F(0, \"all-progress\", &progress,\n+\t\t\t N_(\"historical; same as --progress\"), 1,\n+\t\t\t PARSE_OPT_HIDDEN),\n+\t\tOPT_NOOP_NOARG(0, \"all-progress-implied\"),\n \t\tOPT_INTEGER(0, \"version\", &version,\n \t\t\t N_(\"specify bundle format version\")),\n \t\tOPT_END()\n \t};\n \tchar *bundle_file;\n-\n-\tif (isatty(STDERR_FILENO))\n-\t\tstrvec_push(&pack_opts, \"--progress\");\n-\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n+\tint ret;\n \n \targc = parse_options_cmd_bundle(argc, argv, prefix,\n \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n \t/* bundle internals use argv[1] as further parameters */\n \n+\tif (progress)\n+\t\tstrvec_push(&pack_opts, \"--progress\");\n+\telse\n+\t\tstrvec_push(&pack_opts, \"--quiet\");\n+\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n+\n \tif (!startup_info->have_repository)\n \t\tdie(_(\"Need a repository to create a bundle.\"));\n \tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549976","messageId":"20260807-b4-pks-odb-generate-pack-v1-5-7dec431ae7cd@pks.im","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","subject":"[PATCH 5/5] bundle: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-07T10:45:11Z","receivedAt":"2026-08-07T10:45:55Z","isPatch":true,"body":"git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\ndata that gets appended to the bundle header. While bundles are not\npart of the wire protocol, they are a transfer mechanism for packs all\nthe same, so convert them to use the pack generation interface of the\nobject database as well.\n\nThis makes the pack generator the single spawn point for all pack\nstreams that leave the repository, leaving only local maintenance tasks\nlike git-repack(1) with direct knowledge of git-pack-objects(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/bundle.c | 10 +--------\n bundle.c | 68 +++++++++++++++++++++++++++++---------------------------\n bundle.h | 3 +--\n 3 files changed, 37 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex bfafadc984..de86e092a6 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -69,7 +69,6 @@ static int parse_options_cmd_bundle(int argc,\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\t struct repository *repo UNUSED) {\n-\tstruct strvec pack_opts = STRVEC_INIT;\n \tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n \tstruct option options[] = {\n@@ -92,16 +91,9 @@ static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n \t/* bundle internals use argv[1] as further parameters */\n \n-\tif (progress)\n-\t\tstrvec_push(&pack_opts, \"--progress\");\n-\telse\n-\t\tstrvec_push(&pack_opts, \"--quiet\");\n-\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n-\n \tif (!startup_info->have_repository)\n \t\tdie(_(\"Need a repository to create a bundle.\"));\n-\tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n-\tstrvec_clear(&pack_opts);\n+\tret = !!create_bundle(the_repository, bundle_file, argc, argv, version, progress);\n \tfree(bundle_file);\n \treturn ret;\n }\ndiff --git a/bundle.c b/bundle.c\nindex b64716f252..09afc465c0 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -325,50 +325,52 @@ static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n \n \n /* Write the pack data to bundle_fd */\n-static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *pack_options)\n+static int write_pack_data(int bundle_fd, struct rev_info *revs, int progress)\n {\n-\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n+\tint ret = 0;\n \tint i;\n \n-\tstrvec_pushl(&pack_objects.args,\n-\t\t \"pack-objects\",\n-\t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n-\t\t NULL);\n-\tstrvec_pushv(&pack_objects.args, pack_options->v);\n+\topts.thin = 1;\n+\topts.ofs_delta = 1;\n+\tif (progress)\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n \tif (revs->filter.choice)\n-\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t list_objects_filter_spec(&revs->filter));\n-\tpack_objects.in = -1;\n-\tpack_objects.out = bundle_fd;\n-\tpack_objects.git_cmd = 1;\n+\t\topts.filter_spec = list_objects_filter_spec(&revs->filter);\n \n \t/*\n-\t * start_command() will close our descriptor if it's >1. Duplicate it\n-\t * to avoid surprising the caller.\n+\t * The pack generator will consume our descriptor if it's >1.\n+\t * Duplicate it to avoid surprising the caller.\n \t */\n-\tif (pack_objects.out > 1) {\n-\t\tpack_objects.out = dup(pack_objects.out);\n-\t\tif (pack_objects.out < 0) {\n-\t\t\terror_errno(_(\"unable to dup bundle descriptor\"));\n-\t\t\tchild_process_clear(&pack_objects);\n-\t\t\treturn -1;\n-\t\t}\n+\topts.pack_fd = bundle_fd;\n+\tif (opts.pack_fd > 1) {\n+\t\topts.pack_fd = dup(bundle_fd);\n+\t\tif (opts.pack_fd < 0)\n+\t\t\treturn error_errno(_(\"unable to dup bundle descriptor\"));\n \t}\n \n-\tif (start_command(&pack_objects))\n-\t\treturn error(_(\"Could not spawn pack-objects\"));\n-\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object *object = revs->pending.objects[i].item;\n \t\tif (object->flags & UNINTERESTING)\n-\t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n-\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid), the_hash_algo->hexsz);\n-\t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n+\t\t\toid_array_append(&opts.haves, &object->oid);\n+\t\telse\n+\t\t\toid_array_append(&opts.wants, &object->oid);\n \t}\n-\tclose(pack_objects.in);\n-\tif (finish_command(&pack_objects))\n-\t\treturn error(_(\"pack-objects died\"));\n-\treturn 0;\n+\n+\tif (odb_generate_pack(the_repository->objects, &generator, &opts)) {\n+\t\tret = error(_(\"Could not spawn pack-objects\"));\n+\t\tgoto out;\n+\t}\n+\n+\tif (odb_pack_generator_finish(generator)) {\n+\t\tret = error(_(\"pack-objects died\"));\n+\t\tgoto out;\n+\t}\n+\n+out:\n+\todb_generate_pack_options_release(&opts);\n+\treturn ret;\n }\n \n /*\n@@ -476,7 +478,7 @@ static void write_bundle_prerequisites(struct commit *commit, void *data)\n }\n \n int create_bundle(struct repository *r, const char *path,\n-\t\t int argc, const char **argv, struct strvec *pack_options, int version)\n+\t\t int argc, const char **argv, int version, int progress)\n {\n \tstruct lock_file lock = LOCK_INIT;\n \tint bundle_fd = -1;\n@@ -584,7 +586,7 @@ int create_bundle(struct repository *r, const char *path,\n \t}\n \n \t/* write pack */\n-\tif (write_pack_data(bundle_fd, &revs_copy, pack_options)) {\n+\tif (write_pack_data(bundle_fd, &revs_copy, progress)) {\n \t\tret = -1;\n \t\tgoto out;\n \t}\ndiff --git a/bundle.h b/bundle.h\nindex d664b2f2d6..471da23d1b 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -27,8 +27,7 @@ int read_bundle_header(const char *path, struct bundle_header *header);\n int read_bundle_header_fd(int fd, struct bundle_header *header,\n \t\t\t const char *report_path);\n int create_bundle(struct repository *r, const char *path,\n-\t\t int argc, const char **argv, struct strvec *pack_options,\n-\t\t int version);\n+\t\t int argc, const char **argv, int version, int progress);\n \n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"550051","messageId":"xmqq33wpej49.fsf@gitster.g","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","subject":"Re: [PATCH 0/5] odb: make packfile generation pluggable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-08-07T21:05:58Z","receivedAt":"2026-08-07T21:06:01Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this patch series makes packfile generation pluggable.\n>\n> Note that this series only makes those parts pluggable that are required\n> for the transport layer. The other parts that relate to packfile\n> generation as required by our repository maintenance is kept as-is, as\n> there is a bunch of options there that are way too specific to the\n> \"files\" backend to be portable. This should ultimately not be much of a\n> problem though, as maintenance itself is already pluggable in the first\n> place.\n>\n> It's a bit of a shame though for git-pack-objects(1), which still isn't\n> usable with alternate backends. I tried several times to find good\n> solutions for making it fully pluggable, but due to the backend-specific\n> options it's an utter mess. I want to eventually address this though:\n> same as with git-refs(1), I want to introduce git-objects(1) to care\n> about all things ODB. And as part of that command we can also introduce\n> a command that generates packfiles in a generic fashion, without all the\n> cruft that git-pack-objects(1) has. This is part of a future patch\n> series though.\n>\n> The series is built on top of 2c78326f81 (The 11th batch, 2026-08-05).\n\nWith \"--no-ref-delta\" thing in flight, this will not play well with\nwhat is in 'seen', though.\n"},{"id":"550153","messageId":"anlg2rThlBLavyU8@pks.im","threadId":"66136","inReplyTo":"xmqq33wpej49.fsf@gitster.g","subject":"Re: [PATCH 0/5] odb: make packfile generation pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-10T05:25:46Z","receivedAt":"2026-08-10T05:25:53Z","isPatch":true,"body":"On Fri, Aug 07, 2026 at 02:05:58PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > Hi,\n> >\n> > this patch series makes packfile generation pluggable.\n> >\n> > Note that this series only makes those parts pluggable that are required\n> > for the transport layer. The other parts that relate to packfile\n> > generation as required by our repository maintenance is kept as-is, as\n> > there is a bunch of options there that are way too specific to the\n> > \"files\" backend to be portable. This should ultimately not be much of a\n> > problem though, as maintenance itself is already pluggable in the first\n> > place.\n> >\n> > It's a bit of a shame though for git-pack-objects(1), which still isn't\n> > usable with alternate backends. I tried several times to find good\n> > solutions for making it fully pluggable, but due to the backend-specific\n> > options it's an utter mess. I want to eventually address this though:\n> > same as with git-refs(1), I want to introduce git-objects(1) to care\n> > about all things ODB. And as part of that command we can also introduce\n> > a command that generates packfiles in a generic fashion, without all the\n> > cruft that git-pack-objects(1) has. This is part of a future patch\n> > series though.\n> >\n> > The series is built on top of 2c78326f81 (The 11th batch, 2026-08-05).\n> \n> With \"--no-ref-delta\" thing in flight, this will not play well with\n> what is in 'seen', though.\n\nAh, dang, you're right. I'm not quite sure about the status of that\nseries -- there's been a discussion around whether it is the right fix\nin the first case with Peff, and there wasn't an answer since Peff's\nlast mail.\n\nTaylor, could you maybe share what your plans are? If you want to pursue\nit further I'm happy to add it as a dependency and/or wait a bit.\n\nPatrick\n"},{"id":"550461","messageId":"an0EkMZGEbg6LERc@com-79390","threadId":"66136","inReplyTo":"anlg2rThlBLavyU8@pks.im","subject":"Re: [PATCH 0/5] odb: make packfile generation pluggable","fromName":"Taylor Blau","fromEmail":"ttaylorr@openai.com","sentAt":"2026-08-12T23:41:04Z","receivedAt":"2026-08-12T23:41:14Z","isPatch":true,"body":"On Mon, Aug 10, 2026 at 07:25:46AM +0200, Patrick Steinhardt wrote:\n> > With \"--no-ref-delta\" thing in flight, this will not play well with\n> > what is in 'seen', though.\n>\n> Ah, dang, you're right. I'm not quite sure about the status of that\n> series -- there's been a discussion around whether it is the right fix\n> in the first case with Peff, and there wasn't an answer since Peff's\n> last mail.\n>\n> Taylor, could you maybe share what your plans are? If you want to pursue\n> it further I'm happy to add it as a dependency and/or wait a bit.\n\nStill something that we're working on, though I think that it's fine to\nkick this out of 'seen' for the time being.\n\nThanks,\nTaylor\n"},{"id":"550469","messageId":"an1ajMjVRUsfu-lv@pks.im","threadId":"66136","inReplyTo":"an0EkMZGEbg6LERc@com-79390","subject":"Re: [PATCH 0/5] odb: make packfile generation pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-13T05:47:56Z","receivedAt":"2026-08-13T05:48:03Z","isPatch":true,"body":"On Wed, Aug 12, 2026 at 06:41:04PM -0500, Taylor Blau wrote:\n> On Mon, Aug 10, 2026 at 07:25:46AM +0200, Patrick Steinhardt wrote:\n> > > With \"--no-ref-delta\" thing in flight, this will not play well with\n> > > what is in 'seen', though.\n> >\n> > Ah, dang, you're right. I'm not quite sure about the status of that\n> > series -- there's been a discussion around whether it is the right fix\n> > in the first case with Peff, and there wasn't an answer since Peff's\n> > last mail.\n> >\n> > Taylor, could you maybe share what your plans are? If you want to pursue\n> > it further I'm happy to add it as a dependency and/or wait a bit.\n> \n> Still something that we're working on, though I think that it's fine to\n> kick this out of 'seen' for the time being.\n\nAwesome, thanks for the update.\n\nIn that case, Junio, could you maybe kick out that topic and merge this\none here into seen instead? Thanks!\n\nPatrick\n"},{"id":"550547","messageId":"xmqqzeypsz2s.fsf@gitster.g","threadId":"66136","inReplyTo":"an1ajMjVRUsfu-lv@pks.im","subject":"Re: [PATCH 0/5] odb: make packfile generation pluggable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-08-13T17:35:39Z","receivedAt":"2026-08-13T17:35:42Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> On Wed, Aug 12, 2026 at 06:41:04PM -0500, Taylor Blau wrote:\n>> On Mon, Aug 10, 2026 at 07:25:46AM +0200, Patrick Steinhardt wrote:\n>> > > With \"--no-ref-delta\" thing in flight, this will not play well with\n>> > > what is in 'seen', though.\n>> >\n>> > Ah, dang, you're right. I'm not quite sure about the status of that\n>> > series -- there's been a discussion around whether it is the right fix\n>> > in the first case with Peff, and there wasn't an answer since Peff's\n>> > last mail.\n>> >\n>> > Taylor, could you maybe share what your plans are? If you want to pursue\n>> > it further I'm happy to add it as a dependency and/or wait a bit.\n>> \n>> Still something that we're working on, though I think that it's fine to\n>> kick this out of 'seen' for the time being.\n>\n> Awesome, thanks for the update.\n>\n> In that case, Junio, could you maybe kick out that topic and merge this\n> one here into seen instead? Thanks!\n\nOK. Let's see how it goes.\n\nThanks, both.\n"},{"id":"550551","messageId":"xmqqmrupsxx9.fsf@gitster.g","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-5-7dec431ae7cd@pks.im","subject":"Re: [PATCH 5/5] bundle: generate packfiles via the object database","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-08-13T18:00:34Z","receivedAt":"2026-08-13T18:00:37Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\n> data that gets appended to the bundle header. While bundles are not\n> part of the wire protocol, they are a transfer mechanism for packs all\n> the same, so convert them to use the pack generation interface of the\n> object database as well.\n>\n> This makes the pack generator the single spawn point for all pack\n> streams that leave the repository, leaving only local maintenance tasks\n> like git-repack(1) with direct knowledge of git-pack-objects(1).\n\nNice to see that the series aims for completeness.\n\n> diff --git a/builtin/bundle.c b/builtin/bundle.c\n> index bfafadc984..de86e092a6 100644\n> --- a/builtin/bundle.c\n> +++ b/builtin/bundle.c\n> @@ -69,7 +69,6 @@ static int parse_options_cmd_bundle(int argc,\n> \n> static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n> \t\t\t struct repository *repo UNUSED) {\n> -\tstruct strvec pack_opts = STRVEC_INIT;\n> \tint progress = isatty(STDERR_FILENO);\n> \tint version = -1;\n> \tstruct option options[] = {\n> @@ -92,16 +91,9 @@ static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n> \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n> \t/* bundle internals use argv[1] as further parameters */\n> \n> -\tif (progress)\n> -\t\tstrvec_push(&pack_opts, \"--progress\");\n> -\telse\n> -\t\tstrvec_push(&pack_opts, \"--quiet\");\n> -\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n> -\n> \tif (!startup_info->have_repository)\n> \t\tdie(_(\"Need a repository to create a bundle.\"));\n> -\tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n> -\tstrvec_clear(&pack_opts);\n> +\tret = !!create_bundle(the_repository, bundle_file, argc, argv, version, progress);\n> \tfree(bundle_file);\n> \treturn ret;\n> }\n\nAt this point after we determined startup_info->have_repository is\ntrue, we should be able to rely on \"repo\", not \"the_repository\".\nBut the callchain starting at the create_bundle() function might not\nbe ready yet. Let's keep reading.\n\n> diff --git a/bundle.c b/bundle.c\n> index b64716f252..09afc465c0 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -325,50 +325,52 @@ static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n> \n> \n> /* Write the pack data to bundle_fd */\n> -static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *pack_options)\n> +static int write_pack_data(int bundle_fd, struct rev_info *revs, int progress)\n> {\n> -\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n> +\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n> +\tstruct odb_pack_generator *generator;\n> +\tint ret = 0;\n> \tint i;\n> \n> -\tstrvec_pushl(&pack_objects.args,\n> -\t\t \"pack-objects\",\n> -\t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n> -\t\t NULL);\n> -\tstrvec_pushv(&pack_objects.args, pack_options->v);\n> +\topts.thin = 1;\n> +\topts.ofs_delta = 1;\n> +\tif (progress)\n> +\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n> \tif (revs->filter.choice)\n> -\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n> -\t\t\t list_objects_filter_spec(&revs->filter));\n> -\tpack_objects.in = -1;\n> -\tpack_objects.out = bundle_fd;\n> -\tpack_objects.git_cmd = 1;\n> +\t\topts.filter_spec = list_objects_filter_spec(&revs->filter);\n> \n> \t/*\n> -\t * start_command() will close our descriptor if it's >1. Duplicate it\n> -\t * to avoid surprising the caller.\n> +\t * The pack generator will consume our descriptor if it's >1.\n> +\t * Duplicate it to avoid surprising the caller.\n> \t */\n> -\tif (pack_objects.out > 1) {\n> -\t\tpack_objects.out = dup(pack_objects.out);\n> -\t\tif (pack_objects.out < 0) {\n> -\t\t\terror_errno(_(\"unable to dup bundle descriptor\"));\n> -\t\t\tchild_process_clear(&pack_objects);\n> -\t\t\treturn -1;\n> -\t\t}\n> +\topts.pack_fd = bundle_fd;\n> +\tif (opts.pack_fd > 1) {\n> +\t\topts.pack_fd = dup(bundle_fd);\n> +\t\tif (opts.pack_fd < 0)\n> +\t\t\treturn error_errno(_(\"unable to dup bundle descriptor\"));\n> \t}\n> \n> -\tif (start_command(&pack_objects))\n> -\t\treturn error(_(\"Could not spawn pack-objects\"));\n> -\n> \tfor (i = 0; i < revs->pending.nr; i++) {\n> \t\tstruct object *object = revs->pending.objects[i].item;\n> \t\tif (object->flags & UNINTERESTING)\n> -\t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n> -\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid), the_hash_algo->hexsz);\n> -\t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n> +\t\t\toid_array_append(&opts.haves, &object->oid);\n> +\t\telse\n> +\t\t\toid_array_append(&opts.wants, &object->oid);\n> \t}\n> -\tclose(pack_objects.in);\n> -\tif (finish_command(&pack_objects))\n> -\t\treturn error(_(\"pack-objects died\"));\n> -\treturn 0;\n> +\n> +\tif (odb_generate_pack(the_repository->objects, &generator, &opts)) {\n> +\t\tret = error(_(\"Could not spawn pack-objects\"));\n> +\t\tgoto out;\n> +\t}\n> +\n> +\tif (odb_pack_generator_finish(generator)) {\n> +\t\tret = error(_(\"pack-objects died\"));\n> +\t\tgoto out;\n> +\t}\n> +\n> +out:\n> +\todb_generate_pack_options_release(&opts);\n> +\treturn ret;\n> }\n\nThis function uses the_repository, both directly and through\nthe_hash_algo macro. I think we could use revs->repo here. An\nobvious alternative is to give this function a new parameter \"struct\nrepository *repo\" but then we would have to worry about what should\nhappen when it and revs->repo go out of sync.\n\n> @@ -476,7 +478,7 @@ static void write_bundle_prerequisites(struct commit *commit, void *data)\n> }\n> \n> int create_bundle(struct repository *r, const char *path,\n> -\t\t int argc, const char **argv, struct strvec *pack_options, int version)\n> +\t\t int argc, const char **argv, int version, int progress)\n> {\n> \tstruct lock_file lock = LOCK_INIT;\n> \tint bundle_fd = -1;\n> @@ -584,7 +586,7 @@ int create_bundle(struct repository *r, const char *path,\n> \t}\n> \n> \t/* write pack */\n> -\tif (write_pack_data(bundle_fd, &revs_copy, pack_options)) {\n> +\tif (write_pack_data(bundle_fd, &revs_copy, progress)) {\n> \t\tret = -1;\n> \t\tgoto out;\n> \t}\n> diff --git a/bundle.h b/bundle.h\n> index d664b2f2d6..471da23d1b 100644\n> --- a/bundle.h\n> +++ b/bundle.h\n> @@ -27,8 +27,7 @@ int read_bundle_header(const char *path, struct bundle_header *header);\n> int read_bundle_header_fd(int fd, struct bundle_header *header,\n> \t\t\t const char *report_path);\n> int create_bundle(struct repository *r, const char *path,\n> -\t\t int argc, const char **argv, struct strvec *pack_options,\n> -\t\t int version);\n> +\t\t int argc, const char **argv, int version, int progress);\n> \n> enum verify_bundle_flags {\n> \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n"},{"id":"550593","messageId":"an7GgQLQfleCPr-a@pks.im","threadId":"66136","inReplyTo":"xmqqmrupsxx9.fsf@gitster.g","subject":"Re: [PATCH 5/5] bundle: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-14T07:40:49Z","receivedAt":"2026-08-14T07:40:58Z","isPatch":true,"body":"On Thu, Aug 13, 2026 at 11:00:34AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> > diff --git a/bundle.c b/bundle.c\n> > index b64716f252..09afc465c0 100644\n> > --- a/bundle.c\n> > +++ b/bundle.c\n> > @@ -325,50 +325,52 @@ static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n> > \n> > \n> > /* Write the pack data to bundle_fd */\n> > -static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *pack_options)\n> > +static int write_pack_data(int bundle_fd, struct rev_info *revs, int progress)\n> > {\n> > -\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n> > +\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n> > +\tstruct odb_pack_generator *generator;\n> > +\tint ret = 0;\n> > \tint i;\n> > \n> > -\tstrvec_pushl(&pack_objects.args,\n> > -\t\t \"pack-objects\",\n> > -\t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n> > -\t\t NULL);\n> > -\tstrvec_pushv(&pack_objects.args, pack_options->v);\n> > +\topts.thin = 1;\n> > +\topts.ofs_delta = 1;\n> > +\tif (progress)\n> > +\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n> > \tif (revs->filter.choice)\n> > -\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n> > -\t\t\t list_objects_filter_spec(&revs->filter));\n> > -\tpack_objects.in = -1;\n> > -\tpack_objects.out = bundle_fd;\n> > -\tpack_objects.git_cmd = 1;\n> > +\t\topts.filter_spec = list_objects_filter_spec(&revs->filter);\n> > \n> > \t/*\n> > -\t * start_command() will close our descriptor if it's >1. Duplicate it\n> > -\t * to avoid surprising the caller.\n> > +\t * The pack generator will consume our descriptor if it's >1.\n> > +\t * Duplicate it to avoid surprising the caller.\n> > \t */\n> > -\tif (pack_objects.out > 1) {\n> > -\t\tpack_objects.out = dup(pack_objects.out);\n> > -\t\tif (pack_objects.out < 0) {\n> > -\t\t\terror_errno(_(\"unable to dup bundle descriptor\"));\n> > -\t\t\tchild_process_clear(&pack_objects);\n> > -\t\t\treturn -1;\n> > -\t\t}\n> > +\topts.pack_fd = bundle_fd;\n> > +\tif (opts.pack_fd > 1) {\n> > +\t\topts.pack_fd = dup(bundle_fd);\n> > +\t\tif (opts.pack_fd < 0)\n> > +\t\t\treturn error_errno(_(\"unable to dup bundle descriptor\"));\n> > \t}\n> > \n> > -\tif (start_command(&pack_objects))\n> > -\t\treturn error(_(\"Could not spawn pack-objects\"));\n> > -\n> > \tfor (i = 0; i < revs->pending.nr; i++) {\n> > \t\tstruct object *object = revs->pending.objects[i].item;\n> > \t\tif (object->flags & UNINTERESTING)\n> > -\t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n> > -\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid), the_hash_algo->hexsz);\n> > -\t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n> > +\t\t\toid_array_append(&opts.haves, &object->oid);\n> > +\t\telse\n> > +\t\t\toid_array_append(&opts.wants, &object->oid);\n> > \t}\n> > -\tclose(pack_objects.in);\n> > -\tif (finish_command(&pack_objects))\n> > -\t\treturn error(_(\"pack-objects died\"));\n> > -\treturn 0;\n> > +\n> > +\tif (odb_generate_pack(the_repository->objects, &generator, &opts)) {\n> > +\t\tret = error(_(\"Could not spawn pack-objects\"));\n> > +\t\tgoto out;\n> > +\t}\n> > +\n> > +\tif (odb_pack_generator_finish(generator)) {\n> > +\t\tret = error(_(\"pack-objects died\"));\n> > +\t\tgoto out;\n> > +\t}\n> > +\n> > +out:\n> > +\todb_generate_pack_options_release(&opts);\n> > +\treturn ret;\n> > }\n> \n> This function uses the_repository, both directly and through\n> the_hash_algo macro. I think we could use revs->repo here. An\n> obvious alternative is to give this function a new parameter \"struct\n> repository *repo\" but then we would have to worry about what should\n> happen when it and revs->repo go out of sync.\n\nFair enough.\n\nIdeally, we'd convert the whole file to not use `the_repository` at all\nanymore. But we unfortunately call `get_log_output_encoding()`, which\nimplicitly depends on that function. I think we can still mostly drop\nthe dependency and then just add an `extern` declaration. I'll do so in\nthe next version.\n\nThanks!\n\nPatrick\n"},{"id":"550682","messageId":"20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","subject":"[PATCH v2 0/6] odb: make packfile generation pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-17T05:39:41Z","receivedAt":"2026-08-17T05:39:48Z","isPatch":true,"body":"Hi,\n\nthis patch series makes packfile generation pluggable.\n\nNote that this series only makes those parts pluggable that are required\nfor the transport layer. The other parts that relate to packfile\ngeneration as required by our repository maintenance is kept as-is, as\nthere is a bunch of options there that are way too specific to the\n\"files\" backend to be portable. This should ultimately not be much of a\nproblem though, as maintenance itself is already pluggable in the first\nplace.\n\nIt's a bit of a shame though for git-pack-objects(1), which still isn't\nusable with alternate backends. I tried several times to find good\nsolutions for making it fully pluggable, but due to the backend-specific\noptions it's an utter mess. I want to eventually address this though:\nsame as with git-refs(1), I want to introduce git-objects(1) to care\nabout all things ODB. And as part of that command we can also introduce\na command that generates packfiles in a generic fashion, without all the\ncruft that git-pack-objects(1) has. This is part of a future patch\nseries though.\n\nChanges in v2:\n - Mostly remove the dependencies on `the_repository` in \"bundle.c\".\n - Link to v1: https://patch.msgid.link/20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im\n\nThe series is built on top of 2c78326f81 (The 11th batch, 2026-08-05).\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (6):\n odb: introduce interface to generate packfiles\n upload-pack: generate packfiles via the object database\n send-pack: generate packfiles via the object database\n builtin/bundle: refactor option handling for progress meter\n bundle: get (mostly) rid of `the_repository`\n bundle: generate packfiles via the object database\n\n builtin/bundle.c | 31 ++++------\n bundle.c | 97 ++++++++++++++++++--------------\n bundle.h | 3 +-\n odb.c | 21 +++++++\n odb.h | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-files.c | 144 +++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h | 33 +++++++++++\n send-pack.c | 101 +++++++++++----------------------\n t/t5516-fetch-push.sh | 12 ++--\n upload-pack.c | 125 +++++++++++++++--------------------------\n 10 files changed, 501 insertions(+), 218 deletions(-)\n\nRange-diff versus v1:\n\n1: fb02483cbf = 1: 44fdb3dabf odb: introduce interface to generate packfiles\n2: 1293fa4488 = 2: 01a115f7b1 upload-pack: generate packfiles via the object database\n3: 98bf918183 = 3: 07e2541ca7 send-pack: generate packfiles via the object database\n4: 900536a5fd = 4: dd3f7eaab7 builtin/bundle: refactor option handling for progress meter\n-: ---------- > 5: cf01244c05 bundle: get (mostly) rid of `the_repository`\n5: 37c0dc5d99 ! 6: fa4d4dfdd5 bundle: generate packfiles via the object database\n @@ builtin/bundle.c: static int cmd_bundle_create(int argc, const char **argv, cons\n }\n \n ## bundle.c ##\n -@@ bundle.c: static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n +@@ bundle.c: static int is_tag_in_date_range(struct repository *repo,\n \n \n /* Write the pack data to bundle_fd */\n @@ bundle.c: static int is_tag_in_date_range(struct object *tag, struct rev_info *r\n \t\tstruct object *object = revs->pending.objects[i].item;\n \t\tif (object->flags & UNINTERESTING)\n -\t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n --\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid), the_hash_algo->hexsz);\n +-\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid),\n +-\t\t\t revs->repo->hash_algo->hexsz);\n -\t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n +\t\t\toid_array_append(&opts.haves, &object->oid);\n +\t\telse\n @@ bundle.c: static int is_tag_in_date_range(struct object *tag, struct rev_info *r\n -\t\treturn error(_(\"pack-objects died\"));\n -\treturn 0;\n +\n -+\tif (odb_generate_pack(the_repository->objects, &generator, &opts)) {\n ++\tif (odb_generate_pack(revs->repo->objects, &generator, &opts)) {\n +\t\tret = error(_(\"Could not spawn pack-objects\"));\n +\t\tgoto out;\n +\t}\n\n---\nbase-commit: 2c78326f810173a4f3aefd8021f1e07575412481\nchange-id: 20260807-b4-pks-odb-generate-pack-f30fbcdef3fc\n\n"},{"id":"550683","messageId":"20260817-b4-pks-odb-generate-pack-v2-1-4c8a96ccfdb3@pks.im","threadId":"66136","inReplyTo":"20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im","subject":"[PATCH v2 1/6] odb: introduce interface to generate packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-17T05:39:42Z","receivedAt":"2026-08-17T05:39:50Z","isPatch":true,"body":"Packfiles have two primary use cases:\n\n - They are used to store objects at rest in a Git repository.\n\n - They are used on the transport layer to transfer objects between two\n repositories.\n\nThe first class is closely tied to a given object database backend, and\nas such this use is highly specific to how such a backend decides to\nstore its data. This shows in git-pack-objects(1), which is used by\ngit-repack(1) et al to optimize the object database, which supports lots\nof options that are closely coupled with how data is stored.\n\nBut the second class is quite a lot more generic: we don't care about\nspecifics of how the object database stores its objects, but to generate\nthe packfiles we only care about the object graph itself. Still, this\nuse case is also coupled with git-pack-objects(1).\n\nUnfortunately, because git-pack-objects(1) covers both classes, the\nresult is that it is very hard to port the whole command to properly\nsupport pluggable object databases. There are simply way too many\noptions that an alternative implementation will have a very hard time to\nsupport in the first place.\n\nAnd despite being hard to implement, it's also quite unnecessary to\nimplement those backend-specific options. Optimizing the object database\nhas already been made pluggable, and an alternative implementation is\nunlikely to care about cruft packs, unpacked objects, keep packs and the\nlike. But we still need to make at least _parts_ of the packfile\ngeneration pluggable so that backends can generate packfiles for the\ntransport layer itself.\n\nIntroduce a new interface that lets backends generate a new packfile and\nimplement that interface for the \"files\" backend. The options supported\nby the callback are exactly the set of options that are required for the\ntransport layer, but nothing more.\n\nThis means that git-pack-objects(1) itself cannot be ported over to this\nnew interface, but as explained above that's a hard feat to pull off due\nto the backend-specific features. Ideally though, we should expose the\nability to generate arbitrary packfiles using this interface. The intent\nof this is to eventually introduce a git-objects(1) subcommand (similar\nto git-refs(1)) that exposes generic interfaces for accessing everything\nrelated to the object database. In that case, we are able to expose only\nthose options that are generic.\n\nSubsequent commits will convert git-upload-pack(1), git-send-pack(1) and\ngit-bundle(1) to use this interface.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 21 ++++++++\n odb.h | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-files.c | 144 ++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h | 33 ++++++++++++\n 4 files changed, 350 insertions(+)\n\ndiff --git a/odb.c b/odb.c\nindex caf1d0f542..cd9d5b48bc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1046,6 +1046,27 @@ bool odb_optimize_required(struct object_database *odb,\n \treturn odb_source_optimize_required(odb->sources, opts);\n }\n \n+void odb_generate_pack_options_release(struct odb_generate_pack_options *opts)\n+{\n+\toid_array_clear(&opts->wants);\n+\toid_array_clear(&opts->haves);\n+\toid_array_clear(&opts->shallows);\n+}\n+\n+int odb_generate_pack(struct object_database *odb,\n+\t\t struct odb_pack_generator **out,\n+\t\t const struct odb_generate_pack_options *opts)\n+{\n+\tif (!odb->sources->generate_pack)\n+\t\treturn error(_(\"primary object source does not support generating packfiles\"));\n+\treturn odb_source_generate_pack(odb->sources, out, opts);\n+}\n+\n+int odb_pack_generator_finish(struct odb_pack_generator *generator)\n+{\n+\treturn generator->finish(generator);\n+}\n+\n struct object_database *odb_new(struct repository *repo,\n \t\t\t\tconst char *primary_source,\n \t\t\t\tconst char *secondary_sources)\ndiff --git a/odb.h b/odb.h\nindex fca67e8253..fc1442f243 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -2,6 +2,7 @@\n #define ODB_H\n \n #include \"object.h\"\n+#include \"oid-array.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -677,6 +678,157 @@ int odb_write_object_stream(struct object_database *odb,\n \t\t\t struct odb_write_stream *stream, size_t len,\n \t\t\t struct object_id *oid);\n \n+/*\n+ * Options for generating a packfile via `odb_generate_pack()`.\n+ */\n+struct odb_generate_pack_options {\n+\t/* Tips of the object graph that shall be packed. */\n+\tstruct oid_array wants;\n+\n+\t/*\n+\t * Boundary of the object graph. Objects reachable from any of these\n+\t * tips are expected to already be available to whoever consumes the\n+\t * pack and shall thus not be packed.\n+\t */\n+\tstruct oid_array haves;\n+\n+\t/*\n+\t * The shallow boundary that shall be used when computing object\n+\t * reachability. When set, any shallow information of the repository\n+\t * itself shall be ignored in favor of these objects.\n+\t */\n+\tstruct oid_array shallows;\n+\n+\t/*\n+\t * Pre-expanded object filter specification that limits the set of\n+\t * objects that shall be packed. May be `NULL` in case no filter shall\n+\t * be applied.\n+\t */\n+\tconst char *filter_spec;\n+\n+\t/*\n+\t * Protocols that may be used to offload objects via packfile URIs.\n+\t * May be `NULL` in case packfile URIs shall not be used.\n+\t */\n+\tconst struct string_list *uri_protocols;\n+\n+\t/*\n+\t * Hook command that shall be executed instead of the internal\n+\t * machinery to generate the pack. It is up to the specific backend\n+\t * whether or not this hook is supported. May be `NULL` in case no\n+\t * hook shall be executed.\n+\t */\n+\tconst char *pack_objects_hook;\n+\n+\t/*\n+\t * File descriptor that the generated pack shall be written to. If set\n+\t * to `-1`, a pipe will be created and exposed via the pack generator's\n+\t * `out` field. If set to `0`, the pack will be written to the standard\n+\t * output stream. Otherwise, the provided descriptor will be written to\n+\t * and is consumed by the generator.\n+\t */\n+\tint pack_fd;\n+\n+\t/*\n+\t * File descriptor that progress output shall be written to. The same\n+\t * semantics as for `pack_fd` apply, except that `0` will cause the\n+\t * generator to write to stderr instead of stdout.\n+\t */\n+\tint progress_fd;\n+\n+\t/* Whether to print progress or not. */\n+\tenum {\n+\t\t/* Don't print progress output. */\n+\t\tODB_GENERATE_PACK_PROGRESS_NONE,\n+\n+\t\t/*\n+\t\t * Print progress while computing the packfile, but stop\n+\t\t * printing progress once starting to write it.\n+\t\t */\n+\t\tODB_GENERATE_PACK_PROGRESS_STANDARD,\n+\n+\t\t/*\n+\t\t * Similar to STANDARD, but also print progress when writing\n+\t\t * the packfile.\n+\t\t */\n+\t\tODB_GENERATE_PACK_PROGRESS_VERBOSE,\n+\t} progress;\n+\n+\t/* Allow the pack to contain deltas against unpacked objects. */\n+\tunsigned thin:1;\n+\n+\t/* Use offset deltas instead of reference deltas. */\n+\tunsigned ofs_delta:1;\n+\n+\t/* Include unasked-for annotated tags of packed objects. */\n+\tunsigned include_tag:1;\n+\n+\t/* The generated pack is destined for a shallow consumer. */\n+\tunsigned shallow:1;\n+\n+\t/* Allow objects that may be missing due to a promisor remote. */\n+\tunsigned missing_allow_promisor:1;\n+\n+\t/* Do not use bitmap indices when computing reachability. */\n+\tunsigned disable_bitmaps:1;\n+};\n+\n+#define ODB_GENERATE_PACK_OPTIONS_INIT { \\\n+\t.wants = OID_ARRAY_INIT, \\\n+\t.haves = OID_ARRAY_INIT, \\\n+\t.shallows = OID_ARRAY_INIT, \\\n+\t.pack_fd = -1, \\\n+}\n+\n+/* Release resources associated with the options. */\n+void odb_generate_pack_options_release(struct odb_generate_pack_options *opts);\n+\n+/*\n+ * A handle for an ongoing packfile generation as started via\n+ * `odb_generate_pack()`.\n+ */\n+struct odb_pack_generator {\n+\t/*\n+\t * File descriptor from which the generated pack can be read. Only set\n+\t * when the pack generation was started with `pack_fd == -1`. The\n+\t * caller is responsible for closing the descriptor.\n+\t */\n+\tint out;\n+\n+\t/*\n+\t * File descriptor from which progress output can be read. Only set\n+\t * when the pack generation was started with `progress_fd == -1`. The\n+\t * caller is responsible for closing the descriptor.\n+\t */\n+\tint err;\n+\n+\t/*\n+\t * Callback function to finish this generator. This callback is\n+\t * expected to wait for the packfile generation to complete and to then\n+\t * free the generator itself.\n+\t */\n+\tint (*finish)(struct odb_pack_generator *);\n+};\n+\n+/*\n+ * Start generating a packfile from the object database with the given\n+ * options. The pack is generated asynchronously; the caller is expected to\n+ * consume the file descriptors exposed via the pack generator and to then\n+ * wait for completion via `odb_pack_generator_finish()`.\n+ *\n+ * Returns 0 on success and populates the `out` pointer with the pack\n+ * generator. Returns a negative error code otherwise.\n+ */\n+int odb_generate_pack(struct object_database *odb,\n+\t\t struct odb_pack_generator **out,\n+\t\t const struct odb_generate_pack_options *opts);\n+\n+/*\n+ * Wait for the packfile generation to complete and free the pack generator.\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int odb_pack_generator_finish(struct odb_pack_generator *generator);\n+\n void parse_alternates(const char *string,\n \t\t int sep,\n \t\t const char *relative_base,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5a68af7d84..64a0417be7 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -4,6 +4,7 @@\n #include \"chdir-notify.h\"\n #include \"config.h\"\n #include \"gettext.h\"\n+#include \"hex.h\"\n #include \"lockfile.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n@@ -729,6 +730,148 @@ int odb_source_files_optimize(struct odb_source *source,\n \treturn ret;\n }\n \n+struct odb_pack_generator_files {\n+\tstruct odb_pack_generator base;\n+\tstruct child_process cp;\n+};\n+\n+static int odb_pack_generator_files_finish(struct odb_pack_generator *_generator)\n+{\n+\tstruct odb_pack_generator_files *generator =\n+\t\t(struct odb_pack_generator_files *)_generator;\n+\tint ret;\n+\n+\tret = finish_command(&generator->cp);\n+\tfree(generator);\n+\n+\tif (ret) {\n+\t\t/*\n+\t\t * On failure, pack-objects is expected to have written a\n+\t\t * useful error message to its standard error stream already.\n+\t\t * Death by signal is worth mentioning, though, with the\n+\t\t * exception of SIGPIPE: that is a normal occurrence when the\n+\t\t * consumer of the pack hangs up.\n+\t\t */\n+\t\tif (ret > 128 && ret - 128 == SIGPIPE)\n+\t\t\treturn -1;\n+\t\tif (ret > 128)\n+\t\t\terror(_(\"pack-objects died of signal %d\"), ret - 128);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_source_files_generate_pack(struct odb_source *source UNUSED,\n+\t\t\t\t\t struct odb_pack_generator **out,\n+\t\t\t\t\t const struct odb_generate_pack_options *opts)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tstruct odb_pack_generator_files *generator;\n+\tFILE *in;\n+\n+\t/*\n+\t * The hook is expected to spawn \"$hook git pack-objects \"\n+\t * and to behave like git-pack-objects(1) would have. This can for\n+\t * example be used to serve precomputed packfiles.\n+\t */\n+\tif (opts->pack_objects_hook) {\n+\t\tstrvec_push(&cp.args, opts->pack_objects_hook);\n+\t\tstrvec_push(&cp.args, \"git\");\n+\t\tcp.use_shell = 1;\n+\t} else {\n+\t\tcp.git_cmd = 1;\n+\t}\n+\n+\t/*\n+\t * The caller-provided shallow boundary overrides any shallow state\n+\t * that the repository itself may have, so the shallow file needs to\n+\t * be neutralized.\n+\t */\n+\tif (opts->shallows.nr) {\n+\t\tstrvec_push(&cp.args, \"--shallow-file\");\n+\t\tstrvec_push(&cp.args, \"\");\n+\t}\n+\tstrvec_push(&cp.args, \"pack-objects\");\n+\tstrvec_push(&cp.args, \"--revs\");\n+\tstrvec_push(&cp.args, \"--stdout\");\n+\tif (opts->thin)\n+\t\tstrvec_push(&cp.args, \"--thin\");\n+\tif (opts->shallow)\n+\t\tstrvec_push(&cp.args, \"--shallow\");\n+\tif (opts->ofs_delta)\n+\t\tstrvec_push(&cp.args, \"--delta-base-offset\");\n+\tif (opts->include_tag)\n+\t\tstrvec_push(&cp.args, \"--include-tag\");\n+\tif (opts->missing_allow_promisor)\n+\t\tstrvec_push(&cp.args, \"--missing=allow-promisor\");\n+\tif (opts->disable_bitmaps)\n+\t\tstrvec_push(&cp.args, \"--no-use-bitmap-index\");\n+\tswitch (opts->progress) {\n+\tcase ODB_GENERATE_PACK_PROGRESS_NONE:\n+\t\tstrvec_push(&cp.args, \"--quiet\");\n+\t\tbreak;\n+\tcase ODB_GENERATE_PACK_PROGRESS_STANDARD:\n+\t\tstrvec_push(&cp.args, \"--progress\");\n+\t\tbreak;\n+\tcase ODB_GENERATE_PACK_PROGRESS_VERBOSE:\n+\t\tstrvec_push(&cp.args, \"--all-progress\");\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown progress option %d\", opts->progress);\n+\t}\n+\tif (opts->filter_spec)\n+\t\tstrvec_pushf(&cp.args, \"--filter=%s\", opts->filter_spec);\n+\tif (opts->uri_protocols)\n+\t\tfor (size_t i = 0; i < opts->uri_protocols->nr; i++)\n+\t\t\tstrvec_pushf(&cp.args, \"--uri-protocol=%s\",\n+\t\t\t\t opts->uri_protocols->items[i].string);\n+\n+\tcp.in = -1;\n+\tcp.out = opts->pack_fd;\n+\tcp.err = opts->progress_fd;\n+\tcp.clean_on_exit = 1;\n+\n+\tif (start_command(&cp))\n+\t\treturn error(_(\"could not spawn pack-objects\"));\n+\n+\t/*\n+\t * Feed the objects to pack-objects. This is safe to do synchronously\n+\t * because pack-objects consumes all of its standard input before it\n+\t * starts to generate the pack.\n+\t */\n+\tin = xfdopen(cp.in, \"w\");\n+\tfor (size_t i = 0; i < opts->shallows.nr; i++)\n+\t\tfprintf(in, \"--shallow %s\\n\", oid_to_hex(&opts->shallows.oid[i]));\n+\tfor (size_t i = 0; i < opts->wants.nr; i++)\n+\t\tfprintf(in, \"%s\\n\", oid_to_hex(&opts->wants.oid[i]));\n+\tfprintf(in, \"--not\\n\");\n+\tfor (size_t i = 0; i < opts->haves.nr; i++)\n+\t\tfprintf(in, \"%s\\n\", oid_to_hex(&opts->haves.oid[i]));\n+\tfprintf(in, \"\\n\");\n+\tfflush(in);\n+\tif (ferror(in)) {\n+\t\terror(_(\"error writing to pack-objects\"));\n+\t\tfclose(in);\n+\t\tif (opts->pack_fd < 0)\n+\t\t\tclose(cp.out);\n+\t\tif (opts->progress_fd < 0)\n+\t\t\tclose(cp.err);\n+\t\tfinish_command(&cp);\n+\t\treturn -1;\n+\t}\n+\tfclose(in);\n+\n+\tCALLOC_ARRAY(generator, 1);\n+\tgenerator->base.out = opts->pack_fd < 0 ? cp.out : -1;\n+\tgenerator->base.err = opts->progress_fd < 0 ? cp.err : -1;\n+\tgenerator->base.finish = odb_pack_generator_files_finish;\n+\tgenerator->cp = cp;\n+\n+\t*out = &generator->base;\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t bool local)\n@@ -756,6 +899,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.write_alternate = odb_source_files_write_alternate;\n \tfiles->base.optimize = odb_source_files_optimize;\n \tfiles->base.optimize_required = odb_source_files_optimize_required;\n+\tfiles->base.generate_pack = odb_source_files_generate_pack;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex d69f8e2d1c..e2129766fc 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -278,6 +278,23 @@ struct odb_source {\n \t */\n \tbool (*optimize_required)(struct odb_source *source,\n \t\t\t\t const struct odb_optimize_options *opts);\n+\n+\t/*\n+\t * This callback is expected to start generating a packfile with the\n+\t * given options. The pack shall be generated asynchronously so that\n+\t * the caller can consume the pack data and progress output while the\n+\t * pack is being generated.\n+\t *\n+\t * This callback is optional. Sources that cannot generate packfiles\n+\t * shall leave it unset.\n+\t *\n+\t * The callback is expected to return 0 on success and populate the\n+\t * `out` pointer with the pack generator, a negative error code\n+\t * otherwise.\n+\t */\n+\tint (*generate_pack)(struct odb_source *source,\n+\t\t\t struct odb_pack_generator **out,\n+\t\t\t const struct odb_generate_pack_options *opts);\n };\n \n /*\n@@ -520,4 +537,20 @@ static inline bool odb_source_optimize_required(struct odb_source *source,\n \treturn source->optimize_required(source, opts);\n }\n \n+/*\n+ * Start generating a packfile from the given source with the given options.\n+ * The pack is generated asynchronously; the caller is expected to consume the\n+ * file descriptors exposed via the pack generator and to then wait for\n+ * completion via `odb_pack_generator_finish()`.\n+ *\n+ * Returns 0 on success and populates the `out` pointer with the pack\n+ * generator, a negative error code otherwise.\n+ */\n+static inline int odb_source_generate_pack(struct odb_source *source,\n+\t\t\t\t\t struct odb_pack_generator **out,\n+\t\t\t\t\t const struct odb_generate_pack_options *opts)\n+{\n+\treturn source->generate_pack(source, out, opts);\n+}\n+\n #endif\n\n-- \n2.55.0.739.g4f2b995119.dirty\n\n"},{"id":"550684","messageId":"20260817-b4-pks-odb-generate-pack-v2-2-4c8a96ccfdb3@pks.im","threadId":"66136","inReplyTo":"20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im","subject":"[PATCH v2 2/6] upload-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-17T05:39:43Z","receivedAt":"2026-08-17T05:39:53Z","isPatch":true,"body":"When serving a fetch, git-upload-pack(1) spawns git-pack-objects(1)\ndirectly to generate the packfile that gets sent to the client. This\nhard-codes the assumption that the object database is able to serve\npackfiles via git-pack-objects(1), which is specific to the \"files\"\nbackend.\n\nConvert git-upload-pack(1) to instead use the pack generation interface\nof the object database.\n\nSigned-off-by: Patrick Steinhardt \n---\n upload-pack.c | 125 +++++++++++++++++++++-------------------------------------\n 1 file changed, 45 insertions(+), 80 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex a52856d869..75a857eaa8 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -197,11 +197,11 @@ static void send_client_data(int fd, const char *data, ssize_t sz,\n \twrite_or_die(fd, data, sz);\n }\n \n-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n+static int append_one_shallow(const struct commit_graft *graft, void *cb_data)\n {\n-\tFILE *fp = cb_data;\n+\tstruct oid_array *shallows = cb_data;\n \tif (graft->nr_parent == -1)\n-\t\tfprintf(fp, \"--shallow %s\\n\", oid_to_hex(&graft->oid));\n+\t\toid_array_append(shallows, &graft->oid);\n \treturn 0;\n }\n \n@@ -299,7 +299,8 @@ static int relay_pack_data(int pack_objects_out, struct output_state *os,\n static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t\t const struct string_list *uri_protocols)\n {\n-\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n \tstruct output_state *output_state = xcalloc(1, sizeof(struct output_state));\n \tchar progress[128];\n \tchar abort_msg[] = \"aborting due to possible repository \"\n@@ -307,78 +308,42 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \tuint64_t last_sent_ms = 0;\n \tssize_t sz;\n \tint i;\n-\tFILE *pipe_fd;\n-\n-\tif (!pack_data->pack_objects_hook)\n-\t\tpack_objects.git_cmd = 1;\n-\telse {\n-\t\tstrvec_push(&pack_objects.args, pack_data->pack_objects_hook);\n-\t\tstrvec_push(&pack_objects.args, \"git\");\n-\t\tpack_objects.use_shell = 1;\n-\t}\n \n \tif (pack_data->shallow_nr) {\n-\t\tstrvec_push(&pack_objects.args, \"--shallow-file\");\n-\t\tstrvec_push(&pack_objects.args, \"\");\n-\t}\n-\tstrvec_push(&pack_objects.args, \"pack-objects\");\n-\tstrvec_push(&pack_objects.args, \"--revs\");\n-\tif (pack_data->use_thin_pack)\n-\t\tstrvec_push(&pack_objects.args, \"--thin\");\n-\n-\tstrvec_push(&pack_objects.args, \"--stdout\");\n-\tif (pack_data->shallow_nr)\n-\t\tstrvec_push(&pack_objects.args, \"--shallow\");\n-\tif (!pack_data->no_progress)\n-\t\tstrvec_push(&pack_objects.args, \"--progress\");\n-\tif (pack_data->use_ofs_delta)\n-\t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n-\tif (pack_data->use_include_tag)\n-\t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n-\tif (repo_has_accepted_promisor_remote(the_repository))\n-\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n-\tif (pack_data->filter_options.choice) {\n-\t\tconst char *spec =\n-\t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\", spec);\n-\t}\n-\tif (uri_protocols) {\n-\t\tfor (i = 0; i < uri_protocols->nr; i++)\n-\t\t\tstrvec_pushf(&pack_objects.args, \"--uri-protocol=%s\",\n-\t\t\t\t\t uri_protocols->items[i].string);\n+\t\tfor_each_commit_graft(append_one_shallow, &opts.shallows);\n+\t\topts.shallow = 1;\n \t}\n-\n-\tpack_objects.in = -1;\n-\tpack_objects.out = -1;\n-\tpack_objects.err = -1;\n-\tpack_objects.clean_on_exit = 1;\n-\n-\tif (start_command(&pack_objects))\n-\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n-\n-\tpipe_fd = xfdopen(pack_objects.in, \"w\");\n-\n-\tif (pack_data->shallow_nr)\n-\t\tfor_each_commit_graft(write_one_shallow, pipe_fd);\n-\n \tfor (i = 0; i < pack_data->want_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->want_obj.objects[i].item->oid));\n-\tfprintf(pipe_fd, \"--not\\n\");\n+\t\toid_array_append(&opts.wants,\n+\t\t\t\t &pack_data->want_obj.objects[i].item->oid);\n \tfor (i = 0; i < pack_data->have_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->have_obj.objects[i].item->oid));\n+\t\toid_array_append(&opts.haves,\n+\t\t\t\t &pack_data->have_obj.objects[i].item->oid);\n \tfor (i = 0; i < pack_data->extra_edge_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->extra_edge_obj.objects[i].item->oid));\n-\tfprintf(pipe_fd, \"\\n\");\n-\tfflush(pipe_fd);\n-\tfclose(pipe_fd);\n-\n-\t/* We read from pack_objects.err to capture stderr output for\n-\t * progress bar, and pack_objects.out to capture the pack data.\n-\t */\n+\t\toid_array_append(&opts.haves,\n+\t\t\t\t &pack_data->extra_edge_obj.objects[i].item->oid);\n+\n+\topts.thin = pack_data->use_thin_pack;\n+\tif (!pack_data->no_progress)\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_STANDARD;\n+\topts.ofs_delta = pack_data->use_ofs_delta;\n+\topts.include_tag = pack_data->use_include_tag;\n+\topts.missing_allow_promisor = repo_has_accepted_promisor_remote(the_repository);\n+\tif (pack_data->filter_options.choice)\n+\t\topts.filter_spec = expand_list_objects_filter_spec(&pack_data->filter_options);\n+\topts.uri_protocols = uri_protocols;\n+\topts.pack_objects_hook = pack_data->pack_objects_hook;\n+\topts.pack_fd = -1;\n+\topts.progress_fd = -1;\n+\n+\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n+\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n+\todb_generate_pack_options_release(&opts);\n \n+\t/*\n+\t * We read from generator->err to capture stderr output for the\n+\t * progress bar, and generator->out to capture the pack data.\n+\t */\n \twhile (1) {\n \t\tuint64_t now_ms = getnanotime() / 1000000;\n \t\tstruct pollfd pfd[2];\n@@ -393,14 +358,14 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tpollsize = 0;\n \t\tpe = pu = -1;\n \n-\t\tif (0 <= pack_objects.out) {\n-\t\t\tpfd[pollsize].fd = pack_objects.out;\n+\t\tif (0 <= generator->out) {\n+\t\t\tpfd[pollsize].fd = generator->out;\n \t\t\tpfd[pollsize].events = POLLIN;\n \t\t\tpu = pollsize;\n \t\t\tpollsize++;\n \t\t}\n-\t\tif (0 <= pack_objects.err) {\n-\t\t\tpfd[pollsize].fd = pack_objects.err;\n+\t\tif (0 <= generator->err) {\n+\t\t\tpfd[pollsize].fd = generator->err;\n \t\t\tpfd[pollsize].events = POLLIN;\n \t\t\tpe = pollsize;\n \t\t\tpollsize++;\n@@ -437,15 +402,15 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t\t/* Status ready; we ship that in the side-band\n \t\t\t * or dump to the standard error.\n \t\t\t */\n-\t\t\tsz = xread(pack_objects.err, progress,\n+\t\t\tsz = xread(generator->err, progress,\n \t\t\t\t sizeof(progress));\n \t\t\tif (0 < sz) {\n \t\t\t\tsend_client_data(2, progress, sz,\n \t\t\t\t\t\t pack_data->use_sideband);\n \t\t\t\tlast_sent_ms = now_ms;\n \t\t\t} else if (sz == 0) {\n-\t\t\t\tclose(pack_objects.err);\n-\t\t\t\tpack_objects.err = -1;\n+\t\t\t\tclose(generator->err);\n+\t\t\t\tgenerator->err = -1;\n \t\t\t}\n \t\t\telse\n \t\t\t\tgoto fail;\n@@ -455,15 +420,15 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \n \t\tif (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) {\n \t\t\tbool did_send_data;\n-\t\t\tint result = relay_pack_data(pack_objects.out,\n+\t\t\tint result = relay_pack_data(generator->out,\n \t\t\t\t\t\t output_state,\n \t\t\t\t\t\t pack_data->use_sideband,\n \t\t\t\t\t\t !!uri_protocols,\n \t\t\t\t\t\t &did_send_data);\n \n \t\t\tif (result == 0) {\n-\t\t\t\tclose(pack_objects.out);\n-\t\t\t\tpack_objects.out = -1;\n+\t\t\t\tclose(generator->out);\n+\t\t\t\tgenerator->out = -1;\n \t\t\t} else if (result < 0) {\n \t\t\t\tgoto fail;\n \t\t\t}\n@@ -498,7 +463,7 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t}\n \t}\n \n-\tif (finish_command(&pack_objects)) {\n+\tif (odb_pack_generator_finish(generator)) {\n \t\terror(\"git upload-pack: git-pack-objects died with error.\");\n \t\tgoto fail;\n \t}\n\n-- \n2.55.0.739.g4f2b995119.dirty\n\n"},{"id":"550685","messageId":"20260817-b4-pks-odb-generate-pack-v2-3-4c8a96ccfdb3@pks.im","threadId":"66136","inReplyTo":"20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im","subject":"[PATCH v2 3/6] send-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-17T05:39:44Z","receivedAt":"2026-08-17T05:39:57Z","isPatch":true,"body":"When pushing, git-send-pack(1) spawns git-pack-objects(1) directly to\ngenerate the packfile that gets sent to the remote. Same as with\ngit-upload-pack(1), which has been adapted in the preceding commit,\nthis hard-codes the assumption that objects can be packed via\ngit-pack-objects(1), which is specific to the \"files\" backend.\n\nConvert git-send-pack(1) to use the pack generation interface of the\nobject database instead.\n\nNote that this requires us to adapt t5516 because the parameters passed\nto git-pack-objects(1) are changing:\n\n - The order of arguments changes.\n\n - We pass \"--quiet\" instead of \"-q\".\n\n - We don't pass \"--all-progress-implied\" anymore when not generating\n output.\n\nAll of these changes are benign though and should not result in a change\nin behaviour.\n\nSigned-off-by: Patrick Steinhardt \n---\n send-pack.c | 101 +++++++++++++++++---------------------------------\n t/t5516-fetch-push.sh | 12 +++---\n 2 files changed, 40 insertions(+), 73 deletions(-)\n\ndiff --git a/send-pack.c b/send-pack.c\nindex 3bb5afc687..f20460fbf4 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -42,16 +42,17 @@ int option_parse_push_signed(const struct option *opt,\n \tdie(\"bad %s argument: %s\", opt->long_name, arg);\n }\n \n-static void feed_object(struct repository *r,\n-\t\t\tconst struct object_id *oid, FILE *fh, int negative)\n+static void append_negative_object(struct repository *r,\n+\t\t\t\t struct oid_array *haves,\n+\t\t\t\t const struct object_id *oid)\n {\n-\tif (negative && !odb_has_object(r->objects, oid, 0))\n+\t/*\n+\t * The remote end may have advertised objects that we do not have in\n+\t * our object database. Skip those, as we cannot use them as boundary.\n+\t */\n+\tif (!odb_has_object(r->objects, oid, 0))\n \t\treturn;\n-\n-\tif (negative)\n-\t\tputc('^', fh);\n-\tfputs(oid_to_hex(oid), fh);\n-\tputc('\\n', fh);\n+\toid_array_append(haves, oid);\n }\n \n /*\n@@ -62,92 +63,58 @@ static int pack_objects(struct repository *r,\n \t\t\tstruct oid_array *negotiated,\n \t\t\tstruct send_pack_args *args)\n {\n-\t/*\n-\t * The child becomes pack-objects --revs; we feed\n-\t * the revision parameters to it via its stdin and\n-\t * let its stdout go back to the other end.\n-\t */\n-\tstruct child_process po = CHILD_PROCESS_INIT;\n-\tFILE *po_in;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n \tint rc;\n \n \ttrace2_region_enter(\"send_pack\", \"pack_objects\", r);\n-\tstrvec_push(&po.args, \"pack-objects\");\n-\tstrvec_push(&po.args, \"--all-progress-implied\");\n-\tstrvec_push(&po.args, \"--revs\");\n-\tstrvec_push(&po.args, \"--stdout\");\n-\tif (args->use_thin_pack)\n-\t\tstrvec_push(&po.args, \"--thin\");\n-\tif (args->use_ofs_delta)\n-\t\tstrvec_push(&po.args, \"--delta-base-offset\");\n-\tif (args->quiet || !args->progress)\n-\t\tstrvec_push(&po.args, \"-q\");\n+\n+\topts.thin = args->use_thin_pack;\n+\topts.ofs_delta = args->use_ofs_delta;\n \tif (args->progress)\n-\t\tstrvec_push(&po.args, \"--progress\");\n-\tif (is_repository_shallow(r))\n-\t\tstrvec_push(&po.args, \"--shallow\");\n-\tif (args->disable_bitmaps)\n-\t\tstrvec_push(&po.args, \"--no-use-bitmap-index\");\n-\tpo.in = -1;\n-\tpo.out = args->stateless_rpc ? -1 : fd;\n-\tpo.git_cmd = 1;\n-\tpo.clean_on_exit = 1;\n-\tif (start_command(&po))\n-\t\tdie_errno(\"git pack-objects failed\");\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n+\topts.shallow = is_repository_shallow(r);\n+\topts.disable_bitmaps = args->disable_bitmaps;\n \n \t/*\n-\t * We feed the pack-objects we just spawned with revision\n-\t * parameters by writing to the pipe.\n+\t * The pack is either written directly to the remote's descriptor, or,\n+\t * in the case of a stateless RPC, read back from a pipe so that we\n+\t * can wrap the pack data into pkt-lines.\n \t */\n-\tpo_in = xfdopen(po.in, \"w\");\n+\topts.pack_fd = args->stateless_rpc ? -1 : fd;\n+\n \tfor (size_t i = 0; i < advertised->nr; i++)\n-\t\tfeed_object(r, &advertised->oid[i], po_in, 1);\n+\t\tappend_negative_object(r, &opts.haves, &advertised->oid[i]);\n \tfor (size_t i = 0; i < negotiated->nr; i++)\n-\t\tfeed_object(r, &negotiated->oid[i], po_in, 1);\n+\t\tappend_negative_object(r, &opts.haves, &negotiated->oid[i]);\n \n \twhile (refs) {\n \t\tif (!is_null_oid(&refs->old_oid))\n-\t\t\tfeed_object(r, &refs->old_oid, po_in, 1);\n+\t\t\tappend_negative_object(r, &opts.haves, &refs->old_oid);\n \t\tif (!is_null_oid(&refs->new_oid))\n-\t\t\tfeed_object(r, &refs->new_oid, po_in, 0);\n+\t\t\toid_array_append(&opts.wants, &refs->new_oid);\n \t\trefs = refs->next;\n \t}\n \n-\tfflush(po_in);\n-\tif (ferror(po_in))\n-\t\tdie_errno(\"error writing to pack-objects\");\n-\tfclose(po_in);\n+\tif (odb_generate_pack(r->objects, &generator, &opts))\n+\t\tdie(\"git pack-objects failed\");\n+\todb_generate_pack_options_release(&opts);\n \n \tif (args->stateless_rpc) {\n \t\tchar *buf = xmalloc(LARGE_PACKET_MAX);\n \t\twhile (1) {\n-\t\t\tssize_t n = xread(po.out, buf, LARGE_PACKET_MAX);\n+\t\t\tssize_t n = xread(generator->out, buf, LARGE_PACKET_MAX);\n \t\t\tif (n <= 0)\n \t\t\t\tbreak;\n \t\t\tsend_sideband(fd, -1, buf, n, LARGE_PACKET_MAX);\n \t\t}\n \t\tfree(buf);\n-\t\tclose(po.out);\n-\t\tpo.out = -1;\n+\t\tclose(generator->out);\n \t}\n \n-\trc = finish_command(&po);\n-\tif (rc) {\n-\t\t/*\n-\t\t * For a normal non-zero exit, we assume pack-objects wrote\n-\t\t * something useful to stderr. For death by signal, though,\n-\t\t * we should mention it to the user. The exception is SIGPIPE\n-\t\t * (141), because that's a normal occurrence if the remote end\n-\t\t * hangs up (and we'll report that by trying to read the unpack\n-\t\t * status).\n-\t\t */\n-\t\tif (rc > 128 && rc != 141)\n-\t\t\terror(\"pack-objects died of signal %d\", rc - 128);\n-\t\ttrace2_region_leave(\"send_pack\", \"pack_objects\", r);\n-\t\treturn -1;\n-\t}\n+\trc = odb_pack_generator_finish(generator);\n \ttrace2_region_leave(\"send_pack\", \"pack_objects\", r);\n-\treturn 0;\n+\treturn rc;\n }\n \n static int receive_unpack_status(struct packet_reader *reader)\n@@ -768,7 +735,7 @@ int send_pack(struct repository *r,\n \t\t\tgoto out;\n \t\t}\n \t\tif (!args->stateless_rpc)\n-\t\t\t/* Closed by pack_objects() via start_command() */\n+\t\t\t/* Consumed by the pack generator in pack_objects() */\n \t\t\tfd[1] = -1;\n \t}\n \tif (args->stateless_rpc && cmds_sent)\ndiff --git a/t/t5516-fetch-push.sh b/t/t5516-fetch-push.sh\nindex f3b3efc47f..b982b209bf 100755\n--- a/t/t5516-fetch-push.sh\n+++ b/t/t5516-fetch-push.sh\n@@ -1903,20 +1903,20 @@ test_expect_success 'push with config push.useBitmaps' '\n \ttest_unconfig push.useBitmaps &&\n \tGIT_TRACE2_EVENT=\"$PWD/default\" \\\n \tgit push --quiet testrepo main:test &&\n-\ttest_subcommand git pack-objects --all-progress-implied --revs --stdout \\\n-\t\t--thin --delta-base-offset -q \n---\n builtin/bundle.c | 33 ++++++++++++++++-----------------\n 1 file changed, 16 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 1e170e9278..bfafadc984 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -70,35 +70,34 @@ static int parse_options_cmd_bundle(int argc,\n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\t struct repository *repo UNUSED) {\n \tstruct strvec pack_opts = STRVEC_INIT;\n+\tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n-\tint ret;\n \tstruct option options[] = {\n-\t\tOPT_PASSTHRU_ARGV('q', \"quiet\", &pack_opts, NULL,\n-\t\t\t\t N_(\"do not show progress meter\"),\n-\t\t\t\t PARSE_OPT_NOARG),\n-\t\tOPT_PASSTHRU_ARGV(0, \"progress\", &pack_opts, NULL,\n-\t\t\t\t N_(\"show progress meter\"),\n-\t\t\t\t PARSE_OPT_NOARG),\n-\t\tOPT_PASSTHRU_ARGV(0, \"all-progress\", &pack_opts, NULL,\n-\t\t\t\t N_(\"historical; same as --progress\"),\n-\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n-\t\tOPT_PASSTHRU_ARGV(0, \"all-progress-implied\", &pack_opts, NULL,\n-\t\t\t\t N_(\"historical; does nothing\"),\n-\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n+\t\tOPT_NEGBIT('q', \"quiet\", &progress,\n+\t\t\t N_(\"do not show progress meter\"), 1),\n+\t\tOPT_BIT(0, \"progress\", &progress,\n+\t\t\tN_(\"show progress meter\"), 1),\n+\t\tOPT_BIT_F(0, \"all-progress\", &progress,\n+\t\t\t N_(\"historical; same as --progress\"), 1,\n+\t\t\t PARSE_OPT_HIDDEN),\n+\t\tOPT_NOOP_NOARG(0, \"all-progress-implied\"),\n \t\tOPT_INTEGER(0, \"version\", &version,\n \t\t\t N_(\"specify bundle format version\")),\n \t\tOPT_END()\n \t};\n \tchar *bundle_file;\n-\n-\tif (isatty(STDERR_FILENO))\n-\t\tstrvec_push(&pack_opts, \"--progress\");\n-\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n+\tint ret;\n \n \targc = parse_options_cmd_bundle(argc, argv, prefix,\n \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n \t/* bundle internals use argv[1] as further parameters */\n \n+\tif (progress)\n+\t\tstrvec_push(&pack_opts, \"--progress\");\n+\telse\n+\t\tstrvec_push(&pack_opts, \"--quiet\");\n+\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n+\n \tif (!startup_info->have_repository)\n \t\tdie(_(\"Need a repository to create a bundle.\"));\n \tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n\n-- \n2.55.0.739.g4f2b995119.dirty\n\n"},{"id":"550687","messageId":"20260817-b4-pks-odb-generate-pack-v2-5-4c8a96ccfdb3@pks.im","threadId":"66136","inReplyTo":"20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im","subject":"[PATCH v2 5/6] bundle: get (mostly) rid of `the_repository`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-17T05:39:46Z","receivedAt":"2026-08-17T05:40:03Z","isPatch":true,"body":"Refactor \"bundle.c\" so that we don't depend on `the_repository` anymore.\nThis conversion is trivial for most of the part, as we already have a\nrepository available in all calling conexts.\n\nThe only exception is that we use `get_log_output_encoding()`, which\nimplicitly depends on `the_repository`. Add an `extern` declaration for\nthis function so that we can drop `USE_THE_REPOSITORY_VARIABLE` and not\naccidentally introduce more uses of `the_repository`.\n\nSigned-off-by: Patrick Steinhardt \n---\n bundle.c | 32 +++++++++++++++++++++-----------\n 1 file changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex b64716f252..a9330bf0d3 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -1,4 +1,3 @@\n-#define USE_THE_REPOSITORY_VARIABLE\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"git-compat-util.h\"\n@@ -21,6 +20,13 @@\n #include \"connected.h\"\n #include \"write-or-die.h\"\n \n+/*\n+ * NEEDSWORK: this function implicitly depends on `the_repository` and is not\n+ * available because we dropped USE_THE_REPOSITORY_VARIABLE. We can remove the\n+ * declaration once it's accessible via `repo_config_values`.\n+ */\n+extern const char *get_log_output_encoding(void);\n+\n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n static struct {\n@@ -294,7 +300,8 @@ int list_bundle_refs(struct bundle_header *header, int argc, const char **argv)\n \treturn list_refs(&header->references, argc, argv);\n }\n \n-static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n+static int is_tag_in_date_range(struct repository *repo,\n+\t\t\t\tstruct object *tag, struct rev_info *revs)\n {\n \tsize_t size;\n \tenum object_type type;\n@@ -305,7 +312,7 @@ static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n \tif (revs->max_age == -1 && revs->min_age == -1)\n \t\tgoto out;\n \n-\tbuf = odb_read_object(the_repository->objects, &tag->oid, &type, &size);\n+\tbuf = odb_read_object(repo->objects, &tag->oid, &type, &size);\n \tif (!buf)\n \t\tgoto out;\n \tline = memmem(buf, size, \"\\ntagger \", 8);\n@@ -362,7 +369,8 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\tstruct object *object = revs->pending.objects[i].item;\n \t\tif (object->flags & UNINTERESTING)\n \t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n-\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid), the_hash_algo->hexsz);\n+\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid),\n+\t\t\t revs->repo->hash_algo->hexsz);\n \t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n \t}\n \tclose(pack_objects.in);\n@@ -395,10 +403,10 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \n \t\tif (e->item->flags & UNINTERESTING)\n \t\t\tcontinue;\n-\t\tif (repo_dwim_ref(the_repository, e->name, strlen(e->name),\n+\t\tif (repo_dwim_ref(revs->repo, e->name, strlen(e->name),\n \t\t\t\t &oid, &ref, 0) != 1)\n \t\t\tgoto skip_write_ref;\n-\t\tif (refs_read_ref_full(get_main_ref_store(the_repository), e->name, RESOLVE_REF_READING, &oid, &flag))\n+\t\tif (refs_read_ref_full(get_main_ref_store(revs->repo), e->name, RESOLVE_REF_READING, &oid, &flag))\n \t\t\tflag = 0;\n \t\tdisplay_ref = (flag & REF_ISSYMREF) ? e->name : ref;\n \n@@ -406,7 +414,7 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \t\t\tgoto skip_write_ref;\n \n \t\tif (e->item->type == OBJ_TAG &&\n-\t\t\t\t!is_tag_in_date_range(e->item, revs)) {\n+\t\t\t\t!is_tag_in_date_range(revs->repo, e->item, revs)) {\n \t\t\te->item->flags |= UNINTERESTING;\n \t\t\tgoto skip_write_ref;\n \t\t}\n@@ -428,7 +436,8 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \n \t\tref_count++;\n \t\tstrset_add(&objects, display_ref);\n-\t\twrite_or_die(bundle_fd, oid_to_hex(&e->item->oid), the_hash_algo->hexsz);\n+\t\twrite_or_die(bundle_fd, oid_to_hex(&e->item->oid),\n+\t\t\t revs->repo->hash_algo->hexsz);\n \t\twrite_or_die(bundle_fd, \" \", 1);\n \t\twrite_or_die(bundle_fd, display_ref, strlen(display_ref));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n@@ -507,7 +516,7 @@ int create_bundle(struct repository *r, const char *path,\n \t * SHA1.\n \t * 2. @filter is required because we parsed an object filter.\n \t */\n-\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1_LEGACY] || revs.filter.choice)\n+\tif (r->hash_algo != &hash_algos[GIT_HASH_SHA1_LEGACY] || revs.filter.choice)\n \t\tmin_version = 3;\n \n \tif (argc > 1) {\n@@ -528,14 +537,15 @@ int create_bundle(struct repository *r, const char *path,\n \tif (version < 2 || version > 3) {\n \t\tdie(_(\"unsupported bundle version %d\"), version);\n \t} else if (version < min_version) {\n-\t\tdie(_(\"cannot write bundle version %d with algorithm %s\"), version, the_hash_algo->name);\n+\t\tdie(_(\"cannot write bundle version %d with algorithm %s\"), version,\n+\t\t r->hash_algo->name);\n \t} else if (version == 2) {\n \t\twrite_or_die(bundle_fd, v2_bundle_signature, strlen(v2_bundle_signature));\n \t} else {\n \t\tconst char *capability = \"@object-format=\";\n \t\twrite_or_die(bundle_fd, v3_bundle_signature, strlen(v3_bundle_signature));\n \t\twrite_or_die(bundle_fd, capability, strlen(capability));\n-\t\twrite_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));\n+\t\twrite_or_die(bundle_fd, r->hash_algo->name, strlen(r->hash_algo->name));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n \n \t\tif (revs.filter.choice) {\n\n-- \n2.55.0.739.g4f2b995119.dirty\n\n"},{"id":"550688","messageId":"20260817-b4-pks-odb-generate-pack-v2-6-4c8a96ccfdb3@pks.im","threadId":"66136","inReplyTo":"20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im","subject":"[PATCH v2 6/6] bundle: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-17T05:39:47Z","receivedAt":"2026-08-17T05:40:06Z","isPatch":true,"body":"git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\ndata that gets appended to the bundle header. While bundles are not\npart of the wire protocol, they are a transfer mechanism for packs all\nthe same, so convert them to use the pack generation interface of the\nobject database as well.\n\nThis makes the pack generator the single spawn point for all pack\nstreams that leave the repository, leaving only local maintenance tasks\nlike git-repack(1) with direct knowledge of git-pack-objects(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/bundle.c | 10 +-------\n bundle.c | 69 ++++++++++++++++++++++++++++----------------------------\n bundle.h | 3 +--\n 3 files changed, 37 insertions(+), 45 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex bfafadc984..de86e092a6 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -69,7 +69,6 @@ static int parse_options_cmd_bundle(int argc,\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\t struct repository *repo UNUSED) {\n-\tstruct strvec pack_opts = STRVEC_INIT;\n \tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n \tstruct option options[] = {\n@@ -92,16 +91,9 @@ static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n \t/* bundle internals use argv[1] as further parameters */\n \n-\tif (progress)\n-\t\tstrvec_push(&pack_opts, \"--progress\");\n-\telse\n-\t\tstrvec_push(&pack_opts, \"--quiet\");\n-\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n-\n \tif (!startup_info->have_repository)\n \t\tdie(_(\"Need a repository to create a bundle.\"));\n-\tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n-\tstrvec_clear(&pack_opts);\n+\tret = !!create_bundle(the_repository, bundle_file, argc, argv, version, progress);\n \tfree(bundle_file);\n \treturn ret;\n }\ndiff --git a/bundle.c b/bundle.c\nindex a9330bf0d3..f55a521b2a 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -332,51 +332,52 @@ static int is_tag_in_date_range(struct repository *repo,\n \n \n /* Write the pack data to bundle_fd */\n-static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *pack_options)\n+static int write_pack_data(int bundle_fd, struct rev_info *revs, int progress)\n {\n-\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n+\tint ret = 0;\n \tint i;\n \n-\tstrvec_pushl(&pack_objects.args,\n-\t\t \"pack-objects\",\n-\t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n-\t\t NULL);\n-\tstrvec_pushv(&pack_objects.args, pack_options->v);\n+\topts.thin = 1;\n+\topts.ofs_delta = 1;\n+\tif (progress)\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n \tif (revs->filter.choice)\n-\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t list_objects_filter_spec(&revs->filter));\n-\tpack_objects.in = -1;\n-\tpack_objects.out = bundle_fd;\n-\tpack_objects.git_cmd = 1;\n+\t\topts.filter_spec = list_objects_filter_spec(&revs->filter);\n \n \t/*\n-\t * start_command() will close our descriptor if it's >1. Duplicate it\n-\t * to avoid surprising the caller.\n+\t * The pack generator will consume our descriptor if it's >1.\n+\t * Duplicate it to avoid surprising the caller.\n \t */\n-\tif (pack_objects.out > 1) {\n-\t\tpack_objects.out = dup(pack_objects.out);\n-\t\tif (pack_objects.out < 0) {\n-\t\t\terror_errno(_(\"unable to dup bundle descriptor\"));\n-\t\t\tchild_process_clear(&pack_objects);\n-\t\t\treturn -1;\n-\t\t}\n+\topts.pack_fd = bundle_fd;\n+\tif (opts.pack_fd > 1) {\n+\t\topts.pack_fd = dup(bundle_fd);\n+\t\tif (opts.pack_fd < 0)\n+\t\t\treturn error_errno(_(\"unable to dup bundle descriptor\"));\n \t}\n \n-\tif (start_command(&pack_objects))\n-\t\treturn error(_(\"Could not spawn pack-objects\"));\n-\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object *object = revs->pending.objects[i].item;\n \t\tif (object->flags & UNINTERESTING)\n-\t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n-\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid),\n-\t\t\t revs->repo->hash_algo->hexsz);\n-\t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n+\t\t\toid_array_append(&opts.haves, &object->oid);\n+\t\telse\n+\t\t\toid_array_append(&opts.wants, &object->oid);\n \t}\n-\tclose(pack_objects.in);\n-\tif (finish_command(&pack_objects))\n-\t\treturn error(_(\"pack-objects died\"));\n-\treturn 0;\n+\n+\tif (odb_generate_pack(revs->repo->objects, &generator, &opts)) {\n+\t\tret = error(_(\"Could not spawn pack-objects\"));\n+\t\tgoto out;\n+\t}\n+\n+\tif (odb_pack_generator_finish(generator)) {\n+\t\tret = error(_(\"pack-objects died\"));\n+\t\tgoto out;\n+\t}\n+\n+out:\n+\todb_generate_pack_options_release(&opts);\n+\treturn ret;\n }\n \n /*\n@@ -485,7 +486,7 @@ static void write_bundle_prerequisites(struct commit *commit, void *data)\n }\n \n int create_bundle(struct repository *r, const char *path,\n-\t\t int argc, const char **argv, struct strvec *pack_options, int version)\n+\t\t int argc, const char **argv, int version, int progress)\n {\n \tstruct lock_file lock = LOCK_INIT;\n \tint bundle_fd = -1;\n@@ -594,7 +595,7 @@ int create_bundle(struct repository *r, const char *path,\n \t}\n \n \t/* write pack */\n-\tif (write_pack_data(bundle_fd, &revs_copy, pack_options)) {\n+\tif (write_pack_data(bundle_fd, &revs_copy, progress)) {\n \t\tret = -1;\n \t\tgoto out;\n \t}\ndiff --git a/bundle.h b/bundle.h\nindex d664b2f2d6..471da23d1b 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -27,8 +27,7 @@ int read_bundle_header(const char *path, struct bundle_header *header);\n int read_bundle_header_fd(int fd, struct bundle_header *header,\n \t\t\t const char *report_path);\n int create_bundle(struct repository *r, const char *path,\n-\t\t int argc, const char **argv, struct strvec *pack_options,\n-\t\t int version);\n+\t\t int argc, const char **argv, int version, int progress);\n \n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n\n-- \n2.55.0.739.g4f2b995119.dirty\n\n"},{"id":"550709","messageId":"xmqqik5866di.fsf@gitster.g","threadId":"66136","inReplyTo":"20260817-b4-pks-odb-generate-pack-v2-5-4c8a96ccfdb3@pks.im","subject":"Re: [PATCH v2 5/6] bundle: get (mostly) rid of `the_repository`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-08-17T16:47:53Z","receivedAt":"2026-08-17T16:47:56Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> Refactor \"bundle.c\" so that we don't depend on `the_repository` anymore.\n> This conversion is trivial for most of the part, as we already have a\n> repository available in all calling conexts.\n>\n> The only exception is that we use `get_log_output_encoding()`, which\n> implicitly depends on `the_repository`. Add an `extern` declaration for\n> this function so that we can drop `USE_THE_REPOSITORY_VARIABLE` and not\n> accidentally introduce more uses of `the_repository`.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> bundle.c | 32 +++++++++++++++++++++-----------\n> 1 file changed, 21 insertions(+), 11 deletions(-)\n>\n> diff --git a/bundle.c b/bundle.c\n> index b64716f252..a9330bf0d3 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -1,4 +1,3 @@\n> -#define USE_THE_REPOSITORY_VARIABLE\n> #define DISABLE_SIGN_COMPARE_WARNINGS\n> \n> #include \"git-compat-util.h\"\n> @@ -21,6 +20,13 @@\n> #include \"connected.h\"\n> #include \"write-or-die.h\"\n> \n> +/*\n> + * NEEDSWORK: this function implicitly depends on `the_repository` and is not\n> + * available because we dropped USE_THE_REPOSITORY_VARIABLE. We can remove the\n> + * declaration once it's accessible via `repo_config_values`.\n> + */\n> +extern const char *get_log_output_encoding(void);\n> +\n\nDoesn't this defeat the whole \"drop #define USE_THE_REPOSITORY_VARIABLE\nas a mark that we are done with this file and no longer need to\nworry about it going forward because we won't be able to compile if\nsomebody adds a new use?\" premise? \n\nWe want to omit the above two hunks, even though the rest of the\npatch look perfectly good.\n\nThanks.\n\n"},{"id":"550732","messageId":"aoPtDyISRa0mVXRa@pks.im","threadId":"66136","inReplyTo":"xmqqik5866di.fsf@gitster.g","subject":"Re: [PATCH v2 5/6] bundle: get (mostly) rid of `the_repository`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-18T05:26:39Z","receivedAt":"2026-08-18T05:26:47Z","isPatch":true,"body":"On Mon, Aug 17, 2026 at 09:47:53AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > Refactor \"bundle.c\" so that we don't depend on `the_repository` anymore.\n> > This conversion is trivial for most of the part, as we already have a\n> > repository available in all calling conexts.\n> >\n> > The only exception is that we use `get_log_output_encoding()`, which\n> > implicitly depends on `the_repository`. Add an `extern` declaration for\n> > this function so that we can drop `USE_THE_REPOSITORY_VARIABLE` and not\n> > accidentally introduce more uses of `the_repository`.\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > bundle.c | 32 +++++++++++++++++++++-----------\n> > 1 file changed, 21 insertions(+), 11 deletions(-)\n> >\n> > diff --git a/bundle.c b/bundle.c\n> > index b64716f252..a9330bf0d3 100644\n> > --- a/bundle.c\n> > +++ b/bundle.c\n> > @@ -1,4 +1,3 @@\n> > -#define USE_THE_REPOSITORY_VARIABLE\n> > #define DISABLE_SIGN_COMPARE_WARNINGS\n> > \n> > #include \"git-compat-util.h\"\n> > @@ -21,6 +20,13 @@\n> > #include \"connected.h\"\n> > #include \"write-or-die.h\"\n> > \n> > +/*\n> > + * NEEDSWORK: this function implicitly depends on `the_repository` and is not\n> > + * available because we dropped USE_THE_REPOSITORY_VARIABLE. We can remove the\n> > + * declaration once it's accessible via `repo_config_values`.\n> > + */\n> > +extern const char *get_log_output_encoding(void);\n> > +\n> \n> Doesn't this defeat the whole \"drop #define USE_THE_REPOSITORY_VARIABLE\n> as a mark that we are done with this file and no longer need to\n> worry about it going forward because we won't be able to compile if\n> somebody adds a new use?\" premise?\n\nYes and no. By removing the define early it allows us to not reintroduce\nnew references to `the_repository` by accident, but carve out a single\nexception for one of the functions that still depends on it. The\nalternative would be to not do that, and if so there is no guarantee\nwhatsoever that we won't introduce more references to `the_repository`\nin this file.\n\nSo I'm still leaning towards keeping this as-is, but I don't feel very\nstrongly about this. Let me know in case that argument doesn't sway you\nand I'll adapt.\n\nThanks!\n\nPatrick\n"},{"id":"550831","messageId":"CABPp-BG3_xvbXtt5BucyOy-dHXqX569d4FBfyZwbLiAb-qRPXA@mail.gmail.com","threadId":"66136","inReplyTo":"20260817-b4-pks-odb-generate-pack-v2-1-4c8a96ccfdb3@pks.im","subject":"Re: [PATCH v2 1/6] odb: introduce interface to generate packfiles","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-08-19T16:56:56Z","receivedAt":"2026-08-19T16:57:08Z","isPatch":true,"body":"On Sun, Aug 16, 2026 at 10:40 PM Patrick Steinhardt wrote:\n>\n> +static int odb_source_files_generate_pack(struct odb_source *source UNUSED,\n> + struct odb_pack_generator **out,\n> + const struct odb_generate_pack_options *opts)\n> +{\n> + struct child_process cp = CHILD_PROCESS_INIT;\n> + struct odb_pack_generator_files *generator;\n> + FILE *in;\n[...]\n> + cp.clean_on_exit = 1;\n> +\n> + if (start_command(&cp))\n> + return error(_(\"could not spawn pack-objects\"));\n[...]\n> + CALLOC_ARRAY(generator, 1);\n> + generator->base.out = opts->pack_fd < 0 ? cp.out : -1;\n> + generator->base.err = opts->progress_fd < 0 ? cp.err : -1;\n> + generator->base.finish = odb_pack_generator_files_finish;\n> + generator->cp = cp;\n> +\n> + *out = &generator->base;\n> + return 0;\n> +}\n\nDoes this have a use-after-scope bug lurking here, due to the\ncombination of clean_on_exit = 1 (which makes a copy of &cp for later\nuse), and the fact that cp is a function-local? If I'm reading the\ncode right, start_command() calls mark_child_for_cleanup(), which does\n\n p->process = process; /* where process is &cp */\n\nand then cleanup_children() accesses various fields under p->process.\nYou do copy the necessary fields from cp to generator->cp, but\n&generator->cp was not passed to start_command(), so p->process points\nto the function-local cp.\n\nI think the normal teardown path happens to be fine despite this\nissue: when odb_pack_generator_files_finish() calls\nfinish_command(&generator->cp), it clears the child by matching pid\n(which was copied separately from p->process), so the stale pointer\nnever gets dereferenced in the successful path. But with an\nabnormal-exit, which is where clean_on_exit comes into play, then\ncleanup_children() will be called and start attempting to read\np->process, which now points to some long-reclaimed function stack\nspace.\n"},{"id":"550843","messageId":"aoYkfl3Q2_8bmijh@denethor","threadId":"66136","inReplyTo":"20260817-b4-pks-odb-generate-pack-v2-6-4c8a96ccfdb3@pks.im","subject":"Re: [PATCH v2 6/6] bundle: generate packfiles via the object database","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-19T21:52:05Z","receivedAt":"2026-08-19T21:52:10Z","isPatch":true,"body":"On 26/08/17 07:39AM, Patrick Steinhardt wrote:\n> git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\n> data that gets appended to the bundle header. While bundles are not\n> part of the wire protocol, they are a transfer mechanism for packs all\n> the same, so convert them to use the pack generation interface of the\n> object database as well.\n\nJust to clarify, so the intent here is that git-bundle(1) can be used\none a repository using a different ODB backend and still generate a\nbundle correct? The bundle would still ultimately use a packfile as the\ncommon language format though.\n\n-Justin\n"},{"id":"550865","messageId":"aoaYKPoPRwobCLGZ@pks.im","threadId":"66136","inReplyTo":"aoYkfl3Q2_8bmijh@denethor","subject":"Re: [PATCH v2 6/6] bundle: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T06:01:12Z","receivedAt":"2026-08-20T06:01:18Z","isPatch":true,"body":"On Wed, Aug 19, 2026 at 04:52:05PM -0500, Justin Tobler wrote:\n> On 26/08/17 07:39AM, Patrick Steinhardt wrote:\n> > git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\n> > data that gets appended to the bundle header. While bundles are not\n> > part of the wire protocol, they are a transfer mechanism for packs all\n> > the same, so convert them to use the pack generation interface of the\n> > object database as well.\n> \n> Just to clarify, so the intent here is that git-bundle(1) can be used\n> one a repository using a different ODB backend and still generate a\n> bundle correct? The bundle would still ultimately use a packfile as the\n> common language format though.\n\nYes, exactly. The bundle format would of course remain completely unchanged.\n\nPatrick\n"},{"id":"550866","messageId":"aoaYL_BinFtgdJ5N@pks.im","threadId":"66136","inReplyTo":"CABPp-BG3_xvbXtt5BucyOy-dHXqX569d4FBfyZwbLiAb-qRPXA@mail.gmail.com","subject":"Re: [PATCH v2 1/6] odb: introduce interface to generate packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T06:01:19Z","receivedAt":"2026-08-20T06:01:24Z","isPatch":true,"body":"On Wed, Aug 19, 2026 at 09:56:56AM -0700, Elijah Newren wrote:\n> On Sun, Aug 16, 2026 at 10:40 PM Patrick Steinhardt wrote:\n> >\n> > +static int odb_source_files_generate_pack(struct odb_source *source UNUSED,\n> > + struct odb_pack_generator **out,\n> > + const struct odb_generate_pack_options *opts)\n> > +{\n> > + struct child_process cp = CHILD_PROCESS_INIT;\n> > + struct odb_pack_generator_files *generator;\n> > + FILE *in;\n> [...]\n> > + cp.clean_on_exit = 1;\n> > +\n> > + if (start_command(&cp))\n> > + return error(_(\"could not spawn pack-objects\"));\n> [...]\n> > + CALLOC_ARRAY(generator, 1);\n> > + generator->base.out = opts->pack_fd < 0 ? cp.out : -1;\n> > + generator->base.err = opts->progress_fd < 0 ? cp.err : -1;\n> > + generator->base.finish = odb_pack_generator_files_finish;\n> > + generator->cp = cp;\n> > +\n> > + *out = &generator->base;\n> > + return 0;\n> > +}\n> \n> Does this have a use-after-scope bug lurking here, due to the\n> combination of clean_on_exit = 1 (which makes a copy of &cp for later\n> use), and the fact that cp is a function-local? If I'm reading the\n> code right, start_command() calls mark_child_for_cleanup(), which does\n> \n> p->process = process; /* where process is &cp */\n> \n> and then cleanup_children() accesses various fields under p->process.\n> You do copy the necessary fields from cp to generator->cp, but\n> &generator->cp was not passed to start_command(), so p->process points\n> to the function-local cp.\n\nOh, that's a very good catch indeed. Out of curiosity, how did you end\nup discovering this? Did you just happen to remember that we store the\npointer out of scope or did the copy make you have a deeper look?\n\n> I think the normal teardown path happens to be fine despite this\n> issue: when odb_pack_generator_files_finish() calls\n> finish_command(&generator->cp), it clears the child by matching pid\n> (which was copied separately from p->process), so the stale pointer\n> never gets dereferenced in the successful path. But with an\n> abnormal-exit, which is where clean_on_exit comes into play, then\n> cleanup_children() will be called and start attempting to read\n> p->process, which now points to some long-reclaimed function stack\n> space.\n\nYeah, it's a bug waiting to happen. Will fix, thanks!\n\nPatrick\n"},{"id":"550872","messageId":"20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","subject":"[PATCH v3 0/6] odb: make packfile generation pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T07:55:24Z","receivedAt":"2026-08-20T07:55:37Z","isPatch":true,"body":"Hi,\n\nthis patch series makes packfile generation pluggable.\n\nNote that this series only makes those parts pluggable that are required\nfor the transport layer. The other parts that relate to packfile\ngeneration as required by our repository maintenance is kept as-is, as\nthere is a bunch of options there that are way too specific to the\n\"files\" backend to be portable. This should ultimately not be much of a\nproblem though, as maintenance itself is already pluggable in the first\nplace.\n\nIt's a bit of a shame though for git-pack-objects(1), which still isn't\nusable with alternate backends. I tried several times to find good\nsolutions for making it fully pluggable, but due to the backend-specific\noptions it's an utter mess. I want to eventually address this though:\nsame as with git-refs(1), I want to introduce git-objects(1) to care\nabout all things ODB. And as part of that command we can also introduce\na command that generates packfiles in a generic fashion, without all the\ncruft that git-pack-objects(1) has. This is part of a future patch\nseries though.\n\nChanges in v3:\n - Fix a use-after-scope bug on abnormal exit when child processes are\n cleaned up via `mark_child_for_cleanup()`, as noticed by Elijah.\n - Link to v2: https://patch.msgid.link/20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im\n\nChanges in v2:\n - Mostly remove the dependencies on `the_repository` in \"bundle.c\".\n - Link to v1: https://patch.msgid.link/20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im\n\nThe series is built on top of 2c78326f81 (The 11th batch, 2026-08-05).\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (6):\n odb: introduce interface to generate packfiles\n upload-pack: generate packfiles via the object database\n send-pack: generate packfiles via the object database\n builtin/bundle: refactor option handling for progress meter\n bundle: get (mostly) rid of `the_repository`\n bundle: generate packfiles via the object database\n\n builtin/bundle.c | 31 ++++------\n bundle.c | 97 ++++++++++++++++++--------------\n bundle.h | 3 +-\n odb.c | 21 +++++++\n odb.h | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-files.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h | 33 +++++++++++\n send-pack.c | 101 +++++++++++----------------------\n t/t5516-fetch-push.sh | 12 ++--\n upload-pack.c | 125 +++++++++++++++--------------------------\n 10 files changed, 506 insertions(+), 218 deletions(-)\n\nRange-diff versus v2:\n\n1: 34a7d13ccb ! 1: d1282ffc64 odb: introduce interface to generate packfiles\n @@ odb/source-files.c: int odb_source_files_optimize(struct odb_source *source,\n +\t\t\t\t\t struct odb_pack_generator **out,\n +\t\t\t\t\t const struct odb_generate_pack_options *opts)\n +{\n -+\tstruct child_process cp = CHILD_PROCESS_INIT;\n +\tstruct odb_pack_generator_files *generator;\n ++\tstruct child_process *cp;\n +\tFILE *in;\n +\n ++\tCALLOC_ARRAY(generator, 1);\n ++\tchild_process_init(&generator->cp);\n ++\tcp = &generator->cp;\n ++\n +\t/*\n +\t * The hook is expected to spawn \"$hook git pack-objects \"\n +\t * and to behave like git-pack-objects(1) would have. This can for\n +\t * example be used to serve precomputed packfiles.\n +\t */\n +\tif (opts->pack_objects_hook) {\n -+\t\tstrvec_push(&cp.args, opts->pack_objects_hook);\n -+\t\tstrvec_push(&cp.args, \"git\");\n -+\t\tcp.use_shell = 1;\n ++\t\tstrvec_push(&cp->args, opts->pack_objects_hook);\n ++\t\tstrvec_push(&cp->args, \"git\");\n ++\t\tcp->use_shell = 1;\n +\t} else {\n -+\t\tcp.git_cmd = 1;\n ++\t\tcp->git_cmd = 1;\n +\t}\n +\n +\t/*\n @@ odb/source-files.c: int odb_source_files_optimize(struct odb_source *source,\n +\t * be neutralized.\n +\t */\n +\tif (opts->shallows.nr) {\n -+\t\tstrvec_push(&cp.args, \"--shallow-file\");\n -+\t\tstrvec_push(&cp.args, \"\");\n ++\t\tstrvec_push(&cp->args, \"--shallow-file\");\n ++\t\tstrvec_push(&cp->args, \"\");\n +\t}\n -+\tstrvec_push(&cp.args, \"pack-objects\");\n -+\tstrvec_push(&cp.args, \"--revs\");\n -+\tstrvec_push(&cp.args, \"--stdout\");\n ++\tstrvec_push(&cp->args, \"pack-objects\");\n ++\tstrvec_push(&cp->args, \"--revs\");\n ++\tstrvec_push(&cp->args, \"--stdout\");\n +\tif (opts->thin)\n -+\t\tstrvec_push(&cp.args, \"--thin\");\n ++\t\tstrvec_push(&cp->args, \"--thin\");\n +\tif (opts->shallow)\n -+\t\tstrvec_push(&cp.args, \"--shallow\");\n ++\t\tstrvec_push(&cp->args, \"--shallow\");\n +\tif (opts->ofs_delta)\n -+\t\tstrvec_push(&cp.args, \"--delta-base-offset\");\n ++\t\tstrvec_push(&cp->args, \"--delta-base-offset\");\n +\tif (opts->include_tag)\n -+\t\tstrvec_push(&cp.args, \"--include-tag\");\n ++\t\tstrvec_push(&cp->args, \"--include-tag\");\n +\tif (opts->missing_allow_promisor)\n -+\t\tstrvec_push(&cp.args, \"--missing=allow-promisor\");\n ++\t\tstrvec_push(&cp->args, \"--missing=allow-promisor\");\n +\tif (opts->disable_bitmaps)\n -+\t\tstrvec_push(&cp.args, \"--no-use-bitmap-index\");\n ++\t\tstrvec_push(&cp->args, \"--no-use-bitmap-index\");\n +\tswitch (opts->progress) {\n +\tcase ODB_GENERATE_PACK_PROGRESS_NONE:\n -+\t\tstrvec_push(&cp.args, \"--quiet\");\n ++\t\tstrvec_push(&cp->args, \"--quiet\");\n +\t\tbreak;\n +\tcase ODB_GENERATE_PACK_PROGRESS_STANDARD:\n -+\t\tstrvec_push(&cp.args, \"--progress\");\n ++\t\tstrvec_push(&cp->args, \"--progress\");\n +\t\tbreak;\n +\tcase ODB_GENERATE_PACK_PROGRESS_VERBOSE:\n -+\t\tstrvec_push(&cp.args, \"--all-progress\");\n ++\t\tstrvec_push(&cp->args, \"--all-progress\");\n +\t\tbreak;\n +\tdefault:\n +\t\tBUG(\"unknown progress option %d\", opts->progress);\n +\t}\n +\tif (opts->filter_spec)\n -+\t\tstrvec_pushf(&cp.args, \"--filter=%s\", opts->filter_spec);\n ++\t\tstrvec_pushf(&cp->args, \"--filter=%s\", opts->filter_spec);\n +\tif (opts->uri_protocols)\n +\t\tfor (size_t i = 0; i < opts->uri_protocols->nr; i++)\n -+\t\t\tstrvec_pushf(&cp.args, \"--uri-protocol=%s\",\n ++\t\t\tstrvec_pushf(&cp->args, \"--uri-protocol=%s\",\n +\t\t\t\t opts->uri_protocols->items[i].string);\n +\n -+\tcp.in = -1;\n -+\tcp.out = opts->pack_fd;\n -+\tcp.err = opts->progress_fd;\n -+\tcp.clean_on_exit = 1;\n ++\tcp->in = -1;\n ++\tcp->out = opts->pack_fd;\n ++\tcp->err = opts->progress_fd;\n ++\tcp->clean_on_exit = 1;\n +\n -+\tif (start_command(&cp))\n ++\tif (start_command(cp)) {\n ++\t\tfree(generator);\n +\t\treturn error(_(\"could not spawn pack-objects\"));\n ++\t}\n +\n +\t/*\n +\t * Feed the objects to pack-objects. This is safe to do synchronously\n +\t * because pack-objects consumes all of its standard input before it\n +\t * starts to generate the pack.\n +\t */\n -+\tin = xfdopen(cp.in, \"w\");\n ++\tin = xfdopen(cp->in, \"w\");\n +\tfor (size_t i = 0; i < opts->shallows.nr; i++)\n +\t\tfprintf(in, \"--shallow %s\\n\", oid_to_hex(&opts->shallows.oid[i]));\n +\tfor (size_t i = 0; i < opts->wants.nr; i++)\n @@ odb/source-files.c: int odb_source_files_optimize(struct odb_source *source,\n +\t\terror(_(\"error writing to pack-objects\"));\n +\t\tfclose(in);\n +\t\tif (opts->pack_fd < 0)\n -+\t\t\tclose(cp.out);\n ++\t\t\tclose(cp->out);\n +\t\tif (opts->progress_fd < 0)\n -+\t\t\tclose(cp.err);\n -+\t\tfinish_command(&cp);\n ++\t\t\tclose(cp->err);\n ++\t\tfinish_command(cp);\n ++\t\tfree(generator);\n +\t\treturn -1;\n +\t}\n +\tfclose(in);\n +\n -+\tCALLOC_ARRAY(generator, 1);\n -+\tgenerator->base.out = opts->pack_fd < 0 ? cp.out : -1;\n -+\tgenerator->base.err = opts->progress_fd < 0 ? cp.err : -1;\n ++\tgenerator->base.out = opts->pack_fd < 0 ? cp->out : -1;\n ++\tgenerator->base.err = opts->progress_fd < 0 ? cp->err : -1;\n +\tgenerator->base.finish = odb_pack_generator_files_finish;\n -+\tgenerator->cp = cp;\n +\n +\t*out = &generator->base;\n +\treturn 0;\n2: c7234030ad = 2: b17dfd945b upload-pack: generate packfiles via the object database\n3: 9866e8af3d = 3: 8e9be66b36 send-pack: generate packfiles via the object database\n4: 1cd0c10438 = 4: 3dfc5df91d builtin/bundle: refactor option handling for progress meter\n5: 621c9bb411 = 5: 9f938bce19 bundle: get (mostly) rid of `the_repository`\n6: 5c1ee3d116 = 6: 41395b1444 bundle: generate packfiles via the object database\n\n---\nbase-commit: 2c78326f810173a4f3aefd8021f1e07575412481\nchange-id: 20260807-b4-pks-odb-generate-pack-f30fbcdef3fc\n\n"},{"id":"550873","messageId":"20260820-b4-pks-odb-generate-pack-v3-1-bc42252f6169@pks.im","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im","subject":"[PATCH v3 1/6] odb: introduce interface to generate packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T07:55:25Z","receivedAt":"2026-08-20T07:55:39Z","isPatch":true,"body":"Packfiles have two primary use cases:\n\n - They are used to store objects at rest in a Git repository.\n\n - They are used on the transport layer to transfer objects between two\n repositories.\n\nThe first class is closely tied to a given object database backend, and\nas such this use is highly specific to how such a backend decides to\nstore its data. This shows in git-pack-objects(1), which is used by\ngit-repack(1) et al to optimize the object database, which supports lots\nof options that are closely coupled with how data is stored.\n\nBut the second class is quite a lot more generic: we don't care about\nspecifics of how the object database stores its objects, but to generate\nthe packfiles we only care about the object graph itself. Still, this\nuse case is also coupled with git-pack-objects(1).\n\nUnfortunately, because git-pack-objects(1) covers both classes, the\nresult is that it is very hard to port the whole command to properly\nsupport pluggable object databases. There are simply way too many\noptions that an alternative implementation will have a very hard time to\nsupport in the first place.\n\nAnd despite being hard to implement, it's also quite unnecessary to\nimplement those backend-specific options. Optimizing the object database\nhas already been made pluggable, and an alternative implementation is\nunlikely to care about cruft packs, unpacked objects, keep packs and the\nlike. But we still need to make at least _parts_ of the packfile\ngeneration pluggable so that backends can generate packfiles for the\ntransport layer itself.\n\nIntroduce a new interface that lets backends generate a new packfile and\nimplement that interface for the \"files\" backend. The options supported\nby the callback are exactly the set of options that are required for the\ntransport layer, but nothing more.\n\nThis means that git-pack-objects(1) itself cannot be ported over to this\nnew interface, but as explained above that's a hard feat to pull off due\nto the backend-specific features. Ideally though, we should expose the\nability to generate arbitrary packfiles using this interface. The intent\nof this is to eventually introduce a git-objects(1) subcommand (similar\nto git-refs(1)) that exposes generic interfaces for accessing everything\nrelated to the object database. In that case, we are able to expose only\nthose options that are generic.\n\nSubsequent commits will convert git-upload-pack(1), git-send-pack(1) and\ngit-bundle(1) to use this interface.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 21 ++++++++\n odb.h | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-files.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h | 33 ++++++++++++\n 4 files changed, 355 insertions(+)\n\ndiff --git a/odb.c b/odb.c\nindex caf1d0f542..cd9d5b48bc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1046,6 +1046,27 @@ bool odb_optimize_required(struct object_database *odb,\n \treturn odb_source_optimize_required(odb->sources, opts);\n }\n \n+void odb_generate_pack_options_release(struct odb_generate_pack_options *opts)\n+{\n+\toid_array_clear(&opts->wants);\n+\toid_array_clear(&opts->haves);\n+\toid_array_clear(&opts->shallows);\n+}\n+\n+int odb_generate_pack(struct object_database *odb,\n+\t\t struct odb_pack_generator **out,\n+\t\t const struct odb_generate_pack_options *opts)\n+{\n+\tif (!odb->sources->generate_pack)\n+\t\treturn error(_(\"primary object source does not support generating packfiles\"));\n+\treturn odb_source_generate_pack(odb->sources, out, opts);\n+}\n+\n+int odb_pack_generator_finish(struct odb_pack_generator *generator)\n+{\n+\treturn generator->finish(generator);\n+}\n+\n struct object_database *odb_new(struct repository *repo,\n \t\t\t\tconst char *primary_source,\n \t\t\t\tconst char *secondary_sources)\ndiff --git a/odb.h b/odb.h\nindex fca67e8253..fc1442f243 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -2,6 +2,7 @@\n #define ODB_H\n \n #include \"object.h\"\n+#include \"oid-array.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -677,6 +678,157 @@ int odb_write_object_stream(struct object_database *odb,\n \t\t\t struct odb_write_stream *stream, size_t len,\n \t\t\t struct object_id *oid);\n \n+/*\n+ * Options for generating a packfile via `odb_generate_pack()`.\n+ */\n+struct odb_generate_pack_options {\n+\t/* Tips of the object graph that shall be packed. */\n+\tstruct oid_array wants;\n+\n+\t/*\n+\t * Boundary of the object graph. Objects reachable from any of these\n+\t * tips are expected to already be available to whoever consumes the\n+\t * pack and shall thus not be packed.\n+\t */\n+\tstruct oid_array haves;\n+\n+\t/*\n+\t * The shallow boundary that shall be used when computing object\n+\t * reachability. When set, any shallow information of the repository\n+\t * itself shall be ignored in favor of these objects.\n+\t */\n+\tstruct oid_array shallows;\n+\n+\t/*\n+\t * Pre-expanded object filter specification that limits the set of\n+\t * objects that shall be packed. May be `NULL` in case no filter shall\n+\t * be applied.\n+\t */\n+\tconst char *filter_spec;\n+\n+\t/*\n+\t * Protocols that may be used to offload objects via packfile URIs.\n+\t * May be `NULL` in case packfile URIs shall not be used.\n+\t */\n+\tconst struct string_list *uri_protocols;\n+\n+\t/*\n+\t * Hook command that shall be executed instead of the internal\n+\t * machinery to generate the pack. It is up to the specific backend\n+\t * whether or not this hook is supported. May be `NULL` in case no\n+\t * hook shall be executed.\n+\t */\n+\tconst char *pack_objects_hook;\n+\n+\t/*\n+\t * File descriptor that the generated pack shall be written to. If set\n+\t * to `-1`, a pipe will be created and exposed via the pack generator's\n+\t * `out` field. If set to `0`, the pack will be written to the standard\n+\t * output stream. Otherwise, the provided descriptor will be written to\n+\t * and is consumed by the generator.\n+\t */\n+\tint pack_fd;\n+\n+\t/*\n+\t * File descriptor that progress output shall be written to. The same\n+\t * semantics as for `pack_fd` apply, except that `0` will cause the\n+\t * generator to write to stderr instead of stdout.\n+\t */\n+\tint progress_fd;\n+\n+\t/* Whether to print progress or not. */\n+\tenum {\n+\t\t/* Don't print progress output. */\n+\t\tODB_GENERATE_PACK_PROGRESS_NONE,\n+\n+\t\t/*\n+\t\t * Print progress while computing the packfile, but stop\n+\t\t * printing progress once starting to write it.\n+\t\t */\n+\t\tODB_GENERATE_PACK_PROGRESS_STANDARD,\n+\n+\t\t/*\n+\t\t * Similar to STANDARD, but also print progress when writing\n+\t\t * the packfile.\n+\t\t */\n+\t\tODB_GENERATE_PACK_PROGRESS_VERBOSE,\n+\t} progress;\n+\n+\t/* Allow the pack to contain deltas against unpacked objects. */\n+\tunsigned thin:1;\n+\n+\t/* Use offset deltas instead of reference deltas. */\n+\tunsigned ofs_delta:1;\n+\n+\t/* Include unasked-for annotated tags of packed objects. */\n+\tunsigned include_tag:1;\n+\n+\t/* The generated pack is destined for a shallow consumer. */\n+\tunsigned shallow:1;\n+\n+\t/* Allow objects that may be missing due to a promisor remote. */\n+\tunsigned missing_allow_promisor:1;\n+\n+\t/* Do not use bitmap indices when computing reachability. */\n+\tunsigned disable_bitmaps:1;\n+};\n+\n+#define ODB_GENERATE_PACK_OPTIONS_INIT { \\\n+\t.wants = OID_ARRAY_INIT, \\\n+\t.haves = OID_ARRAY_INIT, \\\n+\t.shallows = OID_ARRAY_INIT, \\\n+\t.pack_fd = -1, \\\n+}\n+\n+/* Release resources associated with the options. */\n+void odb_generate_pack_options_release(struct odb_generate_pack_options *opts);\n+\n+/*\n+ * A handle for an ongoing packfile generation as started via\n+ * `odb_generate_pack()`.\n+ */\n+struct odb_pack_generator {\n+\t/*\n+\t * File descriptor from which the generated pack can be read. Only set\n+\t * when the pack generation was started with `pack_fd == -1`. The\n+\t * caller is responsible for closing the descriptor.\n+\t */\n+\tint out;\n+\n+\t/*\n+\t * File descriptor from which progress output can be read. Only set\n+\t * when the pack generation was started with `progress_fd == -1`. The\n+\t * caller is responsible for closing the descriptor.\n+\t */\n+\tint err;\n+\n+\t/*\n+\t * Callback function to finish this generator. This callback is\n+\t * expected to wait for the packfile generation to complete and to then\n+\t * free the generator itself.\n+\t */\n+\tint (*finish)(struct odb_pack_generator *);\n+};\n+\n+/*\n+ * Start generating a packfile from the object database with the given\n+ * options. The pack is generated asynchronously; the caller is expected to\n+ * consume the file descriptors exposed via the pack generator and to then\n+ * wait for completion via `odb_pack_generator_finish()`.\n+ *\n+ * Returns 0 on success and populates the `out` pointer with the pack\n+ * generator. Returns a negative error code otherwise.\n+ */\n+int odb_generate_pack(struct object_database *odb,\n+\t\t struct odb_pack_generator **out,\n+\t\t const struct odb_generate_pack_options *opts);\n+\n+/*\n+ * Wait for the packfile generation to complete and free the pack generator.\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int odb_pack_generator_finish(struct odb_pack_generator *generator);\n+\n void parse_alternates(const char *string,\n \t\t int sep,\n \t\t const char *relative_base,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5a68af7d84..a33e01fbed 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -4,6 +4,7 @@\n #include \"chdir-notify.h\"\n #include \"config.h\"\n #include \"gettext.h\"\n+#include \"hex.h\"\n #include \"lockfile.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n@@ -729,6 +730,153 @@ int odb_source_files_optimize(struct odb_source *source,\n \treturn ret;\n }\n \n+struct odb_pack_generator_files {\n+\tstruct odb_pack_generator base;\n+\tstruct child_process cp;\n+};\n+\n+static int odb_pack_generator_files_finish(struct odb_pack_generator *_generator)\n+{\n+\tstruct odb_pack_generator_files *generator =\n+\t\t(struct odb_pack_generator_files *)_generator;\n+\tint ret;\n+\n+\tret = finish_command(&generator->cp);\n+\tfree(generator);\n+\n+\tif (ret) {\n+\t\t/*\n+\t\t * On failure, pack-objects is expected to have written a\n+\t\t * useful error message to its standard error stream already.\n+\t\t * Death by signal is worth mentioning, though, with the\n+\t\t * exception of SIGPIPE: that is a normal occurrence when the\n+\t\t * consumer of the pack hangs up.\n+\t\t */\n+\t\tif (ret > 128 && ret - 128 == SIGPIPE)\n+\t\t\treturn -1;\n+\t\tif (ret > 128)\n+\t\t\terror(_(\"pack-objects died of signal %d\"), ret - 128);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_source_files_generate_pack(struct odb_source *source UNUSED,\n+\t\t\t\t\t struct odb_pack_generator **out,\n+\t\t\t\t\t const struct odb_generate_pack_options *opts)\n+{\n+\tstruct odb_pack_generator_files *generator;\n+\tstruct child_process *cp;\n+\tFILE *in;\n+\n+\tCALLOC_ARRAY(generator, 1);\n+\tchild_process_init(&generator->cp);\n+\tcp = &generator->cp;\n+\n+\t/*\n+\t * The hook is expected to spawn \"$hook git pack-objects \"\n+\t * and to behave like git-pack-objects(1) would have. This can for\n+\t * example be used to serve precomputed packfiles.\n+\t */\n+\tif (opts->pack_objects_hook) {\n+\t\tstrvec_push(&cp->args, opts->pack_objects_hook);\n+\t\tstrvec_push(&cp->args, \"git\");\n+\t\tcp->use_shell = 1;\n+\t} else {\n+\t\tcp->git_cmd = 1;\n+\t}\n+\n+\t/*\n+\t * The caller-provided shallow boundary overrides any shallow state\n+\t * that the repository itself may have, so the shallow file needs to\n+\t * be neutralized.\n+\t */\n+\tif (opts->shallows.nr) {\n+\t\tstrvec_push(&cp->args, \"--shallow-file\");\n+\t\tstrvec_push(&cp->args, \"\");\n+\t}\n+\tstrvec_push(&cp->args, \"pack-objects\");\n+\tstrvec_push(&cp->args, \"--revs\");\n+\tstrvec_push(&cp->args, \"--stdout\");\n+\tif (opts->thin)\n+\t\tstrvec_push(&cp->args, \"--thin\");\n+\tif (opts->shallow)\n+\t\tstrvec_push(&cp->args, \"--shallow\");\n+\tif (opts->ofs_delta)\n+\t\tstrvec_push(&cp->args, \"--delta-base-offset\");\n+\tif (opts->include_tag)\n+\t\tstrvec_push(&cp->args, \"--include-tag\");\n+\tif (opts->missing_allow_promisor)\n+\t\tstrvec_push(&cp->args, \"--missing=allow-promisor\");\n+\tif (opts->disable_bitmaps)\n+\t\tstrvec_push(&cp->args, \"--no-use-bitmap-index\");\n+\tswitch (opts->progress) {\n+\tcase ODB_GENERATE_PACK_PROGRESS_NONE:\n+\t\tstrvec_push(&cp->args, \"--quiet\");\n+\t\tbreak;\n+\tcase ODB_GENERATE_PACK_PROGRESS_STANDARD:\n+\t\tstrvec_push(&cp->args, \"--progress\");\n+\t\tbreak;\n+\tcase ODB_GENERATE_PACK_PROGRESS_VERBOSE:\n+\t\tstrvec_push(&cp->args, \"--all-progress\");\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown progress option %d\", opts->progress);\n+\t}\n+\tif (opts->filter_spec)\n+\t\tstrvec_pushf(&cp->args, \"--filter=%s\", opts->filter_spec);\n+\tif (opts->uri_protocols)\n+\t\tfor (size_t i = 0; i < opts->uri_protocols->nr; i++)\n+\t\t\tstrvec_pushf(&cp->args, \"--uri-protocol=%s\",\n+\t\t\t\t opts->uri_protocols->items[i].string);\n+\n+\tcp->in = -1;\n+\tcp->out = opts->pack_fd;\n+\tcp->err = opts->progress_fd;\n+\tcp->clean_on_exit = 1;\n+\n+\tif (start_command(cp)) {\n+\t\tfree(generator);\n+\t\treturn error(_(\"could not spawn pack-objects\"));\n+\t}\n+\n+\t/*\n+\t * Feed the objects to pack-objects. This is safe to do synchronously\n+\t * because pack-objects consumes all of its standard input before it\n+\t * starts to generate the pack.\n+\t */\n+\tin = xfdopen(cp->in, \"w\");\n+\tfor (size_t i = 0; i < opts->shallows.nr; i++)\n+\t\tfprintf(in, \"--shallow %s\\n\", oid_to_hex(&opts->shallows.oid[i]));\n+\tfor (size_t i = 0; i < opts->wants.nr; i++)\n+\t\tfprintf(in, \"%s\\n\", oid_to_hex(&opts->wants.oid[i]));\n+\tfprintf(in, \"--not\\n\");\n+\tfor (size_t i = 0; i < opts->haves.nr; i++)\n+\t\tfprintf(in, \"%s\\n\", oid_to_hex(&opts->haves.oid[i]));\n+\tfprintf(in, \"\\n\");\n+\tfflush(in);\n+\tif (ferror(in)) {\n+\t\terror(_(\"error writing to pack-objects\"));\n+\t\tfclose(in);\n+\t\tif (opts->pack_fd < 0)\n+\t\t\tclose(cp->out);\n+\t\tif (opts->progress_fd < 0)\n+\t\t\tclose(cp->err);\n+\t\tfinish_command(cp);\n+\t\tfree(generator);\n+\t\treturn -1;\n+\t}\n+\tfclose(in);\n+\n+\tgenerator->base.out = opts->pack_fd < 0 ? cp->out : -1;\n+\tgenerator->base.err = opts->progress_fd < 0 ? cp->err : -1;\n+\tgenerator->base.finish = odb_pack_generator_files_finish;\n+\n+\t*out = &generator->base;\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t bool local)\n@@ -756,6 +904,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.write_alternate = odb_source_files_write_alternate;\n \tfiles->base.optimize = odb_source_files_optimize;\n \tfiles->base.optimize_required = odb_source_files_optimize_required;\n+\tfiles->base.generate_pack = odb_source_files_generate_pack;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex d69f8e2d1c..e2129766fc 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -278,6 +278,23 @@ struct odb_source {\n \t */\n \tbool (*optimize_required)(struct odb_source *source,\n \t\t\t\t const struct odb_optimize_options *opts);\n+\n+\t/*\n+\t * This callback is expected to start generating a packfile with the\n+\t * given options. The pack shall be generated asynchronously so that\n+\t * the caller can consume the pack data and progress output while the\n+\t * pack is being generated.\n+\t *\n+\t * This callback is optional. Sources that cannot generate packfiles\n+\t * shall leave it unset.\n+\t *\n+\t * The callback is expected to return 0 on success and populate the\n+\t * `out` pointer with the pack generator, a negative error code\n+\t * otherwise.\n+\t */\n+\tint (*generate_pack)(struct odb_source *source,\n+\t\t\t struct odb_pack_generator **out,\n+\t\t\t const struct odb_generate_pack_options *opts);\n };\n \n /*\n@@ -520,4 +537,20 @@ static inline bool odb_source_optimize_required(struct odb_source *source,\n \treturn source->optimize_required(source, opts);\n }\n \n+/*\n+ * Start generating a packfile from the given source with the given options.\n+ * The pack is generated asynchronously; the caller is expected to consume the\n+ * file descriptors exposed via the pack generator and to then wait for\n+ * completion via `odb_pack_generator_finish()`.\n+ *\n+ * Returns 0 on success and populates the `out` pointer with the pack\n+ * generator, a negative error code otherwise.\n+ */\n+static inline int odb_source_generate_pack(struct odb_source *source,\n+\t\t\t\t\t struct odb_pack_generator **out,\n+\t\t\t\t\t const struct odb_generate_pack_options *opts)\n+{\n+\treturn source->generate_pack(source, out, opts);\n+}\n+\n #endif\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550874","messageId":"20260820-b4-pks-odb-generate-pack-v3-2-bc42252f6169@pks.im","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im","subject":"[PATCH v3 2/6] upload-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T07:55:26Z","receivedAt":"2026-08-20T07:55:41Z","isPatch":true,"body":"When serving a fetch, git-upload-pack(1) spawns git-pack-objects(1)\ndirectly to generate the packfile that gets sent to the client. This\nhard-codes the assumption that the object database is able to serve\npackfiles via git-pack-objects(1), which is specific to the \"files\"\nbackend.\n\nConvert git-upload-pack(1) to instead use the pack generation interface\nof the object database.\n\nSigned-off-by: Patrick Steinhardt \n---\n upload-pack.c | 125 +++++++++++++++++++++-------------------------------------\n 1 file changed, 45 insertions(+), 80 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex a52856d869..75a857eaa8 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -197,11 +197,11 @@ static void send_client_data(int fd, const char *data, ssize_t sz,\n \twrite_or_die(fd, data, sz);\n }\n \n-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n+static int append_one_shallow(const struct commit_graft *graft, void *cb_data)\n {\n-\tFILE *fp = cb_data;\n+\tstruct oid_array *shallows = cb_data;\n \tif (graft->nr_parent == -1)\n-\t\tfprintf(fp, \"--shallow %s\\n\", oid_to_hex(&graft->oid));\n+\t\toid_array_append(shallows, &graft->oid);\n \treturn 0;\n }\n \n@@ -299,7 +299,8 @@ static int relay_pack_data(int pack_objects_out, struct output_state *os,\n static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t\t const struct string_list *uri_protocols)\n {\n-\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n \tstruct output_state *output_state = xcalloc(1, sizeof(struct output_state));\n \tchar progress[128];\n \tchar abort_msg[] = \"aborting due to possible repository \"\n@@ -307,78 +308,42 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \tuint64_t last_sent_ms = 0;\n \tssize_t sz;\n \tint i;\n-\tFILE *pipe_fd;\n-\n-\tif (!pack_data->pack_objects_hook)\n-\t\tpack_objects.git_cmd = 1;\n-\telse {\n-\t\tstrvec_push(&pack_objects.args, pack_data->pack_objects_hook);\n-\t\tstrvec_push(&pack_objects.args, \"git\");\n-\t\tpack_objects.use_shell = 1;\n-\t}\n \n \tif (pack_data->shallow_nr) {\n-\t\tstrvec_push(&pack_objects.args, \"--shallow-file\");\n-\t\tstrvec_push(&pack_objects.args, \"\");\n-\t}\n-\tstrvec_push(&pack_objects.args, \"pack-objects\");\n-\tstrvec_push(&pack_objects.args, \"--revs\");\n-\tif (pack_data->use_thin_pack)\n-\t\tstrvec_push(&pack_objects.args, \"--thin\");\n-\n-\tstrvec_push(&pack_objects.args, \"--stdout\");\n-\tif (pack_data->shallow_nr)\n-\t\tstrvec_push(&pack_objects.args, \"--shallow\");\n-\tif (!pack_data->no_progress)\n-\t\tstrvec_push(&pack_objects.args, \"--progress\");\n-\tif (pack_data->use_ofs_delta)\n-\t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n-\tif (pack_data->use_include_tag)\n-\t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n-\tif (repo_has_accepted_promisor_remote(the_repository))\n-\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n-\tif (pack_data->filter_options.choice) {\n-\t\tconst char *spec =\n-\t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\", spec);\n-\t}\n-\tif (uri_protocols) {\n-\t\tfor (i = 0; i < uri_protocols->nr; i++)\n-\t\t\tstrvec_pushf(&pack_objects.args, \"--uri-protocol=%s\",\n-\t\t\t\t\t uri_protocols->items[i].string);\n+\t\tfor_each_commit_graft(append_one_shallow, &opts.shallows);\n+\t\topts.shallow = 1;\n \t}\n-\n-\tpack_objects.in = -1;\n-\tpack_objects.out = -1;\n-\tpack_objects.err = -1;\n-\tpack_objects.clean_on_exit = 1;\n-\n-\tif (start_command(&pack_objects))\n-\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n-\n-\tpipe_fd = xfdopen(pack_objects.in, \"w\");\n-\n-\tif (pack_data->shallow_nr)\n-\t\tfor_each_commit_graft(write_one_shallow, pipe_fd);\n-\n \tfor (i = 0; i < pack_data->want_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->want_obj.objects[i].item->oid));\n-\tfprintf(pipe_fd, \"--not\\n\");\n+\t\toid_array_append(&opts.wants,\n+\t\t\t\t &pack_data->want_obj.objects[i].item->oid);\n \tfor (i = 0; i < pack_data->have_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->have_obj.objects[i].item->oid));\n+\t\toid_array_append(&opts.haves,\n+\t\t\t\t &pack_data->have_obj.objects[i].item->oid);\n \tfor (i = 0; i < pack_data->extra_edge_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->extra_edge_obj.objects[i].item->oid));\n-\tfprintf(pipe_fd, \"\\n\");\n-\tfflush(pipe_fd);\n-\tfclose(pipe_fd);\n-\n-\t/* We read from pack_objects.err to capture stderr output for\n-\t * progress bar, and pack_objects.out to capture the pack data.\n-\t */\n+\t\toid_array_append(&opts.haves,\n+\t\t\t\t &pack_data->extra_edge_obj.objects[i].item->oid);\n+\n+\topts.thin = pack_data->use_thin_pack;\n+\tif (!pack_data->no_progress)\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_STANDARD;\n+\topts.ofs_delta = pack_data->use_ofs_delta;\n+\topts.include_tag = pack_data->use_include_tag;\n+\topts.missing_allow_promisor = repo_has_accepted_promisor_remote(the_repository);\n+\tif (pack_data->filter_options.choice)\n+\t\topts.filter_spec = expand_list_objects_filter_spec(&pack_data->filter_options);\n+\topts.uri_protocols = uri_protocols;\n+\topts.pack_objects_hook = pack_data->pack_objects_hook;\n+\topts.pack_fd = -1;\n+\topts.progress_fd = -1;\n+\n+\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n+\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n+\todb_generate_pack_options_release(&opts);\n \n+\t/*\n+\t * We read from generator->err to capture stderr output for the\n+\t * progress bar, and generator->out to capture the pack data.\n+\t */\n \twhile (1) {\n \t\tuint64_t now_ms = getnanotime() / 1000000;\n \t\tstruct pollfd pfd[2];\n@@ -393,14 +358,14 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tpollsize = 0;\n \t\tpe = pu = -1;\n \n-\t\tif (0 <= pack_objects.out) {\n-\t\t\tpfd[pollsize].fd = pack_objects.out;\n+\t\tif (0 <= generator->out) {\n+\t\t\tpfd[pollsize].fd = generator->out;\n \t\t\tpfd[pollsize].events = POLLIN;\n \t\t\tpu = pollsize;\n \t\t\tpollsize++;\n \t\t}\n-\t\tif (0 <= pack_objects.err) {\n-\t\t\tpfd[pollsize].fd = pack_objects.err;\n+\t\tif (0 <= generator->err) {\n+\t\t\tpfd[pollsize].fd = generator->err;\n \t\t\tpfd[pollsize].events = POLLIN;\n \t\t\tpe = pollsize;\n \t\t\tpollsize++;\n@@ -437,15 +402,15 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t\t/* Status ready; we ship that in the side-band\n \t\t\t * or dump to the standard error.\n \t\t\t */\n-\t\t\tsz = xread(pack_objects.err, progress,\n+\t\t\tsz = xread(generator->err, progress,\n \t\t\t\t sizeof(progress));\n \t\t\tif (0 < sz) {\n \t\t\t\tsend_client_data(2, progress, sz,\n \t\t\t\t\t\t pack_data->use_sideband);\n \t\t\t\tlast_sent_ms = now_ms;\n \t\t\t} else if (sz == 0) {\n-\t\t\t\tclose(pack_objects.err);\n-\t\t\t\tpack_objects.err = -1;\n+\t\t\t\tclose(generator->err);\n+\t\t\t\tgenerator->err = -1;\n \t\t\t}\n \t\t\telse\n \t\t\t\tgoto fail;\n@@ -455,15 +420,15 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \n \t\tif (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) {\n \t\t\tbool did_send_data;\n-\t\t\tint result = relay_pack_data(pack_objects.out,\n+\t\t\tint result = relay_pack_data(generator->out,\n \t\t\t\t\t\t output_state,\n \t\t\t\t\t\t pack_data->use_sideband,\n \t\t\t\t\t\t !!uri_protocols,\n \t\t\t\t\t\t &did_send_data);\n \n \t\t\tif (result == 0) {\n-\t\t\t\tclose(pack_objects.out);\n-\t\t\t\tpack_objects.out = -1;\n+\t\t\t\tclose(generator->out);\n+\t\t\t\tgenerator->out = -1;\n \t\t\t} else if (result < 0) {\n \t\t\t\tgoto fail;\n \t\t\t}\n@@ -498,7 +463,7 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t}\n \t}\n \n-\tif (finish_command(&pack_objects)) {\n+\tif (odb_pack_generator_finish(generator)) {\n \t\terror(\"git upload-pack: git-pack-objects died with error.\");\n \t\tgoto fail;\n \t}\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550875","messageId":"20260820-b4-pks-odb-generate-pack-v3-3-bc42252f6169@pks.im","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im","subject":"[PATCH v3 3/6] send-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T07:55:27Z","receivedAt":"2026-08-20T07:55:44Z","isPatch":true,"body":"When pushing, git-send-pack(1) spawns git-pack-objects(1) directly to\ngenerate the packfile that gets sent to the remote. Same as with\ngit-upload-pack(1), which has been adapted in the preceding commit,\nthis hard-codes the assumption that objects can be packed via\ngit-pack-objects(1), which is specific to the \"files\" backend.\n\nConvert git-send-pack(1) to use the pack generation interface of the\nobject database instead.\n\nNote that this requires us to adapt t5516 because the parameters passed\nto git-pack-objects(1) are changing:\n\n - The order of arguments changes.\n\n - We pass \"--quiet\" instead of \"-q\".\n\n - We don't pass \"--all-progress-implied\" anymore when not generating\n output.\n\nAll of these changes are benign though and should not result in a change\nin behaviour.\n\nSigned-off-by: Patrick Steinhardt \n---\n send-pack.c | 101 +++++++++++++++++---------------------------------\n t/t5516-fetch-push.sh | 12 +++---\n 2 files changed, 40 insertions(+), 73 deletions(-)\n\ndiff --git a/send-pack.c b/send-pack.c\nindex 3bb5afc687..f20460fbf4 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -42,16 +42,17 @@ int option_parse_push_signed(const struct option *opt,\n \tdie(\"bad %s argument: %s\", opt->long_name, arg);\n }\n \n-static void feed_object(struct repository *r,\n-\t\t\tconst struct object_id *oid, FILE *fh, int negative)\n+static void append_negative_object(struct repository *r,\n+\t\t\t\t struct oid_array *haves,\n+\t\t\t\t const struct object_id *oid)\n {\n-\tif (negative && !odb_has_object(r->objects, oid, 0))\n+\t/*\n+\t * The remote end may have advertised objects that we do not have in\n+\t * our object database. Skip those, as we cannot use them as boundary.\n+\t */\n+\tif (!odb_has_object(r->objects, oid, 0))\n \t\treturn;\n-\n-\tif (negative)\n-\t\tputc('^', fh);\n-\tfputs(oid_to_hex(oid), fh);\n-\tputc('\\n', fh);\n+\toid_array_append(haves, oid);\n }\n \n /*\n@@ -62,92 +63,58 @@ static int pack_objects(struct repository *r,\n \t\t\tstruct oid_array *negotiated,\n \t\t\tstruct send_pack_args *args)\n {\n-\t/*\n-\t * The child becomes pack-objects --revs; we feed\n-\t * the revision parameters to it via its stdin and\n-\t * let its stdout go back to the other end.\n-\t */\n-\tstruct child_process po = CHILD_PROCESS_INIT;\n-\tFILE *po_in;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n \tint rc;\n \n \ttrace2_region_enter(\"send_pack\", \"pack_objects\", r);\n-\tstrvec_push(&po.args, \"pack-objects\");\n-\tstrvec_push(&po.args, \"--all-progress-implied\");\n-\tstrvec_push(&po.args, \"--revs\");\n-\tstrvec_push(&po.args, \"--stdout\");\n-\tif (args->use_thin_pack)\n-\t\tstrvec_push(&po.args, \"--thin\");\n-\tif (args->use_ofs_delta)\n-\t\tstrvec_push(&po.args, \"--delta-base-offset\");\n-\tif (args->quiet || !args->progress)\n-\t\tstrvec_push(&po.args, \"-q\");\n+\n+\topts.thin = args->use_thin_pack;\n+\topts.ofs_delta = args->use_ofs_delta;\n \tif (args->progress)\n-\t\tstrvec_push(&po.args, \"--progress\");\n-\tif (is_repository_shallow(r))\n-\t\tstrvec_push(&po.args, \"--shallow\");\n-\tif (args->disable_bitmaps)\n-\t\tstrvec_push(&po.args, \"--no-use-bitmap-index\");\n-\tpo.in = -1;\n-\tpo.out = args->stateless_rpc ? -1 : fd;\n-\tpo.git_cmd = 1;\n-\tpo.clean_on_exit = 1;\n-\tif (start_command(&po))\n-\t\tdie_errno(\"git pack-objects failed\");\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n+\topts.shallow = is_repository_shallow(r);\n+\topts.disable_bitmaps = args->disable_bitmaps;\n \n \t/*\n-\t * We feed the pack-objects we just spawned with revision\n-\t * parameters by writing to the pipe.\n+\t * The pack is either written directly to the remote's descriptor, or,\n+\t * in the case of a stateless RPC, read back from a pipe so that we\n+\t * can wrap the pack data into pkt-lines.\n \t */\n-\tpo_in = xfdopen(po.in, \"w\");\n+\topts.pack_fd = args->stateless_rpc ? -1 : fd;\n+\n \tfor (size_t i = 0; i < advertised->nr; i++)\n-\t\tfeed_object(r, &advertised->oid[i], po_in, 1);\n+\t\tappend_negative_object(r, &opts.haves, &advertised->oid[i]);\n \tfor (size_t i = 0; i < negotiated->nr; i++)\n-\t\tfeed_object(r, &negotiated->oid[i], po_in, 1);\n+\t\tappend_negative_object(r, &opts.haves, &negotiated->oid[i]);\n \n \twhile (refs) {\n \t\tif (!is_null_oid(&refs->old_oid))\n-\t\t\tfeed_object(r, &refs->old_oid, po_in, 1);\n+\t\t\tappend_negative_object(r, &opts.haves, &refs->old_oid);\n \t\tif (!is_null_oid(&refs->new_oid))\n-\t\t\tfeed_object(r, &refs->new_oid, po_in, 0);\n+\t\t\toid_array_append(&opts.wants, &refs->new_oid);\n \t\trefs = refs->next;\n \t}\n \n-\tfflush(po_in);\n-\tif (ferror(po_in))\n-\t\tdie_errno(\"error writing to pack-objects\");\n-\tfclose(po_in);\n+\tif (odb_generate_pack(r->objects, &generator, &opts))\n+\t\tdie(\"git pack-objects failed\");\n+\todb_generate_pack_options_release(&opts);\n \n \tif (args->stateless_rpc) {\n \t\tchar *buf = xmalloc(LARGE_PACKET_MAX);\n \t\twhile (1) {\n-\t\t\tssize_t n = xread(po.out, buf, LARGE_PACKET_MAX);\n+\t\t\tssize_t n = xread(generator->out, buf, LARGE_PACKET_MAX);\n \t\t\tif (n <= 0)\n \t\t\t\tbreak;\n \t\t\tsend_sideband(fd, -1, buf, n, LARGE_PACKET_MAX);\n \t\t}\n \t\tfree(buf);\n-\t\tclose(po.out);\n-\t\tpo.out = -1;\n+\t\tclose(generator->out);\n \t}\n \n-\trc = finish_command(&po);\n-\tif (rc) {\n-\t\t/*\n-\t\t * For a normal non-zero exit, we assume pack-objects wrote\n-\t\t * something useful to stderr. For death by signal, though,\n-\t\t * we should mention it to the user. The exception is SIGPIPE\n-\t\t * (141), because that's a normal occurrence if the remote end\n-\t\t * hangs up (and we'll report that by trying to read the unpack\n-\t\t * status).\n-\t\t */\n-\t\tif (rc > 128 && rc != 141)\n-\t\t\terror(\"pack-objects died of signal %d\", rc - 128);\n-\t\ttrace2_region_leave(\"send_pack\", \"pack_objects\", r);\n-\t\treturn -1;\n-\t}\n+\trc = odb_pack_generator_finish(generator);\n \ttrace2_region_leave(\"send_pack\", \"pack_objects\", r);\n-\treturn 0;\n+\treturn rc;\n }\n \n static int receive_unpack_status(struct packet_reader *reader)\n@@ -768,7 +735,7 @@ int send_pack(struct repository *r,\n \t\t\tgoto out;\n \t\t}\n \t\tif (!args->stateless_rpc)\n-\t\t\t/* Closed by pack_objects() via start_command() */\n+\t\t\t/* Consumed by the pack generator in pack_objects() */\n \t\t\tfd[1] = -1;\n \t}\n \tif (args->stateless_rpc && cmds_sent)\ndiff --git a/t/t5516-fetch-push.sh b/t/t5516-fetch-push.sh\nindex f3b3efc47f..b982b209bf 100755\n--- a/t/t5516-fetch-push.sh\n+++ b/t/t5516-fetch-push.sh\n@@ -1903,20 +1903,20 @@ test_expect_success 'push with config push.useBitmaps' '\n \ttest_unconfig push.useBitmaps &&\n \tGIT_TRACE2_EVENT=\"$PWD/default\" \\\n \tgit push --quiet testrepo main:test &&\n-\ttest_subcommand git pack-objects --all-progress-implied --revs --stdout \\\n-\t\t--thin --delta-base-offset -q \n---\n builtin/bundle.c | 33 ++++++++++++++++-----------------\n 1 file changed, 16 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 1e170e9278..bfafadc984 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -70,35 +70,34 @@ static int parse_options_cmd_bundle(int argc,\n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\t struct repository *repo UNUSED) {\n \tstruct strvec pack_opts = STRVEC_INIT;\n+\tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n-\tint ret;\n \tstruct option options[] = {\n-\t\tOPT_PASSTHRU_ARGV('q', \"quiet\", &pack_opts, NULL,\n-\t\t\t\t N_(\"do not show progress meter\"),\n-\t\t\t\t PARSE_OPT_NOARG),\n-\t\tOPT_PASSTHRU_ARGV(0, \"progress\", &pack_opts, NULL,\n-\t\t\t\t N_(\"show progress meter\"),\n-\t\t\t\t PARSE_OPT_NOARG),\n-\t\tOPT_PASSTHRU_ARGV(0, \"all-progress\", &pack_opts, NULL,\n-\t\t\t\t N_(\"historical; same as --progress\"),\n-\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n-\t\tOPT_PASSTHRU_ARGV(0, \"all-progress-implied\", &pack_opts, NULL,\n-\t\t\t\t N_(\"historical; does nothing\"),\n-\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n+\t\tOPT_NEGBIT('q', \"quiet\", &progress,\n+\t\t\t N_(\"do not show progress meter\"), 1),\n+\t\tOPT_BIT(0, \"progress\", &progress,\n+\t\t\tN_(\"show progress meter\"), 1),\n+\t\tOPT_BIT_F(0, \"all-progress\", &progress,\n+\t\t\t N_(\"historical; same as --progress\"), 1,\n+\t\t\t PARSE_OPT_HIDDEN),\n+\t\tOPT_NOOP_NOARG(0, \"all-progress-implied\"),\n \t\tOPT_INTEGER(0, \"version\", &version,\n \t\t\t N_(\"specify bundle format version\")),\n \t\tOPT_END()\n \t};\n \tchar *bundle_file;\n-\n-\tif (isatty(STDERR_FILENO))\n-\t\tstrvec_push(&pack_opts, \"--progress\");\n-\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n+\tint ret;\n \n \targc = parse_options_cmd_bundle(argc, argv, prefix,\n \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n \t/* bundle internals use argv[1] as further parameters */\n \n+\tif (progress)\n+\t\tstrvec_push(&pack_opts, \"--progress\");\n+\telse\n+\t\tstrvec_push(&pack_opts, \"--quiet\");\n+\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n+\n \tif (!startup_info->have_repository)\n \t\tdie(_(\"Need a repository to create a bundle.\"));\n \tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550877","messageId":"20260820-b4-pks-odb-generate-pack-v3-5-bc42252f6169@pks.im","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im","subject":"[PATCH v3 5/6] bundle: get (mostly) rid of `the_repository`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T07:55:29Z","receivedAt":"2026-08-20T07:55:50Z","isPatch":true,"body":"Refactor \"bundle.c\" so that we don't depend on `the_repository` anymore.\nThis conversion is trivial for most of the part, as we already have a\nrepository available in all calling conexts.\n\nThe only exception is that we use `get_log_output_encoding()`, which\nimplicitly depends on `the_repository`. Add an `extern` declaration for\nthis function so that we can drop `USE_THE_REPOSITORY_VARIABLE` and not\naccidentally introduce more uses of `the_repository`.\n\nSigned-off-by: Patrick Steinhardt \n---\n bundle.c | 32 +++++++++++++++++++++-----------\n 1 file changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex b64716f252..a9330bf0d3 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -1,4 +1,3 @@\n-#define USE_THE_REPOSITORY_VARIABLE\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"git-compat-util.h\"\n@@ -21,6 +20,13 @@\n #include \"connected.h\"\n #include \"write-or-die.h\"\n \n+/*\n+ * NEEDSWORK: this function implicitly depends on `the_repository` and is not\n+ * available because we dropped USE_THE_REPOSITORY_VARIABLE. We can remove the\n+ * declaration once it's accessible via `repo_config_values`.\n+ */\n+extern const char *get_log_output_encoding(void);\n+\n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n static struct {\n@@ -294,7 +300,8 @@ int list_bundle_refs(struct bundle_header *header, int argc, const char **argv)\n \treturn list_refs(&header->references, argc, argv);\n }\n \n-static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n+static int is_tag_in_date_range(struct repository *repo,\n+\t\t\t\tstruct object *tag, struct rev_info *revs)\n {\n \tsize_t size;\n \tenum object_type type;\n@@ -305,7 +312,7 @@ static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n \tif (revs->max_age == -1 && revs->min_age == -1)\n \t\tgoto out;\n \n-\tbuf = odb_read_object(the_repository->objects, &tag->oid, &type, &size);\n+\tbuf = odb_read_object(repo->objects, &tag->oid, &type, &size);\n \tif (!buf)\n \t\tgoto out;\n \tline = memmem(buf, size, \"\\ntagger \", 8);\n@@ -362,7 +369,8 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\tstruct object *object = revs->pending.objects[i].item;\n \t\tif (object->flags & UNINTERESTING)\n \t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n-\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid), the_hash_algo->hexsz);\n+\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid),\n+\t\t\t revs->repo->hash_algo->hexsz);\n \t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n \t}\n \tclose(pack_objects.in);\n@@ -395,10 +403,10 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \n \t\tif (e->item->flags & UNINTERESTING)\n \t\t\tcontinue;\n-\t\tif (repo_dwim_ref(the_repository, e->name, strlen(e->name),\n+\t\tif (repo_dwim_ref(revs->repo, e->name, strlen(e->name),\n \t\t\t\t &oid, &ref, 0) != 1)\n \t\t\tgoto skip_write_ref;\n-\t\tif (refs_read_ref_full(get_main_ref_store(the_repository), e->name, RESOLVE_REF_READING, &oid, &flag))\n+\t\tif (refs_read_ref_full(get_main_ref_store(revs->repo), e->name, RESOLVE_REF_READING, &oid, &flag))\n \t\t\tflag = 0;\n \t\tdisplay_ref = (flag & REF_ISSYMREF) ? e->name : ref;\n \n@@ -406,7 +414,7 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \t\t\tgoto skip_write_ref;\n \n \t\tif (e->item->type == OBJ_TAG &&\n-\t\t\t\t!is_tag_in_date_range(e->item, revs)) {\n+\t\t\t\t!is_tag_in_date_range(revs->repo, e->item, revs)) {\n \t\t\te->item->flags |= UNINTERESTING;\n \t\t\tgoto skip_write_ref;\n \t\t}\n@@ -428,7 +436,8 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \n \t\tref_count++;\n \t\tstrset_add(&objects, display_ref);\n-\t\twrite_or_die(bundle_fd, oid_to_hex(&e->item->oid), the_hash_algo->hexsz);\n+\t\twrite_or_die(bundle_fd, oid_to_hex(&e->item->oid),\n+\t\t\t revs->repo->hash_algo->hexsz);\n \t\twrite_or_die(bundle_fd, \" \", 1);\n \t\twrite_or_die(bundle_fd, display_ref, strlen(display_ref));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n@@ -507,7 +516,7 @@ int create_bundle(struct repository *r, const char *path,\n \t * SHA1.\n \t * 2. @filter is required because we parsed an object filter.\n \t */\n-\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1_LEGACY] || revs.filter.choice)\n+\tif (r->hash_algo != &hash_algos[GIT_HASH_SHA1_LEGACY] || revs.filter.choice)\n \t\tmin_version = 3;\n \n \tif (argc > 1) {\n@@ -528,14 +537,15 @@ int create_bundle(struct repository *r, const char *path,\n \tif (version < 2 || version > 3) {\n \t\tdie(_(\"unsupported bundle version %d\"), version);\n \t} else if (version < min_version) {\n-\t\tdie(_(\"cannot write bundle version %d with algorithm %s\"), version, the_hash_algo->name);\n+\t\tdie(_(\"cannot write bundle version %d with algorithm %s\"), version,\n+\t\t r->hash_algo->name);\n \t} else if (version == 2) {\n \t\twrite_or_die(bundle_fd, v2_bundle_signature, strlen(v2_bundle_signature));\n \t} else {\n \t\tconst char *capability = \"@object-format=\";\n \t\twrite_or_die(bundle_fd, v3_bundle_signature, strlen(v3_bundle_signature));\n \t\twrite_or_die(bundle_fd, capability, strlen(capability));\n-\t\twrite_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));\n+\t\twrite_or_die(bundle_fd, r->hash_algo->name, strlen(r->hash_algo->name));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n \n \t\tif (revs.filter.choice) {\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550878","messageId":"20260820-b4-pks-odb-generate-pack-v3-6-bc42252f6169@pks.im","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im","subject":"[PATCH v3 6/6] bundle: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T07:55:30Z","receivedAt":"2026-08-20T07:55:52Z","isPatch":true,"body":"git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\ndata that gets appended to the bundle header. While bundles are not\npart of the wire protocol, they are a transfer mechanism for packs all\nthe same, so convert them to use the pack generation interface of the\nobject database as well.\n\nThis makes the pack generator the single spawn point for all pack\nstreams that leave the repository, leaving only local maintenance tasks\nlike git-repack(1) with direct knowledge of git-pack-objects(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/bundle.c | 10 +-------\n bundle.c | 69 ++++++++++++++++++++++++++++----------------------------\n bundle.h | 3 +--\n 3 files changed, 37 insertions(+), 45 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex bfafadc984..de86e092a6 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -69,7 +69,6 @@ static int parse_options_cmd_bundle(int argc,\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\t struct repository *repo UNUSED) {\n-\tstruct strvec pack_opts = STRVEC_INIT;\n \tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n \tstruct option options[] = {\n@@ -92,16 +91,9 @@ static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n \t/* bundle internals use argv[1] as further parameters */\n \n-\tif (progress)\n-\t\tstrvec_push(&pack_opts, \"--progress\");\n-\telse\n-\t\tstrvec_push(&pack_opts, \"--quiet\");\n-\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n-\n \tif (!startup_info->have_repository)\n \t\tdie(_(\"Need a repository to create a bundle.\"));\n-\tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n-\tstrvec_clear(&pack_opts);\n+\tret = !!create_bundle(the_repository, bundle_file, argc, argv, version, progress);\n \tfree(bundle_file);\n \treturn ret;\n }\ndiff --git a/bundle.c b/bundle.c\nindex a9330bf0d3..f55a521b2a 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -332,51 +332,52 @@ static int is_tag_in_date_range(struct repository *repo,\n \n \n /* Write the pack data to bundle_fd */\n-static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *pack_options)\n+static int write_pack_data(int bundle_fd, struct rev_info *revs, int progress)\n {\n-\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n+\tint ret = 0;\n \tint i;\n \n-\tstrvec_pushl(&pack_objects.args,\n-\t\t \"pack-objects\",\n-\t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n-\t\t NULL);\n-\tstrvec_pushv(&pack_objects.args, pack_options->v);\n+\topts.thin = 1;\n+\topts.ofs_delta = 1;\n+\tif (progress)\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n \tif (revs->filter.choice)\n-\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t list_objects_filter_spec(&revs->filter));\n-\tpack_objects.in = -1;\n-\tpack_objects.out = bundle_fd;\n-\tpack_objects.git_cmd = 1;\n+\t\topts.filter_spec = list_objects_filter_spec(&revs->filter);\n \n \t/*\n-\t * start_command() will close our descriptor if it's >1. Duplicate it\n-\t * to avoid surprising the caller.\n+\t * The pack generator will consume our descriptor if it's >1.\n+\t * Duplicate it to avoid surprising the caller.\n \t */\n-\tif (pack_objects.out > 1) {\n-\t\tpack_objects.out = dup(pack_objects.out);\n-\t\tif (pack_objects.out < 0) {\n-\t\t\terror_errno(_(\"unable to dup bundle descriptor\"));\n-\t\t\tchild_process_clear(&pack_objects);\n-\t\t\treturn -1;\n-\t\t}\n+\topts.pack_fd = bundle_fd;\n+\tif (opts.pack_fd > 1) {\n+\t\topts.pack_fd = dup(bundle_fd);\n+\t\tif (opts.pack_fd < 0)\n+\t\t\treturn error_errno(_(\"unable to dup bundle descriptor\"));\n \t}\n \n-\tif (start_command(&pack_objects))\n-\t\treturn error(_(\"Could not spawn pack-objects\"));\n-\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object *object = revs->pending.objects[i].item;\n \t\tif (object->flags & UNINTERESTING)\n-\t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n-\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid),\n-\t\t\t revs->repo->hash_algo->hexsz);\n-\t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n+\t\t\toid_array_append(&opts.haves, &object->oid);\n+\t\telse\n+\t\t\toid_array_append(&opts.wants, &object->oid);\n \t}\n-\tclose(pack_objects.in);\n-\tif (finish_command(&pack_objects))\n-\t\treturn error(_(\"pack-objects died\"));\n-\treturn 0;\n+\n+\tif (odb_generate_pack(revs->repo->objects, &generator, &opts)) {\n+\t\tret = error(_(\"Could not spawn pack-objects\"));\n+\t\tgoto out;\n+\t}\n+\n+\tif (odb_pack_generator_finish(generator)) {\n+\t\tret = error(_(\"pack-objects died\"));\n+\t\tgoto out;\n+\t}\n+\n+out:\n+\todb_generate_pack_options_release(&opts);\n+\treturn ret;\n }\n \n /*\n@@ -485,7 +486,7 @@ static void write_bundle_prerequisites(struct commit *commit, void *data)\n }\n \n int create_bundle(struct repository *r, const char *path,\n-\t\t int argc, const char **argv, struct strvec *pack_options, int version)\n+\t\t int argc, const char **argv, int version, int progress)\n {\n \tstruct lock_file lock = LOCK_INIT;\n \tint bundle_fd = -1;\n@@ -594,7 +595,7 @@ int create_bundle(struct repository *r, const char *path,\n \t}\n \n \t/* write pack */\n-\tif (write_pack_data(bundle_fd, &revs_copy, pack_options)) {\n+\tif (write_pack_data(bundle_fd, &revs_copy, progress)) {\n \t\tret = -1;\n \t\tgoto out;\n \t}\ndiff --git a/bundle.h b/bundle.h\nindex d664b2f2d6..471da23d1b 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -27,8 +27,7 @@ int read_bundle_header(const char *path, struct bundle_header *header);\n int read_bundle_header_fd(int fd, struct bundle_header *header,\n \t\t\t const char *report_path);\n int create_bundle(struct repository *r, const char *path,\n-\t\t int argc, const char **argv, struct strvec *pack_options,\n-\t\t int version);\n+\t\t int argc, const char **argv, int version, int progress);\n \n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550885","messageId":"CAOLa=ZSYyfOCs8Dr0Xdhv-=Q=j0z7vtfYqopDb07XuAm2PU84g@mail.gmail.com","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-1-bc42252f6169@pks.im","subject":"Re: [PATCH v3 1/6] odb: introduce interface to generate packfiles","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-20T10:16:37Z","receivedAt":"2026-08-20T10:16:50Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> Packfiles have two primary use cases:\n>\n> - They are used to store objects at rest in a Git repository.\n>\n> - They are used on the transport layer to transfer objects between two\n> repositories.\n>\n> The first class is closely tied to a given object database backend, and\n> as such this use is highly specific to how such a backend decides to\n> store its data. This shows in git-pack-objects(1), which is used by\n> git-repack(1) et al to optimize the object database, which supports lots\n> of options that are closely coupled with how data is stored.\n>\n> But the second class is quite a lot more generic: we don't care about\n> specifics of how the object database stores its objects, but to generate\n> the packfiles we only care about the object graph itself. Still, this\n> use case is also coupled with git-pack-objects(1).\n>\n> Unfortunately, because git-pack-objects(1) covers both classes, the\n> result is that it is very hard to port the whole command to properly\n> support pluggable object databases. There are simply way too many\n> options that an alternative implementation will have a very hard time to\n> support in the first place.\n>\n> And despite being hard to implement, it's also quite unnecessary to\n> implement those backend-specific options. Optimizing the object database\n> has already been made pluggable, and an alternative implementation is\n> unlikely to care about cruft packs, unpacked objects, keep packs and the\n> like. But we still need to make at least _parts_ of the packfile\n> generation pluggable so that backends can generate packfiles for the\n> transport layer itself.\n>\n> Introduce a new interface that lets backends generate a new packfile and\n> implement that interface for the \"files\" backend. The options supported\n> by the callback are exactly the set of options that are required for the\n> transport layer, but nothing more.\n>\n\nOkay so we also provide the implementation for the files odb here, which\ncalls 'git-pack-objects(1)' to generate the packfile.\n\nThe changes look in order.\n\n[snip]\n\n> diff --git a/odb/source.h b/odb/source.h\n> index d69f8e2d1c..e2129766fc 100644\n> --- a/odb/source.h\n> +++ b/odb/source.h\n> @@ -278,6 +278,23 @@ struct odb_source {\n> \t */\n> \tbool (*optimize_required)(struct odb_source *source,\n> \t\t\t\t const struct odb_optimize_options *opts);\n> +\n> +\t/*\n> +\t * This callback is expected to start generating a packfile with the\n> +\t * given options. The pack shall be generated asynchronously so that\n> +\t * the caller can consume the pack data and progress output while the\n> +\t * pack is being generated.\n> +\t *\n> +\t * This callback is optional. Sources that cannot generate packfiles\n> +\t * shall leave it unset.\n> +\t *\n> +\t * The callback is expected to return 0 on success and populate the\n> +\t * `out` pointer with the pack generator, a negative error code\n> +\t * otherwise.\n> +\t */\n> +\tint (*generate_pack)(struct odb_source *source,\n> +\t\t\t struct odb_pack_generator **out,\n> +\t\t\t const struct odb_generate_pack_options *opts);\n> };\n>\n\nNit: I see that `source` is unused anyways, do we need to pass it in? Or\nis just for consistency?\n\n> /*\n> @@ -520,4 +537,20 @@ static inline bool odb_source_optimize_required(struct odb_source *source,\n> \treturn source->optimize_required(source, opts);\n> }\n>\n> +/*\n> + * Start generating a packfile from the given source with the given options.\n> + * The pack is generated asynchronously; the caller is expected to consume the\n> + * file descriptors exposed via the pack generator and to then wait for\n> + * completion via `odb_pack_generator_finish()`.\n> + *\n> + * Returns 0 on success and populates the `out` pointer with the pack\n> + * generator, a negative error code otherwise.\n> + */\n> +static inline int odb_source_generate_pack(struct odb_source *source,\n> +\t\t\t\t\t struct odb_pack_generator **out,\n> +\t\t\t\t\t const struct odb_generate_pack_options *opts)\n> +{\n> +\treturn source->generate_pack(source, out, opts);\n> +}\n> +\n> #endif\n>\n> --\n> 2.55.0.822.g20453c30eb.dirty\n"},{"id":"550886","messageId":"CAOLa=ZQcZ93R6wRyDiQtyATBNfj_6Eu0zXtEx7kbfzihvyP5qg@mail.gmail.com","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-2-bc42252f6169@pks.im","subject":"Re: [PATCH v3 2/6] upload-pack: generate packfiles via the object database","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-20T10:24:09Z","receivedAt":"2026-08-20T10:24:12Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> When serving a fetch, git-upload-pack(1) spawns git-pack-objects(1)\n> directly to generate the packfile that gets sent to the client. This\n> hard-codes the assumption that the object database is able to serve\n> packfiles via git-pack-objects(1), which is specific to the \"files\"\n> backend.\n>\n\nNaive question, the previous patch says that only the primary odb source\nwill be used to generate the packfile and we added the implementation\nfor the files backend.\n\nDoes this mean that this will only work if the files backend is the\nprimary backend?\n\n> Convert git-upload-pack(1) to instead use the pack generation interface\n> of the object database.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> upload-pack.c | 125 +++++++++++++++++++++-------------------------------------\n> 1 file changed, 45 insertions(+), 80 deletions(-)\n>\n> diff --git a/upload-pack.c b/upload-pack.c\n> index a52856d869..75a857eaa8 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -197,11 +197,11 @@ static void send_client_data(int fd, const char *data, ssize_t sz,\n> \twrite_or_die(fd, data, sz);\n> }\n>\n> -static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n> +static int append_one_shallow(const struct commit_graft *graft, void *cb_data)\n> {\n> -\tFILE *fp = cb_data;\n> +\tstruct oid_array *shallows = cb_data;\n> \tif (graft->nr_parent == -1)\n> -\t\tfprintf(fp, \"--shallow %s\\n\", oid_to_hex(&graft->oid));\n> +\t\toid_array_append(shallows, &graft->oid);\n> \treturn 0;\n> }\n>\n\nOkay makes sense, we now append to an array\n\n> @@ -299,7 +299,8 @@ static int relay_pack_data(int pack_objects_out, struct output_state *os,\n> static void create_pack_file(struct upload_pack_data *pack_data,\n> \t\t\t const struct string_list *uri_protocols)\n> {\n> -\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n> +\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n> +\tstruct odb_pack_generator *generator;\n> \tstruct output_state *output_state = xcalloc(1, sizeof(struct output_state));\n> \tchar progress[128];\n> \tchar abort_msg[] = \"aborting due to possible repository \"\n> @@ -307,78 +308,42 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n> \tuint64_t last_sent_ms = 0;\n> \tssize_t sz;\n> \tint i;\n> -\tFILE *pipe_fd;\n> -\n> -\tif (!pack_data->pack_objects_hook)\n> -\t\tpack_objects.git_cmd = 1;\n> -\telse {\n> -\t\tstrvec_push(&pack_objects.args, pack_data->pack_objects_hook);\n> -\t\tstrvec_push(&pack_objects.args, \"git\");\n> -\t\tpack_objects.use_shell = 1;\n> -\t}\n>\n> \tif (pack_data->shallow_nr) {\n> -\t\tstrvec_push(&pack_objects.args, \"--shallow-file\");\n> -\t\tstrvec_push(&pack_objects.args, \"\");\n> -\t}\n> -\tstrvec_push(&pack_objects.args, \"pack-objects\");\n> -\tstrvec_push(&pack_objects.args, \"--revs\");\n> -\tif (pack_data->use_thin_pack)\n> -\t\tstrvec_push(&pack_objects.args, \"--thin\");\n> -\n> -\tstrvec_push(&pack_objects.args, \"--stdout\");\n> -\tif (pack_data->shallow_nr)\n> -\t\tstrvec_push(&pack_objects.args, \"--shallow\");\n> -\tif (!pack_data->no_progress)\n> -\t\tstrvec_push(&pack_objects.args, \"--progress\");\n> -\tif (pack_data->use_ofs_delta)\n> -\t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n> -\tif (pack_data->use_include_tag)\n> -\t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n> -\tif (repo_has_accepted_promisor_remote(the_repository))\n> -\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n> -\tif (pack_data->filter_options.choice) {\n> -\t\tconst char *spec =\n> -\t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n> -\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\", spec);\n> -\t}\n> -\tif (uri_protocols) {\n> -\t\tfor (i = 0; i < uri_protocols->nr; i++)\n> -\t\t\tstrvec_pushf(&pack_objects.args, \"--uri-protocol=%s\",\n> -\t\t\t\t\t uri_protocols->items[i].string);\n> +\t\tfor_each_commit_graft(append_one_shallow, &opts.shallows);\n> +\t\topts.shallow = 1;\n> \t}\n> -\n> -\tpack_objects.in = -1;\n> -\tpack_objects.out = -1;\n> -\tpack_objects.err = -1;\n> -\tpack_objects.clean_on_exit = 1;\n> -\n> -\tif (start_command(&pack_objects))\n> -\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n> -\n> -\tpipe_fd = xfdopen(pack_objects.in, \"w\");\n> -\n> -\tif (pack_data->shallow_nr)\n> -\t\tfor_each_commit_graft(write_one_shallow, pipe_fd);\n> -\n> \tfor (i = 0; i < pack_data->want_obj.nr; i++)\n> -\t\tfprintf(pipe_fd, \"%s\\n\",\n> -\t\t\toid_to_hex(&pack_data->want_obj.objects[i].item->oid));\n> -\tfprintf(pipe_fd, \"--not\\n\");\n> +\t\toid_array_append(&opts.wants,\n> +\t\t\t\t &pack_data->want_obj.objects[i].item->oid);\n> \tfor (i = 0; i < pack_data->have_obj.nr; i++)\n> -\t\tfprintf(pipe_fd, \"%s\\n\",\n> -\t\t\toid_to_hex(&pack_data->have_obj.objects[i].item->oid));\n> +\t\toid_array_append(&opts.haves,\n> +\t\t\t\t &pack_data->have_obj.objects[i].item->oid);\n> \tfor (i = 0; i < pack_data->extra_edge_obj.nr; i++)\n> -\t\tfprintf(pipe_fd, \"%s\\n\",\n> -\t\t\toid_to_hex(&pack_data->extra_edge_obj.objects[i].item->oid));\n> -\tfprintf(pipe_fd, \"\\n\");\n> -\tfflush(pipe_fd);\n> -\tfclose(pipe_fd);\n> -\n> -\t/* We read from pack_objects.err to capture stderr output for\n> -\t * progress bar, and pack_objects.out to capture the pack data.\n> -\t */\n> +\t\toid_array_append(&opts.haves,\n> +\t\t\t\t &pack_data->extra_edge_obj.objects[i].item->oid);\n> +\n> +\topts.thin = pack_data->use_thin_pack;\n> +\tif (!pack_data->no_progress)\n> +\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_STANDARD;\n> +\topts.ofs_delta = pack_data->use_ofs_delta;\n> +\topts.include_tag = pack_data->use_include_tag;\n> +\topts.missing_allow_promisor = repo_has_accepted_promisor_remote(the_repository);\n> +\tif (pack_data->filter_options.choice)\n> +\t\topts.filter_spec = expand_list_objects_filter_spec(&pack_data->filter_options);\n> +\topts.uri_protocols = uri_protocols;\n> +\topts.pack_objects_hook = pack_data->pack_objects_hook;\n> +\topts.pack_fd = -1;\n> +\topts.progress_fd = -1;\n> +\n> +\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n> +\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n\nNit: should we still talk about 'forking' here? As far as upload-pack is\nconsidered, it handed over the task to the odb, 'forking' is an internal\nimplementation detail.\n\n[snip]\n\nrest looks good!\n"},{"id":"550888","messageId":"CAOLa=ZT-Tw2gMVCBS7b62VSkSJAHyVwO7dffsFW3Q4QzQe1JZg@mail.gmail.com","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-4-bc42252f6169@pks.im","subject":"Re: [PATCH v3 4/6] builtin/bundle: refactor option handling for progress meter","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-20T11:17:30Z","receivedAt":"2026-08-20T11:17:33Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> The git-bundle(1) command has a couple of command line options that\n> relate to whether or not progress should be reported. These options\n> match the options that git-pack-objects(1) expects, and consequently\n> they mostly get passed through to it directly.\n>\n> This results in somewhat of a confusing interface: there are four\n> different options that relate to whether or not progress should be\n> displayed and how verbose it should be. But in reality, there's really\n> only two modes:\n>\n> - \"--progress\" and \"--all-progress\" result in the same outcome, which\n> is also documented as such.\n>\n> - \"--all-progress-implied\" does nothing as we pass that argument to\n> git-pack-objects(1) unconditionally anyway.\n>\n> So in the end, the options only control whether or not progress should\n> be displayed at all, nothing else.\n>\n> Refactor the interface to instead use a simple `progress` boolean. This\n> makes argument handling a lot more straight-forward and it prepares us\n> for the next commit, where we're migrating git-bundle(1) to the generic\n> interface for generating a packfile.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/bundle.c | 33 ++++++++++++++++-----------------\n> 1 file changed, 16 insertions(+), 17 deletions(-)\n>\n> diff --git a/builtin/bundle.c b/builtin/bundle.c\n> index 1e170e9278..bfafadc984 100644\n> --- a/builtin/bundle.c\n> +++ b/builtin/bundle.c\n> @@ -70,35 +70,34 @@ static int parse_options_cmd_bundle(int argc,\n> static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n> \t\t\t struct repository *repo UNUSED) {\n> \tstruct strvec pack_opts = STRVEC_INIT;\n> +\tint progress = isatty(STDERR_FILENO);\n> \tint version = -1;\n> -\tint ret;\n> \tstruct option options[] = {\n> -\t\tOPT_PASSTHRU_ARGV('q', \"quiet\", &pack_opts, NULL,\n> -\t\t\t\t N_(\"do not show progress meter\"),\n> -\t\t\t\t PARSE_OPT_NOARG),\n> -\t\tOPT_PASSTHRU_ARGV(0, \"progress\", &pack_opts, NULL,\n> -\t\t\t\t N_(\"show progress meter\"),\n> -\t\t\t\t PARSE_OPT_NOARG),\n> -\t\tOPT_PASSTHRU_ARGV(0, \"all-progress\", &pack_opts, NULL,\n> -\t\t\t\t N_(\"historical; same as --progress\"),\n> -\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n> -\t\tOPT_PASSTHRU_ARGV(0, \"all-progress-implied\", &pack_opts, NULL,\n> -\t\t\t\t N_(\"historical; does nothing\"),\n> -\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n> +\t\tOPT_NEGBIT('q', \"quiet\", &progress,\n> +\t\t\t N_(\"do not show progress meter\"), 1),\n> +\t\tOPT_BIT(0, \"progress\", &progress,\n> +\t\t\tN_(\"show progress meter\"), 1),\n> +\t\tOPT_BIT_F(0, \"all-progress\", &progress,\n> +\t\t\t N_(\"historical; same as --progress\"), 1,\n> +\t\t\t PARSE_OPT_HIDDEN),\n> +\t\tOPT_NOOP_NOARG(0, \"all-progress-implied\"),\n> \t\tOPT_INTEGER(0, \"version\", &version,\n> \t\t\t N_(\"specify bundle format version\")),\n> \t\tOPT_END()\n> \t};\n>\n\nThis is much nicer to read.\n\n> \tchar *bundle_file;\n> -\n> -\tif (isatty(STDERR_FILENO))\n> -\t\tstrvec_push(&pack_opts, \"--progress\");\n> -\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n> +\tint ret;\n>\n> \targc = parse_options_cmd_bundle(argc, argv, prefix,\n> \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n> \t/* bundle internals use argv[1] as further parameters */\n>\n> +\tif (progress)\n> +\t\tstrvec_push(&pack_opts, \"--progress\");\n> +\telse\n> +\t\tstrvec_push(&pack_opts, \"--quiet\");\n> +\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n> +\n>\n\nTangent: While trying to understand this patch, I noticed that we only\nlist the '-q' shortform for '--quiet' in the 'git-pack-objects(1)'\ndocumentation.\n\n> \tif (!startup_info->have_repository)\n> \t\tdie(_(\"Need a repository to create a bundle.\"));\n> \tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n>\n> --\n> 2.55.0.822.g20453c30eb.dirty\n"},{"id":"550889","messageId":"CAOLa=ZQ7-_=T1NSXY433oME8OoddJOuLX0wmdbk2ocQ0JTAuKQ@mail.gmail.com","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-6-bc42252f6169@pks.im","subject":"Re: [PATCH v3 6/6] bundle: generate packfiles via the object database","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-20T11:19:45Z","receivedAt":"2026-08-20T11:19:49Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\n> data that gets appended to the bundle header. While bundles are not\n> part of the wire protocol, they are a transfer mechanism for packs all\n> the same, so convert them to use the pack generation interface of the\n> object database as well.\n>\n> This makes the pack generator the single spawn point for all pack\n> streams that leave the repository, leaving only local maintenance tasks\n> like git-repack(1) with direct knowledge of git-pack-objects(1).\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/bundle.c | 10 +-------\n> bundle.c | 69 ++++++++++++++++++++++++++++----------------------------\n> bundle.h | 3 +--\n> 3 files changed, 37 insertions(+), 45 deletions(-)\n>\n> diff --git a/builtin/bundle.c b/builtin/bundle.c\n> index bfafadc984..de86e092a6 100644\n> --- a/builtin/bundle.c\n> +++ b/builtin/bundle.c\n> @@ -69,7 +69,6 @@ static int parse_options_cmd_bundle(int argc,\n>\n> static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n> \t\t\t struct repository *repo UNUSED) {\n\nThis '{' should be on the next line, but that's not on you :)\n\n[snip]\n\nThe rest looks good.\n"},{"id":"550890","messageId":"CAOLa=ZRsVjRrwzAf==SmevATf+OWoHdnHwUbvi1=M6foBRzLnA@mail.gmail.com","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im","subject":"Re: [PATCH v3 0/6] odb: make packfile generation pluggable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-20T11:20:15Z","receivedAt":"2026-08-20T11:20:20Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this patch series makes packfile generation pluggable.\n>\n> Note that this series only makes those parts pluggable that are required\n> for the transport layer. The other parts that relate to packfile\n> generation as required by our repository maintenance is kept as-is, as\n> there is a bunch of options there that are way too specific to the\n> \"files\" backend to be portable. This should ultimately not be much of a\n> problem though, as maintenance itself is already pluggable in the first\n> place.\n>\n> It's a bit of a shame though for git-pack-objects(1), which still isn't\n> usable with alternate backends. I tried several times to find good\n> solutions for making it fully pluggable, but due to the backend-specific\n> options it's an utter mess. I want to eventually address this though:\n> same as with git-refs(1), I want to introduce git-objects(1) to care\n> about all things ODB. And as part of that command we can also introduce\n> a command that generates packfiles in a generic fashion, without all the\n> cruft that git-pack-objects(1) has. This is part of a future patch\n> series though.\n>\n> Changes in v3:\n> - Fix a use-after-scope bug on abnormal exit when child processes are\n> cleaned up via `mark_child_for_cleanup()`, as noticed by Elijah.\n> - Link to v2: https://patch.msgid.link/20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im\n>\n\nDropping in to review the new version, the changes look good!\n\n[snip]\n"},{"id":"550891","messageId":"aobnP3bJ1SQpYoFa@pks.im","threadId":"66136","inReplyTo":"CAOLa=ZQ7-_=T1NSXY433oME8OoddJOuLX0wmdbk2ocQ0JTAuKQ@mail.gmail.com","subject":"Re: [PATCH v3 6/6] bundle: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T11:38:39Z","receivedAt":"2026-08-20T11:38:47Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 07:19:45AM -0400, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\n> > data that gets appended to the bundle header. While bundles are not\n> > part of the wire protocol, they are a transfer mechanism for packs all\n> > the same, so convert them to use the pack generation interface of the\n> > object database as well.\n> >\n> > This makes the pack generator the single spawn point for all pack\n> > streams that leave the repository, leaving only local maintenance tasks\n> > like git-repack(1) with direct knowledge of git-pack-objects(1).\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > builtin/bundle.c | 10 +-------\n> > bundle.c | 69 ++++++++++++++++++++++++++++----------------------------\n> > bundle.h | 3 +--\n> > 3 files changed, 37 insertions(+), 45 deletions(-)\n> >\n> > diff --git a/builtin/bundle.c b/builtin/bundle.c\n> > index bfafadc984..de86e092a6 100644\n> > --- a/builtin/bundle.c\n> > +++ b/builtin/bundle.c\n> > @@ -69,7 +69,6 @@ static int parse_options_cmd_bundle(int argc,\n> >\n> > static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n> > \t\t\t struct repository *repo UNUSED) {\n> \n> This '{' should be on the next line, but that's not on you :)\n\nI can sneak in a small fixup. Doesn't hurt, I guess.\n\nPatrick\n"},{"id":"550892","messageId":"aobnSsQVOCSoR4kE@pks.im","threadId":"66136","inReplyTo":"CAOLa=ZSYyfOCs8Dr0Xdhv-=Q=j0z7vtfYqopDb07XuAm2PU84g@mail.gmail.com","subject":"Re: [PATCH v3 1/6] odb: introduce interface to generate packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T11:38:50Z","receivedAt":"2026-08-20T11:38:54Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 03:16:37AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> > diff --git a/odb/source.h b/odb/source.h\n> > index d69f8e2d1c..e2129766fc 100644\n> > --- a/odb/source.h\n> > +++ b/odb/source.h\n> > @@ -278,6 +278,23 @@ struct odb_source {\n> > \t */\n> > \tbool (*optimize_required)(struct odb_source *source,\n> > \t\t\t\t const struct odb_optimize_options *opts);\n> > +\n> > +\t/*\n> > +\t * This callback is expected to start generating a packfile with the\n> > +\t * given options. The pack shall be generated asynchronously so that\n> > +\t * the caller can consume the pack data and progress output while the\n> > +\t * pack is being generated.\n> > +\t *\n> > +\t * This callback is optional. Sources that cannot generate packfiles\n> > +\t * shall leave it unset.\n> > +\t *\n> > +\t * The callback is expected to return 0 on success and populate the\n> > +\t * `out` pointer with the pack generator, a negative error code\n> > +\t * otherwise.\n> > +\t */\n> > +\tint (*generate_pack)(struct odb_source *source,\n> > +\t\t\t struct odb_pack_generator **out,\n> > +\t\t\t const struct odb_generate_pack_options *opts);\n> > };\n> >\n> \n> Nit: I see that `source` is unused anyways, do we need to pass it in? Or\n> is just for consistency?\n\nOur specific implementation does not use it, but others might want. So\nit's mostly for consistency to give the callback enough context.\n\nPatrick\n"},{"id":"550893","messageId":"aobnT6mmINHBmV4g@pks.im","threadId":"66136","inReplyTo":"CAOLa=ZQcZ93R6wRyDiQtyATBNfj_6Eu0zXtEx7kbfzihvyP5qg@mail.gmail.com","subject":"Re: [PATCH v3 2/6] upload-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T11:38:55Z","receivedAt":"2026-08-20T11:39:00Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 06:24:09AM -0400, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > When serving a fetch, git-upload-pack(1) spawns git-pack-objects(1)\n> > directly to generate the packfile that gets sent to the client. This\n> > hard-codes the assumption that the object database is able to serve\n> > packfiles via git-pack-objects(1), which is specific to the \"files\"\n> > backend.\n> >\n> \n> Naive question, the previous patch says that only the primary odb source\n> will be used to generate the packfile and we added the implementation\n> for the files backend.\n> \n> Does this mean that this will only work if the files backend is the\n> primary backend?\n\nThe primary backend is the one that will generate packs in the first\nplace. For now, the only primary backend that we ever have is the\n\"files\" backend. But if we ever add a different backend then that would\nof course implement its own implementation for generating packs.\n\nSo at the status quo: yes, but with the added infrastructure it's now\npluggable and can be implemented by other backends, too.\n\n> > diff --git a/upload-pack.c b/upload-pack.c\n> > index a52856d869..75a857eaa8 100644\n> > --- a/upload-pack.c\n> > +++ b/upload-pack.c\n[snip]\n> > +\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n> > +\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n> \n> Nit: should we still talk about 'forking' here? As far as upload-pack is\n> considered, it handed over the task to the odb, 'forking' is an internal\n> implementation detail.\n\nFair, we should probably just say \"unable to pack objects\" here.\n\nPatrick\n"},{"id":"550894","messageId":"aobnos_v8xko0w57@pks.im","threadId":"66136","inReplyTo":"CAOLa=ZRsVjRrwzAf==SmevATf+OWoHdnHwUbvi1=M6foBRzLnA@mail.gmail.com","subject":"Re: [PATCH v3 0/6] odb: make packfile generation pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-20T11:40:18Z","receivedAt":"2026-08-20T11:40:24Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 07:20:15AM -0400, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > Hi,\n> >\n> > this patch series makes packfile generation pluggable.\n> >\n> > Note that this series only makes those parts pluggable that are required\n> > for the transport layer. The other parts that relate to packfile\n> > generation as required by our repository maintenance is kept as-is, as\n> > there is a bunch of options there that are way too specific to the\n> > \"files\" backend to be portable. This should ultimately not be much of a\n> > problem though, as maintenance itself is already pluggable in the first\n> > place.\n> >\n> > It's a bit of a shame though for git-pack-objects(1), which still isn't\n> > usable with alternate backends. I tried several times to find good\n> > solutions for making it fully pluggable, but due to the backend-specific\n> > options it's an utter mess. I want to eventually address this though:\n> > same as with git-refs(1), I want to introduce git-objects(1) to care\n> > about all things ODB. And as part of that command we can also introduce\n> > a command that generates packfiles in a generic fashion, without all the\n> > cruft that git-pack-objects(1) has. This is part of a future patch\n> > series though.\n> >\n> > Changes in v3:\n> > - Fix a use-after-scope bug on abnormal exit when child processes are\n> > cleaned up via `mark_child_for_cleanup()`, as noticed by Elijah.\n> > - Link to v2: https://patch.msgid.link/20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im\n> >\n> \n> Dropping in to review the new version, the changes look good!\n\nThanks! I'll wait until tomorrow and then send another version with your\nnits addressed.\n\nPatrick\n"},{"id":"550920","messageId":"xmqqik54soy0.fsf@gitster.g","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-1-bc42252f6169@pks.im","subject":"Re: [PATCH v3 1/6] odb: introduce interface to generate packfiles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-08-20T17:04:55Z","receivedAt":"2026-08-20T17:04:57Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> Packfiles have two primary use cases:\n>\n> - They are used to store objects at rest in a Git repository.\n>\n> - They are used on the transport layer to transfer objects between two\n> repositories.\n>\n> The first class is closely tied to a given object database backend, and\n> as such this use is highly specific to how such a backend decides to\n> store its data. This shows in git-pack-objects(1), which is used by\n> git-repack(1) et al to optimize the object database, which supports lots\n> of options that are closely coupled with how data is stored.\n>\n> But the second class is quite a lot more generic: we don't care about\n> specifics of how the object database stores its objects, but to generate\n> the packfiles we only care about the object graph itself. Still, this\n> use case is also coupled with git-pack-objects(1).\n>\n> Unfortunately, because git-pack-objects(1) covers both classes, the\n> result is that it is very hard to port the whole command to properly\n> support pluggable object databases. There are simply way too many\n> options that an alternative implementation will have a very hard time to\n> support in the first place.\n>\n> And despite being hard to implement, it's also quite unnecessary to\n> implement those backend-specific options. Optimizing the object database\n> has already been made pluggable, and an alternative implementation is\n> unlikely to care about cruft packs, unpacked objects, keep packs and the\n> like. But we still need to make at least _parts_ of the packfile\n> generation pluggable so that backends can generate packfiles for the\n> transport layer itself.\n>\n> Introduce a new interface that lets backends generate a new packfile and\n> implement that interface for the \"files\" backend. The options supported\n> by the callback are exactly the set of options that are required for the\n> transport layer, but nothing more.\n>\n> This means that git-pack-objects(1) itself cannot be ported over to this\n> new interface, but as explained above that's a hard feat to pull off due\n> to the backend-specific features. Ideally though, we should expose the\n> ability to generate arbitrary packfiles using this interface. The intent\n> of this is to eventually introduce a git-objects(1) subcommand (similar\n> to git-refs(1)) that exposes generic interfaces for accessing everything\n> related to the object database. In that case, we are able to expose only\n> those options that are generic.\n>\n> Subsequent commits will convert git-upload-pack(1), git-send-pack(1) and\n> git-bundle(1) to use this interface.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> odb.c | 21 ++++++++\n> odb.h | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n> odb/source-files.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++\n> odb/source.h | 33 ++++++++++++\n> 4 files changed, 355 insertions(+)\n>\n> diff --git a/odb.c b/odb.c\n> index caf1d0f542..cd9d5b48bc 100644\n> --- a/odb.c\n> +++ b/odb.c\n> @@ -1046,6 +1046,27 @@ bool odb_optimize_required(struct object_database *odb,\n> \treturn odb_source_optimize_required(odb->sources, opts);\n> }\n> \n> +void odb_generate_pack_options_release(struct odb_generate_pack_options *opts)\n> +{\n> +\toid_array_clear(&opts->wants);\n> +\toid_array_clear(&opts->haves);\n> +\toid_array_clear(&opts->shallows);\n> +}\n> +\n> +int odb_generate_pack(struct object_database *odb,\n> +\t\t struct odb_pack_generator **out,\n> +\t\t const struct odb_generate_pack_options *opts)\n> +{\n> +\tif (!odb->sources->generate_pack)\n> +\t\treturn error(_(\"primary object source does not support generating packfiles\"));\n> +\treturn odb_source_generate_pack(odb->sources, out, opts);\n> +}\n\nPerhaps a stupid question but the opts->pack_fd is documented:\n\n> +struct odb_generate_pack_options {\n> ...\n> +\t/*\n> +\t * File descriptor that the generated pack shall be written to. If set\n> +\t * to `-1`, a pipe will be created and exposed via the pack generator's\n> +\t * `out` field. If set to `0`, the pack will be written to the standard\n> +\t * output stream. Otherwise, the provided descriptor will be written to\n> +\t * and is consumed by the generator.\n> +\t */\n> +\tint pack_fd;\n> +\n\nHere I assume that \"and is consumed by\" refers to \"generator writes\ninto it and then closes it when it is done\"?\n\nodb_source_generate_pack() delegate to source->generate_pack(),\nwhich I presume goes to odb_source_files_generate_pack(), which in\nturn assigns opts->pack_fd to cp->out and calls start_command(cp) to\nrun pack-objects. The file descriptor is closed when the process\nfinishes.\n\nWhat happens if the odb->sources[0] does not support .generate_pack?\nShould opts->pack_fd be \"consumed\" here to avoid leaking it, or we\ndo not have to worry about it because the caller will soon exit\nitself?\n"},{"id":"550947","messageId":"CAOLa=ZQLEg-Ufo0QUEpf2sxuJ=G=8zvT1+deDg7JzxNQTQivLg@mail.gmail.com","threadId":"66136","inReplyTo":"aobnT6mmINHBmV4g@pks.im","subject":"Re: [PATCH v3 2/6] upload-pack: generate packfiles via the object database","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-20T21:11:13Z","receivedAt":"2026-08-20T21:11:15Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> On Thu, Aug 20, 2026 at 06:24:09AM -0400, Karthik Nayak wrote:\n>> Patrick Steinhardt writes:\n>>\n>> > When serving a fetch, git-upload-pack(1) spawns git-pack-objects(1)\n>> > directly to generate the packfile that gets sent to the client. This\n>> > hard-codes the assumption that the object database is able to serve\n>> > packfiles via git-pack-objects(1), which is specific to the \"files\"\n>> > backend.\n>> >\n>>\n>> Naive question, the previous patch says that only the primary odb source\n>> will be used to generate the packfile and we added the implementation\n>> for the files backend.\n>>\n>> Does this mean that this will only work if the files backend is the\n>> primary backend?\n>\n> The primary backend is the one that will generate packs in the first\n> place. For now, the only primary backend that we ever have is the\n> \"files\" backend. But if we ever add a different backend then that would\n> of course implement its own implementation for generating packs.\n>\n> So at the status quo: yes, but with the added infrastructure it's now\n> pluggable and can be implemented by other backends, too.\n>\n\nOkay that makes sense. Thanks!\n\n>> > diff --git a/upload-pack.c b/upload-pack.c\n>> > index a52856d869..75a857eaa8 100644\n>> > --- a/upload-pack.c\n>> > +++ b/upload-pack.c\n> [snip]\n>> > +\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n>> > +\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n>>\n>> Nit: should we still talk about 'forking' here? As far as upload-pack is\n>> considered, it handed over the task to the odb, 'forking' is an internal\n>> implementation detail.\n>\n> Fair, we should probably just say \"unable to pack objects\" here.\n>\n> Patrick\n"},{"id":"550977","messageId":"CABPp-BHSFW38sF4dZkqZuGaRASVRj2FVG2NN1OTA7-Dd6Pt6rw@mail.gmail.com","threadId":"66136","inReplyTo":"aoaYL_BinFtgdJ5N@pks.im","subject":"Re: [PATCH v2 1/6] odb: introduce interface to generate packfiles","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-08-21T04:41:03Z","receivedAt":"2026-08-21T04:41:16Z","isPatch":true,"body":"On Wed, Aug 19, 2026 at 11:01 PM Patrick Steinhardt wrote:\n>\n> On Wed, Aug 19, 2026 at 09:56:56AM -0700, Elijah Newren wrote:\n> > On Sun, Aug 16, 2026 at 10:40 PM Patrick Steinhardt wrote:\n> > >\n> > > +static int odb_source_files_generate_pack(struct odb_source *source UNUSED,\n> > > + struct odb_pack_generator **out,\n> > > + const struct odb_generate_pack_options *opts)\n> > > +{\n> > > + struct child_process cp = CHILD_PROCESS_INIT;\n> > > + struct odb_pack_generator_files *generator;\n> > > + FILE *in;\n> > [...]\n> > > + cp.clean_on_exit = 1;\n> > > +\n> > > + if (start_command(&cp))\n> > > + return error(_(\"could not spawn pack-objects\"));\n> > [...]\n> > > + CALLOC_ARRAY(generator, 1);\n> > > + generator->base.out = opts->pack_fd < 0 ? cp.out : -1;\n> > > + generator->base.err = opts->progress_fd < 0 ? cp.err : -1;\n> > > + generator->base.finish = odb_pack_generator_files_finish;\n> > > + generator->cp = cp;\n> > > +\n> > > + *out = &generator->base;\n> > > + return 0;\n> > > +}\n> >\n> > Does this have a use-after-scope bug lurking here, due to the\n> > combination of clean_on_exit = 1 (which makes a copy of &cp for later\n> > use), and the fact that cp is a function-local? If I'm reading the\n> > code right, start_command() calls mark_child_for_cleanup(), which does\n> >\n> > p->process = process; /* where process is &cp */\n> >\n> > and then cleanup_children() accesses various fields under p->process.\n> > You do copy the necessary fields from cp to generator->cp, but\n> > &generator->cp was not passed to start_command(), so p->process points\n> > to the function-local cp.\n>\n> Oh, that's a very good catch indeed. Out of curiosity, how did you end\n> up discovering this? Did you just happen to remember that we store the\n> pointer out of scope or did the copy make you have a deeper look?\n\nNeither. Went to review the series, but I was worried I'd be missing\ncontext from not reviewing earlier odb refactorings. Used AI to help\norient me and give me its own findings from reviewing your patches.\n(AI will sometimes spot things I miss in a review, though it'll also\nmiss some things I catch.) And sometimes I iterate with AI to dig\ninto various areas. Anyway, it flagged the potential problem, and I\ndug in to make sure it didn't look like a hallucination before\ncleaning it up and passing it on. I'm still looking through your\nother patches in this series, but should finish soon.\n\nOn a related note, one of my local patches happens to have a semantic\nconflict with this series (namely 3/6), which piqued my interest.\nI'll submit it soon, using your series as a base so I can submit my\npatch with the conflict fixed.\n"},{"id":"550980","messageId":"CABPp-BE63m2sB4-18JUiYDK+UXaCq9z_=A8JAutvjn155_HWZA@mail.gmail.com","threadId":"66136","inReplyTo":"aoPtDyISRa0mVXRa@pks.im","subject":"Re: [PATCH v2 5/6] bundle: get (mostly) rid of `the_repository`","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-08-21T05:42:49Z","receivedAt":"2026-08-21T05:43:02Z","isPatch":true,"body":"On Mon, Aug 17, 2026 at 10:26 PM Patrick Steinhardt wrote:\n>\n> On Mon, Aug 17, 2026 at 09:47:53AM -0700, Junio C Hamano wrote:\n> > Patrick Steinhardt writes:\n> >\n> > > Refactor \"bundle.c\" so that we don't depend on `the_repository` anymore.\n> > > This conversion is trivial for most of the part, as we already have a\n> > > repository available in all calling conexts.\n> > >\n> > > The only exception is that we use `get_log_output_encoding()`, which\n> > > implicitly depends on `the_repository`. Add an `extern` declaration for\n> > > this function so that we can drop `USE_THE_REPOSITORY_VARIABLE` and not\n> > > accidentally introduce more uses of `the_repository`.\n> > >\n> > > Signed-off-by: Patrick Steinhardt \n> > > ---\n> > > bundle.c | 32 +++++++++++++++++++++-----------\n> > > 1 file changed, 21 insertions(+), 11 deletions(-)\n> > >\n> > > diff --git a/bundle.c b/bundle.c\n> > > index b64716f252..a9330bf0d3 100644\n> > > --- a/bundle.c\n> > > +++ b/bundle.c\n> > > @@ -1,4 +1,3 @@\n> > > -#define USE_THE_REPOSITORY_VARIABLE\n> > > #define DISABLE_SIGN_COMPARE_WARNINGS\n> > >\n> > > #include \"git-compat-util.h\"\n> > > @@ -21,6 +20,13 @@\n> > > #include \"connected.h\"\n> > > #include \"write-or-die.h\"\n> > >\n> > > +/*\n> > > + * NEEDSWORK: this function implicitly depends on `the_repository` and is not\n> > > + * available because we dropped USE_THE_REPOSITORY_VARIABLE. We can remove the\n> > > + * declaration once it's accessible via `repo_config_values`.\n> > > + */\n> > > +extern const char *get_log_output_encoding(void);\n> > > +\n> >\n> > Doesn't this defeat the whole \"drop #define USE_THE_REPOSITORY_VARIABLE\n> > as a mark that we are done with this file and no longer need to\n> > worry about it going forward because we won't be able to compile if\n> > somebody adds a new use?\" premise?\n>\n> Yes and no. By removing the define early it allows us to not reintroduce\n> new references to `the_repository` by accident, but carve out a single\n> exception for one of the functions that still depends on it. The\n> alternative would be to not do that, and if so there is no guarantee\n> whatsoever that we won't introduce more references to `the_repository`\n> in this file.\n>\n> So I'm still leaning towards keeping this as-is, but I don't feel very\n> strongly about this. Let me know in case that argument doesn't sway you\n> and I'll adapt.\n\nWould it make more sense to do this the way replay.c does:\n\n#define USE_THE_REPOSITORY_VARIABLE\n\n/*\n * We technically need USE_THE_REPOSITORY_VARIABLE for , but\n * do not want to use the_repository.\n */\n#define the_repository DO_NOT_USE_THE_REPOSITORY\n\n\nand remove the declaration of get_log_output_encoding() that you\nadded? Alternatively, should replay.c be adapted to the way you are\ndoing it here?\n"},{"id":"550982","messageId":"CABPp-BHAeb5Q6kWw8e0fz9+avKyJL0_k7cUzRhesHScJjB3Xfw@mail.gmail.com","threadId":"66136","inReplyTo":"20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im","subject":"Re: [PATCH v3 0/6] odb: make packfile generation pluggable","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-08-21T06:05:23Z","receivedAt":"2026-08-21T06:05:35Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 12:55 AM Patrick Steinhardt wrote:\n>\n> Hi,\n>\n> this patch series makes packfile generation pluggable.\n>\n> Note that this series only makes those parts pluggable that are required\n> for the transport layer. The other parts that relate to packfile\n> generation as required by our repository maintenance is kept as-is, as\n> there is a bunch of options there that are way too specific to the\n> \"files\" backend to be portable. This should ultimately not be much of a\n> problem though, as maintenance itself is already pluggable in the first\n> place.\n>\n> It's a bit of a shame though for git-pack-objects(1), which still isn't\n> usable with alternate backends. I tried several times to find good\n> solutions for making it fully pluggable, but due to the backend-specific\n> options it's an utter mess. I want to eventually address this though:\n> same as with git-refs(1), I want to introduce git-objects(1) to care\n> about all things ODB. And as part of that command we can also introduce\n> a command that generates packfiles in a generic fashion, without all the\n> cruft that git-pack-objects(1) has. This is part of a future patch\n> series though.\n\nSo, big picture, today there are three callers that spawn \"git\npack-objects --revs --stdout ...\" by hand to produce a pack for\ntransfer: upload-pack, send-pack, and bundle. Each hand-rolls a\nchild_process, feeds a rev list on stdin, and drains the pack from the\nchild's stdout. This series hoists that shared machinery into a new\nobject-database interface, decoupling the transport use of packs from\nthe storage use. I like it.\n\n> Changes in v3:\n> - Fix a use-after-scope bug on abnormal exit when child processes are\n> cleaned up via `mark_child_for_cleanup()`, as noticed by Elijah.\n> - Link to v2: https://patch.msgid.link/20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im\n\nThanks, the fix in 1/6 addresses what I raised on v2.\n\nI read through the series -- I did have an alternative suggestion for\n5/6 (which I posted on v2 5/6 since there was already a thread there),\nbut otherwise I didn't spot anything beyond what other reviewers\nalready raised.\n"},{"id":"550983","messageId":"aofwHhFeeWgh_3FY@pks.im","threadId":"66136","inReplyTo":"xmqqik54soy0.fsf@gitster.g","subject":"Re: [PATCH v3 1/6] odb: introduce interface to generate packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:28:46Z","receivedAt":"2026-08-21T06:28:57Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 10:04:55AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> > diff --git a/odb.c b/odb.c\n> > index caf1d0f542..cd9d5b48bc 100644\n> > --- a/odb.c\n> > +++ b/odb.c\n> > @@ -1046,6 +1046,27 @@ bool odb_optimize_required(struct object_database *odb,\n> > \treturn odb_source_optimize_required(odb->sources, opts);\n> > }\n> > \n> > +void odb_generate_pack_options_release(struct odb_generate_pack_options *opts)\n> > +{\n> > +\toid_array_clear(&opts->wants);\n> > +\toid_array_clear(&opts->haves);\n> > +\toid_array_clear(&opts->shallows);\n> > +}\n> > +\n> > +int odb_generate_pack(struct object_database *odb,\n> > +\t\t struct odb_pack_generator **out,\n> > +\t\t const struct odb_generate_pack_options *opts)\n> > +{\n> > +\tif (!odb->sources->generate_pack)\n> > +\t\treturn error(_(\"primary object source does not support generating packfiles\"));\n> > +\treturn odb_source_generate_pack(odb->sources, out, opts);\n> > +}\n> \n> Perhaps a stupid question but the opts->pack_fd is documented:\n> \n> > +struct odb_generate_pack_options {\n> > ...\n> > +\t/*\n> > +\t * File descriptor that the generated pack shall be written to. If set\n> > +\t * to `-1`, a pipe will be created and exposed via the pack generator's\n> > +\t * `out` field. If set to `0`, the pack will be written to the standard\n> > +\t * output stream. Otherwise, the provided descriptor will be written to\n> > +\t * and is consumed by the generator.\n> > +\t */\n> > +\tint pack_fd;\n> > +\n> \n> Here I assume that \"and is consumed by\" refers to \"generator writes\n> into it and then closes it when it is done\"?\n\nYes.\n\n> odb_source_generate_pack() delegate to source->generate_pack(),\n> which I presume goes to odb_source_files_generate_pack(), which in\n> turn assigns opts->pack_fd to cp->out and calls start_command(cp) to\n> run pack-objects. The file descriptor is closed when the process\n> finishes.\n\nExactly.\n\n> What happens if the odb->sources[0] does not support .generate_pack?\n\nIf it does not support generating packs then Git would crash as this is\na non-optional callback. All sources that could be our primary source\nthough do support it, and the expectation is that any future backends\nwould know how to implement it, too.\n\n> Should opts->pack_fd be \"consumed\" here to avoid leaking it, or we\n> do not have to worry about it because the caller will soon exit\n> itself?\n\nSo this case here should not ever happen -- if we don't have the\ncallback, then there's nothing that can even set `pack_fd` and we should\ndie.\n\nPatrick\n"},{"id":"550984","messageId":"20260821-b4-pks-odb-generate-pack-v4-0-074e8bd641f8@pks.im","threadId":"66136","inReplyTo":"20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im","subject":"[PATCH v4 0/6] odb: make packfile generation pluggable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:30:00Z","receivedAt":"2026-08-21T06:30:10Z","isPatch":true,"body":"Hi,\n\nthis patch series makes packfile generation pluggable.\n\nNote that this series only makes those parts pluggable that are required\nfor the transport layer. The other parts that relate to packfile\ngeneration as required by our repository maintenance is kept as-is, as\nthere is a bunch of options there that are way too specific to the\n\"files\" backend to be portable. This should ultimately not be much of a\nproblem though, as maintenance itself is already pluggable in the first\nplace.\n\nIt's a bit of a shame though for git-pack-objects(1), which still isn't\nusable with alternate backends. I tried several times to find good\nsolutions for making it fully pluggable, but due to the backend-specific\noptions it's an utter mess. I want to eventually address this though:\nsame as with git-refs(1), I want to introduce git-objects(1) to care\nabout all things ODB. And as part of that command we can also introduce\na command that generates packfiles in a generic fashion, without all the\ncruft that git-pack-objects(1) has. This is part of a future patch\nseries though.\n\nChanges in v4:\n - Improve an error message.\n - Sneak in a small stylistic fix while at it.\n - Link to v3: https://patch.msgid.link/20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im\n\nChanges in v3:\n - Fix a use-after-scope bug on abnormal exit when child processes are\n cleaned up via `mark_child_for_cleanup()`, as noticed by Elijah.\n - Link to v2: https://patch.msgid.link/20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im\n\nChanges in v2:\n - Mostly remove the dependencies on `the_repository` in \"bundle.c\".\n - Link to v1: https://patch.msgid.link/20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im\n\nThe series is built on top of 2c78326f81 (The 11th batch, 2026-08-05).\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (6):\n odb: introduce interface to generate packfiles\n upload-pack: generate packfiles via the object database\n send-pack: generate packfiles via the object database\n builtin/bundle: refactor option handling for progress meter\n bundle: get (mostly) rid of `the_repository`\n bundle: generate packfiles via the object database\n\n builtin/bundle.c | 34 +++++------\n bundle.c | 97 ++++++++++++++++++--------------\n bundle.h | 3 +-\n odb.c | 21 +++++++\n odb.h | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-files.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h | 33 +++++++++++\n send-pack.c | 101 +++++++++++----------------------\n t/t5516-fetch-push.sh | 12 ++--\n upload-pack.c | 125 +++++++++++++++--------------------------\n 10 files changed, 508 insertions(+), 219 deletions(-)\n\nRange-diff versus v3:\n\n1: 4a56334af1 = 1: 33039a0ab8 odb: introduce interface to generate packfiles\n2: 1ff0eaf6b7 ! 2: 7093fcee83 upload-pack: generate packfiles via the object database\n @@ upload-pack.c: static void create_pack_file(struct upload_pack_data *pack_data,\n -\t */\n +\t\toid_array_append(&opts.haves,\n +\t\t\t\t &pack_data->extra_edge_obj.objects[i].item->oid);\n -+\n + \n +\topts.thin = pack_data->use_thin_pack;\n +\tif (!pack_data->no_progress)\n +\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_STANDARD;\n @@ upload-pack.c: static void create_pack_file(struct upload_pack_data *pack_data,\n +\topts.progress_fd = -1;\n +\n +\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n -+\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n ++\t\tdie(\"git upload-pack: unable to generate pack\");\n +\todb_generate_pack_options_release(&opts);\n - \n ++\n +\t/*\n +\t * We read from generator->err to capture stderr output for the\n +\t * progress bar, and generator->out to capture the pack data.\n3: 22a19a9a70 = 3: 0a2ca04c01 send-pack: generate packfiles via the object database\n4: 5d2275c90b = 4: 2d339ee7b7 builtin/bundle: refactor option handling for progress meter\n5: 0f00e6d234 = 5: 3cf0210247 bundle: get (mostly) rid of `the_repository`\n6: ae6af210ff ! 6: d3345e4407 bundle: generate packfiles via the object database\n @@ Commit message\n \n ## builtin/bundle.c ##\n @@ builtin/bundle.c: static int parse_options_cmd_bundle(int argc,\n + }\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n - \t\t\t struct repository *repo UNUSED) {\n +-\t\t\t struct repository *repo UNUSED) {\n -\tstruct strvec pack_opts = STRVEC_INIT;\n ++\t\t\t struct repository *repo UNUSED)\n ++{\n \tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n \tstruct option options[] = {\n\n---\nbase-commit: 2c78326f810173a4f3aefd8021f1e07575412481\nchange-id: 20260807-b4-pks-odb-generate-pack-f30fbcdef3fc\n\n"},{"id":"550985","messageId":"20260821-b4-pks-odb-generate-pack-v4-1-074e8bd641f8@pks.im","threadId":"66136","inReplyTo":"20260821-b4-pks-odb-generate-pack-v4-0-074e8bd641f8@pks.im","subject":"[PATCH v4 1/6] odb: introduce interface to generate packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:30:01Z","receivedAt":"2026-08-21T06:30:12Z","isPatch":true,"body":"Packfiles have two primary use cases:\n\n - They are used to store objects at rest in a Git repository.\n\n - They are used on the transport layer to transfer objects between two\n repositories.\n\nThe first class is closely tied to a given object database backend, and\nas such this use is highly specific to how such a backend decides to\nstore its data. This shows in git-pack-objects(1), which is used by\ngit-repack(1) et al to optimize the object database, which supports lots\nof options that are closely coupled with how data is stored.\n\nBut the second class is quite a lot more generic: we don't care about\nspecifics of how the object database stores its objects, but to generate\nthe packfiles we only care about the object graph itself. Still, this\nuse case is also coupled with git-pack-objects(1).\n\nUnfortunately, because git-pack-objects(1) covers both classes, the\nresult is that it is very hard to port the whole command to properly\nsupport pluggable object databases. There are simply way too many\noptions that an alternative implementation will have a very hard time to\nsupport in the first place.\n\nAnd despite being hard to implement, it's also quite unnecessary to\nimplement those backend-specific options. Optimizing the object database\nhas already been made pluggable, and an alternative implementation is\nunlikely to care about cruft packs, unpacked objects, keep packs and the\nlike. But we still need to make at least _parts_ of the packfile\ngeneration pluggable so that backends can generate packfiles for the\ntransport layer itself.\n\nIntroduce a new interface that lets backends generate a new packfile and\nimplement that interface for the \"files\" backend. The options supported\nby the callback are exactly the set of options that are required for the\ntransport layer, but nothing more.\n\nThis means that git-pack-objects(1) itself cannot be ported over to this\nnew interface, but as explained above that's a hard feat to pull off due\nto the backend-specific features. Ideally though, we should expose the\nability to generate arbitrary packfiles using this interface. The intent\nof this is to eventually introduce a git-objects(1) subcommand (similar\nto git-refs(1)) that exposes generic interfaces for accessing everything\nrelated to the object database. In that case, we are able to expose only\nthose options that are generic.\n\nSubsequent commits will convert git-upload-pack(1), git-send-pack(1) and\ngit-bundle(1) to use this interface.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb.c | 21 ++++++++\n odb.h | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source-files.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++++\n odb/source.h | 33 ++++++++++++\n 4 files changed, 355 insertions(+)\n\ndiff --git a/odb.c b/odb.c\nindex caf1d0f542..cd9d5b48bc 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1046,6 +1046,27 @@ bool odb_optimize_required(struct object_database *odb,\n \treturn odb_source_optimize_required(odb->sources, opts);\n }\n \n+void odb_generate_pack_options_release(struct odb_generate_pack_options *opts)\n+{\n+\toid_array_clear(&opts->wants);\n+\toid_array_clear(&opts->haves);\n+\toid_array_clear(&opts->shallows);\n+}\n+\n+int odb_generate_pack(struct object_database *odb,\n+\t\t struct odb_pack_generator **out,\n+\t\t const struct odb_generate_pack_options *opts)\n+{\n+\tif (!odb->sources->generate_pack)\n+\t\treturn error(_(\"primary object source does not support generating packfiles\"));\n+\treturn odb_source_generate_pack(odb->sources, out, opts);\n+}\n+\n+int odb_pack_generator_finish(struct odb_pack_generator *generator)\n+{\n+\treturn generator->finish(generator);\n+}\n+\n struct object_database *odb_new(struct repository *repo,\n \t\t\t\tconst char *primary_source,\n \t\t\t\tconst char *secondary_sources)\ndiff --git a/odb.h b/odb.h\nindex fca67e8253..fc1442f243 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -2,6 +2,7 @@\n #define ODB_H\n \n #include \"object.h\"\n+#include \"oid-array.h\"\n #include \"oidset.h\"\n #include \"oidmap.h\"\n #include \"string-list.h\"\n@@ -677,6 +678,157 @@ int odb_write_object_stream(struct object_database *odb,\n \t\t\t struct odb_write_stream *stream, size_t len,\n \t\t\t struct object_id *oid);\n \n+/*\n+ * Options for generating a packfile via `odb_generate_pack()`.\n+ */\n+struct odb_generate_pack_options {\n+\t/* Tips of the object graph that shall be packed. */\n+\tstruct oid_array wants;\n+\n+\t/*\n+\t * Boundary of the object graph. Objects reachable from any of these\n+\t * tips are expected to already be available to whoever consumes the\n+\t * pack and shall thus not be packed.\n+\t */\n+\tstruct oid_array haves;\n+\n+\t/*\n+\t * The shallow boundary that shall be used when computing object\n+\t * reachability. When set, any shallow information of the repository\n+\t * itself shall be ignored in favor of these objects.\n+\t */\n+\tstruct oid_array shallows;\n+\n+\t/*\n+\t * Pre-expanded object filter specification that limits the set of\n+\t * objects that shall be packed. May be `NULL` in case no filter shall\n+\t * be applied.\n+\t */\n+\tconst char *filter_spec;\n+\n+\t/*\n+\t * Protocols that may be used to offload objects via packfile URIs.\n+\t * May be `NULL` in case packfile URIs shall not be used.\n+\t */\n+\tconst struct string_list *uri_protocols;\n+\n+\t/*\n+\t * Hook command that shall be executed instead of the internal\n+\t * machinery to generate the pack. It is up to the specific backend\n+\t * whether or not this hook is supported. May be `NULL` in case no\n+\t * hook shall be executed.\n+\t */\n+\tconst char *pack_objects_hook;\n+\n+\t/*\n+\t * File descriptor that the generated pack shall be written to. If set\n+\t * to `-1`, a pipe will be created and exposed via the pack generator's\n+\t * `out` field. If set to `0`, the pack will be written to the standard\n+\t * output stream. Otherwise, the provided descriptor will be written to\n+\t * and is consumed by the generator.\n+\t */\n+\tint pack_fd;\n+\n+\t/*\n+\t * File descriptor that progress output shall be written to. The same\n+\t * semantics as for `pack_fd` apply, except that `0` will cause the\n+\t * generator to write to stderr instead of stdout.\n+\t */\n+\tint progress_fd;\n+\n+\t/* Whether to print progress or not. */\n+\tenum {\n+\t\t/* Don't print progress output. */\n+\t\tODB_GENERATE_PACK_PROGRESS_NONE,\n+\n+\t\t/*\n+\t\t * Print progress while computing the packfile, but stop\n+\t\t * printing progress once starting to write it.\n+\t\t */\n+\t\tODB_GENERATE_PACK_PROGRESS_STANDARD,\n+\n+\t\t/*\n+\t\t * Similar to STANDARD, but also print progress when writing\n+\t\t * the packfile.\n+\t\t */\n+\t\tODB_GENERATE_PACK_PROGRESS_VERBOSE,\n+\t} progress;\n+\n+\t/* Allow the pack to contain deltas against unpacked objects. */\n+\tunsigned thin:1;\n+\n+\t/* Use offset deltas instead of reference deltas. */\n+\tunsigned ofs_delta:1;\n+\n+\t/* Include unasked-for annotated tags of packed objects. */\n+\tunsigned include_tag:1;\n+\n+\t/* The generated pack is destined for a shallow consumer. */\n+\tunsigned shallow:1;\n+\n+\t/* Allow objects that may be missing due to a promisor remote. */\n+\tunsigned missing_allow_promisor:1;\n+\n+\t/* Do not use bitmap indices when computing reachability. */\n+\tunsigned disable_bitmaps:1;\n+};\n+\n+#define ODB_GENERATE_PACK_OPTIONS_INIT { \\\n+\t.wants = OID_ARRAY_INIT, \\\n+\t.haves = OID_ARRAY_INIT, \\\n+\t.shallows = OID_ARRAY_INIT, \\\n+\t.pack_fd = -1, \\\n+}\n+\n+/* Release resources associated with the options. */\n+void odb_generate_pack_options_release(struct odb_generate_pack_options *opts);\n+\n+/*\n+ * A handle for an ongoing packfile generation as started via\n+ * `odb_generate_pack()`.\n+ */\n+struct odb_pack_generator {\n+\t/*\n+\t * File descriptor from which the generated pack can be read. Only set\n+\t * when the pack generation was started with `pack_fd == -1`. The\n+\t * caller is responsible for closing the descriptor.\n+\t */\n+\tint out;\n+\n+\t/*\n+\t * File descriptor from which progress output can be read. Only set\n+\t * when the pack generation was started with `progress_fd == -1`. The\n+\t * caller is responsible for closing the descriptor.\n+\t */\n+\tint err;\n+\n+\t/*\n+\t * Callback function to finish this generator. This callback is\n+\t * expected to wait for the packfile generation to complete and to then\n+\t * free the generator itself.\n+\t */\n+\tint (*finish)(struct odb_pack_generator *);\n+};\n+\n+/*\n+ * Start generating a packfile from the object database with the given\n+ * options. The pack is generated asynchronously; the caller is expected to\n+ * consume the file descriptors exposed via the pack generator and to then\n+ * wait for completion via `odb_pack_generator_finish()`.\n+ *\n+ * Returns 0 on success and populates the `out` pointer with the pack\n+ * generator. Returns a negative error code otherwise.\n+ */\n+int odb_generate_pack(struct object_database *odb,\n+\t\t struct odb_pack_generator **out,\n+\t\t const struct odb_generate_pack_options *opts);\n+\n+/*\n+ * Wait for the packfile generation to complete and free the pack generator.\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int odb_pack_generator_finish(struct odb_pack_generator *generator);\n+\n void parse_alternates(const char *string,\n \t\t int sep,\n \t\t const char *relative_base,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5a68af7d84..a33e01fbed 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -4,6 +4,7 @@\n #include \"chdir-notify.h\"\n #include \"config.h\"\n #include \"gettext.h\"\n+#include \"hex.h\"\n #include \"lockfile.h\"\n #include \"object-file.h\"\n #include \"odb.h\"\n@@ -729,6 +730,153 @@ int odb_source_files_optimize(struct odb_source *source,\n \treturn ret;\n }\n \n+struct odb_pack_generator_files {\n+\tstruct odb_pack_generator base;\n+\tstruct child_process cp;\n+};\n+\n+static int odb_pack_generator_files_finish(struct odb_pack_generator *_generator)\n+{\n+\tstruct odb_pack_generator_files *generator =\n+\t\t(struct odb_pack_generator_files *)_generator;\n+\tint ret;\n+\n+\tret = finish_command(&generator->cp);\n+\tfree(generator);\n+\n+\tif (ret) {\n+\t\t/*\n+\t\t * On failure, pack-objects is expected to have written a\n+\t\t * useful error message to its standard error stream already.\n+\t\t * Death by signal is worth mentioning, though, with the\n+\t\t * exception of SIGPIPE: that is a normal occurrence when the\n+\t\t * consumer of the pack hangs up.\n+\t\t */\n+\t\tif (ret > 128 && ret - 128 == SIGPIPE)\n+\t\t\treturn -1;\n+\t\tif (ret > 128)\n+\t\t\terror(_(\"pack-objects died of signal %d\"), ret - 128);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_source_files_generate_pack(struct odb_source *source UNUSED,\n+\t\t\t\t\t struct odb_pack_generator **out,\n+\t\t\t\t\t const struct odb_generate_pack_options *opts)\n+{\n+\tstruct odb_pack_generator_files *generator;\n+\tstruct child_process *cp;\n+\tFILE *in;\n+\n+\tCALLOC_ARRAY(generator, 1);\n+\tchild_process_init(&generator->cp);\n+\tcp = &generator->cp;\n+\n+\t/*\n+\t * The hook is expected to spawn \"$hook git pack-objects \"\n+\t * and to behave like git-pack-objects(1) would have. This can for\n+\t * example be used to serve precomputed packfiles.\n+\t */\n+\tif (opts->pack_objects_hook) {\n+\t\tstrvec_push(&cp->args, opts->pack_objects_hook);\n+\t\tstrvec_push(&cp->args, \"git\");\n+\t\tcp->use_shell = 1;\n+\t} else {\n+\t\tcp->git_cmd = 1;\n+\t}\n+\n+\t/*\n+\t * The caller-provided shallow boundary overrides any shallow state\n+\t * that the repository itself may have, so the shallow file needs to\n+\t * be neutralized.\n+\t */\n+\tif (opts->shallows.nr) {\n+\t\tstrvec_push(&cp->args, \"--shallow-file\");\n+\t\tstrvec_push(&cp->args, \"\");\n+\t}\n+\tstrvec_push(&cp->args, \"pack-objects\");\n+\tstrvec_push(&cp->args, \"--revs\");\n+\tstrvec_push(&cp->args, \"--stdout\");\n+\tif (opts->thin)\n+\t\tstrvec_push(&cp->args, \"--thin\");\n+\tif (opts->shallow)\n+\t\tstrvec_push(&cp->args, \"--shallow\");\n+\tif (opts->ofs_delta)\n+\t\tstrvec_push(&cp->args, \"--delta-base-offset\");\n+\tif (opts->include_tag)\n+\t\tstrvec_push(&cp->args, \"--include-tag\");\n+\tif (opts->missing_allow_promisor)\n+\t\tstrvec_push(&cp->args, \"--missing=allow-promisor\");\n+\tif (opts->disable_bitmaps)\n+\t\tstrvec_push(&cp->args, \"--no-use-bitmap-index\");\n+\tswitch (opts->progress) {\n+\tcase ODB_GENERATE_PACK_PROGRESS_NONE:\n+\t\tstrvec_push(&cp->args, \"--quiet\");\n+\t\tbreak;\n+\tcase ODB_GENERATE_PACK_PROGRESS_STANDARD:\n+\t\tstrvec_push(&cp->args, \"--progress\");\n+\t\tbreak;\n+\tcase ODB_GENERATE_PACK_PROGRESS_VERBOSE:\n+\t\tstrvec_push(&cp->args, \"--all-progress\");\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown progress option %d\", opts->progress);\n+\t}\n+\tif (opts->filter_spec)\n+\t\tstrvec_pushf(&cp->args, \"--filter=%s\", opts->filter_spec);\n+\tif (opts->uri_protocols)\n+\t\tfor (size_t i = 0; i < opts->uri_protocols->nr; i++)\n+\t\t\tstrvec_pushf(&cp->args, \"--uri-protocol=%s\",\n+\t\t\t\t opts->uri_protocols->items[i].string);\n+\n+\tcp->in = -1;\n+\tcp->out = opts->pack_fd;\n+\tcp->err = opts->progress_fd;\n+\tcp->clean_on_exit = 1;\n+\n+\tif (start_command(cp)) {\n+\t\tfree(generator);\n+\t\treturn error(_(\"could not spawn pack-objects\"));\n+\t}\n+\n+\t/*\n+\t * Feed the objects to pack-objects. This is safe to do synchronously\n+\t * because pack-objects consumes all of its standard input before it\n+\t * starts to generate the pack.\n+\t */\n+\tin = xfdopen(cp->in, \"w\");\n+\tfor (size_t i = 0; i < opts->shallows.nr; i++)\n+\t\tfprintf(in, \"--shallow %s\\n\", oid_to_hex(&opts->shallows.oid[i]));\n+\tfor (size_t i = 0; i < opts->wants.nr; i++)\n+\t\tfprintf(in, \"%s\\n\", oid_to_hex(&opts->wants.oid[i]));\n+\tfprintf(in, \"--not\\n\");\n+\tfor (size_t i = 0; i < opts->haves.nr; i++)\n+\t\tfprintf(in, \"%s\\n\", oid_to_hex(&opts->haves.oid[i]));\n+\tfprintf(in, \"\\n\");\n+\tfflush(in);\n+\tif (ferror(in)) {\n+\t\terror(_(\"error writing to pack-objects\"));\n+\t\tfclose(in);\n+\t\tif (opts->pack_fd < 0)\n+\t\t\tclose(cp->out);\n+\t\tif (opts->progress_fd < 0)\n+\t\t\tclose(cp->err);\n+\t\tfinish_command(cp);\n+\t\tfree(generator);\n+\t\treturn -1;\n+\t}\n+\tfclose(in);\n+\n+\tgenerator->base.out = opts->pack_fd < 0 ? cp->out : -1;\n+\tgenerator->base.err = opts->progress_fd < 0 ? cp->err : -1;\n+\tgenerator->base.finish = odb_pack_generator_files_finish;\n+\n+\t*out = &generator->base;\n+\treturn 0;\n+}\n+\n struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t bool local)\n@@ -756,6 +904,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,\n \tfiles->base.write_alternate = odb_source_files_write_alternate;\n \tfiles->base.optimize = odb_source_files_optimize;\n \tfiles->base.optimize_required = odb_source_files_optimize_required;\n+\tfiles->base.generate_pack = odb_source_files_generate_pack;\n \n \t/*\n \t * Ideally, we would only ever store absolute paths in the source. This\ndiff --git a/odb/source.h b/odb/source.h\nindex d69f8e2d1c..e2129766fc 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -278,6 +278,23 @@ struct odb_source {\n \t */\n \tbool (*optimize_required)(struct odb_source *source,\n \t\t\t\t const struct odb_optimize_options *opts);\n+\n+\t/*\n+\t * This callback is expected to start generating a packfile with the\n+\t * given options. The pack shall be generated asynchronously so that\n+\t * the caller can consume the pack data and progress output while the\n+\t * pack is being generated.\n+\t *\n+\t * This callback is optional. Sources that cannot generate packfiles\n+\t * shall leave it unset.\n+\t *\n+\t * The callback is expected to return 0 on success and populate the\n+\t * `out` pointer with the pack generator, a negative error code\n+\t * otherwise.\n+\t */\n+\tint (*generate_pack)(struct odb_source *source,\n+\t\t\t struct odb_pack_generator **out,\n+\t\t\t const struct odb_generate_pack_options *opts);\n };\n \n /*\n@@ -520,4 +537,20 @@ static inline bool odb_source_optimize_required(struct odb_source *source,\n \treturn source->optimize_required(source, opts);\n }\n \n+/*\n+ * Start generating a packfile from the given source with the given options.\n+ * The pack is generated asynchronously; the caller is expected to consume the\n+ * file descriptors exposed via the pack generator and to then wait for\n+ * completion via `odb_pack_generator_finish()`.\n+ *\n+ * Returns 0 on success and populates the `out` pointer with the pack\n+ * generator, a negative error code otherwise.\n+ */\n+static inline int odb_source_generate_pack(struct odb_source *source,\n+\t\t\t\t\t struct odb_pack_generator **out,\n+\t\t\t\t\t const struct odb_generate_pack_options *opts)\n+{\n+\treturn source->generate_pack(source, out, opts);\n+}\n+\n #endif\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550986","messageId":"20260821-b4-pks-odb-generate-pack-v4-2-074e8bd641f8@pks.im","threadId":"66136","inReplyTo":"20260821-b4-pks-odb-generate-pack-v4-0-074e8bd641f8@pks.im","subject":"[PATCH v4 2/6] upload-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:30:02Z","receivedAt":"2026-08-21T06:30:14Z","isPatch":true,"body":"When serving a fetch, git-upload-pack(1) spawns git-pack-objects(1)\ndirectly to generate the packfile that gets sent to the client. This\nhard-codes the assumption that the object database is able to serve\npackfiles via git-pack-objects(1), which is specific to the \"files\"\nbackend.\n\nConvert git-upload-pack(1) to instead use the pack generation interface\nof the object database.\n\nSigned-off-by: Patrick Steinhardt \n---\n upload-pack.c | 125 +++++++++++++++++++++-------------------------------------\n 1 file changed, 45 insertions(+), 80 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex a52856d869..22573ad365 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -197,11 +197,11 @@ static void send_client_data(int fd, const char *data, ssize_t sz,\n \twrite_or_die(fd, data, sz);\n }\n \n-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n+static int append_one_shallow(const struct commit_graft *graft, void *cb_data)\n {\n-\tFILE *fp = cb_data;\n+\tstruct oid_array *shallows = cb_data;\n \tif (graft->nr_parent == -1)\n-\t\tfprintf(fp, \"--shallow %s\\n\", oid_to_hex(&graft->oid));\n+\t\toid_array_append(shallows, &graft->oid);\n \treturn 0;\n }\n \n@@ -299,7 +299,8 @@ static int relay_pack_data(int pack_objects_out, struct output_state *os,\n static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t\t const struct string_list *uri_protocols)\n {\n-\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n \tstruct output_state *output_state = xcalloc(1, sizeof(struct output_state));\n \tchar progress[128];\n \tchar abort_msg[] = \"aborting due to possible repository \"\n@@ -307,78 +308,42 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \tuint64_t last_sent_ms = 0;\n \tssize_t sz;\n \tint i;\n-\tFILE *pipe_fd;\n-\n-\tif (!pack_data->pack_objects_hook)\n-\t\tpack_objects.git_cmd = 1;\n-\telse {\n-\t\tstrvec_push(&pack_objects.args, pack_data->pack_objects_hook);\n-\t\tstrvec_push(&pack_objects.args, \"git\");\n-\t\tpack_objects.use_shell = 1;\n-\t}\n \n \tif (pack_data->shallow_nr) {\n-\t\tstrvec_push(&pack_objects.args, \"--shallow-file\");\n-\t\tstrvec_push(&pack_objects.args, \"\");\n-\t}\n-\tstrvec_push(&pack_objects.args, \"pack-objects\");\n-\tstrvec_push(&pack_objects.args, \"--revs\");\n-\tif (pack_data->use_thin_pack)\n-\t\tstrvec_push(&pack_objects.args, \"--thin\");\n-\n-\tstrvec_push(&pack_objects.args, \"--stdout\");\n-\tif (pack_data->shallow_nr)\n-\t\tstrvec_push(&pack_objects.args, \"--shallow\");\n-\tif (!pack_data->no_progress)\n-\t\tstrvec_push(&pack_objects.args, \"--progress\");\n-\tif (pack_data->use_ofs_delta)\n-\t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n-\tif (pack_data->use_include_tag)\n-\t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n-\tif (repo_has_accepted_promisor_remote(the_repository))\n-\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n-\tif (pack_data->filter_options.choice) {\n-\t\tconst char *spec =\n-\t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\", spec);\n-\t}\n-\tif (uri_protocols) {\n-\t\tfor (i = 0; i < uri_protocols->nr; i++)\n-\t\t\tstrvec_pushf(&pack_objects.args, \"--uri-protocol=%s\",\n-\t\t\t\t\t uri_protocols->items[i].string);\n+\t\tfor_each_commit_graft(append_one_shallow, &opts.shallows);\n+\t\topts.shallow = 1;\n \t}\n-\n-\tpack_objects.in = -1;\n-\tpack_objects.out = -1;\n-\tpack_objects.err = -1;\n-\tpack_objects.clean_on_exit = 1;\n-\n-\tif (start_command(&pack_objects))\n-\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n-\n-\tpipe_fd = xfdopen(pack_objects.in, \"w\");\n-\n-\tif (pack_data->shallow_nr)\n-\t\tfor_each_commit_graft(write_one_shallow, pipe_fd);\n-\n \tfor (i = 0; i < pack_data->want_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->want_obj.objects[i].item->oid));\n-\tfprintf(pipe_fd, \"--not\\n\");\n+\t\toid_array_append(&opts.wants,\n+\t\t\t\t &pack_data->want_obj.objects[i].item->oid);\n \tfor (i = 0; i < pack_data->have_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->have_obj.objects[i].item->oid));\n+\t\toid_array_append(&opts.haves,\n+\t\t\t\t &pack_data->have_obj.objects[i].item->oid);\n \tfor (i = 0; i < pack_data->extra_edge_obj.nr; i++)\n-\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\toid_to_hex(&pack_data->extra_edge_obj.objects[i].item->oid));\n-\tfprintf(pipe_fd, \"\\n\");\n-\tfflush(pipe_fd);\n-\tfclose(pipe_fd);\n-\n-\t/* We read from pack_objects.err to capture stderr output for\n-\t * progress bar, and pack_objects.out to capture the pack data.\n-\t */\n+\t\toid_array_append(&opts.haves,\n+\t\t\t\t &pack_data->extra_edge_obj.objects[i].item->oid);\n \n+\topts.thin = pack_data->use_thin_pack;\n+\tif (!pack_data->no_progress)\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_STANDARD;\n+\topts.ofs_delta = pack_data->use_ofs_delta;\n+\topts.include_tag = pack_data->use_include_tag;\n+\topts.missing_allow_promisor = repo_has_accepted_promisor_remote(the_repository);\n+\tif (pack_data->filter_options.choice)\n+\t\topts.filter_spec = expand_list_objects_filter_spec(&pack_data->filter_options);\n+\topts.uri_protocols = uri_protocols;\n+\topts.pack_objects_hook = pack_data->pack_objects_hook;\n+\topts.pack_fd = -1;\n+\topts.progress_fd = -1;\n+\n+\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n+\t\tdie(\"git upload-pack: unable to generate pack\");\n+\todb_generate_pack_options_release(&opts);\n+\n+\t/*\n+\t * We read from generator->err to capture stderr output for the\n+\t * progress bar, and generator->out to capture the pack data.\n+\t */\n \twhile (1) {\n \t\tuint64_t now_ms = getnanotime() / 1000000;\n \t\tstruct pollfd pfd[2];\n@@ -393,14 +358,14 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tpollsize = 0;\n \t\tpe = pu = -1;\n \n-\t\tif (0 <= pack_objects.out) {\n-\t\t\tpfd[pollsize].fd = pack_objects.out;\n+\t\tif (0 <= generator->out) {\n+\t\t\tpfd[pollsize].fd = generator->out;\n \t\t\tpfd[pollsize].events = POLLIN;\n \t\t\tpu = pollsize;\n \t\t\tpollsize++;\n \t\t}\n-\t\tif (0 <= pack_objects.err) {\n-\t\t\tpfd[pollsize].fd = pack_objects.err;\n+\t\tif (0 <= generator->err) {\n+\t\t\tpfd[pollsize].fd = generator->err;\n \t\t\tpfd[pollsize].events = POLLIN;\n \t\t\tpe = pollsize;\n \t\t\tpollsize++;\n@@ -437,15 +402,15 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t\t/* Status ready; we ship that in the side-band\n \t\t\t * or dump to the standard error.\n \t\t\t */\n-\t\t\tsz = xread(pack_objects.err, progress,\n+\t\t\tsz = xread(generator->err, progress,\n \t\t\t\t sizeof(progress));\n \t\t\tif (0 < sz) {\n \t\t\t\tsend_client_data(2, progress, sz,\n \t\t\t\t\t\t pack_data->use_sideband);\n \t\t\t\tlast_sent_ms = now_ms;\n \t\t\t} else if (sz == 0) {\n-\t\t\t\tclose(pack_objects.err);\n-\t\t\t\tpack_objects.err = -1;\n+\t\t\t\tclose(generator->err);\n+\t\t\t\tgenerator->err = -1;\n \t\t\t}\n \t\t\telse\n \t\t\t\tgoto fail;\n@@ -455,15 +420,15 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \n \t\tif (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) {\n \t\t\tbool did_send_data;\n-\t\t\tint result = relay_pack_data(pack_objects.out,\n+\t\t\tint result = relay_pack_data(generator->out,\n \t\t\t\t\t\t output_state,\n \t\t\t\t\t\t pack_data->use_sideband,\n \t\t\t\t\t\t !!uri_protocols,\n \t\t\t\t\t\t &did_send_data);\n \n \t\t\tif (result == 0) {\n-\t\t\t\tclose(pack_objects.out);\n-\t\t\t\tpack_objects.out = -1;\n+\t\t\t\tclose(generator->out);\n+\t\t\t\tgenerator->out = -1;\n \t\t\t} else if (result < 0) {\n \t\t\t\tgoto fail;\n \t\t\t}\n@@ -498,7 +463,7 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\t}\n \t}\n \n-\tif (finish_command(&pack_objects)) {\n+\tif (odb_pack_generator_finish(generator)) {\n \t\terror(\"git upload-pack: git-pack-objects died with error.\");\n \t\tgoto fail;\n \t}\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550987","messageId":"20260821-b4-pks-odb-generate-pack-v4-3-074e8bd641f8@pks.im","threadId":"66136","inReplyTo":"20260821-b4-pks-odb-generate-pack-v4-0-074e8bd641f8@pks.im","subject":"[PATCH v4 3/6] send-pack: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:30:03Z","receivedAt":"2026-08-21T06:30:16Z","isPatch":true,"body":"When pushing, git-send-pack(1) spawns git-pack-objects(1) directly to\ngenerate the packfile that gets sent to the remote. Same as with\ngit-upload-pack(1), which has been adapted in the preceding commit,\nthis hard-codes the assumption that objects can be packed via\ngit-pack-objects(1), which is specific to the \"files\" backend.\n\nConvert git-send-pack(1) to use the pack generation interface of the\nobject database instead.\n\nNote that this requires us to adapt t5516 because the parameters passed\nto git-pack-objects(1) are changing:\n\n - The order of arguments changes.\n\n - We pass \"--quiet\" instead of \"-q\".\n\n - We don't pass \"--all-progress-implied\" anymore when not generating\n output.\n\nAll of these changes are benign though and should not result in a change\nin behaviour.\n\nSigned-off-by: Patrick Steinhardt \n---\n send-pack.c | 101 +++++++++++++++++---------------------------------\n t/t5516-fetch-push.sh | 12 +++---\n 2 files changed, 40 insertions(+), 73 deletions(-)\n\ndiff --git a/send-pack.c b/send-pack.c\nindex 3bb5afc687..f20460fbf4 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -42,16 +42,17 @@ int option_parse_push_signed(const struct option *opt,\n \tdie(\"bad %s argument: %s\", opt->long_name, arg);\n }\n \n-static void feed_object(struct repository *r,\n-\t\t\tconst struct object_id *oid, FILE *fh, int negative)\n+static void append_negative_object(struct repository *r,\n+\t\t\t\t struct oid_array *haves,\n+\t\t\t\t const struct object_id *oid)\n {\n-\tif (negative && !odb_has_object(r->objects, oid, 0))\n+\t/*\n+\t * The remote end may have advertised objects that we do not have in\n+\t * our object database. Skip those, as we cannot use them as boundary.\n+\t */\n+\tif (!odb_has_object(r->objects, oid, 0))\n \t\treturn;\n-\n-\tif (negative)\n-\t\tputc('^', fh);\n-\tfputs(oid_to_hex(oid), fh);\n-\tputc('\\n', fh);\n+\toid_array_append(haves, oid);\n }\n \n /*\n@@ -62,92 +63,58 @@ static int pack_objects(struct repository *r,\n \t\t\tstruct oid_array *negotiated,\n \t\t\tstruct send_pack_args *args)\n {\n-\t/*\n-\t * The child becomes pack-objects --revs; we feed\n-\t * the revision parameters to it via its stdin and\n-\t * let its stdout go back to the other end.\n-\t */\n-\tstruct child_process po = CHILD_PROCESS_INIT;\n-\tFILE *po_in;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n \tint rc;\n \n \ttrace2_region_enter(\"send_pack\", \"pack_objects\", r);\n-\tstrvec_push(&po.args, \"pack-objects\");\n-\tstrvec_push(&po.args, \"--all-progress-implied\");\n-\tstrvec_push(&po.args, \"--revs\");\n-\tstrvec_push(&po.args, \"--stdout\");\n-\tif (args->use_thin_pack)\n-\t\tstrvec_push(&po.args, \"--thin\");\n-\tif (args->use_ofs_delta)\n-\t\tstrvec_push(&po.args, \"--delta-base-offset\");\n-\tif (args->quiet || !args->progress)\n-\t\tstrvec_push(&po.args, \"-q\");\n+\n+\topts.thin = args->use_thin_pack;\n+\topts.ofs_delta = args->use_ofs_delta;\n \tif (args->progress)\n-\t\tstrvec_push(&po.args, \"--progress\");\n-\tif (is_repository_shallow(r))\n-\t\tstrvec_push(&po.args, \"--shallow\");\n-\tif (args->disable_bitmaps)\n-\t\tstrvec_push(&po.args, \"--no-use-bitmap-index\");\n-\tpo.in = -1;\n-\tpo.out = args->stateless_rpc ? -1 : fd;\n-\tpo.git_cmd = 1;\n-\tpo.clean_on_exit = 1;\n-\tif (start_command(&po))\n-\t\tdie_errno(\"git pack-objects failed\");\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n+\topts.shallow = is_repository_shallow(r);\n+\topts.disable_bitmaps = args->disable_bitmaps;\n \n \t/*\n-\t * We feed the pack-objects we just spawned with revision\n-\t * parameters by writing to the pipe.\n+\t * The pack is either written directly to the remote's descriptor, or,\n+\t * in the case of a stateless RPC, read back from a pipe so that we\n+\t * can wrap the pack data into pkt-lines.\n \t */\n-\tpo_in = xfdopen(po.in, \"w\");\n+\topts.pack_fd = args->stateless_rpc ? -1 : fd;\n+\n \tfor (size_t i = 0; i < advertised->nr; i++)\n-\t\tfeed_object(r, &advertised->oid[i], po_in, 1);\n+\t\tappend_negative_object(r, &opts.haves, &advertised->oid[i]);\n \tfor (size_t i = 0; i < negotiated->nr; i++)\n-\t\tfeed_object(r, &negotiated->oid[i], po_in, 1);\n+\t\tappend_negative_object(r, &opts.haves, &negotiated->oid[i]);\n \n \twhile (refs) {\n \t\tif (!is_null_oid(&refs->old_oid))\n-\t\t\tfeed_object(r, &refs->old_oid, po_in, 1);\n+\t\t\tappend_negative_object(r, &opts.haves, &refs->old_oid);\n \t\tif (!is_null_oid(&refs->new_oid))\n-\t\t\tfeed_object(r, &refs->new_oid, po_in, 0);\n+\t\t\toid_array_append(&opts.wants, &refs->new_oid);\n \t\trefs = refs->next;\n \t}\n \n-\tfflush(po_in);\n-\tif (ferror(po_in))\n-\t\tdie_errno(\"error writing to pack-objects\");\n-\tfclose(po_in);\n+\tif (odb_generate_pack(r->objects, &generator, &opts))\n+\t\tdie(\"git pack-objects failed\");\n+\todb_generate_pack_options_release(&opts);\n \n \tif (args->stateless_rpc) {\n \t\tchar *buf = xmalloc(LARGE_PACKET_MAX);\n \t\twhile (1) {\n-\t\t\tssize_t n = xread(po.out, buf, LARGE_PACKET_MAX);\n+\t\t\tssize_t n = xread(generator->out, buf, LARGE_PACKET_MAX);\n \t\t\tif (n <= 0)\n \t\t\t\tbreak;\n \t\t\tsend_sideband(fd, -1, buf, n, LARGE_PACKET_MAX);\n \t\t}\n \t\tfree(buf);\n-\t\tclose(po.out);\n-\t\tpo.out = -1;\n+\t\tclose(generator->out);\n \t}\n \n-\trc = finish_command(&po);\n-\tif (rc) {\n-\t\t/*\n-\t\t * For a normal non-zero exit, we assume pack-objects wrote\n-\t\t * something useful to stderr. For death by signal, though,\n-\t\t * we should mention it to the user. The exception is SIGPIPE\n-\t\t * (141), because that's a normal occurrence if the remote end\n-\t\t * hangs up (and we'll report that by trying to read the unpack\n-\t\t * status).\n-\t\t */\n-\t\tif (rc > 128 && rc != 141)\n-\t\t\terror(\"pack-objects died of signal %d\", rc - 128);\n-\t\ttrace2_region_leave(\"send_pack\", \"pack_objects\", r);\n-\t\treturn -1;\n-\t}\n+\trc = odb_pack_generator_finish(generator);\n \ttrace2_region_leave(\"send_pack\", \"pack_objects\", r);\n-\treturn 0;\n+\treturn rc;\n }\n \n static int receive_unpack_status(struct packet_reader *reader)\n@@ -768,7 +735,7 @@ int send_pack(struct repository *r,\n \t\t\tgoto out;\n \t\t}\n \t\tif (!args->stateless_rpc)\n-\t\t\t/* Closed by pack_objects() via start_command() */\n+\t\t\t/* Consumed by the pack generator in pack_objects() */\n \t\t\tfd[1] = -1;\n \t}\n \tif (args->stateless_rpc && cmds_sent)\ndiff --git a/t/t5516-fetch-push.sh b/t/t5516-fetch-push.sh\nindex f3b3efc47f..b982b209bf 100755\n--- a/t/t5516-fetch-push.sh\n+++ b/t/t5516-fetch-push.sh\n@@ -1903,20 +1903,20 @@ test_expect_success 'push with config push.useBitmaps' '\n \ttest_unconfig push.useBitmaps &&\n \tGIT_TRACE2_EVENT=\"$PWD/default\" \\\n \tgit push --quiet testrepo main:test &&\n-\ttest_subcommand git pack-objects --all-progress-implied --revs --stdout \\\n-\t\t--thin --delta-base-offset -q \n---\n builtin/bundle.c | 33 ++++++++++++++++-----------------\n 1 file changed, 16 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 1e170e9278..bfafadc984 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -70,35 +70,34 @@ static int parse_options_cmd_bundle(int argc,\n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\t struct repository *repo UNUSED) {\n \tstruct strvec pack_opts = STRVEC_INIT;\n+\tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n-\tint ret;\n \tstruct option options[] = {\n-\t\tOPT_PASSTHRU_ARGV('q', \"quiet\", &pack_opts, NULL,\n-\t\t\t\t N_(\"do not show progress meter\"),\n-\t\t\t\t PARSE_OPT_NOARG),\n-\t\tOPT_PASSTHRU_ARGV(0, \"progress\", &pack_opts, NULL,\n-\t\t\t\t N_(\"show progress meter\"),\n-\t\t\t\t PARSE_OPT_NOARG),\n-\t\tOPT_PASSTHRU_ARGV(0, \"all-progress\", &pack_opts, NULL,\n-\t\t\t\t N_(\"historical; same as --progress\"),\n-\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n-\t\tOPT_PASSTHRU_ARGV(0, \"all-progress-implied\", &pack_opts, NULL,\n-\t\t\t\t N_(\"historical; does nothing\"),\n-\t\t\t\t PARSE_OPT_NOARG | PARSE_OPT_HIDDEN),\n+\t\tOPT_NEGBIT('q', \"quiet\", &progress,\n+\t\t\t N_(\"do not show progress meter\"), 1),\n+\t\tOPT_BIT(0, \"progress\", &progress,\n+\t\t\tN_(\"show progress meter\"), 1),\n+\t\tOPT_BIT_F(0, \"all-progress\", &progress,\n+\t\t\t N_(\"historical; same as --progress\"), 1,\n+\t\t\t PARSE_OPT_HIDDEN),\n+\t\tOPT_NOOP_NOARG(0, \"all-progress-implied\"),\n \t\tOPT_INTEGER(0, \"version\", &version,\n \t\t\t N_(\"specify bundle format version\")),\n \t\tOPT_END()\n \t};\n \tchar *bundle_file;\n-\n-\tif (isatty(STDERR_FILENO))\n-\t\tstrvec_push(&pack_opts, \"--progress\");\n-\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n+\tint ret;\n \n \targc = parse_options_cmd_bundle(argc, argv, prefix,\n \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n \t/* bundle internals use argv[1] as further parameters */\n \n+\tif (progress)\n+\t\tstrvec_push(&pack_opts, \"--progress\");\n+\telse\n+\t\tstrvec_push(&pack_opts, \"--quiet\");\n+\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n+\n \tif (!startup_info->have_repository)\n \t\tdie(_(\"Need a repository to create a bundle.\"));\n \tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550989","messageId":"20260821-b4-pks-odb-generate-pack-v4-5-074e8bd641f8@pks.im","threadId":"66136","inReplyTo":"20260821-b4-pks-odb-generate-pack-v4-0-074e8bd641f8@pks.im","subject":"[PATCH v4 5/6] bundle: get (mostly) rid of `the_repository`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:30:05Z","receivedAt":"2026-08-21T06:30:22Z","isPatch":true,"body":"Refactor \"bundle.c\" so that we don't depend on `the_repository` anymore.\nThis conversion is trivial for most of the part, as we already have a\nrepository available in all calling conexts.\n\nThe only exception is that we use `get_log_output_encoding()`, which\nimplicitly depends on `the_repository`. Add an `extern` declaration for\nthis function so that we can drop `USE_THE_REPOSITORY_VARIABLE` and not\naccidentally introduce more uses of `the_repository`.\n\nSigned-off-by: Patrick Steinhardt \n---\n bundle.c | 32 +++++++++++++++++++++-----------\n 1 file changed, 21 insertions(+), 11 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex b64716f252..a9330bf0d3 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -1,4 +1,3 @@\n-#define USE_THE_REPOSITORY_VARIABLE\n #define DISABLE_SIGN_COMPARE_WARNINGS\n \n #include \"git-compat-util.h\"\n@@ -21,6 +20,13 @@\n #include \"connected.h\"\n #include \"write-or-die.h\"\n \n+/*\n+ * NEEDSWORK: this function implicitly depends on `the_repository` and is not\n+ * available because we dropped USE_THE_REPOSITORY_VARIABLE. We can remove the\n+ * declaration once it's accessible via `repo_config_values`.\n+ */\n+extern const char *get_log_output_encoding(void);\n+\n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n static struct {\n@@ -294,7 +300,8 @@ int list_bundle_refs(struct bundle_header *header, int argc, const char **argv)\n \treturn list_refs(&header->references, argc, argv);\n }\n \n-static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n+static int is_tag_in_date_range(struct repository *repo,\n+\t\t\t\tstruct object *tag, struct rev_info *revs)\n {\n \tsize_t size;\n \tenum object_type type;\n@@ -305,7 +312,7 @@ static int is_tag_in_date_range(struct object *tag, struct rev_info *revs)\n \tif (revs->max_age == -1 && revs->min_age == -1)\n \t\tgoto out;\n \n-\tbuf = odb_read_object(the_repository->objects, &tag->oid, &type, &size);\n+\tbuf = odb_read_object(repo->objects, &tag->oid, &type, &size);\n \tif (!buf)\n \t\tgoto out;\n \tline = memmem(buf, size, \"\\ntagger \", 8);\n@@ -362,7 +369,8 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\tstruct object *object = revs->pending.objects[i].item;\n \t\tif (object->flags & UNINTERESTING)\n \t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n-\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid), the_hash_algo->hexsz);\n+\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid),\n+\t\t\t revs->repo->hash_algo->hexsz);\n \t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n \t}\n \tclose(pack_objects.in);\n@@ -395,10 +403,10 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \n \t\tif (e->item->flags & UNINTERESTING)\n \t\t\tcontinue;\n-\t\tif (repo_dwim_ref(the_repository, e->name, strlen(e->name),\n+\t\tif (repo_dwim_ref(revs->repo, e->name, strlen(e->name),\n \t\t\t\t &oid, &ref, 0) != 1)\n \t\t\tgoto skip_write_ref;\n-\t\tif (refs_read_ref_full(get_main_ref_store(the_repository), e->name, RESOLVE_REF_READING, &oid, &flag))\n+\t\tif (refs_read_ref_full(get_main_ref_store(revs->repo), e->name, RESOLVE_REF_READING, &oid, &flag))\n \t\t\tflag = 0;\n \t\tdisplay_ref = (flag & REF_ISSYMREF) ? e->name : ref;\n \n@@ -406,7 +414,7 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \t\t\tgoto skip_write_ref;\n \n \t\tif (e->item->type == OBJ_TAG &&\n-\t\t\t\t!is_tag_in_date_range(e->item, revs)) {\n+\t\t\t\t!is_tag_in_date_range(revs->repo, e->item, revs)) {\n \t\t\te->item->flags |= UNINTERESTING;\n \t\t\tgoto skip_write_ref;\n \t\t}\n@@ -428,7 +436,8 @@ static int write_bundle_refs(int bundle_fd, struct rev_info *revs)\n \n \t\tref_count++;\n \t\tstrset_add(&objects, display_ref);\n-\t\twrite_or_die(bundle_fd, oid_to_hex(&e->item->oid), the_hash_algo->hexsz);\n+\t\twrite_or_die(bundle_fd, oid_to_hex(&e->item->oid),\n+\t\t\t revs->repo->hash_algo->hexsz);\n \t\twrite_or_die(bundle_fd, \" \", 1);\n \t\twrite_or_die(bundle_fd, display_ref, strlen(display_ref));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n@@ -507,7 +516,7 @@ int create_bundle(struct repository *r, const char *path,\n \t * SHA1.\n \t * 2. @filter is required because we parsed an object filter.\n \t */\n-\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1_LEGACY] || revs.filter.choice)\n+\tif (r->hash_algo != &hash_algos[GIT_HASH_SHA1_LEGACY] || revs.filter.choice)\n \t\tmin_version = 3;\n \n \tif (argc > 1) {\n@@ -528,14 +537,15 @@ int create_bundle(struct repository *r, const char *path,\n \tif (version < 2 || version > 3) {\n \t\tdie(_(\"unsupported bundle version %d\"), version);\n \t} else if (version < min_version) {\n-\t\tdie(_(\"cannot write bundle version %d with algorithm %s\"), version, the_hash_algo->name);\n+\t\tdie(_(\"cannot write bundle version %d with algorithm %s\"), version,\n+\t\t r->hash_algo->name);\n \t} else if (version == 2) {\n \t\twrite_or_die(bundle_fd, v2_bundle_signature, strlen(v2_bundle_signature));\n \t} else {\n \t\tconst char *capability = \"@object-format=\";\n \t\twrite_or_die(bundle_fd, v3_bundle_signature, strlen(v3_bundle_signature));\n \t\twrite_or_die(bundle_fd, capability, strlen(capability));\n-\t\twrite_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));\n+\t\twrite_or_die(bundle_fd, r->hash_algo->name, strlen(r->hash_algo->name));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n \n \t\tif (revs.filter.choice) {\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550990","messageId":"20260821-b4-pks-odb-generate-pack-v4-6-074e8bd641f8@pks.im","threadId":"66136","inReplyTo":"20260821-b4-pks-odb-generate-pack-v4-0-074e8bd641f8@pks.im","subject":"[PATCH v4 6/6] bundle: generate packfiles via the object database","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:30:06Z","receivedAt":"2026-08-21T06:30:25Z","isPatch":true,"body":"git-bundle(1) spawns git-pack-objects(1) directly to generate the pack\ndata that gets appended to the bundle header. While bundles are not\npart of the wire protocol, they are a transfer mechanism for packs all\nthe same, so convert them to use the pack generation interface of the\nobject database as well.\n\nThis makes the pack generator the single spawn point for all pack\nstreams that leave the repository, leaving only local maintenance tasks\nlike git-repack(1) with direct knowledge of git-pack-objects(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/bundle.c | 13 +++--------\n bundle.c | 69 ++++++++++++++++++++++++++++----------------------------\n bundle.h | 3 +--\n 3 files changed, 39 insertions(+), 46 deletions(-)\n\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex bfafadc984..5c6d8e1343 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -68,8 +68,8 @@ static int parse_options_cmd_bundle(int argc,\n }\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n-\t\t\t struct repository *repo UNUSED) {\n-\tstruct strvec pack_opts = STRVEC_INIT;\n+\t\t\t struct repository *repo UNUSED)\n+{\n \tint progress = isatty(STDERR_FILENO);\n \tint version = -1;\n \tstruct option options[] = {\n@@ -92,16 +92,9 @@ static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n \t\t\tbuiltin_bundle_create_usage, options, &bundle_file);\n \t/* bundle internals use argv[1] as further parameters */\n \n-\tif (progress)\n-\t\tstrvec_push(&pack_opts, \"--progress\");\n-\telse\n-\t\tstrvec_push(&pack_opts, \"--quiet\");\n-\tstrvec_push(&pack_opts, \"--all-progress-implied\");\n-\n \tif (!startup_info->have_repository)\n \t\tdie(_(\"Need a repository to create a bundle.\"));\n-\tret = !!create_bundle(the_repository, bundle_file, argc, argv, &pack_opts, version);\n-\tstrvec_clear(&pack_opts);\n+\tret = !!create_bundle(the_repository, bundle_file, argc, argv, version, progress);\n \tfree(bundle_file);\n \treturn ret;\n }\ndiff --git a/bundle.c b/bundle.c\nindex a9330bf0d3..f55a521b2a 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -332,51 +332,52 @@ static int is_tag_in_date_range(struct repository *repo,\n \n \n /* Write the pack data to bundle_fd */\n-static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *pack_options)\n+static int write_pack_data(int bundle_fd, struct rev_info *revs, int progress)\n {\n-\tstruct child_process pack_objects = CHILD_PROCESS_INIT;\n+\tstruct odb_generate_pack_options opts = ODB_GENERATE_PACK_OPTIONS_INIT;\n+\tstruct odb_pack_generator *generator;\n+\tint ret = 0;\n \tint i;\n \n-\tstrvec_pushl(&pack_objects.args,\n-\t\t \"pack-objects\",\n-\t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n-\t\t NULL);\n-\tstrvec_pushv(&pack_objects.args, pack_options->v);\n+\topts.thin = 1;\n+\topts.ofs_delta = 1;\n+\tif (progress)\n+\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_VERBOSE;\n \tif (revs->filter.choice)\n-\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t list_objects_filter_spec(&revs->filter));\n-\tpack_objects.in = -1;\n-\tpack_objects.out = bundle_fd;\n-\tpack_objects.git_cmd = 1;\n+\t\topts.filter_spec = list_objects_filter_spec(&revs->filter);\n \n \t/*\n-\t * start_command() will close our descriptor if it's >1. Duplicate it\n-\t * to avoid surprising the caller.\n+\t * The pack generator will consume our descriptor if it's >1.\n+\t * Duplicate it to avoid surprising the caller.\n \t */\n-\tif (pack_objects.out > 1) {\n-\t\tpack_objects.out = dup(pack_objects.out);\n-\t\tif (pack_objects.out < 0) {\n-\t\t\terror_errno(_(\"unable to dup bundle descriptor\"));\n-\t\t\tchild_process_clear(&pack_objects);\n-\t\t\treturn -1;\n-\t\t}\n+\topts.pack_fd = bundle_fd;\n+\tif (opts.pack_fd > 1) {\n+\t\topts.pack_fd = dup(bundle_fd);\n+\t\tif (opts.pack_fd < 0)\n+\t\t\treturn error_errno(_(\"unable to dup bundle descriptor\"));\n \t}\n \n-\tif (start_command(&pack_objects))\n-\t\treturn error(_(\"Could not spawn pack-objects\"));\n-\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object *object = revs->pending.objects[i].item;\n \t\tif (object->flags & UNINTERESTING)\n-\t\t\twrite_or_die(pack_objects.in, \"^\", 1);\n-\t\twrite_or_die(pack_objects.in, oid_to_hex(&object->oid),\n-\t\t\t revs->repo->hash_algo->hexsz);\n-\t\twrite_or_die(pack_objects.in, \"\\n\", 1);\n+\t\t\toid_array_append(&opts.haves, &object->oid);\n+\t\telse\n+\t\t\toid_array_append(&opts.wants, &object->oid);\n \t}\n-\tclose(pack_objects.in);\n-\tif (finish_command(&pack_objects))\n-\t\treturn error(_(\"pack-objects died\"));\n-\treturn 0;\n+\n+\tif (odb_generate_pack(revs->repo->objects, &generator, &opts)) {\n+\t\tret = error(_(\"Could not spawn pack-objects\"));\n+\t\tgoto out;\n+\t}\n+\n+\tif (odb_pack_generator_finish(generator)) {\n+\t\tret = error(_(\"pack-objects died\"));\n+\t\tgoto out;\n+\t}\n+\n+out:\n+\todb_generate_pack_options_release(&opts);\n+\treturn ret;\n }\n \n /*\n@@ -485,7 +486,7 @@ static void write_bundle_prerequisites(struct commit *commit, void *data)\n }\n \n int create_bundle(struct repository *r, const char *path,\n-\t\t int argc, const char **argv, struct strvec *pack_options, int version)\n+\t\t int argc, const char **argv, int version, int progress)\n {\n \tstruct lock_file lock = LOCK_INIT;\n \tint bundle_fd = -1;\n@@ -594,7 +595,7 @@ int create_bundle(struct repository *r, const char *path,\n \t}\n \n \t/* write pack */\n-\tif (write_pack_data(bundle_fd, &revs_copy, pack_options)) {\n+\tif (write_pack_data(bundle_fd, &revs_copy, progress)) {\n \t\tret = -1;\n \t\tgoto out;\n \t}\ndiff --git a/bundle.h b/bundle.h\nindex d664b2f2d6..471da23d1b 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -27,8 +27,7 @@ int read_bundle_header(const char *path, struct bundle_header *header);\n int read_bundle_header_fd(int fd, struct bundle_header *header,\n \t\t\t const char *report_path);\n int create_bundle(struct repository *r, const char *path,\n-\t\t int argc, const char **argv, struct strvec *pack_options,\n-\t\t int version);\n+\t\t int argc, const char **argv, int version, int progress);\n \n enum verify_bundle_flags {\n \tVERIFY_BUNDLE_VERBOSE = (1 << 0),\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"550992","messageId":"aofxjYAYPGkCrtQ7@pks.im","threadId":"66136","inReplyTo":"CABPp-BHSFW38sF4dZkqZuGaRASVRj2FVG2NN1OTA7-Dd6Pt6rw@mail.gmail.com","subject":"Re: [PATCH v2 1/6] odb: introduce interface to generate packfiles","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:34:53Z","receivedAt":"2026-08-21T06:34:58Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 09:41:03PM -0700, Elijah Newren wrote:\n> On Wed, Aug 19, 2026 at 11:01 PM Patrick Steinhardt wrote:\n> >\n> > On Wed, Aug 19, 2026 at 09:56:56AM -0700, Elijah Newren wrote:\n> > > On Sun, Aug 16, 2026 at 10:40 PM Patrick Steinhardt wrote:\n> > > >\n> > > > +static int odb_source_files_generate_pack(struct odb_source *source UNUSED,\n> > > > + struct odb_pack_generator **out,\n> > > > + const struct odb_generate_pack_options *opts)\n> > > > +{\n> > > > + struct child_process cp = CHILD_PROCESS_INIT;\n> > > > + struct odb_pack_generator_files *generator;\n> > > > + FILE *in;\n> > > [...]\n> > > > + cp.clean_on_exit = 1;\n> > > > +\n> > > > + if (start_command(&cp))\n> > > > + return error(_(\"could not spawn pack-objects\"));\n> > > [...]\n> > > > + CALLOC_ARRAY(generator, 1);\n> > > > + generator->base.out = opts->pack_fd < 0 ? cp.out : -1;\n> > > > + generator->base.err = opts->progress_fd < 0 ? cp.err : -1;\n> > > > + generator->base.finish = odb_pack_generator_files_finish;\n> > > > + generator->cp = cp;\n> > > > +\n> > > > + *out = &generator->base;\n> > > > + return 0;\n> > > > +}\n> > >\n> > > Does this have a use-after-scope bug lurking here, due to the\n> > > combination of clean_on_exit = 1 (which makes a copy of &cp for later\n> > > use), and the fact that cp is a function-local? If I'm reading the\n> > > code right, start_command() calls mark_child_for_cleanup(), which does\n> > >\n> > > p->process = process; /* where process is &cp */\n> > >\n> > > and then cleanup_children() accesses various fields under p->process.\n> > > You do copy the necessary fields from cp to generator->cp, but\n> > > &generator->cp was not passed to start_command(), so p->process points\n> > > to the function-local cp.\n> >\n> > Oh, that's a very good catch indeed. Out of curiosity, how did you end\n> > up discovering this? Did you just happen to remember that we store the\n> > pointer out of scope or did the copy make you have a deeper look?\n> \n> Neither. Went to review the series, but I was worried I'd be missing\n> context from not reviewing earlier odb refactorings. Used AI to help\n> orient me and give me its own findings from reviewing your patches.\n> (AI will sometimes spot things I miss in a review, though it'll also\n> miss some things I catch.) And sometimes I iterate with AI to dig\n> into various areas. Anyway, it flagged the potential problem, and I\n> dug in to make sure it didn't look like a hallucination before\n> cleaning it up and passing it on. I'm still looking through your\n> other patches in this series, but should finish soon.\n\nI agree. For all the pain AI is causing, doing reviews is one of the\nthings where it's helping me a ton. Both by reviewing my own patch\nseries before I send them out, and by reviewing others.\n\nThanks!\n\nPatrick\n"},{"id":"550996","messageId":"aogCTy4-DYYhS-VK@pks.im","threadId":"66136","inReplyTo":"CABPp-BE63m2sB4-18JUiYDK+UXaCq9z_=A8JAutvjn155_HWZA@mail.gmail.com","subject":"Re: [PATCH v2 5/6] bundle: get (mostly) rid of `the_repository`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T07:46:23Z","receivedAt":"2026-08-21T07:46:35Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 10:42:49PM -0700, Elijah Newren wrote:\n> On Mon, Aug 17, 2026 at 10:26 PM Patrick Steinhardt wrote:\n> > On Mon, Aug 17, 2026 at 09:47:53AM -0700, Junio C Hamano wrote:\n> > > Patrick Steinhardt writes:\n> > > > diff --git a/bundle.c b/bundle.c\n> > > > index b64716f252..a9330bf0d3 100644\n> > > > --- a/bundle.c\n> > > > +++ b/bundle.c\n> > > > @@ -1,4 +1,3 @@\n> > > > -#define USE_THE_REPOSITORY_VARIABLE\n> > > > #define DISABLE_SIGN_COMPARE_WARNINGS\n> > > >\n> > > > #include \"git-compat-util.h\"\n> > > > @@ -21,6 +20,13 @@\n> > > > #include \"connected.h\"\n> > > > #include \"write-or-die.h\"\n> > > >\n> > > > +/*\n> > > > + * NEEDSWORK: this function implicitly depends on `the_repository` and is not\n> > > > + * available because we dropped USE_THE_REPOSITORY_VARIABLE. We can remove the\n> > > > + * declaration once it's accessible via `repo_config_values`.\n> > > > + */\n> > > > +extern const char *get_log_output_encoding(void);\n> > > > +\n> > >\n> > > Doesn't this defeat the whole \"drop #define USE_THE_REPOSITORY_VARIABLE\n> > > as a mark that we are done with this file and no longer need to\n> > > worry about it going forward because we won't be able to compile if\n> > > somebody adds a new use?\" premise?\n> >\n> > Yes and no. By removing the define early it allows us to not reintroduce\n> > new references to `the_repository` by accident, but carve out a single\n> > exception for one of the functions that still depends on it. The\n> > alternative would be to not do that, and if so there is no guarantee\n> > whatsoever that we won't introduce more references to `the_repository`\n> > in this file.\n> >\n> > So I'm still leaning towards keeping this as-is, but I don't feel very\n> > strongly about this. Let me know in case that argument doesn't sway you\n> > and I'll adapt.\n> \n> Would it make more sense to do this the way replay.c does:\n> \n> #define USE_THE_REPOSITORY_VARIABLE\n> \n> /*\n> * We technically need USE_THE_REPOSITORY_VARIABLE for , but\n> * do not want to use the_repository.\n> */\n> #define the_repository DO_NOT_USE_THE_REPOSITORY\n> \n> and remove the declaration of get_log_output_encoding() that you\n> added? Alternatively, should replay.c be adapted to the way you are\n> doing it here?\n\nThe benefit of removing `USE_THE_REPOSITORY_VARIABLE` completely over\nstubbing out `the_repository` is that it will also remove a couple of\nfunction declarations that implicitly rely on `the_repository`, like for\nexample `get_log_output_encoding()`. So I think it's a slightly better\nmechainsm over redefining `the_repository`.\n\nPatrick\n"},{"id":"551012","messageId":"CAOLa=ZQMjb1SzYTVVuMF0ajmre_5_q=L6bmSQwYY233f-RiVXA@mail.gmail.com","threadId":"66136","inReplyTo":"20260821-b4-pks-odb-generate-pack-v4-0-074e8bd641f8@pks.im","subject":"Re: [PATCH v4 0/6] odb: make packfile generation pluggable","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-21T12:38:11Z","receivedAt":"2026-08-21T12:38:15Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this patch series makes packfile generation pluggable.\n>\n> Note that this series only makes those parts pluggable that are required\n> for the transport layer. The other parts that relate to packfile\n> generation as required by our repository maintenance is kept as-is, as\n> there is a bunch of options there that are way too specific to the\n> \"files\" backend to be portable. This should ultimately not be much of a\n> problem though, as maintenance itself is already pluggable in the first\n> place.\n>\n> It's a bit of a shame though for git-pack-objects(1), which still isn't\n> usable with alternate backends. I tried several times to find good\n> solutions for making it fully pluggable, but due to the backend-specific\n> options it's an utter mess. I want to eventually address this though:\n> same as with git-refs(1), I want to introduce git-objects(1) to care\n> about all things ODB. And as part of that command we can also introduce\n> a command that generates packfiles in a generic fashion, without all the\n> cruft that git-pack-objects(1) has. This is part of a future patch\n> series though.\n>\n> Changes in v4:\n> - Improve an error message.\n> - Sneak in a small stylistic fix while at it.\n> - Link to v3: https://patch.msgid.link/20260820-b4-pks-odb-generate-pack-v3-0-bc42252f6169@pks.im\n>\n> Changes in v3:\n> - Fix a use-after-scope bug on abnormal exit when child processes are\n> cleaned up via `mark_child_for_cleanup()`, as noticed by Elijah.\n> - Link to v2: https://patch.msgid.link/20260817-b4-pks-odb-generate-pack-v2-0-4c8a96ccfdb3@pks.im\n>\n> Changes in v2:\n> - Mostly remove the dependencies on `the_repository` in \"bundle.c\".\n> - Link to v1: https://patch.msgid.link/20260807-b4-pks-odb-generate-pack-v1-0-7dec431ae7cd@pks.im\n>\n> The series is built on top of 2c78326f81 (The 11th batch, 2026-08-05).\n>\n> Thanks!\n>\n> Patrick\n>\n> ---\n> Patrick Steinhardt (6):\n> odb: introduce interface to generate packfiles\n> upload-pack: generate packfiles via the object database\n> send-pack: generate packfiles via the object database\n> builtin/bundle: refactor option handling for progress meter\n> bundle: get (mostly) rid of `the_repository`\n> bundle: generate packfiles via the object database\n>\n> builtin/bundle.c | 34 +++++------\n> bundle.c | 97 ++++++++++++++++++--------------\n> bundle.h | 3 +-\n> odb.c | 21 +++++++\n> odb.h | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++\n> odb/source-files.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++++\n> odb/source.h | 33 +++++++++++\n> send-pack.c | 101 +++++++++++----------------------\n> t/t5516-fetch-push.sh | 12 ++--\n> upload-pack.c | 125 +++++++++++++++--------------------------\n> 10 files changed, 508 insertions(+), 219 deletions(-)\n>\n> Range-diff versus v3:\n>\n> 1: 4a56334af1 = 1: 33039a0ab8 odb: introduce interface to generate packfiles\n> 2: 1ff0eaf6b7 ! 2: 7093fcee83 upload-pack: generate packfiles via the object database\n> @@ upload-pack.c: static void create_pack_file(struct upload_pack_data *pack_data,\n> -\t */\n> +\t\toid_array_append(&opts.haves,\n> +\t\t\t\t &pack_data->extra_edge_obj.objects[i].item->oid);\n> -+\n> +\n> +\topts.thin = pack_data->use_thin_pack;\n> +\tif (!pack_data->no_progress)\n> +\t\topts.progress = ODB_GENERATE_PACK_PROGRESS_STANDARD;\n> @@ upload-pack.c: static void create_pack_file(struct upload_pack_data *pack_data,\n> +\topts.progress_fd = -1;\n> +\n> +\tif (odb_generate_pack(the_repository->objects, &generator, &opts))\n> -+\t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n> ++\t\tdie(\"git upload-pack: unable to generate pack\");\n> +\todb_generate_pack_options_release(&opts);\n> -\n> ++\n> +\t/*\n> +\t * We read from generator->err to capture stderr output for the\n> +\t * progress bar, and generator->out to capture the pack data.\n> 3: 22a19a9a70 = 3: 0a2ca04c01 send-pack: generate packfiles via the object database\n> 4: 5d2275c90b = 4: 2d339ee7b7 builtin/bundle: refactor option handling for progress meter\n> 5: 0f00e6d234 = 5: 3cf0210247 bundle: get (mostly) rid of `the_repository`\n> 6: ae6af210ff ! 6: d3345e4407 bundle: generate packfiles via the object database\n> @@ Commit message\n>\n> ## builtin/bundle.c ##\n> @@ builtin/bundle.c: static int parse_options_cmd_bundle(int argc,\n> + }\n>\n> static int cmd_bundle_create(int argc, const char **argv, const char *prefix,\n> - \t\t\t struct repository *repo UNUSED) {\n> +-\t\t\t struct repository *repo UNUSED) {\n> -\tstruct strvec pack_opts = STRVEC_INIT;\n> ++\t\t\t struct repository *repo UNUSED)\n> ++{\n> \tint progress = isatty(STDERR_FILENO);\n> \tint version = -1;\n> \tstruct option options[] = {\n>\n> ---\n> base-commit: 2c78326f810173a4f3aefd8021f1e07575412481\n> change-id: 20260807-b4-pks-odb-generate-pack-f30fbcdef3fc\n\nEverything looks good now. Thanks!\n"}]}