{"thread":{"id":"66203","subject":"[PATCH 0/2] fetch-pack: allow parallelizing packfile URI fetches","startedAt":"2026-08-21T12:31:57Z","lastAt":"2026-08-31T05:43:47Z","messageCount":5,"participants":["Patrick Steinhardt","Justin Tobler"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"551008","messageId":"20260821-pks-parallelize-fetching-packfile-uris-v1-0-0df52d9427ce@pks.im","threadId":"66203","inReplyTo":null,"subject":"[PATCH 0/2] fetch-pack: allow parallelizing packfile URI fetches","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T12:31:43Z","receivedAt":"2026-08-21T12:31:57Z","isPatch":true,"body":"Hi,\n\nthis patch series prepares git-fetch(1) and git-clone(1) to handle\nfetches of packfile URIs in parallel. This can significantly speed up\nfetches when the server announces a bunch of packfiles, as shown in the\nbenchmarks in the second patch.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (2):\n      fetch-pack: prepare for threaded fetching of packfile URIs\n      fetch-pack: allow parallelizing packfile URI fetches\n\n Documentation/config/fetch.adoc |   9 ++\n fetch-pack.c                    | 228 ++++++++++++++++++++++++++++++----------\n t/t5702-protocol-v2.sh          |  44 ++++++++\n 3 files changed, 227 insertions(+), 54 deletions(-)\n\n\n---\nbase-commit: 1a3e64c6c4a623626ff0687008732a8e007e2a1c\nchange-id: 20260821-pks-parallelize-fetching-packfile-uris-b1ad24a82fe0\n\n"},{"id":"551010","messageId":"20260821-pks-parallelize-fetching-packfile-uris-v1-1-0df52d9427ce@pks.im","threadId":"66203","inReplyTo":"20260821-pks-parallelize-fetching-packfile-uris-v1-0-0df52d9427ce@pks.im","subject":"[PATCH 1/2] fetch-pack: prepare for threaded fetching of packfile URIs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T12:31:44Z","receivedAt":"2026-08-21T12:31:58Z","isPatch":true,"body":"In the next commit, we're about to add the ability to parallelize\nfetching packfile URIs. Refactor the code to prepare for this by\nsplitting the logic up into three explicit phases:\n\n  1. Preparation phase, where we allocate the state that will be\n     populated by the different threads.\n\n  2. Fetch phase, where we fetch the packfile URIs. This is the part\n     that will be parallelized, and we need to be careful to not access\n     any shared state here.\n\n  3. Aggregation phase, where we aggregate results from the parallel\n     worker threads.\n\nThis should not result in a user-visible change in behaviour.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n fetch-pack.c | 148 +++++++++++++++++++++++++++++++++++++----------------------\n 1 file changed, 94 insertions(+), 54 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 626f799712..6aca0b2588 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1668,6 +1668,98 @@ static void do_check_stateless_delimiter(int stateless_rpc,\n \t\t\t\t  _(\"git fetch-pack: expected response end packet\"));\n }\n \n+struct fetch_packfile_uri_result {\n+\tstruct oidset gitmodules_found;\n+\tchar packhash[GIT_MAX_HEXSZ + 1];\n+\tbool created_keep;\n+};\n+\n+static void fetch_packfile_uri(const char *uri_with_hash,\n+\t\t\t       const struct strvec *index_pack_args,\n+\t\t\t       struct fetch_packfile_uri_result *result)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tconst char *uri = uri_with_hash +\n+\t\tthe_hash_algo->hexsz + 1;\n+\n+\tstrvec_push(&cmd.args, \"http-fetch\");\n+\tstrvec_pushf(&cmd.args, \"--packfile=%.*s\",\n+\t\t     (int) the_hash_algo->hexsz, uri_with_hash);\n+\tfor (size_t j = 0; j < index_pack_args->nr; j++)\n+\t\tstrvec_pushf(&cmd.args, \"--index-pack-arg=%s\",\n+\t\t\t     index_pack_args->v[j]);\n+\tstrvec_push(&cmd.args, uri);\n+\tcmd.git_cmd = 1;\n+\tcmd.no_stdin = 1;\n+\tcmd.out = -1;\n+\tif (start_command(&cmd))\n+\t\tdie(\"fetch-pack: unable to spawn http-fetch\");\n+\n+\tif (read_in_full(cmd.out, result->packhash, 5) != 5 ||\n+\t    (memcmp(result->packhash, \"keep\\t\", 5) &&\n+\t     memcmp(result->packhash, \"pack\\t\", 5)))\n+\t\tdie(\"fetch-pack: expected pack or keep then TAB at start of http-fetch output\");\n+\tresult->created_keep = !memcmp(result->packhash, \"keep\\t\", 5);\n+\n+\tif (read_in_full(cmd.out, result->packhash,\n+\t\t\t the_hash_algo->hexsz + 1) != the_hash_algo->hexsz + 1 ||\n+\t    result->packhash[the_hash_algo->hexsz] != '\\n')\n+\t\tdie(\"fetch-pack: expected hash then LF in http-fetch output\");\n+\tresult->packhash[the_hash_algo->hexsz] = '\\0';\n+\n+\tparse_gitmodules_oids(cmd.out, &result->gitmodules_found);\n+\n+\tclose(cmd.out);\n+\n+\tif (finish_command(&cmd))\n+\t\tdie(\"fetch-pack: unable to finish http-fetch\");\n+\n+\tif (memcmp(uri_with_hash, result->packhash, the_hash_algo->hexsz))\n+\t\tdie(\"fetch-pack: pack downloaded from %s does not match expected hash %.*s\",\n+\t\t    uri, (int) the_hash_algo->hexsz,\n+\t\t    uri_with_hash);\n+}\n+\n+static void fetch_packfile_uris(const struct string_list *packfile_uris,\n+\t\t\t\tconst struct strvec *index_pack_args,\n+\t\t\t\tstruct oidset *gitmodules_found,\n+\t\t\t\tstruct string_list *pack_lockfiles)\n+{\n+\tstruct fetch_packfile_uri_result *results;\n+\n+\t/* Initialize the data. */\n+\tCALLOC_ARRAY(results, packfile_uris->nr);\n+\tfor (size_t i = 0; i < packfile_uris->nr; i++)\n+\t\toidset_init(&results[i].gitmodules_found, 0);\n+\n+\t/* Perform the fetches. */\n+\tfor (size_t i = 0; i < packfile_uris->nr; i++)\n+\t\tfetch_packfile_uri(packfile_uris->items[i].string,\n+\t\t\t\t   index_pack_args, &results[i]);\n+\n+\t/* Aggregate results. */\n+\tfor (size_t i = 0; i < packfile_uris->nr; i++) {\n+\t\tstruct fetch_packfile_uri_result *result = &results[i];\n+\t\tconst struct object_id *oid;\n+\t\tstruct oidset_iter iter;\n+\n+\t\tif (result->created_keep) {\n+\t\t\tchar *lockfile = xstrfmt(\"%s/pack/pack-%s.keep\",\n+\t\t\t\t\t\t repo_get_object_directory(the_repository),\n+\t\t\t\t\t\t result->packhash);\n+\t\t\tstring_list_append_nodup(pack_lockfiles, lockfile);\n+\t\t}\n+\n+\t\toidset_iter_init(&result->gitmodules_found, &iter);\n+\t\twhile ((oid = oidset_iter_next(&iter)))\n+\t\t\toidset_insert(gitmodules_found, oid);\n+\n+\t\toidset_clear(&result->gitmodules_found);\n+\t}\n+\n+\tfree(results);\n+}\n+\n static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t\t    int fd[2],\n \t\t\t\t    const struct ref *orig_ref,\n@@ -1692,7 +1784,6 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstruct object_id common_oid;\n \tint received_ready = 0;\n \tstruct string_list packfile_uris = STRING_LIST_INIT_DUP;\n-\tint i;\n \tstruct strvec index_pack_args = STRVEC_INIT;\n \tconst char *promisor_remote_config;\n \n@@ -1853,59 +1944,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n-\tfor (i = 0; i < packfile_uris.nr; i++) {\n-\t\tbool created_keep;\n-\t\tint j;\n-\t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n-\t\tchar packhash[GIT_MAX_HEXSZ + 1];\n-\t\tconst char *uri = packfile_uris.items[i].string +\n-\t\t\tthe_hash_algo->hexsz + 1;\n-\n-\t\tstrvec_push(&cmd.args, \"http-fetch\");\n-\t\tstrvec_pushf(&cmd.args, \"--packfile=%.*s\",\n-\t\t\t     (int) the_hash_algo->hexsz,\n-\t\t\t     packfile_uris.items[i].string);\n-\t\tfor (j = 0; j < index_pack_args.nr; j++)\n-\t\t\tstrvec_pushf(&cmd.args, \"--index-pack-arg=%s\",\n-\t\t\t\t     index_pack_args.v[j]);\n-\t\tstrvec_push(&cmd.args, uri);\n-\t\tcmd.git_cmd = 1;\n-\t\tcmd.no_stdin = 1;\n-\t\tcmd.out = -1;\n-\t\tif (start_command(&cmd))\n-\t\t\tdie(\"fetch-pack: unable to spawn http-fetch\");\n-\n-\t\tif (read_in_full(cmd.out, packhash, 5) != 5 ||\n-\t\t    (memcmp(packhash, \"keep\\t\", 5) &&\n-\t\t     memcmp(packhash, \"pack\\t\", 5)))\n-\t\t\tdie(\"fetch-pack: expected pack or keep then TAB at start of http-fetch output\");\n-\t\tcreated_keep = !memcmp(packhash, \"keep\\t\", 5);\n-\n-\t\tif (read_in_full(cmd.out, packhash,\n-\t\t\t\t the_hash_algo->hexsz + 1) != the_hash_algo->hexsz + 1 ||\n-\t\t    packhash[the_hash_algo->hexsz] != '\\n')\n-\t\t\tdie(\"fetch-pack: expected hash then LF in http-fetch output\");\n-\t\tpackhash[the_hash_algo->hexsz] = '\\0';\n-\n-\t\tparse_gitmodules_oids(cmd.out, &fsck_options.gitmodules_found);\n-\n-\t\tclose(cmd.out);\n-\n-\t\tif (finish_command(&cmd))\n-\t\t\tdie(\"fetch-pack: unable to finish http-fetch\");\n-\n-\t\tif (memcmp(packfile_uris.items[i].string, packhash,\n-\t\t\t   the_hash_algo->hexsz))\n-\t\t\tdie(\"fetch-pack: pack downloaded from %s does not match expected hash %.*s\",\n-\t\t\t    uri, (int) the_hash_algo->hexsz,\n-\t\t\t    packfile_uris.items[i].string);\n-\n-\t\tif (created_keep)\n-\t\t\tstring_list_append_nodup(pack_lockfiles,\n-\t\t\t\t\t\t xstrfmt(\"%s/pack/pack-%s.keep\",\n-\t\t\t\t\t\t\t repo_get_object_directory(the_repository),\n-\t\t\t\t\t\t\t packhash));\n-\t}\n+\tfetch_packfile_uris(&packfile_uris, &index_pack_args,\n+\t\t\t    &fsck_options.gitmodules_found, pack_lockfiles);\n \tstring_list_clear(&packfile_uris, 0);\n \tstrvec_clear(&index_pack_args);\n \n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"551011","messageId":"20260821-pks-parallelize-fetching-packfile-uris-v1-2-0df52d9427ce@pks.im","threadId":"66203","inReplyTo":"20260821-pks-parallelize-fetching-packfile-uris-v1-0-0df52d9427ce@pks.im","subject":"[PATCH 2/2] fetch-pack: allow parallelizing packfile URI fetches","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T12:31:45Z","receivedAt":"2026-08-21T12:32:00Z","isPatch":true,"body":"When cloning from a server that supports packfile URIs we may see\nmultiple URIs being announced by the server. If so, the expectation is\nthat the client will download all of those packfiles. This is being done\nsequentially, where we fetch one packfile after the other.\n\nIn many cases this should be fine, but there are scenarios where it's\nnot. When packfiles are for example hosted by object storage (think AWS\nS3 or GCS) then the way to achieve high performance is typically to\nparallelize downloading the data as a single connection is often capped\nat a certain bandwidth. Furthermore, when the server announces a bunch\nof smaller packfiles, then the overhead of establishing the connection\nmay eventually add up.\n\nDespite the limitations caused by the network bandwidth and latency, Git\nalso runs git-index-pack(1) on all of the fetched packfiles. This is\nanother task that can be easily parallelized for another speedup.\n\nAll of these limitations can be addressed by parallelizing the fetch.\nIntroduce a new configuration option that allows the user to ask for\nthis: by default we continue to not parallelize the fetch to retain the\nstatus quo. But when configured to 0 (where we auto-detect the number of\ncores) or a value larger than 1 we perform the fetches concurrently.\n\nWith this infrastructure in place we can significantly speed up such\nfetches. Using a local HTTP server demonstrates the speedup when using a\nthrottled connection of 2MB/s and downloading 8x1MB packfiles:\n\n    Benchmark 1: 2MB/s, 8x1MB packfiles, 1 threads\n      Time (mean ± σ):      4.321 s ±  0.003 s    [User: 0.195 s, System: 0.113 s]\n      Range (min … max):    4.318 s …  4.325 s    5 runs\n\n    Benchmark 2: 2MB/s, 8x1MB packfiles, 2 threads\n      Time (mean ± σ):      2.284 s ±  0.241 s    [User: 0.191 s, System: 0.114 s]\n      Range (min … max):    2.173 s …  2.714 s    5 runs\n\n    Benchmark 3: 2MB/s, 8x1MB packfiles, 4 threads\n      Time (mean ± σ):      1.212 s ±  0.238 s    [User: 0.192 s, System: 0.105 s]\n      Range (min … max):    1.102 s …  1.638 s    5 runs\n\n    Benchmark 4: 2MB/s, 8x1MB packfiles, 8 threads\n      Time (mean ± σ):     569.9 ms ±   2.5 ms    [User: 183.4 ms, System: 116.0 ms]\n      Range (min … max):   566.5 ms … 573.4 ms    5 runs\n\n    Summary\n      2MB/s, 8x1MB packfiles, 8 threads ran\n        2.13 ± 0.42 times faster than 2MB/s, 8x1MB packfiles, 4 threads\n        4.01 ± 0.42 times faster than 2MB/s, 8x1MB packfiles, 2 threads\n        7.58 ± 0.03 times faster than 2MB/s, 8x1MB packfiles, 1 threads\n\nQuite unsurprisingly, we scale almost linearly with the number of\nthreads in this case as we're limited by the bandwidth of a single\nconnection. But we can also demonstrate a speedup on an unthrottled\nconnection when downloading slightly larger packfiles:\n\n    Benchmark 1: unthrottled, 8x16MB packfiles, 1 threads\n      Time (mean ± σ):      2.434 s ±  0.031 s    [User: 2.067 s, System: 0.329 s]\n      Range (min … max):    2.381 s …  2.460 s    5 runs\n\n    Benchmark 2: unthrottled, 8x16MB packfiles, 2 threads\n      Time (mean ± σ):      1.353 s ±  0.129 s    [User: 2.025 s, System: 0.328 s]\n      Range (min … max):    1.288 s …  1.583 s    5 runs\n\n    Benchmark 3: unthrottled, 8x16MB packfiles, 4 threads\n      Time (mean ± σ):     702.9 ms ±  23.9 ms    [User: 1732.5 ms, System: 313.9 ms]\n      Range (min … max):   660.7 ms … 718.5 ms    5 runs\n\n    Benchmark 4: unthrottled, 8x16MB packfiles, 8 threads\n      Time (mean ± σ):     455.1 ms ±   7.7 ms    [User: 1730.0 ms, System: 372.3 ms]\n      Range (min … max):   442.8 ms … 462.8 ms    5 runs\n\n    Summary\n      unthrottled, 8x16MB packfiles, 8 threads ran\n        1.54 ± 0.06 times faster than unthrottled, 8x16MB packfiles, 4 threads\n        2.97 ± 0.29 times faster than unthrottled, 8x16MB packfiles, 2 threads\n        5.35 ± 0.11 times faster than unthrottled, 8x16MB packfiles, 1 threads\n\nIn this case, the speedup is caused by us running git-index-pack(1) in\nparallel. The improvement isn't linear, but still quite significant.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/config/fetch.adoc |  9 +++++\n fetch-pack.c                    | 86 +++++++++++++++++++++++++++++++++++++++--\n t/t5702-protocol-v2.sh          | 44 +++++++++++++++++++++\n 3 files changed, 136 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/fetch.adoc b/Documentation/config/fetch.adoc\nindex 00435e9a16..7afe8d7d5c 100644\n--- a/Documentation/config/fetch.adoc\n+++ b/Documentation/config/fetch.adoc\n@@ -94,6 +94,15 @@ A value of 0 will give some reasonable default. If unset, it defaults to 1.\n For submodules, this setting can be overridden using the `submodule.fetchJobs`\n config setting.\n \n+`fetch.packfileURIThreads`::\n+\tSpecifies the number of threads used to download packfiles\n+\tadvertised by the server via the `packfile-uris` capability in\n+\tparallel. Each packfile is downloaded via a separate\n+\tlinkgit:git-http-fetch[1] process.\n++\n+A value of 0 will use a reasonable default based on the number of available\n+CPUs. If unset, it defaults to 1, downloading packfiles sequentially.\n+\n `fetch.writeCommitGraph`::\n \tSet to true to write a commit-graph after every `git fetch` command\n \tthat downloads a pack-file from a remote. Using the `--split` option,\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 6aca0b2588..b9dca9e07f 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -37,6 +37,7 @@\n #include \"mergesort.h\"\n #include \"prio-queue.h\"\n #include \"promisor-remote.h\"\n+#include \"thread-utils.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -53,6 +54,7 @@ static struct shallow_lock shallow_lock;\n static const char *alternate_shallow_file;\n static struct strbuf fsck_msg_types = STRBUF_INIT;\n static struct string_list uri_protocols = STRING_LIST_INIT_DUP;\n+static unsigned int packfile_uri_threads = 1;\n \n /* Remember to update object flag allocation in object.h */\n #define COMPLETE\t(1U << 0)\n@@ -1692,6 +1694,13 @@ static void fetch_packfile_uri(const char *uri_with_hash,\n \tcmd.git_cmd = 1;\n \tcmd.no_stdin = 1;\n \tcmd.out = -1;\n+\n+\t/*\n+\t * Multiple threads may spawn and reap children concurrently in here.\n+\t * This is safe because the child-cleanup bookkeeping in run-command.c,\n+\t * which is not thread-safe, is only ever used when `clean_on_exit` is\n+\t * set.\n+\t */\n \tif (start_command(&cmd))\n \t\tdie(\"fetch-pack: unable to spawn http-fetch\");\n \n@@ -1720,22 +1729,84 @@ static void fetch_packfile_uri(const char *uri_with_hash,\n \t\t    uri_with_hash);\n }\n \n+struct fetch_packfile_uris_state {\n+\tconst struct string_list *packfile_uris;\n+\tconst struct strvec *index_pack_args;\n+\tstruct fetch_packfile_uri_result *results;\n+\tsize_t next;\n+\tpthread_mutex_t lock;\n+};\n+\n+static void *fetch_packfile_uris_thread(void *data)\n+{\n+\tstruct fetch_packfile_uris_state *state = data;\n+\n+\ttrace2_thread_start(\"fetch_packfile_uri\");\n+\n+\tfor (;;) {\n+\t\tsize_t i;\n+\n+\t\tpthread_mutex_lock(&state->lock);\n+\t\ti = state->next++;\n+\t\tpthread_mutex_unlock(&state->lock);\n+\t\tif (i >= state->packfile_uris->nr)\n+\t\t\tbreak;\n+\n+\t\tfetch_packfile_uri(state->packfile_uris->items[i].string,\n+\t\t\t\t   state->index_pack_args,\n+\t\t\t\t   &state->results[i]);\n+\t}\n+\n+\ttrace2_thread_exit();\n+\n+\treturn NULL;\n+}\n+\n static void fetch_packfile_uris(const struct string_list *packfile_uris,\n \t\t\t\tconst struct strvec *index_pack_args,\n \t\t\t\tstruct oidset *gitmodules_found,\n \t\t\t\tstruct string_list *pack_lockfiles)\n {\n+\tunsigned int nr_threads = packfile_uri_threads;\n \tstruct fetch_packfile_uri_result *results;\n \n+\tif (!nr_threads)\n+\t\tnr_threads = online_cpus();\n+\tif (nr_threads > packfile_uris->nr)\n+\t\tnr_threads = packfile_uris->nr;\n+\n \t/* Initialize the data. */\n \tCALLOC_ARRAY(results, packfile_uris->nr);\n \tfor (size_t i = 0; i < packfile_uris->nr; i++)\n \t\toidset_init(&results[i].gitmodules_found, 0);\n \n \t/* Perform the fetches. */\n-\tfor (size_t i = 0; i < packfile_uris->nr; i++)\n-\t\tfetch_packfile_uri(packfile_uris->items[i].string,\n-\t\t\t\t   index_pack_args, &results[i]);\n+\tif (nr_threads > 1) {\n+\t\tstruct fetch_packfile_uris_state state = {\n+\t\t\t.packfile_uris = packfile_uris,\n+\t\t\t.index_pack_args = index_pack_args,\n+\t\t\t.results = results,\n+\t\t};\n+\t\tpthread_t *threads;\n+\n+\t\tpthread_mutex_init(&state.lock, NULL);\n+\t\tALLOC_ARRAY(threads, nr_threads);\n+\n+\t\tfor (size_t i = 0; i < nr_threads; i++)\n+\t\t\tif (pthread_create(&threads[i], NULL,\n+\t\t\t\t\t   fetch_packfile_uris_thread, &state))\n+\t\t\t\tdie(_(\"failed to create thread\"));\n+\t\tfor (size_t i = 0; i < nr_threads; i++)\n+\t\t\tif (pthread_join(threads[i], NULL))\n+\t\t\t\tdie(_(\"failed to join thread\"));\n+\n+\t\tpthread_mutex_destroy(&state.lock);\n+\t\tfree(threads);\n+\t} else {\n+\t\tfor (size_t i = 0; i < packfile_uris->nr; i++)\n+\t\t\tfetch_packfile_uri(packfile_uris->items[i].string,\n+\t\t\t\t\t   index_pack_args, &results[i]);\n+\t}\n \n \t/* Aggregate results. */\n \tfor (size_t i = 0; i < packfile_uris->nr; i++) {\n@@ -2018,6 +2089,15 @@ static void fetch_pack_config(void)\n \t\t}\n \t}\n \n+\tif (!repo_config_get_uint(the_repository, \"fetch.packfileurithreads\",\n+\t\t\t\t  &packfile_uri_threads)) {\n+\t\tif (!HAVE_THREADS && packfile_uri_threads != 1) {\n+\t\t\twarning(_(\"no threads support, ignoring %s\"),\n+\t\t\t\t\"fetch.packfileURIThreads\");\n+\t\t\tpackfile_uri_threads = 1;\n+\t\t}\n+\t}\n+\n \trepo_config(the_repository, fetch_pack_config_cb, NULL);\n }\n \ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 0f05286de8..a43d64ac95 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -1270,6 +1270,50 @@ test_expect_success 'part of packfile response provided as URI' '\n \ttest_line_count = 6 filelist\n '\n \n+test_expect_success 'packfile URIs are downloaded in parallel' '\n+\tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\trm -rf \"$P\" http_child log trace2.txt &&\n+\n+\tgit init \"$P\" &&\n+\tgit -C \"$P\" config \"uploadpack.allowsidebandall\" \"true\" &&\n+\n+\tfor i in one two three\n+\tdo\n+\t\techo blob-$i >\"$P\"/blob-$i &&\n+\t\tgit -C \"$P\" add blob-$i &&\n+\t\tconfigure_exclusion \"$P\" blob-$i >h-$i || return 1\n+\tdone &&\n+\tgit -C \"$P\" commit -m message &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TEST_SIDEBAND_ALL=1 git \\\n+\t\t-c protocol.version=2 \\\n+\t\t-c fetch.uriprotocols=http,https \\\n+\t\t-c fetch.packfileurithreads=2 \\\n+\t\tclone --quiet \"$HTTPD_URL/smart/http_parent\" http_child 2>err &&\n+\n+\t# Ensure that all objects were found.\n+\tfor i in one two three\n+\tdo\n+\t\tgit -C http_child cat-file -e \"$(cat h-$i)\" || return 1\n+\tdone &&\n+\n+\t# Ensure that there are exactly 4 packfiles with associated .idx.\n+\tls http_child/.git/objects/pack/*.pack \\\n+\t    http_child/.git/objects/pack/*.idx >filelist &&\n+\ttest_line_count = 8 filelist &&\n+\n+\tif test_have_prereq PTHREADS\n+\tthen\n+\t\t# Ensure that exactly two worker threads were spawned.\n+\t\tgit grep --no-index --only-matching \"\\\"thread\\\":\\\"th[0-9]*:fetch_packfile_uri\\\"\" trace2.txt >threads &&\n+\t\tsort -u <threads >threads.unique &&\n+\t\ttest_line_count = 2 threads.unique &&\n+\t\ttest_grep ! \"warning: no threads support, ignoring fetch.packfileURIThreads\" err\n+\telse\n+\t\ttest_grep \"warning: no threads support, ignoring fetch.packfileURIThreads\" err\n+\tfi\n+'\n+\n test_expect_success 'packfile URIs with fetch instead of clone' '\n \tP=\"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n \trm -rf \"$P\" http_child log &&\n\n-- \n2.55.0.822.g20453c30eb.dirty\n\n"},{"id":"551492","messageId":"apSpaLf_Pu3G4Nqm@denethor","threadId":"66203","inReplyTo":"20260821-pks-parallelize-fetching-packfile-uris-v1-0-0df52d9427ce@pks.im","subject":"Re: [PATCH 0/2] fetch-pack: allow parallelizing packfile URI fetches","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-31T02:33:52Z","receivedAt":"2026-08-31T02:34:00Z","isPatch":true,"body":"On 26/08/21 02:31PM, Patrick Steinhardt wrote:\n> Hi,\n> \n> this patch series prepares git-fetch(1) and git-clone(1) to handle\n> fetches of packfile URIs in parallel. This can significantly speed up\n> fetches when the server announces a bunch of packfiles, as shown in the\n> benchmarks in the second patch.\n\nSo I've been working on a series to extend the use of the ODB\ntransaction interface to also cover fetch-pack. As part of this, my\ncurrent plan was to also refactor fetching packfile URIs so that they\ncan be written through `odb_transaction_write_pack()`. I like what this\nseries is doing, but I wonder if it might a bit more straightforward if\nwe try to land transaction here first. Otherwise, I think we may end up\nhaving to redo some of these changes to get parallelizing packfile URI\nfetches to work.\n\n-Justin\n"},{"id":"551493","messageId":"apUUiv4SD0-W8QS3@pks.im","threadId":"66203","inReplyTo":"apSpaLf_Pu3G4Nqm@denethor","subject":"Re: [PATCH 0/2] fetch-pack: allow parallelizing packfile URI fetches","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-31T05:43:38Z","receivedAt":"2026-08-31T05:43:47Z","isPatch":true,"body":"On Sun, Aug 30, 2026 at 09:33:52PM -0500, Justin Tobler wrote:\n> On 26/08/21 02:31PM, Patrick Steinhardt wrote:\n> > Hi,\n> > \n> > this patch series prepares git-fetch(1) and git-clone(1) to handle\n> > fetches of packfile URIs in parallel. This can significantly speed up\n> > fetches when the server announces a bunch of packfiles, as shown in the\n> > benchmarks in the second patch.\n> \n> So I've been working on a series to extend the use of the ODB\n> transaction interface to also cover fetch-pack. As part of this, my\n> current plan was to also refactor fetching packfile URIs so that they\n> can be written through `odb_transaction_write_pack()`. I like what this\n> series is doing, but I wonder if it might a bit more straightforward if\n> we try to land transaction here first. Otherwise, I think we may end up\n> having to redo some of these changes to get parallelizing packfile URI\n> fetches to work.\n\nI'm fine to drop this series for now in favor of yours. I'll resend once\nyour series has been merged. Thanks!\n\nPatrick\n"}]}