{"thread":{"id":"66569","subject":"[PATCH/RFC] upload-pack: fix slow fetches from a shallow repository","startedAt":"2026-10-17T22:11:46Z","lastAt":"2026-11-11T08:28:03Z","messageCount":5,"participants":["Harald Nordgren via GitGitGadget","Junio C Hamano","Harald Nordgren"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"555336","messageId":"pull.2455.git.git.1792275106640.gitgitgadget@gmail.com","threadId":"66569","inReplyTo":null,"subject":"[PATCH/RFC] upload-pack: fix slow fetches from a shallow repository","fromName":"Harald Nordgren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-10-17T22:11:46Z","receivedAt":"2026-10-17T22:11:46Z","isPatch":true,"sender":{"key":"haraldnordgren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/9569897?v=4"},"body":"From: Harald Nordgren <haraldnordgren@gmail.com>\n\nFetching from a shallow repository with a long history can hang. In a\nhomebrew-core clone with 900k commits, fetching one new commit into\nanother clone took 7 minutes. The same fetch over protocol v0 takes\nabout a second.\n\nOver protocol v2, the server goes through all of its history to work\nout which of its shallow commits to tell the client about, and a\nhistory with many merges makes it go over the same commits again and\nagain. Protocol v0 lists all shallow commits and lets the client sort\nthem out.\n\nDo the same in v2 when the client does not ask for a depth. This also\nmakes `git clone --reject-shallow` refuse a shallow repository over v2\nin every case, as it already does over v0. For `git fetch --unshallow`,\nwhich still needs to go through the history, visit each commit only\nonce.\n\nSigned-off-by: Harald Nordgren <haraldnordgren@gmail.com>\n---\n    upload-pack: fix slow fetches from a shallow repository\n    \n    Fix git fetch hanging for minutes when the remote is a shallow\n    repository with a long history.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2455%2FHaraldNordgren%2Fupload-pack-shallow-walk-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2455/HaraldNordgren/upload-pack-shallow-walk-v1\nPull-Request: https://github.com/git/git/pull/2455\n\n shallow.c     | 3 ++-\n upload-pack.c | 5 ++---\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/shallow.c b/shallow.c\nindex 8e244a5669..921db5e17a 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -209,7 +209,8 @@ static struct commit_list *get_shallows_or_depth(struct object_array *heads,\n \t\t\t\t*depth_slot = xmalloc(sizeof(int));\n \t\t\t\t**depth_slot = cur_depth;\n \t\t\t} else {\n-\t\t\t\tif (cur_depth >= **depth_slot)\n+\t\t\t\tif (depth == INFINITE_DEPTH ||\n+\t\t\t\t    cur_depth >= **depth_slot)\n \t\t\t\t\tcontinue;\n \t\t\t\t**depth_slot = cur_depth;\n \t\t\t}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 197bcfa279..844bf80575 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1718,9 +1718,8 @@ static void send_shallow_info(struct upload_pack_data *data)\n \n \tpacket_writer_write(&data->writer, \"shallow-info\\n\");\n \n-\tif (!send_shallow_list(data) &&\n-\t    is_repository_shallow(the_repository))\n-\t\tdeepen(data, INFINITE_DEPTH);\n+\tif (!send_shallow_list(data))\n+\t\tadvertise_shallow_grafts(1);\n \n \tpacket_delim(1);\n }\n\nbase-commit: c51408889c3b372814bc20b9431c3b74d0d268eb\n-- \ngitgitgadget\n\n"},{"id":"555414","messageId":"xmqqpkx57rcv.fsf@gitster.g","threadId":"66569","inReplyTo":"pull.2455.git.git.1792275106640.gitgitgadget@gmail.com","subject":"Re: [PATCH/RFC] upload-pack: fix slow fetches from a shallow repository","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-19T13:27:28Z","receivedAt":"2026-10-19T13:27:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Harald Nordgren via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 197bcfa279..844bf80575 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -1718,9 +1718,8 @@ static void send_shallow_info(struct upload_pack_data *data)\n>  \n>  \tpacket_writer_write(&data->writer, \"shallow-info\\n\");\n>  \n> -\tif (!send_shallow_list(data) &&\n> -\t    is_repository_shallow(the_repository))\n> -\t\tdeepen(data, INFINITE_DEPTH);\n> +\tif (!send_shallow_list(data))\n> +\t\tadvertise_shallow_grafts(1);\n\nThe advertise_shallow_grafts() function is written for protocol\nv0 and uses packet_write_fmt() to write raw data directly to the\nnamed file descriptor (in this case, fd #1).\n\nHowever, protocol v2 negotiation uses the packet_writer API.  When\nsideband-all is in use, the response must be multiplexed over the\nsideband channel.  Because the updated code fails to use the\npacket_writer API, 'shallow' lines are written without the '\\001'\nband designator.  The other side then interprets 's' (the first\nbyte of 'shallow', which is 115 in decimal) as the sideband channel\nand dies with a protocol violation.\n\n t/t5702-protocol-v2.sh | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git c/t/t5702-protocol-v2.sh w/t/t5702-protocol-v2.sh\nindex 7e69ba7f84..df3ec8c1c9 100755\n--- c/t/t5702-protocol-v2.sh\n+++ w/t/t5702-protocol-v2.sh\n@@ -15,6 +15,18 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n start_git_daemon --export-all --enable=receive-pack\n daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n \n+test_expect_success 'fetch from shallow server over v2' '\n+\ttest_when_finished \"rm -fr server shallow-server client\" &&\n+\tgit init server &&\n+\ttest_commit -C server one &&\n+\ttest_commit -C server two &&\n+\tgit clone --depth=1 \"file://$(pwd)/server\" shallow-server &&\n+\ttest_commit -C shallow-server three &&\n+\n+\tgit -C shallow-server config uploadpack.allowsidebandall true &&\n+\tgit -c protocol.version=2 clone \"file://$(pwd)/shallow-server\" client\n+'\n+\n test_expect_success 'create repo to be served by git-daemon' '\n \tgit init \"$daemon_parent\" &&\n \ttest_commit -C \"$daemon_parent\" one\n\n\n"},{"id":"555423","messageId":"CAHwyqnWw+SDy357RN2ENMgThBfhZY+V_Vr7zR9HdR3DaGKV0cw@mail.gmail.com","threadId":"66569","inReplyTo":"xmqqpkx57rcv.fsf@gitster.g","subject":"Re: [PATCH/RFC] upload-pack: fix slow fetches from a shallow repository","fromName":"Harald Nordgren","fromEmail":"haraldnordgren@gmail.com","sentAt":"2026-10-19T14:50:51Z","receivedAt":"2026-10-19T14:50:51Z","isPatch":true,"sender":{"key":"haraldnordgren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/9569897?v=4"},"body":"> The advertise_shallow_grafts() function is written for protocol\n> v0 and uses packet_write_fmt() to write raw data directly to the\n> named file descriptor (in this case, fd #1).\n>\n> However, protocol v2 negotiation uses the packet_writer API.  When\n> sideband-all is in use, the response must be multiplexed over the\n> sideband channel.  Because the updated code fails to use the\n> packet_writer API, 'shallow' lines are written without the '\\001'\n> band designator.  The other side then interprets 's' (the first\n> byte of 'shallow', which is 115 in decimal) as the sideband channel\n> and dies with a protocol violation.\n\nThanks for pointing this out! I'll take a look!\n\n\nHarald\n\n"},{"id":"555540","messageId":"pull.2455.v2.git.git.1792503631364.gitgitgadget@gmail.com","threadId":"66569","inReplyTo":"pull.2455.git.git.1792275106640.gitgitgadget@gmail.com","subject":"[PATCH v2] upload-pack: fix slow fetches from a shallow repository","fromName":"Harald Nordgren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-10-20T13:40:31Z","receivedAt":"2026-10-20T13:40:31Z","isPatch":true,"sender":{"key":"haraldnordgren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/9569897?v=4"},"body":"From: Harald Nordgren <haraldnordgren@gmail.com>\n\nFetching from a shallow repository with a long history can hang. In a\nhomebrew-core clone with 900k commits, fetching one new commit into\nanother clone took 7 minutes. The same fetch over protocol v0 takes\nabout a second.\n\nOver protocol v2, the server goes through all of its history to work\nout which of its shallow commits to tell the client about, and a\nhistory with many merges makes it go over the same commits again and\nagain. Protocol v0 lists all shallow commits and lets the client sort\nthem out.\n\nDo the same in v2 when the client does not ask for a depth. This also\nmakes `git clone --reject-shallow` refuse a shallow repository over v2\nin every case, as it already does over v0. For `git fetch --unshallow`,\nwhich still needs to go through the history, visit each commit only\nonce.\n\nSigned-off-by: Harald Nordgren <haraldnordgren@gmail.com>\n---\n    upload-pack: fix slow fetches from a shallow repository\n    \n    Fix git fetch hanging for minutes when the remote is a shallow\n    repository with a long history.\n    \n    Changes in v2:\n    \n     * Fix fetches with sideband-all by writing the shallow lines through\n       the packet writer. Add a test that clones from a shallow repository\n       with uploadpack.allowSidebandAll set.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2455%2FHaraldNordgren%2Fupload-pack-shallow-walk-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2455/HaraldNordgren/upload-pack-shallow-walk-v2\nPull-Request: https://github.com/git/git/pull/2455\n\nRange-diff vs v1:\n\n 1:  4f4192120c ! 1:  8b0cb4c0e0 upload-pack: fix slow fetches from a shallow repository\n     @@ shallow.c: static struct commit_list *get_shallows_or_depth(struct object_array\n       \t\t\t\t**depth_slot = cur_depth;\n       \t\t\t}\n      \n     + ## t/t5537-fetch-shallow.sh ##\n     +@@ t/t5537-fetch-shallow.sh: test_expect_success 'clone from shallow clone' '\n     + \t)\n     + '\n     + \n     ++test_expect_success 'clone from shallow clone with sideband-all' '\n     ++\ttest_config -C shallow uploadpack.allowSidebandAll true &&\n     ++\ttest_when_finished \"rm -rf shallow-sideband-all\" &&\n     ++\tgit clone --no-local shallow shallow-sideband-all &&\n     ++\t(\n     ++\t\tcd shallow-sideband-all &&\n     ++\t\tgit log --format=%s >actual &&\n     ++\t\ttest_write_lines 4 3 >expect &&\n     ++\t\ttest_cmp expect actual\n     ++\t)\n     ++'\n     ++\n     + test_expect_success 'fetch from shallow clone' '\n     + \t(\n     + \tcd shallow &&\n     +\n       ## upload-pack.c ##\n     -@@ upload-pack.c: static void send_shallow_info(struct upload_pack_data *data)\n     +@@ upload-pack.c: static void send_wanted_ref_info(struct upload_pack_data *data)\n     + \tpacket_writer_delim(&data->writer);\n     + }\n       \n     - \tpacket_writer_write(&data->writer, \"shallow-info\\n\");\n     ++static int send_shallow_graft(const struct commit_graft *graft, void *cb_data)\n     ++{\n     ++\tstruct packet_writer *writer = cb_data;\n     ++\n     ++\tif (graft->nr_parent == -1)\n     ++\t\tpacket_writer_write(writer, \"shallow %s\\n\",\n     ++\t\t\t\t    oid_to_hex(&graft->oid));\n     ++\treturn 0;\n     ++}\n     ++\n     + static void send_shallow_info(struct upload_pack_data *data)\n     + {\n     + \t/* No shallow info needs to be sent */\n     +@@ upload-pack.c: static void send_shallow_info(struct upload_pack_data *data)\n       \n     --\tif (!send_shallow_list(data) &&\n     --\t    is_repository_shallow(the_repository))\n     + \tif (!send_shallow_list(data) &&\n     + \t    is_repository_shallow(the_repository))\n      -\t\tdeepen(data, INFINITE_DEPTH);\n     -+\tif (!send_shallow_list(data))\n     -+\t\tadvertise_shallow_grafts(1);\n     ++\t\tfor_each_commit_graft(send_shallow_graft, &data->writer);\n       \n       \tpacket_delim(1);\n       }\n\n\n shallow.c                |  3 ++-\n t/t5537-fetch-shallow.sh | 12 ++++++++++++\n upload-pack.c            | 12 +++++++++++-\n 3 files changed, 25 insertions(+), 2 deletions(-)\n\ndiff --git a/shallow.c b/shallow.c\nindex 8e244a5669..921db5e17a 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -209,7 +209,8 @@ static struct commit_list *get_shallows_or_depth(struct object_array *heads,\n \t\t\t\t*depth_slot = xmalloc(sizeof(int));\n \t\t\t\t**depth_slot = cur_depth;\n \t\t\t} else {\n-\t\t\t\tif (cur_depth >= **depth_slot)\n+\t\t\t\tif (depth == INFINITE_DEPTH ||\n+\t\t\t\t    cur_depth >= **depth_slot)\n \t\t\t\t\tcontinue;\n \t\t\t\t**depth_slot = cur_depth;\n \t\t\t}\ndiff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh\nindex f323ceebd2..140d5c2dba 100755\n--- a/t/t5537-fetch-shallow.sh\n+++ b/t/t5537-fetch-shallow.sh\n@@ -43,6 +43,18 @@ test_expect_success 'clone from shallow clone' '\n \t)\n '\n \n+test_expect_success 'clone from shallow clone with sideband-all' '\n+\ttest_config -C shallow uploadpack.allowSidebandAll true &&\n+\ttest_when_finished \"rm -rf shallow-sideband-all\" &&\n+\tgit clone --no-local shallow shallow-sideband-all &&\n+\t(\n+\t\tcd shallow-sideband-all &&\n+\t\tgit log --format=%s >actual &&\n+\t\ttest_write_lines 4 3 >expect &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'fetch from shallow clone' '\n \t(\n \tcd shallow &&\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 197bcfa279..e6dc19526c 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1709,6 +1709,16 @@ static void send_wanted_ref_info(struct upload_pack_data *data)\n \tpacket_writer_delim(&data->writer);\n }\n \n+static int send_shallow_graft(const struct commit_graft *graft, void *cb_data)\n+{\n+\tstruct packet_writer *writer = cb_data;\n+\n+\tif (graft->nr_parent == -1)\n+\t\tpacket_writer_write(writer, \"shallow %s\\n\",\n+\t\t\t\t    oid_to_hex(&graft->oid));\n+\treturn 0;\n+}\n+\n static void send_shallow_info(struct upload_pack_data *data)\n {\n \t/* No shallow info needs to be sent */\n@@ -1720,7 +1730,7 @@ static void send_shallow_info(struct upload_pack_data *data)\n \n \tif (!send_shallow_list(data) &&\n \t    is_repository_shallow(the_repository))\n-\t\tdeepen(data, INFINITE_DEPTH);\n+\t\tfor_each_commit_graft(send_shallow_graft, &data->writer);\n \n \tpacket_delim(1);\n }\n\nbase-commit: c51408889c3b372814bc20b9431c3b74d0d268eb\n-- \ngitgitgadget\n\n"},{"id":"558162","messageId":"CAHwyqnWZMb4DYf5yxS3CxiRs0ZnZwy-z8-k9JmxGgV5edtmh9g@mail.gmail.com","threadId":"66569","inReplyTo":"pull.2455.v2.git.git.1792503631364.gitgitgadget@gmail.com","subject":"Re: [PATCH v2] upload-pack: fix slow fetches from a shallow repository","fromName":"Harald Nordgren","fromEmail":"haraldnordgren@gmail.com","sentAt":"2026-11-11T08:28:03Z","receivedAt":"2026-11-11T08:28:03Z","isPatch":true,"sender":{"key":"haraldnordgren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/9569897?v=4"},"body":"Hi Junio!\n\nWould you have time to take another look here?\n\n\nHarald\n\n"}]}