{"thread":{"id":"65470","subject":"[GSoC PATCH v2] backfill: add --[no-]progress option","startedAt":"2026-04-12T19:37:04Z","lastAt":"2026-04-15T18:29:01Z","messageCount":5,"participants":["Trieu Huynh","Derrick Stolee","Tian Yuchen"],"isPatch":true,"patchVersion":2,"patchTotal":null},"messages":[{"id":"541443","messageId":"20260412193659.26288-1-viking4@gmail.com","threadId":"65470","inReplyTo":null,"subject":"[GSoC PATCH v2] backfill: add --[no-]progress option","fromName":"Trieu Huynh","fromEmail":"vikingtc4@gmail.com","sentAt":"2026-04-12T19:36:59Z","receivedAt":"2026-04-12T19:37:04Z","isPatch":true,"body":"From: Trieu Huynh <vikingtc4@gmail.com>\n\n'git backfill' does not show an overall progress bar across\nbatches, giving no cross-batch feedback during potentially\nlong-running operations on large repositories.  By contrast,\n'git fetch', 'git gc', and 'git index-pack' all support\n--[no-]progress.\n\nAdd a --[no-]progress option that tracks the total number of\nmissing blobs downloaded across all batches, defaulting to\nshowing progress when stderr is a terminal (matching the\nbehaviour of 'git fetch').\n\nAdd tests to verify that:\n - progress is shown by default on a TTY\n - --progress forces output regardless of TTY\n - --no-progress suppresses output\n\nSigned-off-by: Trieu Huynh <vikingtc4@gmail.com>\n---\n builtin/backfill.c  | 18 +++++++++++++++++-\n t/t5620-backfill.sh | 24 ++++++++++++++++++++++++\n 2 files changed, 41 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/backfill.c b/builtin/backfill.c\nindex d794dd842f..e90c899071 100644\n--- a/builtin/backfill.c\n+++ b/builtin/backfill.c\n@@ -26,7 +26,7 @@\n #include \"path-walk.h\"\n \n static const char * const builtin_backfill_usage[] = {\n-\tN_(\"git backfill [--min-batch-size=<n>] [--[no-]sparse]\"),\n+\tN_(\"git backfill [--min-batch-size=<n>] [--[no-]sparse] [--[no-]progress]\"),\n \tNULL\n };\n \n@@ -36,6 +36,9 @@ struct backfill_context {\n \tsize_t min_batch_size;\n \tint sparse;\n \tstruct rev_info revs;\n+\tint show_progress;\n+\tsize_t nr_downloaded;\n+\tstruct progress *progress;\n };\n \n static void backfill_context_clear(struct backfill_context *ctx)\n@@ -48,6 +51,7 @@ static void download_batch(struct backfill_context *ctx)\n \tpromisor_remote_get_direct(ctx->repo,\n \t\t\t\t   ctx->current_batch.oid,\n \t\t\t\t   ctx->current_batch.nr);\n+\tctx->nr_downloaded += ctx->current_batch.nr;\n \toid_array_clear(&ctx->current_batch);\n \n \t/*\n@@ -55,6 +59,7 @@ static void download_batch(struct backfill_context *ctx)\n \t * avoid possible duplicate downloads of the same objects.\n \t */\n \todb_reprepare(ctx->repo->objects);\n+\tdisplay_progress(ctx->progress, ctx->nr_downloaded);\n }\n \n static int fill_missing_blobs(const char *path UNUSED,\n@@ -121,12 +126,16 @@ int cmd_backfill(int argc, const char **argv, const char *prefix, struct reposit\n \t\t.min_batch_size = 50000,\n \t\t.sparse = -1,\n \t\t.revs = REV_INFO_INIT,\n+\t\t.nr_downloaded = 0,\n+\t\t.show_progress = -1,\n \t};\n \tstruct option options[] = {\n \t\tOPT_UNSIGNED(0, \"min-batch-size\", &ctx.min_batch_size,\n \t\t\t     N_(\"Minimum number of objects to request at a time\")),\n \t\tOPT_BOOL(0, \"sparse\", &ctx.sparse,\n \t\t\t N_(\"Restrict the missing objects to the current sparse-checkout\")),\n+\t\tOPT_BOOL(0, \"progress\", &ctx.show_progress,\n+\t\t\t N_(\"show progress while downloading missing objects\")),\n \t\tOPT_END(),\n \t};\n \tstruct repo_config_values *cfg = repo_config_values(the_repository);\n@@ -150,7 +159,14 @@ int cmd_backfill(int argc, const char **argv, const char *prefix, struct reposit\n \tif (ctx.sparse < 0)\n \t\tctx.sparse = cfg->apply_sparse_checkout;\n \n+\tif (ctx.show_progress < 0)\n+\t\tctx.show_progress = isatty(2);\n+\n+\tif (ctx.show_progress)\n+\t\tctx.progress = start_progress(ctx.repo,\n+\t\t\t\t\t      _(\"Downloading missing blobs\"), 0);\n \tresult = do_backfill(&ctx);\n+\tstop_progress(&ctx.progress);\n \tbackfill_context_clear(&ctx);\n \trelease_revisions(&ctx.revs);\n \treturn result;\ndiff --git a/t/t5620-backfill.sh b/t/t5620-backfill.sh\nindex f3b5e39493..a75b84d8ac 100755\n--- a/t/t5620-backfill.sh\n+++ b/t/t5620-backfill.sh\n@@ -133,6 +133,30 @@ test_expect_success 'do partial clone 2, backfill min batch size' '\n \ttest_line_count = 0 revs2\n '\n \n+test_expect_success TTY 'backfill shows progress on tty by default' '\n+\tgit clone --no-checkout --filter=blob:none \\\n+\t\t--single-branch --branch=main \\\n+\t\t\"file://$(pwd)/srv.bare\" clone-tty &&\n+\ttest_terminal env GIT_PROGRESS_DELAY=0 git -C clone-tty backfill 2>err &&\n+\ttest_grep \"Downloading missing blobs\" err\n+'\n+\n+test_expect_success 'backfill --progress shows progress' '\n+\tgit clone --no-checkout --filter=blob:none \\\n+\t\t--single-branch --branch=main \\\n+\t\t\"file://$(pwd)/srv.bare\" clone-progress &&\n+\tgit -C clone-progress backfill --progress 2>err &&\n+\ttest_grep \"Downloading missing blobs\" err\n+'\n+\n+test_expect_success 'backfill --no-progress suppresses progress' '\n+\tgit clone --no-checkout --filter=blob:none \\\n+\t\t--single-branch --branch=main \\\n+\t\t\"file://$(pwd)/srv.bare\" clone-no-progress &&\n+\tgit -C clone-no-progress backfill --no-progress 2>err &&\n+\ttest_grep ! \"Downloading missing blobs\" err\n+'\n+\n test_expect_success 'backfill --sparse without sparse-checkout fails' '\n \tgit init not-sparse &&\n \ttest_must_fail git -C not-sparse backfill --sparse 2>err &&\n-- \n2.43.0\n\n"},{"id":"541444","messageId":"d2cf741c-a381-42a6-9d26-e38481696adb@gmail.com","threadId":"65470","inReplyTo":"20260412193659.26288-1-viking4@gmail.com","subject":"Re: [GSoC PATCH v2] backfill: add --[no-]progress option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2026-04-12T19:46:17Z","receivedAt":"2026-04-12T19:46:20Z","isPatch":true,"body":"On 4/12/26 3:36 PM, Trieu Huynh wrote:\n> From: Trieu Huynh <vikingtc4@gmail.com>\n> \n> 'git backfill' does not show an overall progress bar across\n> batches, giving no cross-batch feedback during potentially\n> long-running operations on large repositories.  By contrast,\n> 'git fetch', 'git gc', and 'git index-pack' all support\n> --[no-]progress.\n> \n> Add a --[no-]progress option that tracks the total number of\n> missing blobs downloaded across all batches, defaulting to\n> showing progress when stderr is a terminal (matching the\n> behaviour of 'git fetch').\n> \n> Add tests to verify that:\n>   - progress is shown by default on a TTY\n>   - --progress forces output regardless of TTY\n>   - --no-progress suppresses output\n\nI think the tests do show an improvement, but we're missing\nthe interaction with the underlying fetch's progress\nindicators. I don't see any mention of how your backfill\nprogress indicators will work with or against the fetch's\nprogress from the remote and index-pack steps.\n\nFurther, if a user supplies 'git backfill --no-progress'\nthen they are probably saying \"I don't want any progress\nindicators\" and that would signal also that the fetch should\nbe quiet. This is perhaps the key detail that makes your\ncurrent version unable to move forward. It creates an\nimplication that it doesn't follow-through on.\n\nOne way to go about this is to hide the 'git fetch' output\nentirely by passing '--quiet' unconditionally from the\nbackfill command. But this may also be too much for users\nwho want to watch the download statistics from the remote.\n\nPerhaps a way to have a robust set that allows all things\nto interact is to do the following:\n\n1. Add a --[no-]verbose option that is off by default. The\n    implementation sends the --quiet flag to 'git fetch' if\n    --verbose isn't provided from the user. This reduces the\n    noise for the default user.\n\n2. Add a --[no-]progress option as you've provided here.\n\nThe complexity at the end is about what happens when the\nuser provides both --verbose and --progress, which is the\nsituation that this patch is currently in. How do the\nprogress indicators mingle with the verbose fetch output?\n\nThanks,\n-Stolee\n\n"},{"id":"541498","messageId":"wsbnw3am5fq6hpjwmmbguo2c3mnv4qkr3hh7apawch7smns6zx@rxegeuugbnhg","threadId":"65470","inReplyTo":"d2cf741c-a381-42a6-9d26-e38481696adb@gmail.com","subject":"Re: [GSoC PATCH v2] backfill: add --[no-]progress option","fromName":"Trieu Huynh","fromEmail":"vikingtc4@gmail.com","sentAt":"2026-04-13T19:02:43Z","receivedAt":"2026-04-13T19:02:49Z","isPatch":true,"body":"On Sun, Apr 12, 2026 at 03:46:17PM -0400, Derrick Stolee wrote:\n> On 4/12/26 3:36 PM, Trieu Huynh wrote:\n> > From: Trieu Huynh <vikingtc4@gmail.com>\n> > \n> > 'git backfill' does not show an overall progress bar across\n> > batches, giving no cross-batch feedback during potentially\n> > long-running operations on large repositories.  By contrast,\n> > 'git fetch', 'git gc', and 'git index-pack' all support\n> > --[no-]progress.\n> > \n> > Add a --[no-]progress option that tracks the total number of\n> > missing blobs downloaded across all batches, defaulting to\n> > showing progress when stderr is a terminal (matching the\n> > behaviour of 'git fetch').\n> > \n> > Add tests to verify that:\n> >   - progress is shown by default on a TTY\n> >   - --progress forces output regardless of TTY\n> >   - --no-progress suppresses output\n> \n> I think the tests do show an improvement, but we're missing\n> the interaction with the underlying fetch's progress\n> indicators. I don't see any mention of how your backfill\n> progress indicators will work with or against the fetch's\n> progress from the remote and index-pack steps.\nActually, I was missing adding it in the changelog, see below:\nAs-is:\nremote: Enumerating objects: 7391, done.\nremote: Counting objects: 100% (293/293), done.\nremote: Compressing objects: 100% (162/162), done.\nremote: Total 7391 (delta 249), reused 131 (delta 131), pack-reused 7098 (from 1)\nReceiving objects: 100% (7391/7391), 4.09 MiB | 10.20 MiB/s, done.\nResolving deltas: 100% (5617/5617), done.\n\nTo-be:\nremote: Enumerating objects: 7391, done.\nremote: Counting objects: 100% (293/293), done.\nremote: Compressing objects: 100% (162/162), done.\nremote: Total 7391 (delta 249), reused 131 (delta 131), pack-reused 7098 (from 1)\nReceiving objects: 100% (7391/7391), 4.09 MiB | 6.46 MiB/s, done.\nResolving deltas: 100% (5618/5618), done.\nDownloading missing blobs: 157594, done.\n> \n> Further, if a user supplies 'git backfill --no-progress'\n> then they are probably saying \"I don't want any progress\n> indicators\" and that would signal also that the fetch should\n> be quiet. This is perhaps the key detail that makes your\n> current version unable to move forward. It creates an\n> implication that it doesn't follow-through on.\nThank you for the point.\nYou are right that the current patch does not address the interaction\nbetween the backfill progress bar and the underlying fetch's own output\n(remote counting/compressing objects, index-pack, etc.). Leaving both\nactive at the same time would produce interleaved and confusing output,\nwhich is worse than no progress at all.\n> \n> One way to go about this is to hide the 'git fetch' output\n> entirely by passing '--quiet' unconditionally from the\n> backfill command. But this may also be too much for users\n> who want to watch the download statistics from the remote.\n> \n> Perhaps a way to have a robust set that allows all things\n> to interact is to do the following:\n> \n> 1. Add a --[no-]verbose option that is off by default. The\n>    implementation sends the --quiet flag to 'git fetch' if\n>    --verbose isn't provided from the user. This reduces the\n>    noise for the default user.\n> \n> 2. Add a --[no-]progress option as you've provided here.\nMake sense to me, will change to implement that way in v3.\n> \n> The complexity at the end is about what happens when the\n> user provides both --verbose and --progress, which is the\n> situation that this patch is currently in. How do the\n> progress indicators mingle with the verbose fetch output?\nIIUC, the fetch output for each batch completes before the progress\nbar updates, so they do not actually interleave. The\n\"Downloading missing blobs\" counter updates in place via carriage return\nduring the run, display until it's done partially, and only prints the\nfinal \"done.\" line at the end, for example:\n\n  remote: Enumerating objects: 50106, done.\n  remote: Counting objects: 100% (780/780), done.\n  ...\n  Receiving objects: 100% (50106/50106), done.\n  remote: Enumerating objects: 50096, done.\n  ...\n  Receiving objects: 100% (50096/50096), done.\n  Downloading missing blobs: 157594, done.\n\nSo --verbose and --progress together produce readable output without\nany special handling needed.\nDoes that direction sound reasonable to you?\n> \n> Thanks,\n> -Stolee\n> \n"},{"id":"541680","messageId":"21c10a52-82f4-4aa0-9027-21bb660b54cc@malon.dev","threadId":"65470","inReplyTo":"20260412193659.26288-1-viking4@gmail.com","subject":"Re: [GSoC PATCH v2] backfill: add --[no-]progress option","fromName":"Tian Yuchen","fromEmail":"cat@malon.dev","sentAt":"2026-04-15T17:04:58Z","receivedAt":"2026-04-15T17:05:14Z","isPatch":true,"body":"On 4/13/26 03:36, Trieu Huynh wrote:\n> @@ -133,6 +133,30 @@ test_expect_success 'do partial clone 2, backfill min batch size' '\n>   \ttest_line_count = 0 revs2\n>   '\n>   \n> +test_expect_success TTY 'backfill shows progress on tty by default' '\n> +\tgit clone --no-checkout --filter=blob:none \\ \n> +\t\t--single-branch --branch=main \\\n\n[1]\n\n> +\t\t\"file://$(pwd)/srv.bare\" clone-tty &&\n> +\ttest_terminal env GIT_PROGRESS_DELAY=0 git -C clone-tty backfill 2>err &&\n> +\ttest_grep \"Downloading missing blobs\" err\n> +'\n> +\n> +test_expect_success 'backfill --progress shows progress' '\n> +\tgit clone --no-checkout --filter=blob:none \\\n> +\t\t--single-branch --branch=main \\\n\n[1]\n\n> +\t\t\"file://$(pwd)/srv.bare\" clone-progress &&\n> +\tgit -C clone-progress backfill --progress 2>err &&\n> +\ttest_grep \"Downloading missing blobs\" err\n> +'\n> +\n> +test_expect_success 'backfill --no-progress suppresses progress' '\n> +\tgit clone --no-checkout --filter=blob:none \\\n> +\t\t--single-branch --branch=main \\\n\n[1]\n\n> +\t\t\"file://$(pwd)/srv.bare\" clone-no-progress &&\n> +\tgit -C clone-no-progress backfill --no-progress 2>err &&\n> +\ttest_grep ! \"Downloading missing blobs\" err\n\n[2]\n\n> +'\n> +\n>   test_expect_success 'backfill --sparse without sparse-checkout fails' '\n>   \tgit init not-sparse &&\n>   \ttest_must_fail git -C not-sparse backfill --sparse 2>err &&\n\n[1] I reckon you can reuse the git-cloned repository; there’s no need to \nclone in every test. It's up to you ;-)\n\n[2] You mentioned that you want test script to verify that \n'--no-progress suppresses output', but are you referring to the output \nbrought by the '--progress' parameter itself, or *all* output?\n\nI believe the second scenario is a bit more meaningful. If that is the \ncase, then the matching condition 'Downloading missing blobs' is clearly \na necessary but insufficient condition. The output from the internal \ncall to 'git fetch' within 'git backfill' will not be matched, which \nresults in a false negative.\n\nRegards, Yuchen\n\n\n"},{"id":"541689","messageId":"1988d824-0ff4-41a4-bc10-1b4e030878ef@gmail.com","threadId":"65470","inReplyTo":"wsbnw3am5fq6hpjwmmbguo2c3mnv4qkr3hh7apawch7smns6zx@rxegeuugbnhg","subject":"Re: [GSoC PATCH v2] backfill: add --[no-]progress option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2026-04-15T18:28:59Z","receivedAt":"2026-04-15T18:29:01Z","isPatch":true,"body":"On 4/13/2026 3:02 PM, Trieu Huynh wrote:\n> On Sun, Apr 12, 2026 at 03:46:17PM -0400, Derrick Stolee wrote:\n>> On 4/12/26 3:36 PM, Trieu Huynh wrote:\n>>> From: Trieu Huynh <vikingtc4@gmail.com>\n>>>\n>>> 'git backfill' does not show an overall progress bar across\n>>> batches, giving no cross-batch feedback during potentially\n>>> long-running operations on large repositories.  By contrast,\n>>> 'git fetch', 'git gc', and 'git index-pack' all support\n>>> --[no-]progress.\n>>>\n>>> Add a --[no-]progress option that tracks the total number of\n>>> missing blobs downloaded across all batches, defaulting to\n>>> showing progress when stderr is a terminal (matching the\n>>> behaviour of 'git fetch').\n>>>\n>>> Add tests to verify that:\n>>>   - progress is shown by default on a TTY\n>>>   - --progress forces output regardless of TTY\n>>>   - --no-progress suppresses output\n>>\n>> I think the tests do show an improvement, but we're missing\n>> the interaction with the underlying fetch's progress\n>> indicators. I don't see any mention of how your backfill\n>> progress indicators will work with or against the fetch's\n>> progress from the remote and index-pack steps.\n> Actually, I was missing adding it in the changelog, see below:\n> As-is:\n> remote: Enumerating objects: 7391, done.\n> remote: Counting objects: 100% (293/293), done.\n> remote: Compressing objects: 100% (162/162), done.\n> remote: Total 7391 (delta 249), reused 131 (delta 131), pack-reused 7098 (from 1)\n> Receiving objects: 100% (7391/7391), 4.09 MiB | 10.20 MiB/s, done.\n> Resolving deltas: 100% (5617/5617), done.\n> \n> To-be:\n> remote: Enumerating objects: 7391, done.\n> remote: Counting objects: 100% (293/293), done.\n> remote: Compressing objects: 100% (162/162), done.\n> remote: Total 7391 (delta 249), reused 131 (delta 131), pack-reused 7098 (from 1)\n> Receiving objects: 100% (7391/7391), 4.09 MiB | 6.46 MiB/s, done.\n> Resolving deltas: 100% (5618/5618), done.\n> Downloading missing blobs: 157594, done.\n\nThese examples are nice, but only for one batch of objects.\nYou'll need to test with a smaller batch size or a larger\nrepo to get the output I'm looking for.\n\n>> The complexity at the end is about what happens when the\n>> user provides both --verbose and --progress, which is the\n>> situation that this patch is currently in. How do the\n>> progress indicators mingle with the verbose fetch output?\n> IIUC, the fetch output for each batch completes before the progress\n> bar updates, so they do not actually interleave. The\n> \"Downloading missing blobs\" counter updates in place via carriage return\n> during the run, display until it's done partially, and only prints the\n> final \"done.\" line at the end, for example:\n> \n>   remote: Enumerating objects: 50106, done.\n>   remote: Counting objects: 100% (780/780), done.\n>   ...\n>   Receiving objects: 100% (50106/50106), done.\n>   remote: Enumerating objects: 50096, done.\n>   ...\n>   Receiving objects: 100% (50096/50096), done.\n>   Downloading missing blobs: 157594, done.\n> \n> So --verbose and --progress together produce readable output without\n> any special handling needed.\n> Does that direction sound reasonable to you?\n\nI think I'd like to see the full output for multiple batches,\nand then I can decide if the progress indicators make sense\ntogether or if they look confusing.\n\nThanks,\n-Stolee\n\n"}]}