git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 9/9] bundle-uri: fetch a list of bundles

From
Derrick Stolee <derrickstolee@github.com>
Date
Sep 30, 2022, 12:49 UTC
Message-ID
<a91f2868-d65b-3e24-cf18-b483b7807c29@github.com>
In-Reply-To
<20220929215808.1438418-1-jonathantanmy@google.com>
On 9/29/2022 5:58 PM, Jonathan Tan wrote:
Show 41 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>> +static int download_bundle_to_file(struct remote_bundle_info *bundle, void *data)
>> +{
>> +	int res;
>> +	struct bundle_list_context *ctx = data;
>> +
>> +	if (ctx->mode == BUNDLE_MODE_ANY && ctx->count)
>> +		return 0;
>> +
>> +	res = fetch_bundle_uri_internal(ctx->r, bundle, ctx->depth + 1, ctx->list);
>> +
>> +	/*
>> +	 * Only increment count if the download succeeded. If our mode is
>> +	 * BUNDLE_MODE_ANY, then we will want to try other URIs in the
>> +	 * list in case they work instead.
>> +	 */
>> +	if (!res)
>> +		ctx->count++;
>> +	return res;
>> +}
> 
> So this returns nonzero if a download fails...
> 
>> +static int download_bundle_list(struct repository *r,
>> +				struct bundle_list *local_list,
>> +				struct bundle_list *global_list,
>> +				int depth)
>> +{
>> +	struct bundle_list_context ctx = {
>> +		.r = r,
>> +		.list = global_list,
>> +		.depth = depth + 1,
>> +		.mode = local_list->mode,
>> +	};
>> +
>> +	return for_all_bundles_in_list(local_list, download_bundle_to_file, &ctx);
>> +}
> 
> ...and for_all_bundles_in_list does not proceed with the rest of the
> loop if any callback invocation returns nonzero. Don't we need to
> continue retrying the others if the mode is ANY?
You are right! Thanks.
 
Show 17 quoted lines
>> +static int attempt_unbundle(struct remote_bundle_info *info, void *data)
>> +{
>> +	struct attempt_unbundle_context *ctx = data;
>> +
>> +	if (info->unbundled || !unbundle_from_file(ctx->r, info->file)) {
>> +		ctx->success_count++;
>> +		info->unbundled = 1;
>> +	} else {
>> +		ctx->failure_count++;
>> +	}
>> +
>> +	return 0;
>> +}
> 
> Do we need to handle the case in which a file is missing but it's
> expected because the mode is ANY and another file was successfully
> downloaded?

By "file is missing" I think you mean "we never successfully downloaded that file" and I agree that we should skip those bundles. I'll add more tests for ANY mode to hopefully catch these issues.

Show 26 quoted lines
>> +static int unbundle_all_bundles(struct repository *r,
>> +				struct bundle_list *list)
>> +{
>> +	int last_success_count = -1;
>> +	struct attempt_unbundle_context ctx = {
>> +		.r = r,
>> +	};
>> +
>> +	/*
>> +	 * Iterate through all bundles looking for ones that can
>> +	 * successfully unbundle. If any succeed, then perhaps another
>> +	 * will succeed in the next attempt.
>> +	 */
>> +	while (last_success_count < ctx.success_count) {
>> +		last_success_count = ctx.success_count;
>> +
>> +		ctx.success_count = 0;
>> +		ctx.failure_count = 0;
>> +		for_all_bundles_in_list(list, attempt_unbundle, &ctx);
> 
> I think it would have been clearer if the invocation to
> for_all_bundles_in_list were to stop early if a bundle has been
> successfully unbundled, and then you can just run this loop n times,
> instead of needing to reset the success count each time in order to
> check that the latest count is more than the prior one. But this works
> too.

It's a little bit backwards to have the "terminate early with nonzero value" signal "success", but it would work. With careful commenting, I think it's doable.

> I see that there are ALL tests, but could we have an ANY test as well?
Yes, excellent point. They are absolutely necessary.

Thanks, -Stolee

Previous: Jonathan TanNext: Derrick Stolee
Message 43 of 94 in “Bundle URIs III: Parse and download from bundle lists”
  1. 0/7 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Aug 22, 2022
  2. 1/7 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Aug 22, 2022
  3. Junio C HamanoAug 22, 2022
  4. 3/7 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Aug 22, 2022
  5. Junio C HamanoAug 22, 2022
  6. Derrick StoleeAug 23, 2022
  7. Josh SteadmonSep 2, 2022
  8. 6/7 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Aug 22, 2022
  9. 4/7 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Aug 22, 2022
  10. 4/7 bundle-uri: unit test "key=value" parsingTeng Long, Sep 1, 2022
  11. 5/7 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Aug 22, 2022
  12. Junio C HamanoAug 22, 2022
  13. Derrick StoleeAug 23, 2022
  14. Jonathan TanAug 31, 2022
  15. 5/7 bundle-uri: parse bundle list in config formatTeng Long, Sep 1, 2022
  16. 2/7 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Aug 22, 2022
  17. Junio C HamanoAug 22, 2022
  18. Derrick StoleeAug 23, 2022
  19. Jonathan TanAug 31, 2022
  20. Glen ChooAug 31, 2022
  21. 4/7 bundle-uri: unit test "key=value" parsingTeng Long, Sep 1, 2022
  22. 7/7 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Aug 22, 2022
  23. Josh SteadmonSep 2, 2022
  24. 7/7 bundle-uri: fetch a list of bundlesTeng Long, Sep 5, 2022
  25. Derrick StoleeSep 8, 2022
  26. 0/9 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Sep 9, 2022
  27. 1/9 bundle-uri: short-circuit capability parsingDerrick Stolee via GitGitGadget, Sep 9, 2022
  28. Junio C HamanoSep 9, 2022
  29. Derrick StoleeSep 19, 2022
  30. 2/9 bundle-uri: use plain string in find_temp_filename()Derrick Stolee via GitGitGadget, Sep 9, 2022
  31. Junio C HamanoSep 9, 2022
  32. Derrick StoleeSep 19, 2022
  33. Junio C HamanoSep 19, 2022
  34. 4/9 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Sep 9, 2022
  35. Jonathan TanSep 29, 2022
  36. 3/9 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Sep 9, 2022
  37. 5/9 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Sep 9, 2022
  38. 6/9 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Sep 9, 2022
  39. 7/9 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Sep 9, 2022
  40. 8/9 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Sep 9, 2022
  41. 9/9 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Sep 9, 2022
  42. Jonathan TanSep 29, 2022
  43. Derrick StoleeSep 30, 2022
  44. Derrick StoleeSep 26, 2022
  45. Junio C HamanoSep 26, 2022
  46. Jonathan TanSep 29, 2022
  47. Derrick StoleeSep 30, 2022
  48. 0/9 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Oct 4, 2022
  49. 1/9 bundle-uri: use plain string in find_temp_filename()Derrick Stolee via GitGitGadget, Oct 4, 2022
  50. 2/9 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Oct 4, 2022
  51. 4/9 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 4, 2022
  52. 3/9 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Oct 4, 2022
  53. 5/9 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 4, 2022
  54. 6/9 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Oct 4, 2022
  55. 9/9 bundle-uri: suppress stderr from remote-httpsDerrick Stolee via GitGitGadget, Oct 4, 2022
  56. 7/9 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Oct 4, 2022
  57. 8/9 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Oct 4, 2022
  58. Jonathan TanOct 4, 2022
  59. Derrick StoleeOct 7, 2022
  60. 00/11 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Oct 10, 2022
  61. 02/11 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Oct 10, 2022
  62. 01/11 bundle-uri: use plain string in find_temp_filename()Derrick Stolee via GitGitGadget, Oct 10, 2022
  63. 03/11 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Oct 10, 2022
  64. 04/11 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 10, 2022
  65. 05/11 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 10, 2022
  66. 06/11 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Oct 10, 2022
  67. 07/11 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Oct 10, 2022
  68. 08/11 bundle: add flags to verify_bundle(), skip walkDerrick Stolee via GitGitGadget, Oct 10, 2022
  69. Junio C HamanoOct 10, 2022
  70. Derrick StoleeOct 10, 2022
  71. Junio C HamanoOct 10, 2022
  72. Derrick StoleeOct 11, 2022
  73. 09/11 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Oct 10, 2022
  74. 10/11 bundle-uri: quiet failed unbundlingsDerrick Stolee via GitGitGadget, Oct 10, 2022
  75. 11/11 bundle-uri: suppress stderr from remote-httpsDerrick Stolee via GitGitGadget, Oct 10, 2022
  76. 00/12 Bundle URIs III: Parse and download from bundle listsDerrick Stolee via GitGitGadget, Oct 12, 2022
  77. 01/12 bundle-uri: use plain string in find_temp_filename()Derrick Stolee via GitGitGadget, Oct 12, 2022
  78. 02/12 bundle-uri: create bundle_list struct and helpersDerrick Stolee via GitGitGadget, Oct 12, 2022
  79. 03/12 bundle-uri: create base key-value pair parsingDerrick Stolee via GitGitGadget, Oct 12, 2022
  80. 05/12 bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 12, 2022
  81. 04/12 bundle-uri: create "key=value" line parsingÆvar Arnfjörð Bjarmason via GitGitGadget, Oct 12, 2022
  82. 06/12 bundle-uri: parse bundle list in config formatDerrick Stolee via GitGitGadget, Oct 12, 2022
  83. 07/12 bundle-uri: limit recursion depth for bundle listsDerrick Stolee via GitGitGadget, Oct 12, 2022
  84. 10/12 bundle: add flags to verify_bundle()Derrick Stolee via GitGitGadget, Oct 12, 2022
  85. 08/12 bundle: properly clear all revision flagsDerrick Stolee via GitGitGadget, Oct 12, 2022
  86. Junio C HamanoOct 12, 2022
  87. 09/12 bundle-uri: fetch a list of bundlesDerrick Stolee via GitGitGadget, Oct 12, 2022
  88. Junio C HamanoOct 26, 2022
  89. 12/12 bundle-uri: suppress stderr from remote-httpsDerrick Stolee via GitGitGadget, Oct 12, 2022
  90. Junio C HamanoOct 26, 2022
  91. 11/12 bundle-uri: quiet failed unbundlingsDerrick Stolee via GitGitGadget, Oct 12, 2022
  92. Junio C HamanoOct 12, 2022
  93. Derrick StoleeOct 26, 2022
  94. Junio C HamanoOct 26, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.