git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 1/8] maintenance: add prefetch task

From
JTJonathan Tan <jonathantanmy@google.com>
Date
Sep 22, 2020, 23:05 UTC
Message-ID
<20200922230535.903375-1-jonathantanmy@google.com>
In-Reply-To
<da64c51a8182ec13aeed8f0157079fb29a09ee85.1598380599.git.gitgitgadget@gmail.com>
Show 17 quoted lines
> From: Derrick Stolee <dstolee@microsoft.com>
> 
> When working with very large repositories, an incremental 'git fetch'
> command can download a large amount of data. If there are many other
> users pushing to a common repo, then this data can rival the initial
> pack-file size of a 'git clone' of a medium-size repo.
> 
> Users may want to keep the data on their local repos as close as
> possible to the data on the remote repos by fetching periodically in
> the background. This can break up a large daily fetch into several
> smaller hourly fetches.
> 
> The task is called "prefetch" because it is work done in advance
> of a foreground fetch to make that 'git fetch' command much faster.
> 
> However, if we simply ran 'git fetch <remote>' in the background,
> then the user running a foregroudn 'git fetch <remote>' would lose
-> foreground

I have some more minor comments that I will send as individual replies, but overall, the patch set looks good to me.

Show 27 quoted lines
> +static int append_remote(struct remote *remote, void *cbdata)
> +{
> +	struct string_list *remotes = (struct string_list *)cbdata;
> +
> +	string_list_append(remotes, remote->name);
> +	return 0;
> +}
> +
> +static int maintenance_task_prefetch(struct maintenance_run_opts *opts)
> +{
> +	int result = 0;
> +	struct string_list_item *item;
> +	struct string_list remotes = STRING_LIST_INIT_DUP;
> +
> +	if (for_each_remote(append_remote, &remotes)) {
> +		error(_("failed to fill remotes"));
> +		result = 1;
> +		goto cleanup;
> +	}
> +
> +	for_each_string_list_item(item, &remotes)
> +		result |= fetch_remote(item->string, opts);
> +
> +cleanup:
> +	string_list_clear(&remotes, 0);
> +	return result;
> +}

I was wondering why the generation of the list and the iteration was split up, but I see that you want to attempt to fetch each remote even if one of them fails.

Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee via GitGitGadget
Message 35 of 66 in “Maintenance II: prefetch, loose-objects, incremental-repack tasks”
  1. 0/9 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 6, 2020
  2. 8/9 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 6, 2020
  3. Son Luong NgocAug 6, 2020
  4. Derrick StoleeAug 6, 2020
  5. 6/9 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 6, 2020
  6. 9/9 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 6, 2020
  7. 7/9 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  8. 5/9 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 6, 2020
  9. 4/9 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 6, 2020
  10. 3/9 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  11. Emily ShafferAug 12, 2020
  12. Derrick StoleeAug 14, 2020
  13. 2/9 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  14. Emily ShafferAug 12, 2020
  15. Derrick StoleeAug 14, 2020
  16. 1/9 fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano via GitGitGadget, Aug 6, 2020
  17. Emily ShafferAug 12, 2020
  18. Junio C HamanoAug 13, 2020
  19. Jonathan NiederAug 13, 2020
  20. fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano, Aug 13, 2020
  21. Derrick StoleeAug 14, 2020
  22. Junio C HamanoAug 14, 2020
  23. 0/9 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 18, 2020
  24. 1/9 fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano via GitGitGadget, Aug 18, 2020
  25. 3/9 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  26. 5/9 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 18, 2020
  27. 2/9 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  28. 8/9 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 18, 2020
  29. 9/9 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 18, 2020
  30. 7/9 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  31. 6/9 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 18, 2020
  32. 4/9 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 18, 2020
  33. 0/8 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 25, 2020
  34. 1/8 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  35. Jonathan TanSep 22, 2020
  36. 2/8 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  37. Jonathan TanSep 22, 2020
  38. Derrick StoleeSep 24, 2020
  39. 3/8 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 25, 2020
  40. Jonathan TanSep 22, 2020
  41. Derrick StoleeSep 24, 2020
  42. 4/8 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 25, 2020
  43. Jonathan TanSep 22, 2020
  44. Derrick StoleeSep 24, 2020
  45. 5/8 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 25, 2020
  46. 6/8 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  47. Jonathan TanSep 22, 2020
  48. Derrick StoleeSep 24, 2020
  49. Jonathan TanSep 24, 2020
  50. 8/8 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 25, 2020
  51. Jonathan TanSep 22, 2020
  52. 7/8 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 25, 2020
  53. Junio C HamanoAug 25, 2020
  54. Son Luong NgocAug 26, 2020
  55. Derrick StoleeAug 26, 2020
  56. 0/8 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Sep 25, 2020
  57. 1/8 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Sep 25, 2020
  58. 3/8 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Sep 25, 2020
  59. Junio C HamanoSep 25, 2020
  60. Derrick StoleeSep 25, 2020
  61. 4/8 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Sep 25, 2020
  62. 2/8 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Sep 25, 2020
  63. 5/8 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Sep 25, 2020
  64. 7/8 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Sep 25, 2020
  65. 8/8 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Sep 25, 2020
  66. 6/8 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Sep 25, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.