git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 0/7] Lazy fetch with subprocess

From
Junio C Hamano <gitster@pobox.com>
Date
Aug 18, 2020, 19:56 UTC
Message-ID
<xmqqimdfybh5.fsf@gitster.c.googlers.com>
In-Reply-To
<cover.1597722941.git.jonathantanmy@google.com>
Jonathan Tan <jonathantanmy@google.com> writes:
> These patches are based on jc/no-update-fetch-head, once again.

FWIW, that topic no longer exists (it has been swallowed by Derrick's maintenance topics). As that topic is not near 'next' yet, we might want to kick the patch out of it into a separate jc/no-update-fetch-head topic again, base this series and also maintenance topics on top of it.

Show 33 quoted lines
> 2:  00ad7dd875 = 2:  9f277f1631 fetch: allow refspecs specified through stdin
> 3:  8b4a522a13 ! 3:  fda9f834f6 fetch: avoid reading submodule config until needed
>     @@ Metadata
>       ## Commit message ##
>          fetch: avoid reading submodule config until needed
>      
>     -    Teach "git fetch" to avoid reading the submodule config until necessary.
>     -    This allows users to avoid the lazy-fetching of this potentially missing
>     -    config file by specifying the --recurse-submodules=no command line
>     -    option.
>     +    In "fetch", there are two parameters submodule_fetch_jobs_config and
>     +    recurse_submodules that can be set in a variety of ways: through
>     +    .gitmodules, through .git/config, and through the command line.
>     +    Currently "fetch" handles this by first reading .gitmodules, then
>     +    reading .git/config (allowing it to overwrite existing values), then
>     +    reading the command line (allowing it to overwrite existing values).
>     +
>     +    Notice that we can avoid reading .gitmodules if .git/config and/or the
>     +    command line already provides us with what we need. In addition, if
>     +    recurse_submodules is found to be "no", we do not need the value of
>     +    submodule_fetch_jobs_config.
>     +
>     +    Avoiding reading .gitmodules is especially important when we use "git
>     +    fetch" to perform lazy fetches in a partial clone because the
>     +    .gitmodules file itself might need to be lazy fetched (and otherwise
>     +    causing an infinite loop).
>     +
>     +    In light of all this, avoid reading .gitmodules until necessary. When
>     +    reading it, we may only need one of the two parameters it provides, so
>     +    teach fetch_config_from_gitmodules() to support NULL arguments. With
>     +    this patch, users (including Git itself when invoking "git fetch" to
>     +    lazy-fetch) will be able to guarantee avoiding reading .gitmodules by
>     +    passing --recurse-submodules=no.
Quite sensible.
Show 12 quoted lines
> 4:  77bc83e7f2 ! 4:  a5554cd27f fetch: only populate existing_refs if needed
>     @@ Metadata
>       ## Commit message ##
>          fetch: only populate existing_refs if needed
>      
>     -    When fetching tags, Git only writes tags that do not already exist in
>     -    the client repository. This necessitates an iteration over all the refs,
>     -    but fetch performs this iteration even if no tags are fetched.
>     +    In "fetch", get_ref_map() iterates over all refs to populate
>     +    "existing_refs" in order to populate peer_ref->old_oid in the returned
>     +    refmap, even if the refmap has no peer_ref set - which is the case when
>     +    only literal hashes (i.e. no refs by name) are fetched.

Much better---the previous round gave us a wrong impression that the change is about the behaviour when fetching tags, but the updated explanation makes it clear that the primary use case is to avoid tag-following while directly fetching objects by names, not via refs.

Previous: Jonathan TanNext: Junio C Hamano
Message 54 of 57 in “Modify fetch-pack to no longer die on error?”
  1. Modify fetch-pack to no longer die on error?Jonathan Tan, Jul 24, 2020
  2. Junio C HamanoJul 24, 2020
  3. Junio C HamanoJul 24, 2020
  4. Jeff KingJul 25, 2020
  5. Junio C HamanoJul 25, 2020
  6. Jeff KingJul 27, 2020
  7. Jonathan TanJul 28, 2020
  8. Jeff KingJul 28, 2020
  9. Jonathan TanJul 29, 2020
  10. Jeff KingJul 29, 2020
  11. Junio C HamanoJul 29, 2020
  12. Jonathan TanJul 29, 2020
  13. 0/7 Lazy fetch with subprocessJonathan Tan, Aug 5, 2020
  14. 1/7 fetch-pack: allow NULL negotiator->add_tipJonathan Tan, Aug 5, 2020
  15. Junio C HamanoAug 5, 2020
  16. Jonathan TanAug 7, 2020
  17. 3/7 negotiator/null: add null fetch negotiatorJonathan Tan, Aug 5, 2020
  18. 2/7 fetch-pack: allow NULL negotiator->known_commonJonathan Tan, Aug 5, 2020
  19. Junio C HamanoAug 5, 2020
  20. Junio C HamanoAug 5, 2020
  21. Jonathan TanAug 7, 2020
  22. 4/7 fetch: --stdinJonathan Tan, Aug 5, 2020
  23. Junio C HamanoAug 5, 2020
  24. Jonathan TanAug 7, 2020
  25. Junio C HamanoAug 7, 2020
  26. Jonathan TanAug 7, 2020
  27. 5/7 fetch: submodule configJonathan Tan, Aug 5, 2020
  28. 7/7 promisor-remote: use subprocess to fetchJonathan Tan, Aug 5, 2020
  29. 6/7 fetch: only populate existing_refs if neededJonathan Tan, Aug 5, 2020
  30. 0/7 Lazy fetch with subprocessJonathan Tan, Aug 11, 2020
  31. 1/7 negotiator/null: add null fetch negotiatorJonathan Tan, Aug 11, 2020
  32. Derrick StoleeAug 12, 2020
  33. Junio C HamanoAug 12, 2020
  34. Jonathan TanAug 12, 2020
  35. 2/7 fetch: allow refspecs specified through stdinJonathan Tan, Aug 11, 2020
  36. 3/7 fetch: avoid reading submodule config until neededJonathan Tan, Aug 11, 2020
  37. Junio C HamanoAug 12, 2020
  38. 4/7 fetch: only populate existing_refs if neededJonathan Tan, Aug 11, 2020
  39. Junio C HamanoAug 12, 2020
  40. 5/7 fetch-pack: do not lazy-fetch during ref iterationJonathan Tan, Aug 11, 2020
  41. Junio C HamanoAug 12, 2020
  42. 6/7 promisor-remote: lazy-fetch objects in subprocessJonathan Tan, Aug 11, 2020
  43. Junio C HamanoAug 12, 2020
  44. 7/7 fetch-pack: remove no_dependents codeJonathan Tan, Aug 11, 2020
  45. Derrick StoleeAug 12, 2020
  46. 0/7 Lazy fetch with subprocessJonathan Tan, Aug 18, 2020
  47. 1/7 negotiator/noop: add noop fetch negotiatorJonathan Tan, Aug 18, 2020
  48. 2/7 fetch: allow refspecs specified through stdinJonathan Tan, Aug 18, 2020
  49. 4/7 fetch: only populate existing_refs if neededJonathan Tan, Aug 18, 2020
  50. 5/7 fetch-pack: do not lazy-fetch during ref iterationJonathan Tan, Aug 18, 2020
  51. 3/7 fetch: avoid reading submodule config until neededJonathan Tan, Aug 18, 2020
  52. 6/7 promisor-remote: lazy-fetch objects in subprocessJonathan Tan, Aug 18, 2020
  53. 7/7 fetch-pack: remove no_dependents codeJonathan Tan, Aug 18, 2020
  54. Junio C HamanoAug 18, 2020
  55. Junio C HamanoAug 18, 2020
  56. fixup! promisor-remote: lazy-fetch objects in subprocessJonathan Tan, Aug 18, 2020
  57. Junio C HamanoAug 18, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.