git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 2/3] builtin/fetch: skip unnecessary tasks when using --negotiate-only

From
Glen Choo <chooglen@google.com>
Date
Dec 22, 2021, 20:27 UTC
Message-ID
<kl6l4k70z8po.fsf@chooglen-macbookpro.roam.corp.google.com>
In-Reply-To
<xmqqpmpojv48.fsf@gitster.g>
Junio C Hamano <gitster@pobox.com> writes:
Show 58 quoted lines
> Glen Choo <chooglen@google.com> writes:
>
>> I would have come to same conclusion if I agreed that we should recurse
>> into submodules even if no objects are fetched. When I first wrote this
>> patch, I was convinced that "no new objects" implies "no need to update
>> submodules" (see my response at [1]), but I'm not sure any more and I'd
>> like to check my understanding.
>
> For example, there is a "everything_local()" optimization in the
> fetch-pack codepath where the following steps happen:
>
>  (1) we look at the current value of the remote refs we are fetching
>      from their ls_refs output (let's call it "new tips");
>
>  (2) we notice that all of these objects happen to exist in our
>      object store.
>
>  (3) we make sure that we do not see any "missing links" if we run a
>      reachability traversal that starts from these objects and our
>      existing refs, and stops when the traversal intersect.
>
> When the last step finds that all objects necessary to point at
> these "new tips" with our refs safely, then we have no reason to
> perform physical transfer of objects.  Yet, we'd update our refs
> to the "new tips".
>
> This can happen in a number of ways.  
>
> Imagine that you have a clone of https://github.com/git/git/ for
> only its 'main' branch (i.e. a single-branch clone).  If you then
> say "git fetch origin maint:maint", we'll learn that the tip of
> their 'maint' branch points at a commit, we look into our object
> store, find that there is no missing object to reach from it to the
> part of the object graph that is reachable from our refs (i.e. my
> 'maint' is always an ancestor of my 'main'), and we find that there
> is no reason to transfer any object.  Yet we will carete a new ref
> and point at the commit.
>
> Or if you did "git branch -d" locally, making objects unreachable in
> your object store, and then fetch from your upstream, which had fast
> forwarded to the contents of the branch you just deleted.
>
> Or they rewound and rebuilt their branches since you fetched the
> last time, and then they realized their mistake and now their refs
> point at a commit that you have already seen but are different from
> what your remote-tracking branches point at now.
>
> Or you are using Derrick's "prefetch" (in "git maintenance run") and
> a background process already downloaded the objects needed for the
> branch you are fetching in the past.
>
> Depending on what happened when these objects were pre-fetched, such
> a real fetch that did not have to perform an object transfer may
> likely to need to adjust things in the submodule repository.
> "prefetch" is designed not to disrupt and to be invisible to the
> normal operation as much as possible, so I would expect that it
> won't do any priming of the submodules based on what it prefetched
> for the superproject, for example.

Thanks for providing concrete examples; that's a lot more convincing than my hypothetical.

> So in short, physical object transfer can be optimized out, even
> when the external world view, i.e. where in the history graph the
> refs point at, changes and makes it necessary to check in the
> submodule repositories.
Yes, you're right. I'll move the jump accordingly :)
Previous: Junio C HamanoNext: Glen Choo
Message 20 of 50 in “builtin/fetch: skip unnecessary tasks when using --negotiate-only”
  1. builtin/fetch: skip unnecessary tasks when using --negotiate-onlyGlen Choo, Dec 7, 2021
  2. Jonathan TanDec 9, 2021
  3. Glen ChooDec 9, 2021
  4. Jonathan TanDec 13, 2021
  5. Glen ChooDec 16, 2021
  6. builtin/fetch: skip unnecessary tasks when using --negotiate-onlyGlen Choo, Dec 17, 2021
  7. Junio C HamanoDec 17, 2021
  8. Glen ChooDec 20, 2021
  9. Junio C HamanoDec 20, 2021
  10. Glen ChooDec 20, 2021
  11. Junio C HamanoDec 20, 2021
  12. Glen ChooDec 21, 2021
  13. Glen ChooDec 21, 2021
  14. 0/3 builtin/fetch: skip unnecessary tasks when using --negotiate-onlyGlen Choo, Dec 22, 2021
  15. 1/3 builtin/fetch: use goto cleanup in cmd_fetch()Glen Choo, Dec 22, 2021
  16. 2/3 builtin/fetch: skip unnecessary tasks when using --negotiate-onlyGlen Choo, Dec 22, 2021
  17. Junio C HamanoDec 22, 2021
  18. Glen ChooDec 22, 2021
  19. Junio C HamanoDec 22, 2021
  20. Glen ChooDec 22, 2021
  21. 3/3 builtin/fetch: die on --negotiate-only and --recurse-submodulesGlen Choo, Dec 22, 2021
  22. Junio C HamanoDec 22, 2021
  23. Jonathan TanDec 23, 2021
  24. 0/3 fetch: skip unnecessary tasks when using --negotiate-onlyGlen Choo, Jan 13, 2022
  25. 1/3 fetch: use goto cleanup in cmd_fetch()Glen Choo, Jan 13, 2022
  26. 2/3 fetch: skip tasks related to fetching objectsGlen Choo, Jan 13, 2022
  27. 3/3 fetch --negotiate-only: do not update submodulesGlen Choo, Jan 13, 2022
  28. Junio C HamanoJan 13, 2022
  29. 0/3 fetch: skip unnecessary tasks when using --negotiate-onlyGlen Choo, Jan 18, 2022
  30. 1/3 fetch: use goto cleanup in cmd_fetch()Glen Choo, Jan 18, 2022
  31. 2/3 fetch: skip tasks related to fetching objectsGlen Choo, Jan 18, 2022
  32. 3/3 fetch --negotiate-only: do not update submodulesGlen Choo, Jan 18, 2022
  33. Junio C HamanoJan 18, 2022
  34. Glen ChooJan 18, 2022
  35. Junio C HamanoJan 19, 2022
  36. 0/3 fetch: skip unnecessary tasks when using --negotiate-onlyGlen Choo, Jan 19, 2022
  37. 1/3 fetch: use goto cleanup in cmd_fetch()Glen Choo, Jan 19, 2022
  38. 2/3 fetch: skip tasks related to fetching objectsGlen Choo, Jan 19, 2022
  39. 3/3 fetch --negotiate-only: do not update submodulesGlen Choo, Jan 19, 2022
  40. Jiang XinJan 20, 2022
  41. Glen ChooJan 20, 2022
  42. 0/3 fetch: skip unnecessary tasks when using --negotiate-onlyGlen Choo, Jan 20, 2022
  43. 1/3 fetch: use goto cleanup in cmd_fetch()Glen Choo, Jan 20, 2022
  44. 2/3 fetch: skip tasks related to fetching objectsGlen Choo, Jan 20, 2022
  45. 3/3 fetch --negotiate-only: do not update submodulesGlen Choo, Jan 20, 2022
  46. Junio C HamanoJan 20, 2022
  47. Glen ChooJan 20, 2022
  48. Re* [PATCH v7 0/3] fetch: skip unnecessary tasks when using --negotiate-onlyJunio C Hamano, Jan 20, 2022
  49. Glen ChooJan 20, 2022
  50. Jiang XinJan 21, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.