git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: An endless loop fetching issue with partial clone, alternates and commit graph

From
HTHaiyng Tan <haiyangtand@gmail.com>
Date
Jun 14, 2022, 07:25 UTC
Message-ID
<CANe9W27GVn-w1WSZNTxh5SKEMzHGEZQCF48vmbvMi4AUEg12yQ@mail.gmail.com>
On Mon, 13 Jun 2022 00:17:07 +0800, Han Xin wrote:
Show 14 quoted lines
> We found an issue that could create an endless loop where alternates
> objects are used improperly.
>
> While do fetching in a partial cloned repository with a commit graph,
> deref_without_lazy_fetch_extended() will call lookup_commit_in_graph()
> to find the commit object. We can found the code in commit-graph.c:
>
>      struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id)
>      {
>           …
>           if (!search_commit_pos_in_graph(id, repo->objects->commit_graph, &pos))
>                return NULL;
>           if (!repo_has_object_file(repo, id))
>                return NULL;
Show 65 quoted lines
> If we found the object in the commit graph, but missing it in the repository,
> we will go into an endless loop:
>      git fetch -> deref_without_lazy_fetch_extended() ->
>           lookup_commit_in_graph() -> repo_has_object_file() ->
>                promisor_remote_get_direct() -> fetch_objects() ->
>                     git fetch
>
> I know that the reason for this issue is due to improper use of
> alternates, we can ensure that objects will not be lost by maintaining
> all the references. But shouldn't we do something about this unusual
> usage, it will cause a fetch bombardment of the remote git service.
>
> We can reproduce this issue with the following test case, it will
> generate a lot of git processes, please be careful to stop it.
> ———————————————————————————
> #!/bin/sh
>
> test_description='test for an endless loop fetching’
>
> GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
> export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
>
> . ./test-lib.sh
>
> test_expect_success 'setup’ ‘
>     git init --bare dest.git &&
>     test_commit one &&
>    git checkout -b testbranch &&
>    test_commit two &&
>    git push dest.git --all
> '
>
> test_expect_success 'prepare a alternates repository without testbranch' '
>    git clone -b $GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME dest.git alternates &&
>    oid=$(git -C alternates rev-parse refs/remotes/origin/testbranch) &&
>    git -C alternates update-ref -d refs/remotes/origin/testbranch &&
>    git -C alternates gc --prune=now
> '
>
> test_expect_success 'prepare a repository with commit-graph' '
>    git init source &&
>    echo "$(pwd)/dest.git/objects" >source/.git/objects/info/alternates &&
>    git -C source remote add origin "$(pwd)/dest.git" &&
>    git -C source config remote.origin.promisor true &&
>    git -C source config remote.origin.partialclonefilter blob:none &&
>    git -C source fetch origin &&
>    (
>        cd source &&
>        test_commit three &&
>        git -c gc.writeCommitGraph=true gc
>    )
> '
>
> test_expect_success 'change alternates' '
>    echo "$(pwd)/alternates/.git/objects" >source/.git/objects/info/alternates &&
>    # this will bring an endless loop fetching
>    git -C source fetch origin $oid
> '
>
> test_done
>
> ------------------------------------------------------
>
> Thanks
> -Han Xin

I think it's caused by using lazy-fetch in deref_without_lazy_fetch_extended(). In lookup_commit_in_graph(), lazy-fetch is initiated by repo_has_object_file() used. has_object() should be used, it's no-lazy-fetch.

Next: Taylor Blau
Message 1 of 50 in “Re: An endless loop fetching issue with partial clone, alternates and commit graph”
  1. Haiyng TanJun 14, 2022
  2. Taylor BlauJun 15, 2022
  3. 0/2 Re: An endless loop fetching issue with partial clone, alternates and commit graphHan Xin, Jun 16, 2022
  4. 1/2 commit-graph.c: add "flags" to lookup_commit_in_graph()Han Xin, Jun 16, 2022
  5. 2/2 fetch-pack.c: pass "oi_flags" to lookup_commit_in_graph()Han Xin, Jun 16, 2022
  6. Jonathan TanJun 17, 2022
  7. commit-graph.c: no lazy fetch in lookup_commit_in_graph()Han Xin, Jun 18, 2022
  8. Patrick SteinhardtJun 20, 2022
  9. 欣韩Jun 20, 2022
  10. Patrick SteinhardtJun 20, 2022
  11. Jonathan TanJun 21, 2022
  12. Han XinJun 22, 2022
  13. 0/2 commit-graph.c: no lazy fetch in lookup_commit_in_graph()Han Xin, Jun 24, 2022
  14. 1/2 test-lib.sh: add limited processes to test-libHan Xin, Jun 24, 2022
  15. Junio C HamanoJun 24, 2022
  16. Han XinJun 25, 2022
  17. Junio C HamanoJun 27, 2022
  18. 2/2 commit-graph.c: no lazy fetch in lookup_commit_in_graph()Han Xin, Jun 24, 2022
  19. Junio C HamanoJun 24, 2022
  20. Han XinJun 25, 2022
  21. Han XinJun 25, 2022
  22. 0/2 no lazy fetch in lookup_commit_in_graph()Han Xin, Jun 28, 2022
  23. 1/2 test-lib.sh: add limited processes to test-libHan Xin, Jun 28, 2022
  24. 2/2 commit-graph.c: no lazy fetch in lookup_commit_in_graph()Han Xin, Jun 28, 2022
  25. Ævar Arnfjörð BjarmasonJun 28, 2022
  26. Junio C HamanoJun 28, 2022
  27. Johannes SchindelinJun 30, 2022
  28. Ævar Arnfjörð BjarmasonJun 30, 2022
  29. Junio C HamanoJun 30, 2022
  30. Ævar Arnfjörð BjarmasonJun 30, 2022
  31. Johannes SchindelinJul 1, 2022
  32. Junio C HamanoJul 1, 2022
  33. Han XinJun 29, 2022
  34. test name conflict + js/ci-github-workflow-markup regression (was: [PATCH v3 0/2] no lazy fetch in lookup_commit_in_graph())Ævar Arnfjörð Bjarmason, Jun 30, 2022
  35. 0/1 no lazy fetch in lookup_commit_in_graph()Han Xin, Jul 1, 2022
  36. 1/1 commit-graph.c: no lazy fetch in lookup_commit_in_graph()Han Xin, Jul 1, 2022
  37. Michael J GruberJul 9, 2022
  38. Jeff KingJul 11, 2022
  39. Junio C HamanoJul 11, 2022
  40. Han XinJul 12, 2022
  41. Junio C HamanoJul 12, 2022
  42. Han XinJul 12, 2022
  43. Jeff KingJul 12, 2022
  44. Junio C HamanoJul 12, 2022
  45. 0/1 no lazy fetch in lookup_commit_in_graph()Han Xin, Jul 12, 2022
  46. 1/1 commit-graph.c: no lazy fetch in lookup_commit_in_graph()Han Xin, Jul 12, 2022
  47. Ævar Arnfjörð BjarmasonJul 12, 2022
  48. Han XinJul 13, 2022
  49. Jeff KingJul 12, 2022
  50. t5330: remove run_with_limited_processses()Han Xin, Jul 12, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.