git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/3] rev-list: add support for commits in `--missing`

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 13, 2023, 05:53 UTC
Message-ID
<ZSjbYCXfSUtEIkAt@tanuki>
In-Reply-To
<xmqqy1g7hl2y.fsf@gitster.g>
On Thu, Oct 12, 2023 at 09:17:09AM -0700, Junio C Hamano wrote:
Show 39 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Wouldn't this have the potential to significantly regress performance
> > for all those preexisting users of the `--missing` option? The commit
> > graph is quite an important optimization nowadays, and especially in
> > commands where we potentially walk a lot of commits (like we may do
> > here) it can result in queries that are orders of magnitudes faster.
> 
> The test fails only when GIT_TEST_COMMIT_GRAPH is on, which updates
> the commit-graph every time a commit is made via "git commit" or
> "git merge".
> 
> I'd suggest stepping back and think a bit.
> 
> My assumption has been that the failing test emulates this scenario
> that can happen in real life:
> 
>  * The user creates a new commit.
> 
>  * A commit graph is written (not as part of GIT_TEST_COMMIT_GRAPH
>    that is not realistic, but as part of "maintenance").
> 
>  * The repository loses some objects due to corruption.
> 
>  * Now, "--missing=print" is invoked so that the user can view what
>    are missing.  Or "--missing=allow-primisor" to ensure that the
>    repository does not have missing objects other than the ones that
>    the promisor will give us if we asked again.
> 
>  * But because the connectivity of these objects appear in the
>    commit graph file, we fail to notice that these objects are
>    missing, producing wrong results.  If we disabled commit-graph
>    while traversal (an earlier writing of it was perfectly OK), then
>    "rev-list --missing" would have noticed and reported what the
>    user wanted to know.
> 
> In other words, the "optimization" you value is working to quickly
> produce a wrong result.  Is it "significantly regress"ing if we
> disabled it to obtain the correct result?
It depends, in my opinion. If:
    - Wrong results caused by the commit graph are only introduced with
      this patch series due to the changed behaviour of `--missing`.
    - We disable commit graphs proactively only because of the changed
      behaviour of `--missing`.

Then yes, it does feel wrong to me to disable commit graphs and regress performance for usecases that perviously worked both correct and fast.

Show 11 quoted lines
> My assumption also has been that there is no point in running
> "rev-list --missing" if we know there is no repository corruption,
> and those who run "rev-list --missing" wants to know if the objects
> are really available, i.e. even if commit-graph that is out of sync
> with reality says it exists, if it is not in the object store, they
> would want to know that.
> 
> If you can show me that it is not the case, then I may be pursuaded
> why producing a result that is out of sync with reality _quickly_,
> instead of taking time to produce a result that matches reality, is
> a worthy "optimization" to keep.

Note that I'm not saying that it's fine to return wrong results -- this is of course a bug that needs to be addressed somehow. After all, things working correctly should always trump things working fast. But until now it felt more like we were going into the direction of disabling commit graphs without checking whether there is an alternative solution that allows us to get the best of both worlds, correctness and performance.

So what I'm looking for in this thread is a reason why we _can't_ have that, or at least can't have it without unreasonable amounts of work. We have helpers like `lookup_commit_in_graph()` that are designed to detect stale commit graphs by double-checking whether a commit that has been looked up via the commit graph actually exists in the repository. So I'm wondering whether this could help us address the issue.

If there is a good reason why all of that is not possible then I'm happy to carve in.

Patrick
Previous: Junio C HamanoNext: Patrick Steinhardt
Message 14 of 61 in “rev-list: add support for commits in `--missing`”
  1. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 9, 2023
  2. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 9, 2023
  3. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 9, 2023
  4. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 9, 2023
  5. Junio C HamanoOct 9, 2023
  6. Patrick SteinhardtOct 10, 2023
  7. Junio C HamanoOct 10, 2023
  8. Karthik NayakOct 11, 2023
  9. Junio C HamanoOct 11, 2023
  10. Karthik NayakOct 12, 2023
  11. Patrick SteinhardtOct 12, 2023
  12. Karthik NayakOct 12, 2023
  13. Junio C HamanoOct 12, 2023
  14. Patrick SteinhardtOct 13, 2023
  15. Patrick SteinhardtOct 13, 2023
  16. commit: detect commits that exist in commit-graph but not in the ODBPatrick Steinhardt, Oct 13, 2023
  17. Junio C HamanoOct 13, 2023
  18. Patrick SteinhardtOct 17, 2023
  19. Junio C HamanoOct 17, 2023
  20. Patrick SteinhardtOct 19, 2023
  21. Patrick SteinhardtOct 19, 2023
  22. Junio C HamanoOct 19, 2023
  23. Jeff KingOct 20, 2023
  24. Junio C HamanoOct 20, 2023
  25. Patrick SteinhardtOct 23, 2023
  26. Junio C HamanoOct 13, 2023
  27. Junio C HamanoOct 12, 2023
  28. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 16, 2023
  29. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 16, 2023
  30. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 16, 2023
  31. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 16, 2023
  32. Junio C HamanoOct 16, 2023
  33. Karthik NayakOct 16, 2023
  34. Junio C HamanoOct 16, 2023
  35. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 19, 2023
  36. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 19, 2023
  37. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 19, 2023
  38. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 19, 2023
  39. Junio C HamanoOct 19, 2023
  40. Junio C HamanoOct 19, 2023
  41. Karthik NayakOct 20, 2023
  42. Karthik NayakOct 20, 2023
  43. Junio C HamanoOct 20, 2023
  44. Junio C HamanoOct 20, 2023
  45. Karthik NayakOct 24, 2023
  46. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 24, 2023
  47. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 24, 2023
  48. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 24, 2023
  49. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 24, 2023
  50. Junio C HamanoOct 24, 2023
  51. Junio C HamanoOct 25, 2023
  52. Karthik NayakOct 25, 2023
  53. Patrick SteinhardtOct 25, 2023
  54. Junio C HamanoOct 26, 2023
  55. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 26, 2023
  56. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 26, 2023
  57. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 26, 2023
  58. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 26, 2023
  59. Patrick SteinhardtOct 27, 2023
  60. Karthik NayakOct 27, 2023
  61. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 27, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.