git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] commit: detect commits that exist in commit-graph but not in the ODB

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 23, 2023, 10:15 UTC
Message-ID
<ZTZHxh4v0_fKkZpg@tanuki>
In-Reply-To
<20231020100024.GA2194074@coredump.intra.peff.net>
On Fri, Oct 20, 2023 at 06:00:24AM -0400, Jeff King wrote:
Show 34 quoted lines
> On Thu, Oct 19, 2023 at 10:16:56AM -0700, Junio C Hamano wrote:
> 
> > Patrick Steinhardt <ps@pks.im> writes:
> > 
> > > There's another way to handle this, which is to conditionally enable the
> > > object existence check. This would be less of a performance hit compared
> > > to disabling commit graphs altogether with `--missing`, but still ensure
> > > that repository corruption was detected. Second, it would not regress
> > > performance for all preexisting users of `repo_parse_commit_gently()`.
> > 
> > The above was what I meant to suggest when you demonstrated that the
> > code with additional check is still much more performant than
> > running without the commit-graph optimization, yet has observable
> > impact on performance for normal codepaths that do not need the
> > extra carefulness.
> > 
> > But I wasn't sure if it is easy to plumb the "do we want to double
> > check?  in other words, are we running something like --missing that
> > care the correctness a bit more than usual cases?" bit down from the
> > caller, because this check is so deep in the callchain.
> 
> I wonder if we would want a "be extra careful" flag that is read from
> the environment? That is largely how GIT_REF_PARANOIA works, and then
> particular operations set it (though actually it is the default these
> days, so they no longer do so explicitly).
> 
> I guess that is really like a global variable but even more gross. ;)
> But it is nice that it can cross process boundaries, because "how
> careful do we want to be" may be in the eye of the caller, especially
> for plumbing commands. E.g., even without --missing, you may want
> "rev-list" to be extra careful about checking the existence of commits.
> The caller in check_connected() could arguably turn that on by default
> (the way we used to turn on GIT_REF_PARANOIA for pruning repacks before
> it became the default).

Good point indeed, I also started to ask myself whether we'd want to have a config option. An environment variable `GIT_OBJECT_PARANOIA` would work equally well though and be less "official".

What I like about this idea is that it would also allow us to easily unify the behaviour of `lookup_commit_in_graph()` and the new sanity check in `repo_parse_commit_gently()` that I'm introducing here. I'd personally think that the default for any such toggle should be `true`, also to keep the current behaviour of `lookup_commit_in_graph()`. But changing it would be easy enough.

I'll send a v2 of this series with such a toggle.
Patrick
Previous: Junio C HamanoNext: Junio C Hamano
Message 25 of 61 in “rev-list: add support for commits in `--missing`”
  1. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 9, 2023
  2. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 9, 2023
  3. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 9, 2023
  4. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 9, 2023
  5. Junio C HamanoOct 9, 2023
  6. Patrick SteinhardtOct 10, 2023
  7. Junio C HamanoOct 10, 2023
  8. Karthik NayakOct 11, 2023
  9. Junio C HamanoOct 11, 2023
  10. Karthik NayakOct 12, 2023
  11. Patrick SteinhardtOct 12, 2023
  12. Karthik NayakOct 12, 2023
  13. Junio C HamanoOct 12, 2023
  14. Patrick SteinhardtOct 13, 2023
  15. Patrick SteinhardtOct 13, 2023
  16. commit: detect commits that exist in commit-graph but not in the ODBPatrick Steinhardt, Oct 13, 2023
  17. Junio C HamanoOct 13, 2023
  18. Patrick SteinhardtOct 17, 2023
  19. Junio C HamanoOct 17, 2023
  20. Patrick SteinhardtOct 19, 2023
  21. Patrick SteinhardtOct 19, 2023
  22. Junio C HamanoOct 19, 2023
  23. Jeff KingOct 20, 2023
  24. Junio C HamanoOct 20, 2023
  25. Patrick SteinhardtOct 23, 2023
  26. Junio C HamanoOct 13, 2023
  27. Junio C HamanoOct 12, 2023
  28. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 16, 2023
  29. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 16, 2023
  30. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 16, 2023
  31. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 16, 2023
  32. Junio C HamanoOct 16, 2023
  33. Karthik NayakOct 16, 2023
  34. Junio C HamanoOct 16, 2023
  35. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 19, 2023
  36. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 19, 2023
  37. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 19, 2023
  38. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 19, 2023
  39. Junio C HamanoOct 19, 2023
  40. Junio C HamanoOct 19, 2023
  41. Karthik NayakOct 20, 2023
  42. Karthik NayakOct 20, 2023
  43. Junio C HamanoOct 20, 2023
  44. Junio C HamanoOct 20, 2023
  45. Karthik NayakOct 24, 2023
  46. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 24, 2023
  47. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 24, 2023
  48. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 24, 2023
  49. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 24, 2023
  50. Junio C HamanoOct 24, 2023
  51. Junio C HamanoOct 25, 2023
  52. Karthik NayakOct 25, 2023
  53. Patrick SteinhardtOct 25, 2023
  54. Junio C HamanoOct 26, 2023
  55. 0/3 rev-list: add support for commits in `--missing`Karthik Nayak, Oct 26, 2023
  56. 1/3 revision: rename bit to `do_not_die_on_missing_objects`Karthik Nayak, Oct 26, 2023
  57. 2/3 rev-list: move `show_commit()` to the bottomKarthik Nayak, Oct 26, 2023
  58. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 26, 2023
  59. Patrick SteinhardtOct 27, 2023
  60. Karthik NayakOct 27, 2023
  61. 3/3 rev-list: add commit object support in `--missing` optionKarthik Nayak, Oct 27, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.