git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 5/7] tmp-objdir: new API for creating and removing primary object dirs

From
Jeff King <peff@peff.net>
Date
Oct 1, 2021, 05:27 UTC
Message-ID
<YVacUCcl6USJSY5T@coredump.intra.peff.net>
In-Reply-To
<CABPp-BF20vpC4m5V4JtS91fuA+PMiEN78J9OpVEPDpnKsTcP8A@mail.gmail.com>
On Thu, Sep 30, 2021 at 09:26:37PM -0700, Elijah Newren wrote:
Show 25 quoted lines
> >  - This side-steps all of our usual code for getting object data into
> >    memory. In general, I'd expect this content to not be too enormous,
> >    but it _could_ be if there are many / large blobs in the result. So
> >    we may end up with large maps. Probably not a big deal on modern
> >    64-bit systems. Maybe an issue on 32-bit systems, just because of
> >    virtual address space.
> >
> >    Likewise, we do support systems with NO_MMAP. They'd work here, but
> >    it would probably mean putting all that object data into the heap. I
> >    could live with that, given how rare such systems are these days, and
> >    that it only matters if you're using --remerge-diff with big blobs.
> 
> Um, I'm starting to get uncomfortable with this pretend_object stuff.
> Part of the reason that merge-ort isn't truly "in memory" despite
> attempting to do exactly that, was because for large enough repos with
> enough files modified on both sides, I wasn't comfortable assuming
> that all new files from three-way content merges and all new trees fit
> into memory.  I'm sure we'd be fine with current-day linux kernel
> sized repos.  No big deal.  In fact, most merges probably don't add
> more than a few dozen new files.  But for microsoft-sized repos, and
> with repos tending to grow over time, more so when the tools
> themselves scale nicely (which we've all been working on enabling),
> makes me worry there might be enough new objects within a single merge
> (especially given the recursive inner merges) that we might need to
> worry about this.

I do think we need to consider that the content might be larger than will comfortably fit in memory. But the point of using mmap is that we don't have to care. The OS is taking care of it for us (just like it would in regular object files).

The question is just whether we're comfortable assuming that mmap exists if you're working on such a large repository. I'd guess that big repos are pretty painful with out it (and again, I'm not even clear which systems Git runs on even lack mmap these days). So IMHO this isn't really a blocker for going in this direction.

Show 9 quoted lines
> >    The whole pretend-we-have-this-object thing
> >    may want to likewise make sure we don't write out objects that we
> >    already have in the real odb.
> 
> Right, so I'd have to copy the relevant logic from write_object_file()
> -- I think that means instead of write_object_file()'s calls to
> freshen_packed_object() and freshen_loose_object() that I instead call
> find_pack_entry() and make has_loose_object() in object-file.c not be
> static and then call it.  Does that sound right?

Yes. You _could_ call the same freshening functions, but I think since we know that our objects are throw-away anyway, we do not even need to perform that freshening operation. I think just has_object_file() would be sufficient for your needs.

-Peff
Previous: Elijah NewrenNext: Ævar Arnfjörð Bjarmason
Message 39 of 64 in “Add a new --remerge-diff capability to show & log”
  1. 0/7 Add a new --remerge-diff capability to show & logElijah Newren via GitGitGadget, Aug 31, 2021
  2. 1/7 merge-ort: mark a few more conflict messages as omittableElijah Newren via GitGitGadget, Aug 31, 2021
  3. Junio C HamanoAug 31, 2021
  4. Elijah NewrenSep 1, 2021
  5. Junio C HamanoSep 1, 2021
  6. 2/7 merge-ort: add ability to record conflict messages in a fileElijah Newren via GitGitGadget, Aug 31, 2021
  7. Jeff KingSep 28, 2021
  8. Elijah NewrenSep 29, 2021
  9. Junio C HamanoSep 29, 2021
  10. Elijah NewrenSep 29, 2021
  11. Jeff KingSep 30, 2021
  12. Ævar Arnfjörð BjarmasonSep 30, 2021
  13. Elijah NewrenOct 1, 2021
  14. Jeff KingOct 1, 2021
  15. 3/7 ll-merge: add API for capturing warnings in a strbuf instead of stderrElijah Newren via GitGitGadget, Aug 31, 2021
  16. Jeff KingSep 28, 2021
  17. Junio C HamanoSep 28, 2021
  18. Elijah NewrenSep 29, 2021
  19. 4/7 merge-ort: capture and print ll-merge warnings in our preferred fashionElijah Newren via GitGitGadget, Aug 31, 2021
  20. Jeff KingSep 28, 2021
  21. Ævar Arnfjörð BjarmasonSep 30, 2021
  22. Elijah NewrenOct 1, 2021
  23. Ævar Arnfjörð BjarmasonOct 1, 2021
  24. 5/7 tmp-objdir: new API for creating and removing primary object dirsElijah Newren via GitGitGadget, Aug 31, 2021
  25. Ævar Arnfjörð BjarmasonSep 28, 2021
  26. Elijah NewrenSep 29, 2021
  27. Jeff KingSep 30, 2021
  28. Ævar Arnfjörð BjarmasonSep 30, 2021
  29. Elijah NewrenOct 1, 2021
  30. Jeff KingSep 28, 2021
  31. Junio C HamanoSep 29, 2021
  32. Jeff KingSep 30, 2021
  33. Ævar Arnfjörð BjarmasonSep 30, 2021
  34. Jeff KingSep 30, 2021
  35. Elijah NewrenOct 1, 2021
  36. Ævar Arnfjörð BjarmasonOct 1, 2021
  37. Elijah NewrenOct 1, 2021
  38. Elijah NewrenOct 1, 2021
  39. Jeff KingOct 1, 2021
  40. Ævar Arnfjörð BjarmasonOct 1, 2021
  41. Elijah NewrenSep 29, 2021
  42. Jeff KingSep 30, 2021
  43. Jeff KingSep 30, 2021
  44. Junio C HamanoSep 30, 2021
  45. Elijah NewrenOct 1, 2021
  46. Junio C HamanoOct 1, 2021
  47. Elijah NewrenOct 1, 2021
  48. Jeff KingOct 1, 2021
  49. Neeraj SinghSep 30, 2021
  50. Junio C HamanoSep 30, 2021
  51. Junio C HamanoSep 30, 2021
  52. Elijah NewrenOct 1, 2021
  53. 7/7 doc/diff-options: explain the new --remerge-diff optionElijah Newren via GitGitGadget, Aug 31, 2021
  54. 6/7 show, log: provide a --remerge-diff capabilityElijah Newren via GitGitGadget, Aug 31, 2021
  55. Sergey OrganovAug 31, 2021
  56. Ævar Arnfjörð BjarmasonSep 28, 2021
  57. Elijah NewrenSep 29, 2021
  58. Bagas SanjayaAug 31, 2021
  59. Elijah NewrenAug 31, 2021
  60. Junio C HamanoAug 31, 2021
  61. Elijah NewrenAug 31, 2021
  62. Junio C HamanoSep 1, 2021
  63. Elijah NewrenSep 1, 2021
  64. Junio C HamanoSep 1, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.