git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 00/16] odb: introduce "inmemory" source

From
Junio C Hamano <gitster@pobox.com>
Date
Apr 3, 2026, 15:41 UTC
Message-ID
<xmqqa4vknjab.fsf@gitster.g>
In-Reply-To
<20260403-b4-pks-odb-source-inmemory-v1-0-8b8d1abaa25e@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
> this patch series introduces the second object database source type,
> which is the "inmemory" source.
I cannot read the word without a hyphen, i.e.e.g., "in-memory".
Show 5 quoted lines
> This source may seem somewhat odd at first: it always starts out empty,
> and any object written into it will only exist in memory until the
> process exits. But the source already serves a purpose in our codebase,
> where some commands, for example git-blame(1), write an in-memory
> worktree commit.

Intermediate tree and blob objects you need while making an octopus merge may also benefit from this feature, to stay only in-core without having to get written out to the outside world.

I understand that this is not meant to be used as a "we create them only for ourselves and they are available only to us while we work, but once we are satisfied we make them available to others", which is much better done by creating an on-disk ephemeral object store, write such objects in them, and then decide at the end of the process between discarding the ephemeral store and moving objects from there to the main object store.

Show 7 quoted lines
> Furthermore, I think that going forward it can serve more purposes as we
> now have an easy way to write and read objects that will not get
> persisted. I could see that this may be useful when for example
> re-merging diffs. But eventually, once we have the object storage format
> extension wired up, callers might even want to manually set up an
> in-memory database as the primary ODB for write operations so that no
> data will be persisted in an arbitrary write.
;-)
Show 53 quoted lines
> Last but not least, this patch series also serves the purpose of
> eventually getting rid of the `struct object_info::whence` member.
> Instead, we'll simply yield the ODB source a specific object has been
> read from, together with some backend-specific data, which gives
> strictly more information compared to the status quo.
>
> The series is based on cf2139f8e1 (The 24th batch, 2026-04-01) with
> ps/odb-cleanup at 109bcb7d1d (odb: drop unneeded headers and forward
> decls, 2026-04-01) merged into it.
>
> Thanks!
>
> Patrick
>
> ---
> Patrick Steinhardt (16):
>       odb: introduce "inmemory" source
>       odb/source-inmemory: implement `free()` callback
>       odb: fix unnecessary call to `find_cached_object()`
>       odb/source-inmemory: implement `read_object_info()` callback
>       odb/source-inmemory: implement `read_object_stream()` callback
>       odb/source-inmemory: implement `write_object()` callback
>       odb/source-inmemory: implement `write_object_stream()` callback
>       cbtree: allow using arbitrary wrapper structures for nodes
>       oidtree: add ability to store data
>       odb/source-inmemory: convert to use oidtree
>       odb/source-inmemory: implement `for_each_object()` callback
>       odb/source-inmemory: implement `find_abbrev_len()` callback
>       odb/source-inmemory: implement `count_objects()` callback
>       odb/source-inmemory: implement `freshen_object()` callback
>       odb/source-inmemory: stub out remaining functions
>       odb: generic inmemory source
>
>  Makefile                 |   1 +
>  cbtree.c                 |  25 +++-
>  cbtree.h                 |  11 +-
>  loose.c                  |   2 +-
>  meson.build              |   1 +
>  object-file.c            |   3 +-
>  odb.c                    |  82 ++---------
>  odb.h                    |   4 +-
>  odb/source-inmemory.c    | 375 +++++++++++++++++++++++++++++++++++++++++++++++
>  odb/source-inmemory.h    |  33 +++++
>  odb/source.h             |   3 +
>  oidtree.c                |  66 ++++++---
>  oidtree.h                |  12 +-
>  t/unit-tests/u-oidtree.c |  26 +++-
>  14 files changed, 529 insertions(+), 115 deletions(-)
>
>
> ---
> base-commit: 3d05c3e2906489caa9f12f0af18dc233a6b8032c
> change-id: 20260401-b4-pks-odb-source-inmemory-7b17c83d9e43
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 27 of 85 in “odb: introduce "inmemory" source”
  1. 00/16 odb: introduce "inmemory" sourcePatrick Steinhardt, Apr 3, 2026
  2. 01/16 odb: introduce "inmemory" sourcePatrick Steinhardt, Apr 3, 2026
  3. Justin ToblerApr 8, 2026
  4. Patrick SteinhardtApr 9, 2026
  5. 02/16 odb/source-inmemory: implement `free()` callbackPatrick Steinhardt, Apr 3, 2026
  6. Justin ToblerApr 8, 2026
  7. 03/16 odb: fix unnecessary call to `find_cached_object()`Patrick Steinhardt, Apr 3, 2026
  8. Justin ToblerApr 8, 2026
  9. Patrick SteinhardtApr 9, 2026
  10. 04/16 odb/source-inmemory: implement `read_object_info()` callbackPatrick Steinhardt, Apr 3, 2026
  11. 05/16 odb/source-inmemory: implement `read_object_stream()` callbackPatrick Steinhardt, Apr 3, 2026
  12. Justin ToblerApr 8, 2026
  13. Patrick SteinhardtApr 9, 2026
  14. 06/16 odb/source-inmemory: implement `write_object()` callbackPatrick Steinhardt, Apr 3, 2026
  15. 07/16 odb/source-inmemory: implement `write_object_stream()` callbackPatrick Steinhardt, Apr 3, 2026
  16. Junio C HamanoApr 3, 2026
  17. Patrick SteinhardtApr 8, 2026
  18. 08/16 cbtree: allow using arbitrary wrapper structures for nodesPatrick Steinhardt, Apr 3, 2026
  19. 09/16 oidtree: add ability to store dataPatrick Steinhardt, Apr 3, 2026
  20. 10/16 odb/source-inmemory: convert to use oidtreePatrick Steinhardt, Apr 3, 2026
  21. 11/16 odb/source-inmemory: implement `for_each_object()` callbackPatrick Steinhardt, Apr 3, 2026
  22. 12/16 odb/source-inmemory: implement `find_abbrev_len()` callbackPatrick Steinhardt, Apr 3, 2026
  23. 13/16 odb/source-inmemory: implement `count_objects()` callbackPatrick Steinhardt, Apr 3, 2026
  24. 14/16 odb/source-inmemory: implement `freshen_object()` callbackPatrick Steinhardt, Apr 3, 2026
  25. 15/16 odb/source-inmemory: stub out remaining functionsPatrick Steinhardt, Apr 3, 2026
  26. 16/16 odb: generic inmemory sourcePatrick Steinhardt, Apr 3, 2026
  27. Junio C HamanoApr 3, 2026
  28. Patrick SteinhardtApr 8, 2026
  29. Junio C HamanoApr 8, 2026
  30. Patrick SteinhardtApr 9, 2026
  31. Junio C HamanoApr 9, 2026
  32. Patrick SteinhardtApr 10, 2026
  33. 00/17 odb: introduce "in-memory" sourcePatrick Steinhardt, Apr 9, 2026
  34. 01/17 odb: introduce "in-memory" sourcePatrick Steinhardt, Apr 9, 2026
  35. Karthik NayakApr 9, 2026
  36. Patrick SteinhardtApr 9, 2026
  37. 02/17 odb/source-inmemory: implement `free()` callbackPatrick Steinhardt, Apr 9, 2026
  38. 03/17 odb: fix unnecessary call to `find_cached_object()`Patrick Steinhardt, Apr 9, 2026
  39. 04/17 odb/source-inmemory: implement `read_object_info()` callbackPatrick Steinhardt, Apr 9, 2026
  40. Karthik NayakApr 9, 2026
  41. Patrick SteinhardtApr 9, 2026
  42. Karthik NayakApr 9, 2026
  43. 05/17 odb/source-inmemory: implement `read_object_stream()` callbackPatrick Steinhardt, Apr 9, 2026
  44. Karthik NayakApr 9, 2026
  45. Patrick SteinhardtApr 9, 2026
  46. 06/17 odb/source-inmemory: implement `write_object()` callbackPatrick Steinhardt, Apr 9, 2026
  47. 07/17 odb/source-inmemory: implement `write_object()` callbackPatrick Steinhardt, Apr 9, 2026
  48. Karthik NayakApr 9, 2026
  49. Patrick SteinhardtApr 9, 2026
  50. 08/17 odb/source-inmemory: implement `write_object_stream()` callbackPatrick Steinhardt, Apr 9, 2026
  51. 09/17 cbtree: allow using arbitrary wrapper structures for nodesPatrick Steinhardt, Apr 9, 2026
  52. Karthik NayakApr 9, 2026
  53. Patrick SteinhardtApr 9, 2026
  54. 10/17 oidtree: add ability to store dataPatrick Steinhardt, Apr 9, 2026
  55. 11/17 odb/source-inmemory: convert to use oidtreePatrick Steinhardt, Apr 9, 2026
  56. 12/17 odb/source-inmemory: implement `for_each_object()` callbackPatrick Steinhardt, Apr 9, 2026
  57. 13/17 odb/source-inmemory: implement `find_abbrev_len()` callbackPatrick Steinhardt, Apr 9, 2026
  58. 14/17 odb/source-inmemory: implement `count_objects()` callbackPatrick Steinhardt, Apr 9, 2026
  59. 15/17 odb/source-inmemory: implement `freshen_object()` callbackPatrick Steinhardt, Apr 9, 2026
  60. 16/17 odb/source-inmemory: stub out remaining functionsPatrick Steinhardt, Apr 9, 2026
  61. Junio C HamanoApr 9, 2026
  62. Patrick SteinhardtApr 10, 2026
  63. 17/17 odb: generic in-memory sourcePatrick Steinhardt, Apr 9, 2026
  64. Karthik NayakApr 9, 2026
  65. Patrick SteinhardtApr 9, 2026
  66. 00/17 odb: introduce "in-memory" sourcePatrick Steinhardt, Apr 10, 2026
  67. 01/17 odb: introduce "in-memory" sourcePatrick Steinhardt, Apr 10, 2026
  68. 02/17 odb/source-inmemory: implement `free()` callbackPatrick Steinhardt, Apr 10, 2026
  69. 03/17 odb: fix unnecessary call to `find_cached_object()`Patrick Steinhardt, Apr 10, 2026
  70. 04/17 odb/source-inmemory: implement `read_object_info()` callbackPatrick Steinhardt, Apr 10, 2026
  71. 05/17 odb/source-inmemory: implement `read_object_stream()` callbackPatrick Steinhardt, Apr 10, 2026
  72. 06/17 odb/source-inmemory: implement `write_object()` callbackPatrick Steinhardt, Apr 10, 2026
  73. 07/17 odb/source-inmemory: implement `write_object_stream()` callbackPatrick Steinhardt, Apr 10, 2026
  74. 08/17 cbtree: allow using arbitrary wrapper structures for nodesPatrick Steinhardt, Apr 10, 2026
  75. 09/17 oidtree: add ability to store dataPatrick Steinhardt, Apr 10, 2026
  76. 10/17 odb/source-inmemory: convert to use oidtreePatrick Steinhardt, Apr 10, 2026
  77. 11/17 odb/source-inmemory: implement `for_each_object()` callbackPatrick Steinhardt, Apr 10, 2026
  78. 12/17 odb/source-inmemory: implement `find_abbrev_len()` callbackPatrick Steinhardt, Apr 10, 2026
  79. 13/17 odb/source-inmemory: implement `count_objects()` callbackPatrick Steinhardt, Apr 10, 2026
  80. 14/17 odb/source-inmemory: implement `freshen_object()` callbackPatrick Steinhardt, Apr 10, 2026
  81. 15/17 odb/source-inmemory: stub out remaining functionsPatrick Steinhardt, Apr 10, 2026
  82. 16/17 odb: generic in-memory sourcePatrick Steinhardt, Apr 10, 2026
  83. 17/17 t/unit-tests: add tests for the in-memory object sourcePatrick Steinhardt, Apr 10, 2026
  84. Karthik NayakApr 14, 2026
  85. Karthik NayakApr 14, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.