git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 02/19] streaming: drop the `open()` callback function

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 23, 2025, 18:59 UTC
Message-ID
<20251123-b4-pks-odb-read-stream-v3-2-1a129182822b@pks.im>
In-Reply-To
<20251123-b4-pks-odb-read-stream-v3-0-1a129182822b@pks.im>

When creating a read stream we first populate the structure with the open callback function and then subsequently call the function. This layout is somewhat weird though:

  - The structure needs to be allocated and partially populated with the
    open function before we can properly initialize it.
  - We only ever call the `open()` callback function right after having
    populated the `struct odb_read_stream::open` member, and it's never
    called thereafter again. So it is somewhat pointless to store the
    callback in the first place.

Especially the first point creates a problem for us. In subsequent commits we'll want to fully move construction of the read source into the respective object sources. E.g., the loose object source will be the one that is responsible for creating the structure. But this creates a problem: if we first need to create the structure so that we can call the source-specific callback we cannot fully handle creation of the structure in the source itself.

We could of course work around that and have the loose object source create the structure and populate its `open()` callback, only. But this doesn't really buy us anything due to the second bullet point above.

Instead, drop the callback entirely and refactor `istream_source()` so that we open the streams immediately. This unblocks a subsequent step, where we'll also start to allocate the structure in the source-specific logic.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 streaming.c | 37 +++++++++++++++----------------------
 1 file changed, 15 insertions(+), 22 deletions(-)
diff --git a/streaming.c b/streaming.c
index 1fb4b7c1c0..1bb3f393b8 100644
--- a/streaming.c
+++ b/streaming.c
@@ -14,10 +14,6 @@
 #include "replace-object.h"
 #include "packfile.h"
 
-typedef int (*open_istream_fn)(struct odb_read_stream *,
-			       struct repository *,
-			       const struct object_id *,
-			       enum object_type *);
 typedef int (*close_istream_fn)(struct odb_read_stream *);
 typedef ssize_t (*read_istream_fn)(struct odb_read_stream *, char *, size_t);
 
@@ -34,7 +30,6 @@ struct filtered_istream {
 };
 
 struct odb_read_stream {
-	open_istream_fn open;
 	close_istream_fn close;
 	read_istream_fn read;
 
@@ -437,21 +432,25 @@ static int istream_source(struct odb_read_stream *st,
 
 	switch (oi.whence) {
 	case OI_LOOSE:
-		st->open = open_istream_loose;
+		if (open_istream_loose(st, r, oid, type) < 0)
+			break;
 		return 0;
 	case OI_PACKED:
-		if (!oi.u.packed.is_delta &&
-		    repo_settings_get_big_file_threshold(the_repository) < size) {
-			st->u.in_pack.pack = oi.u.packed.pack;
-			st->u.in_pack.pos = oi.u.packed.offset;
-			st->open = open_istream_pack_non_delta;
-			return 0;
-		}
-		/* fallthru */
-	default:
-		st->open = open_istream_incore;
+		if (oi.u.packed.is_delta ||
+		    repo_settings_get_big_file_threshold(the_repository) >= size)
+			break;
+
+		st->u.in_pack.pack = oi.u.packed.pack;
+		st->u.in_pack.pos = oi.u.packed.offset;
+		if (open_istream_pack_non_delta(st, r, oid, type) < 0)
+			break;
+
 		return 0;
+	default:
+		break;
 	}
+
+	return open_istream_incore(st, r, oid, type);
 }
 
 /****************************************************************
@@ -485,12 +484,6 @@ struct odb_read_stream *open_istream(struct repository *r,
 		return NULL;
 	}
 
-	if (st->open(st, r, real, type)) {
-		if (open_istream_incore(st, r, real, type)) {
-			free(st);
-			return NULL;
-		}
-	}
 	if (filter) {
 		/* Add "&& !is_null_stream_filter(filter)" for performance */
 		struct odb_read_stream *nst = attach_stream_filter(st, filter);
-- 
2.52.0.rc2.482.gaa765fefd0.dirty
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 68 of 85 in “Refactor object read streams to work via object sources”
  1. 00/18 Refactor object read streams to work via object sourcesPatrick Steinhardt, Nov 19, 2025
  2. 01/18 streaming: rename `git_istream` into `odb_read_stream`Patrick Steinhardt, Nov 19, 2025
  3. Justin ToblerNov 19, 2025
  4. Junio C HamanoNov 19, 2025
  5. Patrick SteinhardtNov 21, 2025
  6. 02/18 streaming: drop the `open()` callback functionPatrick Steinhardt, Nov 19, 2025
  7. Karthik NayakNov 19, 2025
  8. Justin ToblerNov 19, 2025
  9. Patrick SteinhardtNov 21, 2025
  10. 03/18 streaming: propagate final object type via the streamPatrick Steinhardt, Nov 19, 2025
  11. Justin ToblerNov 19, 2025
  12. Patrick SteinhardtNov 21, 2025
  13. 04/18 streaming: explicitly pass packfile info when streaming a packed objectPatrick Steinhardt, Nov 19, 2025
  14. 05/18 streaming: allocate stream inside the backend-specific logicPatrick Steinhardt, Nov 19, 2025
  15. Karthik NayakNov 19, 2025
  16. Patrick SteinhardtNov 21, 2025
  17. 06/18 streaming: create structure for in-core object streamsPatrick Steinhardt, Nov 19, 2025
  18. Karthik NayakNov 19, 2025
  19. Patrick SteinhardtNov 21, 2025
  20. 07/18 streaming: create structure for loose object streamsPatrick Steinhardt, Nov 19, 2025
  21. 08/18 streaming: create structure for packed object streamsPatrick Steinhardt, Nov 19, 2025
  22. 09/18 streaming: create structure for filtered object streamsPatrick Steinhardt, Nov 19, 2025
  23. 10/18 streaming: move zlib stream into backendsPatrick Steinhardt, Nov 19, 2025
  24. 11/18 packfile: introduce function to read object info from a storePatrick Steinhardt, Nov 19, 2025
  25. Karthik NayakNov 19, 2025
  26. Patrick SteinhardtNov 21, 2025
  27. 12/18 streaming: rely on object sources to create object streamPatrick Steinhardt, Nov 19, 2025
  28. Karthik NayakNov 19, 2025
  29. 13/18 streaming: get rid of `the_repository`Patrick Steinhardt, Nov 19, 2025
  30. 14/18 streaming: make the `odb_read_stream` definition publicPatrick Steinhardt, Nov 19, 2025
  31. Karthik NayakNov 19, 2025
  32. Patrick SteinhardtNov 21, 2025
  33. 15/18 streaming: move logic to read loose objects streams into backendPatrick Steinhardt, Nov 19, 2025
  34. 16/18 streaming: move logic to read packed objects streams into backendPatrick Steinhardt, Nov 19, 2025
  35. 17/18 streaming: refactor interface to be object-database-centricPatrick Steinhardt, Nov 19, 2025
  36. 18/18 streaming: move into object database subsystemPatrick Steinhardt, Nov 19, 2025
  37. 00/19 Refactor object read streams to work via object sourcesPatrick Steinhardt, Nov 21, 2025
  38. 01/19 streaming: rename `git_istream` into `odb_read_stream`Patrick Steinhardt, Nov 21, 2025
  39. 02/19 streaming: drop the `open()` callback functionPatrick Steinhardt, Nov 21, 2025
  40. Junio C HamanoNov 21, 2025
  41. Patrick SteinhardtNov 23, 2025
  42. 03/19 streaming: propagate final object type via the streamPatrick Steinhardt, Nov 21, 2025
  43. 04/19 streaming: explicitly pass packfile info when streaming a packed objectPatrick Steinhardt, Nov 21, 2025
  44. 05/19 streaming: allocate stream inside the backend-specific logicPatrick Steinhardt, Nov 21, 2025
  45. 06/19 streaming: create structure for in-core object streamsPatrick Steinhardt, Nov 21, 2025
  46. 07/19 streaming: create structure for loose object streamsPatrick Steinhardt, Nov 21, 2025
  47. 08/19 streaming: create structure for packed object streamsPatrick Steinhardt, Nov 21, 2025
  48. 09/19 streaming: create structure for filtered object streamsPatrick Steinhardt, Nov 21, 2025
  49. 10/19 streaming: move zlib stream into backendsPatrick Steinhardt, Nov 21, 2025
  50. 11/19 packfile: introduce function to read object info from a storePatrick Steinhardt, Nov 21, 2025
  51. 12/19 streaming: rely on object sources to create object streamPatrick Steinhardt, Nov 21, 2025
  52. Junio C HamanoNov 21, 2025
  53. Patrick SteinhardtNov 23, 2025
  54. 13/19 streaming: get rid of `the_repository`Patrick Steinhardt, Nov 21, 2025
  55. Junio C HamanoNov 21, 2025
  56. Patrick SteinhardtNov 23, 2025
  57. 14/19 streaming: make the `odb_read_stream` definition publicPatrick Steinhardt, Nov 21, 2025
  58. 15/19 streaming: move logic to read loose objects streams into backendPatrick Steinhardt, Nov 21, 2025
  59. 16/19 streaming: move logic to read packed objects streams into backendPatrick Steinhardt, Nov 21, 2025
  60. 17/19 streaming: refactor interface to be object-database-centricPatrick Steinhardt, Nov 21, 2025
  61. Junio C HamanoNov 22, 2025
  62. Patrick SteinhardtNov 23, 2025
  63. 18/19 streaming: move into object database subsystemPatrick Steinhardt, Nov 21, 2025
  64. Junio C HamanoNov 23, 2025
  65. 19/19 streaming: drop redundant type and size pointersPatrick Steinhardt, Nov 21, 2025
  66. 00/19 Refactor object read streams to work via object sourcesPatrick Steinhardt, Nov 23, 2025
  67. 01/19 streaming: rename `git_istream` into `odb_read_stream`Patrick Steinhardt, Nov 23, 2025
  68. 02/19 streaming: drop the `open()` callback functionPatrick Steinhardt, Nov 23, 2025
  69. 03/19 streaming: propagate final object type via the streamPatrick Steinhardt, Nov 23, 2025
  70. 04/19 streaming: explicitly pass packfile info when streaming a packed objectPatrick Steinhardt, Nov 23, 2025
  71. 05/19 streaming: allocate stream inside the backend-specific logicPatrick Steinhardt, Nov 23, 2025
  72. 06/19 streaming: create structure for in-core object streamsPatrick Steinhardt, Nov 23, 2025
  73. 07/19 streaming: create structure for loose object streamsPatrick Steinhardt, Nov 23, 2025
  74. 08/19 streaming: create structure for packed object streamsPatrick Steinhardt, Nov 23, 2025
  75. 09/19 streaming: create structure for filtered object streamsPatrick Steinhardt, Nov 23, 2025
  76. 10/19 streaming: move zlib stream into backendsPatrick Steinhardt, Nov 23, 2025
  77. 11/19 packfile: introduce function to read object info from a storePatrick Steinhardt, Nov 23, 2025
  78. 12/19 streaming: rely on object sources to create object streamPatrick Steinhardt, Nov 23, 2025
  79. 13/19 streaming: get rid of `the_repository`Patrick Steinhardt, Nov 23, 2025
  80. 14/19 streaming: make the `odb_read_stream` definition publicPatrick Steinhardt, Nov 23, 2025
  81. 15/19 streaming: move logic to read loose objects streams into backendPatrick Steinhardt, Nov 23, 2025
  82. 16/19 streaming: move logic to read packed objects streams into backendPatrick Steinhardt, Nov 23, 2025
  83. 17/19 streaming: refactor interface to be object-database-centricPatrick Steinhardt, Nov 23, 2025
  84. 18/19 streaming: move into object database subsystemPatrick Steinhardt, Nov 23, 2025
  85. 19/19 streaming: drop redundant type and size pointersPatrick Steinhardt, Nov 23, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.