git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 01/16] packfile: introduce a new `struct packfile_store`

From
Patrick Steinhardt <ps@pks.im>
Date
Aug 19, 2025, 08:19 UTC
Message-ID
<20250819-b4-pks-packfiles-store-v1-1-1660842e125a@pks.im>
In-Reply-To
<20250819-b4-pks-packfiles-store-v1-0-1660842e125a@pks.im>

Information about a object database's packfiles is currently distributed across two different structures:

  - `struct packed_git` contains the `next` pointer as well as the
    `mru_head`, both of which serve to store the list of packfiles.
  - `struct object_database` contains several fields that relate to the
    packfiles.

So we don't really have a central data structure that tracks our packfiles, and consequently responsibilities aren't always clear cut. A consequence for the upcoming pluggable object databases is that this makes it very hard to move management of packfiles from the object database level down into the object database source.

Introduce a new `struct packfile_store` which is about to become the single source of truth for managing packfiles. Right now this data structure doesn't yet contain anything, but in subsequent patches we will move all data structures that relate to packfiles and that are currently contained in `struct object_database` into this new home.

Note that this is only a first step: most importantly, we won't (yet) move the `struct packed_git::next` pointer around. This will happen in a subsequent patch series though so that `struct packed_git` will really only host information about the specific packfile it represents.

Further note that the new structure still sits at the wrong level at the end of this patch series: as mentioned, it should eventually sit at the level of the object database source, not at the object database level. But introducing the packfile store now already makes it way easier to eventually push down the now-selfcontained data structure by one level.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c      |  1 +
 odb.h      |  2 ++
 packfile.c | 13 +++++++++++++
 packfile.h | 18 ++++++++++++++++++
 4 files changed, 34 insertions(+)
diff --git a/odb.c b/odb.c
index 2a92a018c4..34b70d0074 100644
--- a/odb.c
+++ b/odb.c
@@ -996,6 +996,7 @@ struct object_database *odb_new(struct repository *repo)
 
 	memset(o, 0, sizeof(*o));
 	o->repo = repo;
+	o->packfiles = packfile_store_new(o);
 	INIT_LIST_HEAD(&o->packed_git_mru);
 	hashmap_init(&o->pack_map, pack_map_entry_cmp, NULL, 0);
 	pthread_mutex_init(&o->replace_mutex, NULL);
diff --git a/odb.h b/odb.h
index 3dfc66d75a..026ba9386d 100644
--- a/odb.h
+++ b/odb.h
@@ -83,6 +83,7 @@ struct odb_source {
 };
 
 struct packed_git;
+struct packfile_store;
 struct cached_object_entry;
 
 /*
@@ -128,6 +129,7 @@ struct object_database {
 	 *
 	 * should only be accessed directly by packfile.c
 	 */
+	struct packfile_store *packfiles;
 
 	struct packed_git *packed_git;
 	/* A most-recently-used ordered version of the packed_git list. */
diff --git a/packfile.c b/packfile.c
index 5d73932f50..8fbf1cfc2d 100644
--- a/packfile.c
+++ b/packfile.c
@@ -2333,3 +2333,16 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l
 	*len = hdr - out;
 	return 0;
 }
+
+struct packfile_store *packfile_store_new(struct object_database *odb)
+{
+	struct packfile_store *store;
+	CALLOC_ARRAY(store, 1);
+	store->odb = odb;
+	return store;
+}
+
+void packfile_store_free(struct packfile_store *store)
+{
+	free(store);
+}
diff --git a/packfile.h b/packfile.h
index f16753f2a9..8d31fd619a 100644
--- a/packfile.h
+++ b/packfile.h
@@ -52,6 +52,24 @@ struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 };
 
+/*
+ * A store that manages packfiles for a given object database.
+ */
+struct packfile_store {
+	struct object_database *odb;
+};
+
+/*
+ * Allocate and initialize a new empty packfile store for the given object
+ * database.
+ */
+struct packfile_store *packfile_store_new(struct object_database *odb);
+
+/*
+ * Free the packfile store and all its associated state.
+ */
+void packfile_store_free(struct packfile_store *store);
+
 static inline int pack_map_entry_cmp(const void *cmp_data UNUSED,
 				     const struct hashmap_entry *entry,
 				     const struct hashmap_entry *entry2,
-- 
2.51.0.261.g7ce5a0a67e.dirty
Previous: Patrick SteinhardtNext: Karthik Nayak
Message 2 of 181 in “packfile: carve out a new packfile store”
  1. 00/16 packfile: carve out a new packfile storePatrick Steinhardt, Aug 19, 2025
  2. 01/16 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  3. Karthik NayakAug 19, 2025
  4. Patrick SteinhardtAug 20, 2025
  5. Junio C HamanoAug 19, 2025
  6. Patrick SteinhardtAug 20, 2025
  7. 02/16 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  8. 03/16 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  9. Karthik NayakAug 19, 2025
  10. Junio C HamanoAug 19, 2025
  11. Karthik NayakAug 20, 2025
  12. Junio C HamanoAug 22, 2025
  13. Documentation: note styling for bit fieldsKarthik Nayak, Aug 26, 2025
  14. Patrick SteinhardtAug 20, 2025
  15. Junio C HamanoAug 20, 2025
  16. 04/16 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  17. 05/16 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  18. Karthik NayakAug 20, 2025
  19. Jeff KingAug 20, 2025
  20. Patrick SteinhardtAug 21, 2025
  21. 06/16 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Aug 19, 2025
  22. Junio C HamanoAug 19, 2025
  23. Patrick SteinhardtAug 20, 2025
  24. 07/16 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Aug 19, 2025
  25. Junio C HamanoAug 19, 2025
  26. 08/16 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Aug 19, 2025
  27. 09/16 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Aug 19, 2025
  28. Karthik NayakAug 20, 2025
  29. 10/16 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Aug 19, 2025
  30. 11/16 packfile: always add packfiles to MRU when adding a packPatrick Steinhardt, Aug 19, 2025
  31. Karthik NayakAug 20, 2025
  32. 12/16 packfile: introduce function to load and add packfilesPatrick Steinhardt, Aug 19, 2025
  33. Karthik NayakAug 20, 2025
  34. Patrick SteinhardtAug 21, 2025
  35. 13/16 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Aug 19, 2025
  36. 14/16 packfile: remove `get_packed_git()`Patrick Steinhardt, Aug 19, 2025
  37. Karthik NayakAug 20, 2025
  38. Patrick SteinhardtAug 21, 2025
  39. Karthik NayakAug 20, 2025
  40. 15/16 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Aug 19, 2025
  41. Karthik NayakAug 20, 2025
  42. Patrick SteinhardtAug 21, 2025
  43. 16/16 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Aug 19, 2025
  44. Junio C HamanoAug 19, 2025
  45. Karthik NayakAug 20, 2025
  46. 00/16 packfile: carve out a new packfile storePatrick Steinhardt, Aug 21, 2025
  47. 01/16 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  48. 02/16 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  49. Taylor BlauAug 25, 2025
  50. Patrick SteinhardtSep 2, 2025
  51. Taylor BlauSep 2, 2025
  52. Junio C HamanoSep 2, 2025
  53. Patrick SteinhardtSep 3, 2025
  54. Taylor BlauSep 11, 2025
  55. Patrick SteinhardtSep 15, 2025
  56. 03/16 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  57. Taylor BlauAug 26, 2025
  58. 04/16 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  59. Taylor BlauAug 26, 2025
  60. 05/16 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  61. 06/16 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Aug 21, 2025
  62. Taylor BlauAug 26, 2025
  63. Patrick SteinhardtSep 2, 2025
  64. 07/16 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Aug 21, 2025
  65. Taylor BlauAug 26, 2025
  66. 08/16 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Aug 21, 2025
  67. Taylor BlauAug 26, 2025
  68. 09/16 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Aug 21, 2025
  69. Taylor BlauAug 26, 2025
  70. Patrick SteinhardtSep 2, 2025
  71. 10/16 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Aug 21, 2025
  72. Taylor BlauAug 26, 2025
  73. Patrick SteinhardtSep 2, 2025
  74. 11/16 packfile: always add packfiles to MRU when adding a packPatrick Steinhardt, Aug 21, 2025
  75. Taylor BlauAug 27, 2025
  76. Patrick SteinhardtSep 2, 2025
  77. 12/16 packfile: introduce function to load and add packfilesPatrick Steinhardt, Aug 21, 2025
  78. Taylor BlauAug 27, 2025
  79. 13/16 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Aug 21, 2025
  80. Taylor BlauAug 27, 2025
  81. 14/16 packfile: remove `get_packed_git()`Patrick Steinhardt, Aug 21, 2025
  82. Taylor BlauAug 27, 2025
  83. Patrick SteinhardtSep 2, 2025
  84. Taylor BlauSep 11, 2025
  85. Patrick SteinhardtSep 15, 2025
  86. 15/16 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Aug 21, 2025
  87. Taylor BlauAug 27, 2025
  88. Patrick SteinhardtSep 2, 2025
  89. Taylor BlauSep 11, 2025
  90. Patrick SteinhardtSep 15, 2025
  91. 16/16 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Aug 21, 2025
  92. 00/15 packfile: carve out a new packfile storePatrick Steinhardt, Sep 2, 2025
  93. 01/15 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  94. Karthik NayakSep 9, 2025
  95. 02/15 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  96. Karthik NayakSep 9, 2025
  97. Patrick SteinhardtSep 9, 2025
  98. 03/15 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  99. 04/15 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  100. Karthik NayakSep 9, 2025
  101. Patrick SteinhardtSep 9, 2025
  102. 05/15 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  103. 06/15 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Sep 2, 2025
  104. 07/15 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Sep 2, 2025
  105. 08/15 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Sep 2, 2025
  106. 09/15 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Sep 2, 2025
  107. 10/15 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Sep 2, 2025
  108. 11/15 packfile: introduce function to load and add packfilesPatrick Steinhardt, Sep 2, 2025
  109. 12/15 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Sep 2, 2025
  110. 13/15 packfile: remove `get_packed_git()`Patrick Steinhardt, Sep 2, 2025
  111. 14/15 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Sep 2, 2025
  112. 15/15 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Sep 2, 2025
  113. Junio C HamanoSep 2, 2025
  114. Taylor BlauSep 11, 2025
  115. Karthik NayakSep 9, 2025
  116. 00/15 packfile: carve out a new packfile storePatrick Steinhardt, Sep 9, 2025
  117. 01/15 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  118. 02/15 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  119. 03/15 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  120. 04/15 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  121. 05/15 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  122. 06/15 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Sep 9, 2025
  123. 07/15 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Sep 9, 2025
  124. 08/15 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Sep 9, 2025
  125. 09/15 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Sep 9, 2025
  126. 10/15 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Sep 9, 2025
  127. 11/15 packfile: introduce function to load and add packfilesPatrick Steinhardt, Sep 9, 2025
  128. 12/15 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Sep 9, 2025
  129. 13/15 packfile: remove `get_packed_git()`Patrick Steinhardt, Sep 9, 2025
  130. Taylor BlauSep 11, 2025
  131. 14/15 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Sep 9, 2025
  132. 15/15 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Sep 9, 2025
  133. Karthik NayakSep 10, 2025
  134. Taylor BlauSep 11, 2025
  135. Taylor BlauSep 11, 2025
  136. Patrick SteinhardtSep 15, 2025
  137. 00/15 packfile: carve out a new packfile storePatrick Steinhardt, Sep 15, 2025
  138. 01/15 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  139. Justin ToblerSep 17, 2025
  140. Patrick SteinhardtSep 23, 2025
  141. Justin ToblerSep 24, 2025
  142. 02/15 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  143. 03/15 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  144. 04/15 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  145. Justin ToblerSep 17, 2025
  146. Patrick SteinhardtSep 23, 2025
  147. 05/15 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  148. Justin ToblerSep 17, 2025
  149. 06/15 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Sep 15, 2025
  150. 07/15 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Sep 15, 2025
  151. 08/15 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  152. 09/15 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Sep 15, 2025
  153. Justin ToblerSep 17, 2025
  154. Patrick SteinhardtSep 23, 2025
  155. 10/15 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  156. 11/15 packfile: introduce function to load and add packfilesPatrick Steinhardt, Sep 15, 2025
  157. 12/15 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Sep 15, 2025
  158. 13/15 packfile: refactor `get_packed_git()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  159. 14/15 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  160. 15/15 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Sep 15, 2025
  161. 00/15 packfile: carve out a new packfile storePatrick Steinhardt, Sep 23, 2025
  162. 01/15 packfile: introduce a new `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  163. 02/15 odb: move list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  164. 03/15 odb: move initialization bit into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  165. 04/15 odb: move packfile map into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  166. 05/15 odb: move MRU list of packfiles into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  167. 06/15 odb: move kept cache into `struct packfile_store`Patrick Steinhardt, Sep 23, 2025
  168. 07/15 packfile: reorder functions to avoid function declarationPatrick Steinhardt, Sep 23, 2025
  169. 08/15 packfile: refactor `prepare_packed_git()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  170. 09/15 packfile: split up responsibilities of `reprepare_packed_git()`Patrick Steinhardt, Sep 23, 2025
  171. 10/15 packfile: refactor `install_packed_git()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  172. 11/15 packfile: introduce function to load and add packfilesPatrick Steinhardt, Sep 23, 2025
  173. 12/15 packfile: move `get_multi_pack_index()` into "midx.c"Patrick Steinhardt, Sep 23, 2025
  174. 13/15 packfile: refactor `get_packed_git()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  175. 14/15 packfile: refactor `get_all_packs()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  176. 15/15 packfile: refactor `get_packed_git_mru()` to work on packfile storePatrick Steinhardt, Sep 23, 2025
  177. Justin ToblerSep 24, 2025
  178. Junio C HamanoSep 25, 2025
  179. Patrick SteinhardtSep 26, 2025
  180. Taylor BlauSep 28, 2025
  181. Patrick SteinhardtSep 29, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.