git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 5/6] packfile: maintain process-wide limits globally

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 20, 2026, 12:02 UTC
Message-ID
<20261120-pks-packfile-global-statistics-v1-5-19beb83b145f@pks.im>
In-Reply-To
<20261120-pks-packfile-global-statistics-v1-0-19beb83b145f@pks.im>

Our packfile subsystem has two limits that are supposed to apply globally across all open packfiles:

  - `pack_max_fds` is the maximum number of open file descriptors for
    packfiles. When we exceed that limit, we close least-recently-used
    packs until we're below it again.
  - "core.packedGitLimit" is the maximum number of packfile data that
    may be mapped into memory. When we exceed that limit, we close the
    least-recently-used windows into our packfiles.

Those limits are supposed to be process-wide, so even across multiple repositories. But our current implementation will only consider packfiles from a given repository or object database, which may not even have any packfiles open when the logic kicks in. Furthermore, the logic assumes that we're using the "files" backend and thus unconditionally downcasts the object database's backend to it.

Introduce a global list of all packfiles. This list is completely independent of the owning repository or object database, and it is agnostic of the actual object database backend that's in use. Use it to maintain the above two limits.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 packfile.c | 29 +++++++++++++----------------
 1 file changed, 13 insertions(+), 16 deletions(-)
diff --git a/packfile.c b/packfile.c
index 077b0395a79..0d5647c74c8 100644
--- a/packfile.c
+++ b/packfile.c
@@ -35,6 +35,7 @@ char *odb_pack_name(struct repository *r, struct strbuf *buf,
 	return buf->buf;
 }
 
+static struct packfile_list all_packfiles;
 static unsigned int pack_used_ctr;
 static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
@@ -195,6 +196,7 @@ static struct packed_git *packfile_new(struct repository *r, int extra)
 	memset(p, 0, sizeof(*p));
 	p->pack_fd = -1;
 	p->repo = r;
+	packfile_list_append(&all_packfiles, p, 1);
 	return p;
 }
 
@@ -243,17 +245,14 @@ static void scan_windows(struct packed_git *p,
 	}
 }
 
-static int unuse_one_window(struct object_database *odb)
+static int unuse_one_window(void)
 {
-	struct odb_source_files *files = odb_source_files_downcast(odb->source);
 	struct packfile_list_entry *e;
 	struct packed_git *lru_p = NULL;
 	struct pack_window *lru_w = NULL, *lru_l = NULL;
 
-	for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
-		for (e = dir->packed->packs.head; e; e = e->next)
-			scan_windows(e->pack, &lru_p, &lru_w, &lru_l);
-	}
+	for (e = all_packfiles.head; e; e = e->next)
+		scan_windows(e->pack, &lru_p, &lru_w, &lru_l);
 
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -407,20 +406,17 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc
 	*accept_windows_inuse = has_windows_inuse;
 }
 
-static int close_one_pack(struct repository *r)
+static int close_one_pack(void)
 {
-	struct odb_source_files *files = odb_source_files_downcast(r->objects->source);
 	struct packfile_list_entry *e;
 	struct packed_git *lru_p = NULL;
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
-		for (e = dir->packed->packs.head; e; e = e->next) {
-			if (e->pack->pack_fd == -1)
-				continue;
-			find_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);
-		}
+	for (e = all_packfiles.head; e; e = e->next) {
+		if (e->pack->pack_fd == -1)
+			continue;
+		find_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);
 	}
 
 	if (lru_p)
@@ -505,7 +501,7 @@ static int packfile_open_pack(struct packed_git *p)
 			pack_max_fds = 1;
 	}
 
-	while (pack_max_fds <= pack_open_fds && close_one_pack(p->repo))
+	while (pack_max_fds <= pack_open_fds && close_one_pack())
 		; /* nothing */
 
 	p->pack_fd = git_open(p->pack_name);
@@ -649,7 +645,7 @@ unsigned char *use_pack(struct packed_git *p,
 			pack_mapped += win->len;
 
 			while (settings->packed_git_limit < pack_mapped &&
-			       unuse_one_window(p->repo->objects))
+			       unuse_one_window())
 				; /* nothing */
 			win->base = xmmap_gently(NULL, win->len,
 				PROT_READ, MAP_PRIVATE,
@@ -754,6 +750,7 @@ struct packed_git *packfile_from_idx(struct repository *r, const char *path,
 
 void packfile_free(struct packed_git *p)
 {
+	packfile_list_remove(&all_packfiles, p);
 	packfile_close(p);
 	free(p);
 }
-- 
2.56.1.831.g720f3f5937.dirty
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 6 of 7 in “packfile: some lifecycle improvements”
  1. 0/6 packfile: some lifecycle improvementsPatrick Steinhardt, Nov 20, 2026
  2. 1/6 builtin/fast-import: stop reporting packfile statisticsPatrick Steinhardt, Nov 20, 2026
  3. 2/6 packfile: rename lifecycle functionsPatrick Steinhardt, Nov 20, 2026
  4. 3/6 packfile: inline `open_packed_git_1()`Patrick Steinhardt, Nov 20, 2026
  5. 4/6 packfile: introduce function to free a packfilePatrick Steinhardt, Nov 20, 2026
  6. 5/6 packfile: maintain process-wide limits globallyPatrick Steinhardt, Nov 20, 2026
  7. 6/6 packfile: skip closing packs marked as do-not-closePatrick Steinhardt, Nov 20, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.