git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 10/17] Unmap individual windows rather than entire files.

From
Shawn O. Pearce <spearce@spearce.org>
Date
Dec 23, 2006, 07:34 UTC
Message-ID
<20061223073423.GK9837@spearce.org>
In-Reply-To
<53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org>

To support multiple windows per packfile we need to unmap only one window at a time from that packfile, leaving any other windows in place and available for reference.

We treat all windows from all packfiles equally; the least recently used, not-in-use window across all packfiles will always be closed first.

If we have unmapped all windows in a packfile then we can also close the packfile's file descriptor as its possible we won't need to map any window from that file in the near future. This decision about when to close the pack file descriptor may need to be revisited in the future after additional testing on several different platforms can be performed.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 sha1_file.c |   45 ++++++++++++++++++++++++++++++---------------
 1 files changed, 30 insertions(+), 15 deletions(-)
diff --git a/sha1_file.c b/sha1_file.c
index 6c34482..49dd4b7 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -450,24 +450,39 @@ static int check_packed_git_idx(const char *path, unsigned long *idx_size_,
 	return 0;
 }
 
-static int unuse_one_packed_git(void)
+static int unuse_one_window(void)
 {
-	struct packed_git *p, *lru = NULL;
+	struct packed_git *p, *lru_p = NULL;
+	struct pack_window *w, *w_l, *lru_w = NULL, *lru_l = NULL;
 
 	for (p = packed_git; p; p = p->next) {
-		if (!p->windows || p->windows->inuse_cnt)
-			continue;
-		if (!lru || p->windows->last_used < lru->windows->last_used)
-			lru = p;
+		for (w_l = NULL, w = p->windows; w; w = w->next) {
+			if (!w->inuse_cnt) {
+				if (!lru_w || w->last_used < lru_w->last_used) {
+					lru_p = p;
+					lru_w = w;
+					lru_l = w_l;
+				}
+			}
+			w_l = w;
+		}
 	}
-	if (!lru)
-		return 0;
-	munmap(lru->windows->base, lru->windows->len);
-	free(lru->windows);
-	lru->windows = NULL;
-	close(p->pack_fd);
-	p->pack_fd = -1;
-	return 1;
+	if (lru_p) {
+		munmap(lru_w->base, lru_w->len);
+		pack_mapped -= lru_w->len;
+		if (lru_l)
+			lru_l->next = lru_w->next;
+		else {
+			lru_p->windows = lru_w->next;
+			if (!lru_p->windows) {
+				close(lru_p->pack_fd);
+				lru_p->pack_fd = -1;
+			}
+		}
+		free(lru_w);
+		return 1;
+	}
+	return 0;
 }
 
 void unuse_pack(struct pack_window **w_cursor)
@@ -532,7 +547,7 @@ unsigned char* use_pack(struct packed_git *p,
 		open_packed_git(p);
 	if (!win) {
 		pack_mapped += p->pack_size;
-		while (packed_git_limit < pack_mapped && unuse_one_packed_git())
+		while (packed_git_limit < pack_mapped && unuse_one_window())
 			; /* nothing */
 		win = xcalloc(1, sizeof(*win));
 		win->len = p->pack_size;
-- 
1.4.4.3.g87d8
Previous: Shawn O. PearceNext: Shawn O. Pearce
Message 10 of 25 in “Replace unpack_entry_gently with unpack_entry.”
  1. 1/17 Replace unpack_entry_gently with unpack_entry.Shawn O. Pearce, Dec 23, 2006
  2. 2/17 Introduce new config option for mmap limit.Shawn O. Pearce, Dec 23, 2006
  3. 3/17 Refactor packed_git to prepare for sliding mmap windows.Shawn O. Pearce, Dec 23, 2006
  4. 4/17 Use off_t for index and pack file lengths.Shawn O. Pearce, Dec 23, 2006
  5. 5/17 Create read_or_die utility routine.Shawn O. Pearce, Dec 23, 2006
  6. 6/17 Refactor how we open pack files to prepare for multiple windows.Shawn O. Pearce, Dec 23, 2006
  7. 7/17 Replace use_packed_git with window cursors.Shawn O. Pearce, Dec 23, 2006
  8. 8/17 Loop over pack_windows when inflating/accessing data.Shawn O. Pearce, Dec 23, 2006
  9. 9/17 Document why header parsing won't exceed a window.Shawn O. Pearce, Dec 23, 2006
  10. 10/17 Unmap individual windows rather than entire files.Shawn O. Pearce, Dec 23, 2006
  11. 11/17 Fully activate the sliding window pack access.Shawn O. Pearce, Dec 23, 2006
  12. Linus TorvaldsDec 23, 2006
  13. Eric BlakeDec 23, 2006
  14. Johannes SchindelinDec 24, 2006
  15. Junio C HamanoDec 23, 2006
  16. Linus TorvaldsDec 23, 2006
  17. Johannes SchindelinDec 24, 2006
  18. Shawn PearceDec 24, 2006
  19. Shawn PearceDec 24, 2006
  20. 12/17 Load core configuration in git-verify-pack.Shawn O. Pearce, Dec 23, 2006
  21. 13/17 Ensure core.packedGitWindowSize cannot be less than 2 pages.Shawn O. Pearce, Dec 23, 2006
  22. 14/17 Improve error message when packfile mmap fails.Shawn O. Pearce, Dec 23, 2006
  23. 15/17 Support unmapping windows on 'temporary' packfiles.Shawn O. Pearce, Dec 23, 2006
  24. 16/17 Create pack_report() as a debugging aid.Shawn O. Pearce, Dec 23, 2006
  25. 17/17 Test suite for sliding window mmap implementation.Shawn O. Pearce, Dec 23, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.