git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH] delta-islands: free island-related data after use

From
EWEric Wong <e@80x24.org>
Date
Nov 16, 2022, 10:50 UTC
Message-ID
<20221116105013.1777440-1-e@80x24.org>

On my use case involving 771 islands of Linux on kernel.org, this reduces memory usage by around 25MB. The bulk of that comes from free_remote_islands, since free_island_regexes only saves around 40k.

This memory is saved early in the memory-intensive pack process, making it available for the remainder of the long process.

Signed-off-by: Eric Wong <e@80x24.org>
---
  Note: I only noticed this when I screwed up up a pack.island RE,
  ended up with hundreds of thousands of islands instead of 771,
  and kept OOM-ing :x
  Memory savings were measured using the following patch which
  relies on a patched LD_PRELOAD-based malloc debugger:
  https://80x24.org/spew/20221116095404.3974691-1-e@80x24.org/raw
  Will try to hunt down more memory savings in the nearish future.
 delta-islands.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
diff --git a/delta-islands.c b/delta-islands.c
index 26f9e99e1a..391e947cc6 100644
--- a/delta-islands.c
+++ b/delta-islands.c
@@ -454,6 +454,31 @@ static void deduplicate_islands(struct repository *r)
 	free(list);
 }
 
+static void free_island_regexes(void)
+{
+	unsigned int i;
+
+	for (i = 0; i < island_regexes_nr; i++)
+		regfree(&island_regexes[i]);
+
+	FREE_AND_NULL(island_regexes);
+	island_regexes_alloc = island_regexes_nr = 0;
+}
+
+static void free_remote_islands(void)
+{
+	const char *island_name;
+	struct remote_island *rl;
+
+	kh_foreach(remote_islands, island_name, rl, {
+		free((void *)island_name);
+		oid_array_clear(&rl->oids);
+		free(rl);
+	});
+	kh_destroy_str(remote_islands);
+	remote_islands = NULL;
+}
+
 void load_delta_islands(struct repository *r, int progress)
 {
 	island_marks = kh_init_oid_map();
@@ -461,7 +486,9 @@ void load_delta_islands(struct repository *r, int progress)
 
 	git_config(island_config_callback, NULL);
 	for_each_ref(find_island_for_ref, NULL);
+	free_island_regexes();
 	deduplicate_islands(r);
+	free_remote_islands();
 
 	if (progress)
 		fprintf(stderr, _("Marked %d islands, done.\n"), island_counter);
Next: Ævar Arnfjörð Bjarmason
Message 1 of 9 in “delta-islands: free island-related data after use”
  1. delta-islands: free island-related data after useEric Wong, Nov 16, 2022
  2. Ævar Arnfjörð BjarmasonNov 16, 2022
  3. delta-islands: free island-related data after useEric Wong, Nov 17, 2022
  4. Ævar Arnfjörð BjarmasonNov 18, 2022
  5. Jeff KingNov 22, 2022
  6. Jeff KingNov 16, 2022
  7. pack-objects memory use observations [was: [PATCH] delta-islands: free island-related data after use]Eric Wong, Feb 1, 2023
  8. Eric WongFeb 1, 2023
  9. Jeff KingFeb 2, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.