git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2] repack: Add option to preserve and prune old pack files

From
JMJames Melvin <jmelvin@codeaurora.org>
Date
Mar 10, 2017, 22:00 UTC
Message-ID
<20170310220020.2666-1-jmelvin@codeaurora.org>

The new --preserve-and-prune option renames old pack files instead of deleting them after repacking and prunes previously preserved pack files.

This option is designed to prevent stale file handle exceptions during git operations which can happen on users of NFS repos when repacking is done on them. The strategy is to preserve old pack files around until the next repack with the hopes that they will become unreferenced by then and not cause any exceptions to running processes when they are finally deleted (pruned).

Signed-off-by: James Melvin <jmelvin@codeaurora.org>
---
 Documentation/git-repack.txt |  4 +++
 builtin/repack.c             | 66 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 69 insertions(+), 1 deletion(-)
diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt
index 26afe6ed5..effd98a43 100644
--- a/Documentation/git-repack.txt
+++ b/Documentation/git-repack.txt
@@ -143,6 +143,10 @@ other objects in that pack they already have locally.
 	being removed. In addition, any unreachable loose objects will
 	be packed (and their loose counterparts removed).
 
+--preserve-and-prune::
+	 Preserve old pack files by renaming them instead of deleting. Prune any
+	 previously preserved pack files before preserving new ones.
+
 Configuration
 -------------
 
diff --git a/builtin/repack.c b/builtin/repack.c
index 677bc7c81..78fad8f1a 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -10,6 +10,7 @@
 
 static int delta_base_offset = 1;
 static int pack_kept_objects = -1;
+static int preserve_and_prune;
 static int write_bitmaps;
 static char *packdir, *packtmp;
 
@@ -108,6 +109,60 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)
 	closedir(dir);
 }
 
+/*
+ * Adds all preserved packs hex strings to the fname list
+ */
+static void get_preserved_pack_filenames(struct string_list *fname_list)
+{
+	DIR *dir;
+	struct dirent *e;
+	char *fname;
+
+	if (!(dir = opendir(packdir)))
+		return;
+
+	while ((e = readdir(dir)) != NULL) {
+		size_t len;
+		if (!strip_suffix(e->d_name, ".old-pack", &len))
+			continue;
+
+		fname = xmemdupz(e->d_name, len);
+		string_list_append_nodup(fname_list, fname);
+	}
+	closedir(dir);
+}
+
+static void remove_preserved_packs(void) {
+	const char *exts[] = {"pack", "idx", "keep", "bitmap"};
+	int i;
+	struct string_list names = STRING_LIST_INIT_DUP;
+	struct string_list_item *item;
+
+	get_preserved_pack_filenames(&names);
+
+	for_each_string_list_item(item, &names) {
+		for (i = 0; i < ARRAY_SIZE(exts); i++) {
+			char *fname;
+			fname = mkpathdup("%s/%s.old-%s",
+					  packdir,
+					  item->string,
+					  exts[i]);
+			if (remove_path(fname))
+				warning(_("failed to remove '%s'"), fname);
+			free(fname);
+		}
+	}
+}
+
+static void preserve_pack(const char *file_path, const char *file_name,  const char *file_ext)
+{
+	char *fname_old;
+
+	fname_old = mkpathdup("%s/%s.old-%s", packdir, file_name, ++file_ext);
+	rename(file_path, fname_old);
+	free(fname_old);
+}
+
 static void remove_redundant_pack(const char *dir_name, const char *base_name)
 {
 	const char *exts[] = {".pack", ".idx", ".keep", ".bitmap"};
@@ -121,7 +176,10 @@ static void remove_redundant_pack(const char *dir_name, const char *base_name)
 	for (i = 0; i < ARRAY_SIZE(exts); i++) {
 		strbuf_setlen(&buf, plen);
 		strbuf_addstr(&buf, exts[i]);
-		unlink(buf.buf);
+		if (preserve_and_prune)
+			preserve_pack(buf.buf, base_name, exts[i]);
+		else
+			unlink(buf.buf);
 	}
 	strbuf_release(&buf);
 }
@@ -194,6 +252,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 				N_("maximum size of each packfile")),
 		OPT_BOOL(0, "pack-kept-objects", &pack_kept_objects,
 				N_("repack objects in packs marked with .keep")),
+		OPT_BOOL(0, "preserve-and-prune", &preserve_and_prune,
+				N_("preserve old pack files by renaming them instead of deleting, prune any "
+						"previously preserved pack files before preserving new ones")),
 		OPT_END()
 	};
 
@@ -404,6 +465,9 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
 
 	/* End of pack replacement. */
 
+	if (preserve_and_prune)
+		remove_preserved_packs();
+
 	if (delete_redundant) {
 		int opts = 0;
 		string_list_sort(&names);
-- 
2.12.0.191.ga15447d9f
Next: Junio C Hamano
Message 1 of 6 in “repack: Add option to preserve and prune old pack files”
  1. repack: Add option to preserve and prune old pack filesJames Melvin, Mar 10, 2017
  2. Junio C HamanoMar 10, 2017
  3. Jeff KingMar 12, 2017
  4. Junio C HamanoMar 12, 2017
  5. Martin FickMar 13, 2017
  6. Ævar Arnfjörð BjarmasonMar 11, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.