git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Question: .idx without .pack causes performance issues?

From
Junio C Hamano <gitster@pobox.com>
Date
Aug 7, 2015, 22:27 UTC
Message-ID
<xmqqwpx6wx74.fsf@gitster.dls.corp.google.com>
In-Reply-To
<CAEtYS8SGnFFHM5BFzAo+Z2BzUGbp47AibA3v6qm_uEboRmfaNQ@mail.gmail.com>
Doug Kelly <dougk.ff7@gmail.com> writes:
Show 8 quoted lines
> So, I think you're right: prune would need to set report_garbage
> appropriately, then call count-objects to clean that up.  If we wanted
> it to *only* care for lone idx files, we would have to string match on
> the message (seems fragile), but perhaps a more observant approach
> would be to add a custom flag to prune to clean *all* garbage in the
> repository, as passed to report_garbage?  Probably wouldn't want to be
> enabled by default, but only on invocation or with careful
> consideration and setting an appropriate config flag.
I was thinking along this line.

Then you would set "report_garbage" to your own function, call prepare_packed_git(), and in your report-garbagte function, collect paths with seen_bits set exactly to PACKDIR_FILE_IDX. By the time prepare_packed_git() returns, you would have a list of paths only with .idx but without .pack, which you can prune.

We can later start pruning other garbage, but one step at a time.
-- >8 --
Subject: prepare_packed_git(): refactor garbage reporting in pack directory

The hook to report "garbage" files in $GIT_OBJECT_DIRECTORY/pack/ could be generic but is too specific to count-object's needs.

Move the part to produce human-readable messages to count-objects, and refine the interface to callback with the "bits" with values defined in the cache.h header file, so that other callers (e.g. prune) can later use the same mechanism to enumerate different kinds of garbage files and do something intelligent about them, other than reporting in textual messages.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 builtin/count-objects.c | 26 ++++++++++++++++++++++++--
 cache.h                 |  7 +++++--
 path.c                  |  2 +-
 sha1_file.c             | 23 ++++++-----------------
 4 files changed, 36 insertions(+), 22 deletions(-)
diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index ad0c799..4c3198e 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -15,9 +15,31 @@ static int verbose;
 static unsigned long loose, packed, packed_loose;
 static off_t loose_size;
 
-static void real_report_garbage(const char *desc, const char *path)
+const char *bits_to_msg(unsigned seen_bits)
+{
+	switch (seen_bits) {
+	case 0:
+		return "no corresponding .idx or .pack";
+	case PACKDIR_FILE_GARBAGE:
+		return "garbage found";
+	case PACKDIR_FILE_PACK:
+		return "no corresponding .idx";
+	case PACKDIR_FILE_IDX:
+		return "no corresponding .pack";
+	case PACKDIR_FILE_PACK|PACKDIR_FILE_IDX:
+	default:
+		return NULL;
+	}
+}
+
+static void real_report_garbage(unsigned seen_bits, const char *path)
 {
 	struct stat st;
+	const char *desc = bits_to_msg(seen_bits);
+
+	if (!desc)
+		return;
+
 	if (!stat(path, &st))
 		size_garbage += st.st_size;
 	warning("%s: %s", desc, path);
@@ -27,7 +49,7 @@ static void real_report_garbage(const char *desc, const char *path)
 static void loose_garbage(const char *path)
 {
 	if (verbose)
-		report_garbage("garbage found", path);
+		report_garbage(PACKDIR_FILE_GARBAGE, path);
 }
 
 static int count_loose(const unsigned char *sha1, const char *path, void *data)
diff --git a/cache.h b/cache.h
index 6bb7119..2d4dedc 100644
--- a/cache.h
+++ b/cache.h
@@ -1212,8 +1212,11 @@ struct pack_entry {
 
 extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
 
-/* A hook for count-objects to report invalid files in pack directory */
-extern void (*report_garbage)(const char *desc, const char *path);
+/* A hook to report invalid files in pack directory */
+#define PACKDIR_FILE_PACK 1
+#define PACKDIR_FILE_IDX 2
+#define PACKDIR_FILE_GARBAGE 4
+extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
 extern void prepare_packed_git(void);
 extern void reprepare_packed_git(void);
diff --git a/path.c b/path.c
index 10f4cbf..75ec236 100644
--- a/path.c
+++ b/path.c
@@ -143,7 +143,7 @@ void report_linked_checkout_garbage(void)
 		strbuf_setlen(&sb, len);
 		strbuf_addstr(&sb, path);
 		if (file_exists(sb.buf))
-			report_garbage("unused in linked checkout", sb.buf);
+			report_garbage(PACKDIR_FILE_GARBAGE, sb.buf);
 	}
 	strbuf_release(&sb);
 }
diff --git a/sha1_file.c b/sha1_file.c
index 1cee438..0c0b652 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1183,27 +1183,16 @@ void install_packed_git(struct packed_git *pack)
 	packed_git = pack;
 }
 
-void (*report_garbage)(const char *desc, const char *path);
+void (*report_garbage)(unsigned seen_bits, const char *path);
 
 static void report_helper(const struct string_list *list,
 			  int seen_bits, int first, int last)
 {
-	const char *msg;
-	switch (seen_bits) {
-	case 0:
-		msg = "no corresponding .idx or .pack";
-		break;
-	case 1:
-		msg = "no corresponding .idx";
-		break;
-	case 2:
-		msg = "no corresponding .pack";
-		break;
-	default:
+	if (seen_bits == (PACKDIR_FILE_PACK|PACKDIR_FILE_IDX))
 		return;
-	}
+
 	for (; first < last; first++)
-		report_garbage(msg, list->items[first].string);
+		report_garbage(seen_bits, list->items[first].string);
 }
 
 static void report_pack_garbage(struct string_list *list)
@@ -1226,7 +1215,7 @@ static void report_pack_garbage(struct string_list *list)
 		if (baselen == -1) {
 			const char *dot = strrchr(path, '.');
 			if (!dot) {
-				report_garbage("garbage found", path);
+				report_garbage(PACKDIR_FILE_GARBAGE, path);
 				continue;
 			}
 			baselen = dot - path + 1;
@@ -1298,7 +1287,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    ends_with(de->d_name, ".keep"))
 			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path.buf);
+			report_garbage(PACKDIR_FILE_GARBAGE, path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
Previous: Doug KellyNext: Doug Kelly
Message 9 of 34 in “Question: .idx without .pack causes performance issues?”
  1. Doug KellyJul 21, 2015
  2. Junio C HamanoJul 21, 2015
  3. Junio C HamanoJul 21, 2015
  4. Junio C HamanoJul 21, 2015
  5. Doug KellyJul 21, 2015
  6. Doug KellyAug 3, 2015
  7. Junio C HamanoAug 4, 2015
  8. Doug KellyAug 7, 2015
  9. Junio C HamanoAug 7, 2015
  10. 1/2 prepare_packed_git(): refactor garbage reporting in pack directoryDoug Kelly, Aug 13, 2015
  11. 2/2 gc: Remove garbage .idx files from pack dirDoug Kelly, Aug 13, 2015
  12. Junio C HamanoAug 17, 2015
  13. Junio C HamanoAug 17, 2015
  14. Eric SunshineAug 13, 2015
  15. Junio C HamanoAug 17, 2015
  16. Junio C HamanoOct 28, 2015
  17. Doug KellyOct 28, 2015
  18. 1/3 prepare_packed_git(): refactor garbage reporting in pack directoryDoug Kelly, Nov 4, 2015
  19. 2/3 t5304: Add test for cleaning pack garbageDoug Kelly, Nov 4, 2015
  20. 3/3 gc: Remove garbage .idx files from pack dirDoug Kelly, Nov 4, 2015
  21. Doug KellyNov 4, 2015
  22. Junio C HamanoNov 4, 2015
  23. Doug KellyNov 4, 2015
  24. Jeff KingNov 4, 2015
  25. Doug KellyNov 4, 2015
  26. Jeff KingNov 4, 2015
  27. Jeff KingDec 30, 2015
  28. Doug KellyJan 13, 2016
  29. Junio C HamanoJan 13, 2016
  30. Doug KellyJan 13, 2016
  31. Jeff KingJan 13, 2016
  32. Jeff KingNov 4, 2015
  33. Doug KellyJul 21, 2015
  34. Fwd: Question: .idx without .pack causes performance issues?Thomas Berg, Nov 11, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.