git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 4/5] everything_local: use "quick" object existence check

From
Jeff King <peff@peff.net>
Date
Nov 20, 2017, 20:29 UTC
Message-ID
<20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net>
In-Reply-To
<20171120202607.tf2pvegqe35mhxjs@sigill.intra.peff.net>

In b495697b82 (fetch-pack: avoid repeatedly re-scanning pack directory, 2013-01-26), we noticed that everything_local() could waste time trying to find and parse objects which we _expect_ to be missing. The solution was to put has_sha1_file() in front of parse_object() to skip the more-expensive parse attempt.

That optimization was negated later when has_sha1_file() learned to do the same re-scan in 45e8a74873 (has_sha1_file: re-check pack directory before giving up, 2013-08-30).

We can restore it by using the "quick" flag to tell has_sha1_file (actually has_object_file these days) that we prefer speed to thoroughness for this call. See also the fixes in 5827a0354 and 0eeb077be7 for prior art and discussion on using the "quick" flag for these cases.

The recently-added performance regression test in p5551 demonstrates the problem. You can see the original fix:

  Test            b495697b82^       b495697b82
  --------------------------------------------------------
  5551.4: fetch   1.68(1.33+0.35)   0.87(0.69+0.18) -48.2%
and then the regression:
  Test            45e8a74873^       45e8a74873
  ---------------------------------------------------------
  5551.4: fetch   0.96(0.77+0.19)   2.55(2.04+0.50) +165.6%
and now our fix:
  Test            HEAD^             HEAD
  --------------------------------------------------------
  5551.4: fetch   7.21(6.58+0.63)   5.47(5.04+0.43) -24.1%

You can also see that other things have gotten a lot slower since 2013. We'll deal with those in separate patches.

Signed-off-by: Jeff King <peff@peff.net>
---
 fetch-pack.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/fetch-pack.c b/fetch-pack.c
index 008b25d3db..9f6b07ad91 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -716,7 +716,8 @@ static int everything_local(struct fetch_pack_args *args,
 	for (ref = *refs; ref; ref = ref->next) {
 		struct object *o;
 
-		if (!has_object_file(&ref->old_oid))
+		if (!has_object_file_with_flags(&ref->old_oid,
+						OBJECT_INFO_QUICK))
 			continue;
 
 		o = parse_object(&ref->old_oid);
-- 
2.15.0.494.g79a8547723
Previous: Jeff KingNext: Jeff King
Message 11 of 25 in “avoiding pointless pack-directory re-scans”
  1. 0/5 avoiding pointless pack-directory re-scansJeff King, Nov 20, 2017
  2. 1/5 p5550: factor our nonsense-pack creationJeff King, Nov 20, 2017
  3. Eric SunshineNov 20, 2017
  4. Jeff KingNov 21, 2017
  5. Stefan BellerNov 22, 2017
  6. Jeff KingNov 22, 2017
  7. Junio C HamanoNov 23, 2017
  8. Jeff KingNov 23, 2017
  9. 2/5 t/perf/lib-pack: use fast-import checkpoint to create packsJeff King, Nov 20, 2017
  10. 3/5 p5551: add a script to test fetch pack-dir rescansJeff King, Nov 20, 2017
  11. 4/5 everything_local: use "quick" object existence checkJeff King, Nov 20, 2017
  12. 5/5 sha1_file: don't re-scan pack directory for null sha1Jeff King, Nov 20, 2017
  13. Stefan BellerNov 20, 2017
  14. Jeff KingNov 20, 2017
  15. Junio C HamanoNov 21, 2017
  16. Jeff KingNov 21, 2017
  17. Junio C HamanoNov 22, 2017
  18. Jeff KingNov 22, 2017
  19. Junio C HamanoNov 23, 2017
  20. Jeff KingNov 24, 2017
  21. Junio C HamanoNov 25, 2017
  22. Junio C HamanoNov 21, 2017
  23. Jeff KingNov 21, 2017
  24. Junio C HamanoNov 22, 2017
  25. Jeff KingNov 22, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.