git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 7/7] diff: avoid useless filespec population

From
Jeff King <peff@peff.net>
Date
Apr 2, 2010, 00:14 UTC
Message-ID
<20100402001424.GG16462@coredump.intra.peff.net>
In-Reply-To
<20100402000159.GA15101@coredump.intra.peff.net>

builtin_diff calls fill_mmfile fairly early, which in turn calls diff_populate_filespec, which actually retrieves the file's blob contents into a buffer. Long ago, this was sensible as we would need to look at the blobs eventually.

These days, however, we may not ever want those blobs if we end up using a textconv cache, and for large binary files (exactly the sort for which you might have a textconv cache), just retrieving the objects can be costly.

This patch just pushes the fill_mmfile call a bit later, so we can avoid populating the filespec in some cases. There is one thing to note that looks like a bug but isn't. We push the fill_mmfile down into the first branch of a conditional. It seems like we would need it on the other branch, too, but we don't; fill_textconv does it for us (in fact, before this, we were just writing over the results of the fill_mmfile on that branch).

Here's a timing sample on a commit with 45 changed jpgs and avis. The result is fully textconv cached, but we still wasted a lot of time just pulling the blobs from storage. The total size of the blobs (source and dest) is about 180M.

  [before]
  $ time git show >/dev/null
  real    0m0.352s
  user    0m0.148s
  sys     0m0.200s
  [after]
  $ time git show >/dev/null
  real    0m0.009s
  user    0m0.004s
  sys     0m0.004s

And that's on a warm cache. On a cold cache, the "after" case is not much worse, but the "before" case has to do an extra 180M of I/O.

Signed-off-by: Jeff King <peff@peff.net>
---
I remember when I first started using git having it go so fast that I
had to double-check that it had actually worked correctly. I had another
of those moments when I ran this. But it correctly processed everything
I threw at it. More stress-testing appreciated. :)
 diff.c |    9 ++++-----
 1 files changed, 4 insertions(+), 5 deletions(-)
diff --git a/diff.c b/diff.c
index 72d8503..4cb6d9a 100644
--- a/diff.c
+++ b/diff.c
@@ -1680,12 +1680,11 @@ static void builtin_diff(const char *name_a,
 		}
 	}
 
-	if (fill_mmfile(&mf1, one) < 0 || fill_mmfile(&mf2, two) < 0)
-		die("unable to read files to diff");
-
 	if (!DIFF_OPT_TST(o, TEXT) &&
-	    ( (diff_filespec_is_binary(one) && !textconv_one) ||
-	      (diff_filespec_is_binary(two) && !textconv_two) )) {
+	    ( (!textconv_one && diff_filespec_is_binary(one)) ||
+	      (!textconv_two && diff_filespec_is_binary(two)) )) {
+		if (fill_mmfile(&mf1, one) < 0 || fill_mmfile(&mf2, two) < 0)
+			die("unable to read files to diff");
 		/* Quite common confusing case */
 		if (mf1.size == mf2.size &&
 		    !memcmp(mf1.ptr, mf2.ptr, mf1.size))
-- 
1.7.0.4.299.gba9d4
Previous: Jeff KingNext: Junio C Hamano
Message 10 of 13 in “textconv caching”
  1. 0/7 textconv cachingJeff King, Apr 2, 2010
  2. 1/7 fix const-correctness of write_sha1_fileJeff King, Apr 2, 2010
  3. 2/7 fix textconv leak in emit_rewrite_diffJeff King, Apr 2, 2010
  4. 3/7 make commit_tree a library functionJeff King, Apr 2, 2010
  5. 4/7 introduce notes-cache interfaceJeff King, Apr 2, 2010
  6. 5/7 textconv: refactor calls to run_textconvJeff King, Apr 2, 2010
  7. 6/7 diff: cache textconv outputJeff King, Apr 2, 2010
  8. Junio C HamanoApr 2, 2010
  9. Jeff KingApr 2, 2010
  10. 7/7 diff: avoid useless filespec populationJeff King, Apr 2, 2010
  11. Junio C HamanoApr 2, 2010
  12. Jeff KingApr 2, 2010
  13. Jeff KingApr 2, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.