git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 6/6] New --dirstat=lines mode, doing dirstat analysis based on diffstat

From
Johan Herland <johan@herland.net>
Date
Apr 26, 2011, 00:01 UTC
Message-ID
<1303776102-9085-7-git-send-email-johan@herland.net>
In-Reply-To
<1303776102-9085-1-git-send-email-johan@herland.net>

This patch adds an alternative implementation of show_dirstat(), called show_dirstat_by_line(), which uses the more expensive diffstat analysis (as opposed to show_dirstat()'s own (relatively inexpensive) analysis) to derive the numbers from which the --dirstat output is computed.

The alternative implementation is controlled by the new "lines" argument to the --dirstat option (or the diff.dirstat config variable).

In linux-2.6.git, running the three different --dirstat modes:
  time git diff v2.6.20..v2.6.30 --dirstat=changes > /dev/null
vs.
  time git diff v2.6.20..v2.6.30 --dirstat=lines > /dev/null
vs.
  time git diff v2.6.20..v2.6.30 --dirstat=files > /dev/null
yields the following average runtimes on my machine:
- "changes" (default): ~6.0 s
- "lines":             ~9.6 s
- "files":             ~0.1 s

So, as expected, there's a considerable performance hit (~60%) by going through the full diffstat analysis as compared to the default "changes" analysis (obviously, "files" is much faster than both). As such, the "lines" mode is probably only useful if you really need the --dirstat numbers to be consistent with the numbers returned from the other --*stat options.

The patch also includes documentation and tests for the new dirstat mode.
Signed-off-by: Johan Herland <johan@herland.net>
---
 Documentation/config.txt       |    7 +++
 Documentation/diff-options.txt |    7 +++
 diff.c                         |   60 +++++++++++++++++++++++-
 diff.h                         |    1 +
 t/t4046-diff-dirstat.sh        |  100 ++++++++++++++++++++++++++++++++++++++++
 5 files changed, 173 insertions(+), 2 deletions(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 10fa89a..47b2423 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -837,6 +837,13 @@ diff.dirstat::
 	the amount of pure code movements within a file.  In other words,
 	rearranging lines in a file is not counted as much as other changes.
 	This is the default `--dirstat` behavior.
+`lines`;;
+	Compute the dirstat numbers by doing the regular line-based diff
+	analysis, and summing the removed/added line counts. This is a more
+	expensive `--dirstat` behavior than the `changes` behavior, but it
+	does count rearranged lines within a file as much as other changes.
+	The resulting output is consistent with what you get from the other
+	`--*stat` options.
 `files`;;
 	Compute the dirstat numbers by counting the number of files changed.
 	Each changed file counts equally in the dirstat analysis. This is
diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
index b6b1448..0b7417b 100644
--- a/Documentation/diff-options.txt
+++ b/Documentation/diff-options.txt
@@ -80,6 +80,13 @@ endif::git-format-patch[]
 	the amount of pure code movements within a file.  In other words,
 	rearranging lines in a file is not counted as much as other changes.
 	This is the default `--dirstat` behavior.
+`lines`;;
+	Compute the dirstat numbers by doing the regular line-based diff
+	analysis, and summing the removed/added line counts. This is a more
+	expensive `--dirstat` behavior than the `changes` behavior, but it
+	does count rearranged lines within a file as much as other changes.
+	The resulting output is consistent with what you get from the other
+	`--*stat` options.
 `files`;;
 	Compute the dirstat numbers by counting the number of files changed.
 	Each changed file counts equally in the dirstat analysis. This is
diff --git a/diff.c b/diff.c
index 4da3b68..c00984f 100644
--- a/diff.c
+++ b/diff.c
@@ -131,10 +131,17 @@ static void dirstat_opt_args(struct diff_options *options, const char *args)
 	while (*p) {
 		if (!prefixcmp(p, "changes")) {
 			p += 7;
+			DIFF_OPT_CLR(options, DIRSTAT_BY_LINE);
+			DIFF_OPT_CLR(options, DIRSTAT_BY_FILE);
+		}
+		else if (!prefixcmp(p, "lines")) {
+			p += 5;
+			DIFF_OPT_SET(options, DIRSTAT_BY_LINE);
 			DIFF_OPT_CLR(options, DIRSTAT_BY_FILE);
 		}
 		else if (!prefixcmp(p, "files")) {
 			p += 5;
+			DIFF_OPT_CLR(options, DIRSTAT_BY_LINE);
 			DIFF_OPT_SET(options, DIRSTAT_BY_FILE);
 		}
 		else if (!prefixcmp(p, "noncumulative")) {
@@ -1677,6 +1684,48 @@ found_damage:
 	gather_dirstat(options, &dir, changed, "", 0);
 }
 
+static void show_dirstat_by_line(struct diffstat_t *data, struct diff_options *options)
+{
+	int i;
+	unsigned long changed;
+	struct dirstat_dir dir;
+
+	if (data->nr == 0)
+		return;
+
+	dir.files = NULL;
+	dir.alloc = 0;
+	dir.nr = 0;
+	dir.percent = options->dirstat_percent;
+	dir.cumulative = DIFF_OPT_TST(options, DIRSTAT_CUMULATIVE);
+
+	changed = 0;
+	for (i = 0; i < data->nr; i++) {
+		struct diffstat_file *file = data->files[i];
+		unsigned long damage = file->added + file->deleted;
+		if (damage && file->is_binary)
+			/*
+			 * binary files counts bytes, not lines. Must find some
+			 * way to normalize binary bytes vs. textual lines.
+			 * The following heuristic is cheap, but beyond ugly...
+			 */
+			damage = damage < 52 ? 1 : damage / 52;
+		ALLOC_GROW(dir.files, dir.nr + 1, dir.alloc);
+		dir.files[dir.nr].name = file->name;
+		dir.files[dir.nr].changed = damage;
+		changed += damage;
+		dir.nr++;
+	}
+
+	/* This can happen even with many files, if everything was renames */
+	if (!changed)
+		return;
+
+	/* Show all directories with more than x% of the changes */
+	qsort(dir.files, dir.nr, sizeof(dir.files[0]), dirstat_compare);
+	gather_dirstat(options, &dir, changed, "", 0);
+}
+
 static void free_diffstat_info(struct diffstat_t *diffstat)
 {
 	int i;
@@ -4081,6 +4130,7 @@ void diff_flush(struct diff_options *options)
 	struct diff_queue_struct *q = &diff_queued_diff;
 	int i, output_format = options->output_format;
 	int separator = 0;
+	int dirstat_by_line = 0;
 
 	/*
 	 * Order: raw, stat, summary, patch
@@ -4101,7 +4151,11 @@ void diff_flush(struct diff_options *options)
 		separator++;
 	}
 
-	if (output_format & (DIFF_FORMAT_DIFFSTAT|DIFF_FORMAT_SHORTSTAT|DIFF_FORMAT_NUMSTAT)) {
+	if (output_format & DIFF_FORMAT_DIRSTAT && DIFF_OPT_TST(options, DIRSTAT_BY_LINE))
+		dirstat_by_line = 1;
+
+	if (output_format & (DIFF_FORMAT_DIFFSTAT|DIFF_FORMAT_SHORTSTAT|DIFF_FORMAT_NUMSTAT) ||
+	    dirstat_by_line) {
 		struct diffstat_t diffstat;
 
 		memset(&diffstat, 0, sizeof(struct diffstat_t));
@@ -4116,10 +4170,12 @@ void diff_flush(struct diff_options *options)
 			show_stats(&diffstat, options);
 		if (output_format & DIFF_FORMAT_SHORTSTAT)
 			show_shortstats(&diffstat, options);
+		if (output_format & DIFF_FORMAT_DIRSTAT)
+			show_dirstat_by_line(&diffstat, options);
 		free_diffstat_info(&diffstat);
 		separator++;
 	}
-	if (output_format & DIFF_FORMAT_DIRSTAT)
+	if ((output_format & DIFF_FORMAT_DIRSTAT) && !dirstat_by_line)
 		show_dirstat(options);
 
 	if (output_format & DIFF_FORMAT_SUMMARY && !is_summary_empty(q)) {
diff --git a/diff.h b/diff.h
index 781c620..5f12049 100644
--- a/diff.h
+++ b/diff.h
@@ -78,6 +78,7 @@ typedef struct strbuf *(*diff_prefix_fn_t)(struct diff_options *opt, void *data)
 #define DIFF_OPT_IGNORE_UNTRACKED_IN_SUBMODULES (1 << 25)
 #define DIFF_OPT_IGNORE_DIRTY_SUBMODULES (1 << 26)
 #define DIFF_OPT_OVERRIDE_SUBMODULE_CONFIG (1 << 27)
+#define DIFF_OPT_DIRSTAT_BY_LINE     (1 << 28)
 
 #define DIFF_OPT_TST(opts, flag)    ((opts)->flags & DIFF_OPT_##flag)
 #define DIFF_OPT_SET(opts, flag)    ((opts)->flags |= DIFF_OPT_##flag)
diff --git a/t/t4046-diff-dirstat.sh b/t/t4046-diff-dirstat.sh
index da4484c..ef9326a 100755
--- a/t/t4046-diff-dirstat.sh
+++ b/t/t4046-diff-dirstat.sh
@@ -770,4 +770,104 @@ test_expect_success 'diff.dirstat=16.7,cumulative,files' '
 	test_cmp expect_diff_dirstat_CC actual_diff_dirstat_CC
 '
 
+cat <<EOF >expect_diff_dirstat
+  10.6% dst/copy/changed/
+  10.6% dst/copy/rearranged/
+  10.6% dst/copy/unchanged/
+  10.6% dst/move/changed/
+  10.6% dst/move/rearranged/
+  10.6% dst/move/unchanged/
+  10.6% src/move/changed/
+  10.6% src/move/rearranged/
+  10.6% src/move/unchanged/
+EOF
+
+cat <<EOF >expect_diff_dirstat_M
+   5.3% changed/
+  26.3% dst/copy/changed/
+  26.3% dst/copy/rearranged/
+  26.3% dst/copy/unchanged/
+   5.3% dst/move/changed/
+   5.3% dst/move/rearranged/
+   5.3% rearranged/
+EOF
+
+cat <<EOF >expect_diff_dirstat_CC
+  16.7% changed/
+  16.7% dst/copy/changed/
+  16.7% dst/copy/rearranged/
+  16.7% dst/move/changed/
+  16.7% dst/move/rearranged/
+  16.7% rearranged/
+EOF
+
+test_expect_success '--dirstat=lines' '
+	git diff --dirstat=lines HEAD^..HEAD >actual_diff_dirstat &&
+	test_cmp expect_diff_dirstat actual_diff_dirstat &&
+	git diff --dirstat=lines -M HEAD^..HEAD >actual_diff_dirstat_M &&
+	test_cmp expect_diff_dirstat_M actual_diff_dirstat_M &&
+	git diff --dirstat=lines -C -C HEAD^..HEAD >actual_diff_dirstat_CC &&
+	test_cmp expect_diff_dirstat_CC actual_diff_dirstat_CC
+'
+
+test_expect_success 'diff.dirstat=lines' '
+	git -c diff.dirstat=lines diff --dirstat HEAD^..HEAD >actual_diff_dirstat &&
+	test_cmp expect_diff_dirstat actual_diff_dirstat &&
+	git -c diff.dirstat=lines diff --dirstat -M HEAD^..HEAD >actual_diff_dirstat_M &&
+	test_cmp expect_diff_dirstat_M actual_diff_dirstat_M &&
+	git -c diff.dirstat=lines diff --dirstat -C -C HEAD^..HEAD >actual_diff_dirstat_CC &&
+	test_cmp expect_diff_dirstat_CC actual_diff_dirstat_CC
+'
+
+cat <<EOF >expect_diff_dirstat
+   2.1% changed/
+  10.6% dst/copy/changed/
+  10.6% dst/copy/rearranged/
+  10.6% dst/copy/unchanged/
+  10.6% dst/move/changed/
+  10.6% dst/move/rearranged/
+  10.6% dst/move/unchanged/
+   2.1% rearranged/
+  10.6% src/move/changed/
+  10.6% src/move/rearranged/
+  10.6% src/move/unchanged/
+EOF
+
+cat <<EOF >expect_diff_dirstat_M
+   5.3% changed/
+  26.3% dst/copy/changed/
+  26.3% dst/copy/rearranged/
+  26.3% dst/copy/unchanged/
+   5.3% dst/move/changed/
+   5.3% dst/move/rearranged/
+   5.3% rearranged/
+EOF
+
+cat <<EOF >expect_diff_dirstat_CC
+  16.7% changed/
+  16.7% dst/copy/changed/
+  16.7% dst/copy/rearranged/
+  16.7% dst/move/changed/
+  16.7% dst/move/rearranged/
+  16.7% rearranged/
+EOF
+
+test_expect_success '--dirstat=lines,0' '
+	git diff --dirstat=lines,0 HEAD^..HEAD >actual_diff_dirstat &&
+	test_cmp expect_diff_dirstat actual_diff_dirstat &&
+	git diff --dirstat=lines,0 -M HEAD^..HEAD >actual_diff_dirstat_M &&
+	test_cmp expect_diff_dirstat_M actual_diff_dirstat_M &&
+	git diff --dirstat=lines,0 -C -C HEAD^..HEAD >actual_diff_dirstat_CC &&
+	test_cmp expect_diff_dirstat_CC actual_diff_dirstat_CC
+'
+
+test_expect_success 'diff.dirstat=0,lines' '
+	git -c diff.dirstat=0,lines diff --dirstat HEAD^..HEAD >actual_diff_dirstat &&
+	test_cmp expect_diff_dirstat actual_diff_dirstat &&
+	git -c diff.dirstat=0,lines diff --dirstat -M HEAD^..HEAD >actual_diff_dirstat_M &&
+	test_cmp expect_diff_dirstat_M actual_diff_dirstat_M &&
+	git -c diff.dirstat=0,lines diff --dirstat -C -C HEAD^..HEAD >actual_diff_dirstat_CC &&
+	test_cmp expect_diff_dirstat_CC actual_diff_dirstat_CC
+'
+
 test_done

-- 
1.7.5.rc1.3.g4d7b
Previous: Junio C HamanoNext: Junio C Hamano
Message 47 of 91 in “BUG? in --dirstat when rearranging lines in a file”
  1. Johan HerlandApr 7, 2011
  2. Linus TorvaldsApr 7, 2011
  3. Junio C HamanoApr 7, 2011
  4. Linus TorvaldsApr 7, 2011
  5. Johan HerlandApr 8, 2011
  6. 1/3 --dirstat: Document shortcomings compared to --stat or regular diffJohan Herland, Apr 8, 2011
  7. Junio C HamanoApr 8, 2011
  8. 2/3 --dirstat-by-file: Make it faster and more correctJohan Herland, Apr 8, 2011
  9. 3/3 Teach --dirstat to not completely ignore rearranged linesJohan Herland, Apr 8, 2011
  10. Linus TorvaldsApr 8, 2011
  11. Junio C HamanoApr 8, 2011
  12. 0/3 --dirstat fixesJohan Herland, Apr 10, 2011
  13. 1/3 --dirstat: Describe non-obvious differences relative to --stat or regular diffJohan Herland, Apr 10, 2011
  14. 2/3 --dirstat-by-file: Make it faster and more correctJohan Herland, Apr 10, 2011
  15. Junio C HamanoApr 11, 2011
  16. 3/3 Teach --dirstat to not completely ignore rearranged lines within a fileJohan Herland, Apr 10, 2011
  17. Junio C HamanoApr 11, 2011
  18. Johan HerlandApr 11, 2011
  19. Junio C HamanoApr 11, 2011
  20. Johan HerlandApr 12, 2011
  21. 4/3 --dirstat: In case of renames, use target filename instead of source filenameJohan Herland, Apr 12, 2011
  22. Linus TorvaldsApr 12, 2011
  23. 5/3 Alternative --dirstat implementation, based on diffstat analysisJohan Herland, Apr 12, 2011
  24. Linus TorvaldsApr 12, 2011
  25. Linus TorvaldsApr 12, 2011
  26. Johan HerlandApr 12, 2011
  27. Linus TorvaldsApr 12, 2011
  28. Junio C HamanoApr 12, 2011
  29. 0/6 --dirstat fixes, part 2Johan Herland, Apr 26, 2011
  30. 1/6 Add several testcases for --dirstat and friendsJohan Herland, Apr 26, 2011
  31. 2/6 Make --dirstat=0 output directories that contribute < 0.1% of changesJohan Herland, Apr 26, 2011
  32. 3/6 Refactor --dirstat parsing; deprecate --cumulative and --dirstat-by-fileJohan Herland, Apr 26, 2011
  33. Junio C HamanoApr 26, 2011
  34. Johan HerlandApr 27, 2011
  35. Junio C HamanoApr 27, 2011
  36. Junio C HamanoApr 27, 2011
  37. Junio C HamanoApr 27, 2011
  38. 4/6 Add config variable for specifying default --dirstat behaviorJohan Herland, Apr 26, 2011
  39. Junio C HamanoApr 26, 2011
  40. Johan HerlandApr 27, 2011
  41. 5/6 Use floating point for --dirstat percentagesJohan Herland, Apr 26, 2011
  42. Junio C HamanoApr 26, 2011
  43. Johan HerlandApr 27, 2011
  44. Junio C HamanoApr 27, 2011
  45. Linus TorvaldsApr 27, 2011
  46. Junio C HamanoApr 27, 2011
  47. 6/6 New --dirstat=lines mode, doing dirstat analysis based on diffstatJohan Herland, Apr 26, 2011
  48. Junio C HamanoApr 26, 2011
  49. Johan HerlandApr 27, 2011
  50. Linus TorvaldsApr 26, 2011
  51. 0/6 --dirstat fixes, part 2Johan Herland, Apr 27, 2011
  52. 1/6 Add several testcases for --dirstat and friendsJohan Herland, Apr 27, 2011
  53. 2/6 Make --dirstat=0 output directories that contribute < 0.1% of changesJohan Herland, Apr 27, 2011
  54. 3/6 Refactor --dirstat parsing; deprecate --cumulative and --dirstat-by-fileJohan Herland, Apr 27, 2011
  55. 4/6 Add config variable for specifying default --dirstat behaviorJohan Herland, Apr 27, 2011
  56. 5/6 Use floating point for --dirstat percentagesJohan Herland, Apr 27, 2011
  57. Linus TorvaldsApr 27, 2011
  58. 6/6 New --dirstat=lines mode, doing dirstat analysis based on diffstatJohan Herland, Apr 27, 2011
  59. 0/6 --dirstat fixes, part 2Johan Herland, Apr 27, 2011
  60. 1/6 Add several testcases for --dirstat and friendsJohan Herland, Apr 27, 2011
  61. 2/6 Make --dirstat=0 output directories that contribute < 0.1% of changesJohan Herland, Apr 27, 2011
  62. 3/6 Refactor --dirstat parsing; deprecate --cumulative and --dirstat-by-fileJohan Herland, Apr 27, 2011
  63. 4/6 Add config variable for specifying default --dirstat behaviorJohan Herland, Apr 27, 2011
  64. 5/6 Allow specifying --dirstat cut-off percentage as a floating point numberJohan Herland, Apr 27, 2011
  65. Linus TorvaldsApr 27, 2011
  66. 5/6 Allow specifying --dirstat cut-off percentage as a floating point numberJohan Herland, Apr 27, 2011
  67. 6/6 New --dirstat=lines mode, doing dirstat analysis based on diffstatJohan Herland, Apr 27, 2011
  68. 0/7 --dirstat fixes, part 2Johan Herland, Apr 28, 2011
  69. 1/7 Add several testcases for --dirstat and friendsJohan Herland, Apr 28, 2011
  70. 2/7 Make --dirstat=0 output directories that contribute < 0.1% of changesJohan Herland, Apr 28, 2011
  71. 3/7 Refactor --dirstat parsing; deprecate --cumulative and --dirstat-by-fileJohan Herland, Apr 28, 2011
  72. 4/7 Add config variable for specifying default --dirstat behaviorJohan Herland, Apr 28, 2011
  73. 5/7 Allow specifying --dirstat cut-off percentage as a floating point numberJohan Herland, Apr 28, 2011
  74. 6/7 New --dirstat=lines mode, doing dirstat analysis based on diffstatJohan Herland, Apr 28, 2011
  75. 7/7 Improve error handling when parsing dirstat parametersJohan Herland, Apr 28, 2011
  76. Junio C HamanoApr 28, 2011
  77. Junio C HamanoApr 28, 2011
  78. Johan HerlandApr 28, 2011
  79. Johan HerlandApr 28, 2011
  80. Junio C HamanoApr 29, 2011
  81. 0/8 --dirstat fixes, part 2Johan Herland, Apr 29, 2011
  82. 1/8 Add several testcases for --dirstat and friendsJohan Herland, Apr 29, 2011
  83. 2/8 Make --dirstat=0 output directories that contribute < 0.1% of changesJohan Herland, Apr 29, 2011
  84. 3/8 Refactor --dirstat parsing; deprecate --cumulative and --dirstat-by-fileJohan Herland, Apr 29, 2011
  85. 4/8 Add config variable for specifying default --dirstat behaviorJohan Herland, Apr 29, 2011
  86. 5/8 Allow specifying --dirstat cut-off percentage as a floating point numberJohan Herland, Apr 29, 2011
  87. 6/8 New --dirstat=lines mode, doing dirstat analysis based on diffstatJohan Herland, Apr 29, 2011
  88. 7/8 Improve error handling when parsing dirstat parametersJohan Herland, Apr 29, 2011
  89. 8/8 Mark dirstat error messages for translationJohan Herland, Apr 29, 2011
  90. Junio C HamanoApr 12, 2011
  91. Linus TorvaldsApr 10, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.