git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 11/16] rev-list: add bitmap mode to speed up lists

From
Vicent Marti <tanoku@gmail.com>
Date
Jun 24, 2013, 23:23 UTC
Message-ID
<1372116193-32762-12-git-send-email-tanoku@gmail.com>
In-Reply-To
<1372116193-32762-1-git-send-email-tanoku@gmail.com>

The bitmap reachability index used to speed up the counting objects phase during `pack-objects` can also be used to optimize a normal rev-list if the only thing required are the SHA1s of the objects during the list.

Calling `git rev-list --use-bitmaps [committish]` is the equivalent of `git rev-list --objects`, but the rev list is performed based on a bitmap result instead of using a manual counting objects phase.

These are some example timings for `torvalds/linux`:
	$ time ../git/git rev-list --objects master > /dev/null
	real    0m25.567s
	user    0m25.148s
	sys     0m0.384s
	$ time ../git/git rev-list --use-bitmaps master > /dev/null
	real    0m0.393s
	user    0m0.356s
	sys     0m0.036s
Additionally, a `--test-bitmap` flag has been added that will perform
the same rev-list manually (i.e. using a normal revwalk) and using
bitmaps, and verify that the results are the same.
---
 builtin/rev-list.c |   28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)
diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index 67701be..905ed08 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -3,6 +3,8 @@
 #include "diff.h"
 #include "revision.h"
 #include "list-objects.h"
+#include "pack.h"
+#include "pack-bitmap.h"
 #include "builtin.h"
 #include "log-tree.h"
 #include "graph.h"
@@ -256,6 +258,17 @@ static int show_bisect_vars(struct rev_list_info *info, int reaches, int all)
 	return 0;
 }
 
+static int show_object_fast(
+	const unsigned char *sha1,
+	enum object_type type,
+	uint32_t hash, int exclude,
+	struct packed_git *found_pack,
+	off_t found_offset)
+{
+	fprintf(stdout, "%ss\n", sha1_to_hex(sha1));
+	return 1;
+}
+
 int cmd_rev_list(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
@@ -264,6 +277,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)
 	int bisect_list = 0;
 	int bisect_show_vars = 0;
 	int bisect_find_all = 0;
+	int use_bitmaps = 0;
 
 	git_config(git_default_config, NULL);
 	init_revisions(&revs, prefix);
@@ -305,8 +319,15 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)
 			bisect_show_vars = 1;
 			continue;
 		}
+		if (!strcmp(arg, "--use-bitmaps")) {
+			use_bitmaps = 1;
+			continue;
+		}
+		if (!strcmp(arg, "--test-bitmap")) {
+			test_bitmap_walk(&revs);
+			return 0;
+		}
 		usage(rev_list_usage);
-
 	}
 	if (revs.commit_format != CMIT_FMT_UNSPECIFIED) {
 		/* The command line has a --pretty  */
@@ -332,6 +353,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)
 	if (bisect_list)
 		revs.limited = 1;
 
+	if (use_bitmaps && !prepare_bitmap_walk(&revs, NULL)) {
+		traverse_bitmap_commit_list(&show_object_fast);
+		return 0;
+	}
+
 	if (prepare_revision_walk(&revs))
 		die("revision walk setup failed");
 	if (revs.tree_objects)
-- 
1.7.9.5
Previous: Vicent MartíNext: Thomas Rast
Message 52 of 64 in “Speed up Counting Objects with bitmap data”
  1. 00/16 Speed up Counting Objects with bitmap dataVicent Marti, Jun 24, 2013
  2. 01/16 list-objects: mark tree as unparsed when we free its bufferVicent Marti, Jun 24, 2013
  3. 02/16 sha1_file: refactor into `find_pack_object_pos`Vicent Marti, Jun 24, 2013
  4. Thomas RastJun 25, 2013
  5. 03/16 pack-objects: use a faster hash tableVicent Marti, Jun 24, 2013
  6. Thomas RastJun 25, 2013
  7. Jeff KingJun 26, 2013
  8. Jeff KingJun 26, 2013
  9. Ramkumar RamachandraJun 25, 2013
  10. Junio C HamanoJun 25, 2013
  11. Vicent MartíJun 25, 2013
  12. 04/16 pack-objects: make `pack_name_hash` globalVicent Marti, Jun 24, 2013
  13. 05/16 revision: allow setting custom limiter functionVicent Marti, Jun 24, 2013
  14. 06/16 sha1_file: export `git_open_noatime`Vicent Marti, Jun 24, 2013
  15. 07/16 compat: add endinanness helpersVicent Marti, Jun 24, 2013
  16. Peter KreftingJun 25, 2013
  17. Vicent MartíJun 25, 2013
  18. Peter KreftingJun 27, 2013
  19. 08/16 ewah: compressed bitmap implementationVicent Marti, Jun 24, 2013
  20. Junio C HamanoJun 25, 2013
  21. Junio C HamanoJun 25, 2013
  22. Thomas RastJun 25, 2013
  23. 09/16 documentation: add documentation for the bitmap formatVicent Marti, Jun 24, 2013
  24. Shawn PearceJun 25, 2013
  25. Vicent MartíJun 25, 2013
  26. Junio C HamanoJun 25, 2013
  27. Vicent MartíJun 25, 2013
  28. Shawn PearceJun 27, 2013
  29. Vicent MartíJun 27, 2013
  30. Jeff KingJun 27, 2013
  31. Shawn PearceJun 27, 2013
  32. Jeff KingJun 27, 2013
  33. Colby RangerJul 1, 2013
  34. Shawn PearceJul 1, 2013
  35. Jeff KingJul 7, 2013
  36. Shawn PearceJul 7, 2013
  37. Jeff KingJun 26, 2013
  38. Colby RangerJun 26, 2013
  39. Colby RangerJun 26, 2013
  40. Colby RangerJun 27, 2013
  41. Shawn PearceJun 27, 2013
  42. Shawn PearceJun 27, 2013
  43. Thomas RastJun 25, 2013
  44. Vicent MartíJun 25, 2013
  45. Thomas RastJun 26, 2013
  46. Thomas RastJun 26, 2013
  47. 10/16 pack-objects: use bitmaps when packing objectsVicent Marti, Jun 24, 2013
  48. Ramkumar RamachandraJun 25, 2013
  49. Thomas RastJun 25, 2013
  50. Junio C HamanoJun 25, 2013
  51. Vicent MartíJun 25, 2013
  52. 11/16 rev-list: add bitmap mode to speed up listsVicent Marti, Jun 24, 2013
  53. Thomas RastJun 25, 2013
  54. Vicent MartíJun 26, 2013
  55. Thomas RastJun 26, 2013
  56. Jeff KingJun 26, 2013
  57. 12/16 pack-objects: implement bitmap writingVicent Marti, Jun 24, 2013
  58. 13/16 repack: consider bitmaps when performing repacksVicent Marti, Jun 24, 2013
  59. Junio C HamanoJun 25, 2013
  60. Vicent MartíJun 25, 2013
  61. 14/16 sha1_file: implement `nth_packed_object_info`Vicent Marti, Jun 24, 2013
  62. 15/16 write-bitmap: implement new git command to write bitmapsVicent Marti, Jun 24, 2013
  63. 16/16 rev-list: Optimize --count using bitmaps tooVicent Marti, Jun 24, 2013
  64. Thomas RastJun 25, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.