git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 7/7] cat-file: add --batch-all-objects option

From
Jeff King <peff@peff.net>
Date
Jun 22, 2015, 10:45 UTC
Message-ID
<20150622104559.GG14475@peff.net>
In-Reply-To
<20150622103321.GB12584@peff.net>

It can sometimes be useful to examine all objects in the repository. Normally this is done with "git rev-list --all --objects", but:

  1. That shows only reachable objects. You may want to look
     at all available objects.
  2. It's slow. We actually open each object to walk the
     graph. If your operation is OK with seeing unreachable
     objects, it's an order of magnitude faster to just
     enumerate the loose directories and pack indices.

You can do this yourself using "ls" and "git show-index", but it's non-obvious. This patch adds an option to "cat-file --batch-check" to operate on all available objects (rather than reading names from stdin).

This is based on a proposal by Charles Bailey to provide a separate "git list-all-objects" command. That is more orthogonal, as it splits enumerating the objects from getting information about them. However, in practice you will either:

  a. Feed the list of objects directly into cat-file anyway,
     so you can find out information about them. Keeping it
     in a single process is more efficient.
  b. Ask the listing process to start telling you more
     information about the objects, in which case you will
     reinvent cat-file's batch-check formatter.

Adding a cat-file option is simple and efficient. And if you really do want just the object names, you can always do:

  git cat-file --batch-check='%(objectname)' --batch-all-objects
Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-cat-file.txt |  8 ++++++++
 builtin/cat-file.c             | 44 ++++++++++++++++++++++++++++++++++++++++--
 t/t1006-cat-file.sh            | 27 ++++++++++++++++++++++++++
 3 files changed, 77 insertions(+), 2 deletions(-)
diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt
index 0058bd4..6831b08 100644
--- a/Documentation/git-cat-file.txt
+++ b/Documentation/git-cat-file.txt
@@ -69,6 +69,14 @@ OPTIONS
 	not be combined with any other options or arguments.  See the
 	section `BATCH OUTPUT` below for details.
 
+--batch-all-objects::
+	Instead of reading a list of objects on stdin, perform the
+	requested batch operation on all objects in the repository and
+	any alternate object stores (not just reachable objects).
+	Requires `--batch` or `--batch-check` be specified. Note that
+	the order of the objects is unspecified, and there may be
+	duplicate entries.
+
 --buffer::
 	Normally batch output is flushed after each object is output, so
 	that a process can interactively read and write from
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 499ccda..95604c4 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -15,6 +15,7 @@ struct batch_options {
 	int follow_symlinks;
 	int print_contents;
 	int buffer_output;
+	int all_objects;
 	const char *format;
 };
 
@@ -257,7 +258,7 @@ static void batch_object_write(const char *obj_name, struct batch_options *opt,
 	struct strbuf buf = STRBUF_INIT;
 
 	if (sha1_object_info_extended(data->sha1, &data->info, LOOKUP_REPLACE_OBJECT) < 0) {
-		printf("%s missing\n", obj_name);
+		printf("%s missing\n", obj_name ? obj_name : sha1_to_hex(data->sha1));
 		fflush(stdout);
 		return;
 	}
@@ -318,6 +319,34 @@ static void batch_one_object(const char *obj_name, struct batch_options *opt,
 	batch_object_write(obj_name, opt, data);
 }
 
+struct object_cb_data {
+	struct batch_options *opt;
+	struct expand_data *expand;
+};
+
+static int batch_object_cb(const unsigned char *sha1,
+			   struct object_cb_data *data)
+{
+	hashcpy(data->expand->sha1, sha1);
+	batch_object_write(NULL, data->opt, data->expand);
+	return 0;
+}
+
+static int batch_loose_object(const unsigned char *sha1,
+			      const char *path,
+			      void *data)
+{
+	return batch_object_cb(sha1, data);
+}
+
+static int batch_packed_object(const unsigned char *sha1,
+			       struct packed_git *pack,
+			       uint32_t pos,
+			       void *data)
+{
+	return batch_object_cb(sha1, data);
+}
+
 static int batch_objects(struct batch_options *opt)
 {
 	struct strbuf buf = STRBUF_INIT;
@@ -345,6 +374,15 @@ static int batch_objects(struct batch_options *opt)
 	if (opt->print_contents)
 		data.info.typep = &data.type;
 
+	if (opt->all_objects) {
+		struct object_cb_data cb;
+		cb.opt = opt;
+		cb.expand = &data;
+		for_each_loose_object(batch_loose_object, &cb, 0);
+		for_each_packed_object(batch_packed_object, &cb, 0);
+		return 0;
+	}
+
 	/*
 	 * We are going to call get_sha1 on a potentially very large number of
 	 * objects. In most large cases, these will be actual object sha1s. The
@@ -436,6 +474,8 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)
 			PARSE_OPT_OPTARG, batch_option_callback },
 		OPT_BOOL(0, "follow-symlinks", &batch.follow_symlinks,
 			 N_("follow in-tree symlinks (used with --batch or --batch-check)")),
+		OPT_BOOL(0, "batch-all-objects", &batch.all_objects,
+			 N_("show all objects with --batch or --batch-check")),
 		OPT_END()
 	};
 
@@ -460,7 +500,7 @@ int cmd_cat_file(int argc, const char **argv, const char *prefix)
 		usage_with_options(cat_file_usage, options);
 	}
 
-	if (batch.follow_symlinks && !batch.enabled) {
+	if ((batch.follow_symlinks || batch.all_objects) && !batch.enabled) {
 		usage_with_options(cat_file_usage, options);
 	}
 
diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh
index 93a4794..2b4220a 100755
--- a/t/t1006-cat-file.sh
+++ b/t/t1006-cat-file.sh
@@ -547,4 +547,31 @@ test_expect_success 'git cat-file --batch --follow-symlink returns correct sha a
 	test_cmp expect actual
 '
 
+test_expect_success 'cat-file --batch-all-objects shows all objects' '
+	# make new repos so we now the full set of objects; we will
+	# also make sure that there are some packed and some loose
+	# objects, some referenced and some not, and that there are
+	# some available only via alternates.
+	git init all-one &&
+	(
+		cd all-one &&
+		echo content >file &&
+		git add file &&
+		git commit -qm base &&
+		git rev-parse HEAD HEAD^{tree} HEAD:file &&
+		git repack -ad &&
+		echo not-cloned | git hash-object -w --stdin
+	) >expect.unsorted &&
+	git clone -s all-one all-two &&
+	(
+		cd all-two &&
+		echo local-unref | git hash-object -w --stdin
+	) >>expect.unsorted &&
+	sort <expect.unsorted >expect &&
+	git -C all-two cat-file --batch-all-objects \
+				--batch-check="%(objectname)" >actual.unsorted &&
+	sort <actual.unsorted >actual &&
+	test_cmp expect actual
+'
+
 test_done

-- 
2.4.4.719.g3984bc6
Previous: Jeff KingNext: Eric Sunshine
Message 38 of 51 in “Improvements to parse-options and a new filter-objects command”
  1. Charles BaileyJun 19, 2015
  2. 1/3 Correct test-parse-options to handle negative intsCharles Bailey, Jun 19, 2015
  3. Junio C HamanoJun 19, 2015
  4. 2/3 Move unsigned long option parsing out of pack-objects.cCharles Bailey, Jun 19, 2015
  5. Remi Galan AlfonsoJun 19, 2015
  6. Charles BaileyJun 19, 2015
  7. Junio C HamanoJun 19, 2015
  8. Junio C HamanoJun 19, 2015
  9. Jakub NarębskiJun 20, 2015
  10. Jakub NarębskiJun 19, 2015
  11. Charles BaileyJun 20, 2015
  12. Junio C HamanoJun 20, 2015
  13. 3/3 Add filter-objects commandCharles Bailey, Jun 19, 2015
  14. Jeff KingJun 19, 2015
  15. Charles BaileyJun 19, 2015
  16. Jeff KingJun 19, 2015
  17. Junio C HamanoJun 19, 2015
  18. John KeepingJun 19, 2015
  19. Charles BaileyJun 19, 2015
  20. Improvements to integer option parsingCharles Bailey, Jun 21, 2015
  21. 1/2 Correct test-parse-options to handle negative intsCharles Bailey, Jun 21, 2015
  22. 2/2 Move unsigned long option parsing out of pack-objects.cCharles Bailey, Jun 21, 2015
  23. Charles BaileyJun 21, 2015
  24. Junio C HamanoJun 22, 2015
  25. Junio C HamanoJun 22, 2015
  26. Junio C HamanoJun 22, 2015
  27. Charles BaileyJun 22, 2015
  28. Fast enumeration of objectsCharles Bailey, Jun 21, 2015
  29. Add list-all-objects commandCharles Bailey, Jun 21, 2015
  30. Jeff KingJun 22, 2015
  31. Jeff KingJun 22, 2015
  32. 1/7 for_each_packed_object: automatically open pack indexJeff King, Jun 22, 2015
  33. 2/7 cat-file: minor style fix in options listJeff King, Jun 22, 2015
  34. 3/7 cat-file: move batch_options definition to top of fileJeff King, Jun 22, 2015
  35. 4/7 cat-file: add --buffer optionJeff King, Jun 22, 2015
  36. 5/7 cat-file: stop returning value from batch_one_objectJeff King, Jun 22, 2015
  37. 6/7 cat-file: split batch_one_object into two stagesJeff King, Jun 22, 2015
  38. 7/7 cat-file: add --batch-all-objects optionJeff King, Jun 22, 2015
  39. Eric SunshineJun 26, 2015
  40. Jeff KingJun 26, 2015
  41. 8/7 cat-file: sort and de-dup output of --batch-all-objectsJeff King, Jun 22, 2015
  42. Charles BaileyJun 22, 2015
  43. Jeff KingJun 22, 2015
  44. Charles BaileyJun 22, 2015
  45. Junio C HamanoJun 22, 2015
  46. Jeff KingJun 22, 2015
  47. Charles BaileyJun 22, 2015
  48. Duy NguyenJun 22, 2015
  49. Jeff KingJun 22, 2015
  50. Jeff KingJun 22, 2015
  51. Junio C HamanoJun 22, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.