git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2] rev-list: exclude promisor objects at walk time

From
Josh Steadmon <steadmon@google.com>
Date
Apr 4, 2019, 22:53 UTC
Message-ID
<9f327d6d8dc5e71eb0039aef3ac76ea16c2adab3.1554417917.git.steadmon@google.com>
In-Reply-To
<6de682d5e48186970644569586fc6613763d5caa.1554312374.git.steadmon@google.com>

For large repositories, enumerating the list of all promisor objects (in order to exclude them from a rev-list walk) can take a significant amount of time).

When --exclude-promisor-objects is passed to rev-list, don't enumerate the promisor objects. Instead, filter them (and any children objects) during the actual graph walk.

Remove the mark_uninteresting() function as it's not used anywhere else.
Helped-By: Jonathan Tan <jonathantanmy@google.com>
Helped-By: Jeff King <peff@peff.net>
Helped-By: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Josh Steadmon <steadmon@google.com>
---

Re-implemented following Jonathan & Jeff's advice (and also previously Jonathan Nieder's, although I didn't understand it at the time). Thanks for the feedback all.

 list-objects.c | 26 ++++++++++++++++++++++++++
 revision.c     | 16 ----------------
 2 files changed, 26 insertions(+), 16 deletions(-)
diff --git a/list-objects.c b/list-objects.c
index dc77361e11..d1eaa0999e 100644
--- a/list-objects.c
+++ b/list-objects.c
@@ -30,6 +30,7 @@ static void process_blob(struct traversal_context *ctx,
 	struct object *obj = &blob->object;
 	size_t pathlen;
 	enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;
+	struct object_info oi = OBJECT_INFO_INIT;
 
 	if (!ctx->revs->blob_objects)
 		return;
@@ -37,6 +38,11 @@ static void process_blob(struct traversal_context *ctx,
 		die("bad blob object");
 	if (obj->flags & (UNINTERESTING | SEEN))
 		return;
+	if (ctx->revs->exclude_promisor_objects &&
+	    !oid_object_info_extended(the_repository, &obj->oid, &oi, 0) &&
+	    oi.whence == OI_PACKED &&
+	    oi.u.packed.pack->pack_promisor)
+		return;
 
 	/*
 	 * Pre-filter known-missing objects when explicitly requested.
@@ -149,6 +155,7 @@ static void process_tree(struct traversal_context *ctx,
 	int baselen = base->len;
 	enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;
 	int failed_parse;
+	struct object_info oi = OBJECT_INFO_INIT;
 
 	if (!revs->tree_objects)
 		return;
@@ -156,6 +163,11 @@ static void process_tree(struct traversal_context *ctx,
 		die("bad tree object");
 	if (obj->flags & (UNINTERESTING | SEEN))
 		return;
+	if (ctx->revs->exclude_promisor_objects &&
+	    !oid_object_info_extended(the_repository, &obj->oid, &oi, 0) &&
+	    oi.whence == OI_PACKED &&
+	    oi.u.packed.pack->pack_promisor)
+		return;
 
 	failed_parse = parse_tree_gently(tree, 1);
 	if (failed_parse) {
@@ -318,6 +330,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,
 				     struct strbuf *base)
 {
 	int i;
+	struct object_info oi = OBJECT_INFO_INIT;
 
 	assert(base->len == 0);
 
@@ -326,6 +339,12 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,
 		struct object *obj = pending->item;
 		const char *name = pending->name;
 		const char *path = pending->path;
+		if (ctx->revs->exclude_promisor_objects &&
+		    !oid_object_info_extended(the_repository, &obj->oid, &oi, 0) &&
+		    oi.whence == OI_PACKED &&
+		    oi.u.packed.pack->pack_promisor)
+			continue;
+
 		if (obj->flags & (UNINTERESTING | SEEN))
 			continue;
 		if (obj->type == OBJ_TAG) {
@@ -353,9 +372,16 @@ static void do_traverse(struct traversal_context *ctx)
 {
 	struct commit *commit;
 	struct strbuf csp; /* callee's scratch pad */
+	struct object_info oi = OBJECT_INFO_INIT;
 	strbuf_init(&csp, PATH_MAX);
 
 	while ((commit = get_revision(ctx->revs)) != NULL) {
+		if (ctx->revs->exclude_promisor_objects &&
+		    !oid_object_info_extended(the_repository, &commit->object.oid, &oi, 0) &&
+		    oi.whence == OI_PACKED &&
+		    oi.u.packed.pack->pack_promisor)
+			continue;
+
 		/*
 		 * an uninteresting boundary commit may not have its tree
 		 * parsed yet, but we are not going to show them anyway
diff --git a/revision.c b/revision.c
index eb8e51bc63..85974e941d 100644
--- a/revision.c
+++ b/revision.c
@@ -3067,17 +3067,6 @@ void reset_revision_walk(void)
 	clear_object_flags(SEEN | ADDED | SHOWN);
 }
 
-static int mark_uninteresting(const struct object_id *oid,
-			      struct packed_git *pack,
-			      uint32_t pos,
-			      void *cb)
-{
-	struct rev_info *revs = cb;
-	struct object *o = parse_object(revs->repo, oid);
-	o->flags |= UNINTERESTING | SEEN;
-	return 0;
-}
-
 define_commit_slab(indegree_slab, int);
 define_commit_slab(author_date_slab, timestamp_t);
 
@@ -3316,11 +3305,6 @@ int prepare_revision_walk(struct rev_info *revs)
 	    (revs->limited && limiting_can_increase_treesame(revs)))
 		revs->treesame.name = "treesame";
 
-	if (revs->exclude_promisor_objects) {
-		for_each_packed_object(mark_uninteresting, revs,
-				       FOR_EACH_OBJECT_PROMISOR_ONLY);
-	}
-
 	if (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)
 		commit_list_sort_by_date(&revs->commits);
 	if (revs->no_walk)

-- 
2.21.0.392.gf8f6787159e-goog
Previous: Jeff KingNext: Jeff King
Message 7 of 27 in “clone: do faster object check for partial clones”
  1. clone: do faster object check for partial clonesJosh Steadmon, Apr 3, 2019
  2. Jonathan TanApr 3, 2019
  3. Jeff KingApr 3, 2019
  4. Jonathan TanApr 3, 2019
  5. Josh SteadmonApr 4, 2019
  6. Jeff KingApr 4, 2019
  7. rev-list: exclude promisor objects at walk timeJosh Steadmon, Apr 4, 2019
  8. Jeff KingApr 4, 2019
  9. Josh SteadmonApr 4, 2019
  10. Jeff KingApr 5, 2019
  11. Josh SteadmonApr 5, 2019
  12. Josh SteadmonApr 8, 2019
  13. rev-list: exclude promisor objects at walk timeJosh Steadmon, Apr 8, 2019
  14. Christian CouderApr 8, 2019
  15. Josh SteadmonApr 8, 2019
  16. Junio C HamanoApr 9, 2019
  17. Jeff KingApr 9, 2019
  18. Junio C HamanoApr 9, 2019
  19. Josh SteadmonApr 9, 2019
  20. SZEDER GáborApr 9, 2019
  21. Josh SteadmonApr 9, 2019
  22. Jeff KingApr 11, 2019
  23. Josh SteadmonApr 12, 2019
  24. Jeff KingApr 13, 2019
  25. Josh SteadmonApr 19, 2019
  26. clone: do faster object check for partial clonesJosh Steadmon, Apr 19, 2019
  27. Jeff KingApr 22, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.