patch, 3 partsrevision: use priority queue for streaming walks

4 messages between May 27, 2026 and May 27, 2026, from Kristofer Karlsson via GitGitGadget.

Plain Markdown or JSON for tools and agents. Diffs are folded; open one to read it.

Kristofer Karlsson via GitGitGadgetMay 27, 2026, 15:49 UTC on lore

This is a follow-up to kk/limit-list-optim (now in master), which replaced the O(N) sorted linked-list insertion in limit_list() with a priority queue. In the review thread for that patch, I mentioned that the same approach could be applied to the streaming (non-limited) walk in get_revision_1(). Junio suggested doing it as a separate topic, and Peff noted he had a local branch (jk/revs-commits-prio-queue) doing a broader conversion of revs->commits to prio_queue entirely.

This series takes a lighter-weight approach: it keeps the linked list for setup and external callers, and adds a separate commit_queue field that the streaming walk drains into on first use. This avoids touching bisect, line-log, and list-objects code, at the cost of a "only one should be non-empty" invariant between the two fields.

Together with the limit_list() change already in master, this eliminates all commit_list_insert_by_date() callers from revision.c.

Patch 1 is a small leak fix -- a missing release_revisions() call in pack-objects that becomes visible once the commit queue uses a dynamically allocated prio_queue array.

Patch 2 introduces a rev_walk_mode enum to replace the repeated if/else chains in get_revision_1(). The function dispatches on walk mode in multiple places (next commit, expand parents, flag clearing) and these chains must stay in sync. The enum resolves the mode once and both dispatch sites switch on the same variable. This is a lighter alternative to the vtable-based refactoring I mentioned before. No functional change.

Patch 3 is the actual conversion of the streaming walk to use a priority queue.

== Why this helps ==

The streaming walk in get_revision_1() inserts newly discovered parent commits into a date-sorted queue. On master, this uses commit_list_insert_by_date(), which walks the linked list to find the insertion point -- O(w) per insert, where w is the queue width (active walk frontier).

In merge-heavy repositories, the walk frontier stays wide:

Repository Commits Peak width Avg width =======================================

monorepo (2.4M) 2,420K 2,653 1,700 linux.git 1,445K 581 235 git.git 82K 188 82

On the monorepo, each of the 2.4M commits requires scanning an average of 1,700 list entries to find the insertion point. With the priority queue, this drops to ~11 heap comparisons.

== Benchmarks ==

All benchmarks: best of 3 runs, same machine, commit-graph present.

Streaming walks (affected by this series):

git rev-list --count HEAD (monorepo, 2.4M commits)

  master:   17.94s
  patched:   3.38s   (5.3x faster)

git rev-list HEAD (monorepo, full output)

  master:   27.72s
  patched:   8.61s   (2.8x faster, I/O-bound fraction unchanged)

Regression checks -- non-merge-heavy repos (streaming path, but frontier stays narrow so O(w) insertion was never the bottleneck):

git rev-list --count HEAD (linux.git, 1.4M commits)

  master:    1.76s
  patched:   1.81s   (no change)

git rev-list HEAD (linux.git, full output)

  master:    4.46s
  patched:   4.52s   (no change)

git rev-list --count HEAD (git.git, 82K commits)

  master:     83ms
  patched:    86ms   (no change)

Regression checks -- other walk modes (not affected by this series):

git rev-list --count HEAD~5000...HEAD (monorepo, limited path)

  master:    7.36s
  patched:   7.02s   (no change)

== Profile breakdown ==

perf profiling of rev-list --count HEAD on the monorepo shows where the time goes:

master (17.94s): commit_list_insert_by_date 79% 14.25s fixed overhead (parse/lookup) 21% 3.69s

patched (3.38s): heap ops (compare + sift) 16% 0.53s fixed overhead (parse/lookup) 84% 2.85s

The queue maintenance itself sped up 27x (14.25s to 0.53s). The overall 5.3x is lower because the fixed costs -- object lookup (17%), commit-graph parsing (14%), memory allocation (10%) -- are roughly constant between the two versions at ~3s.

This means the patch removes the dominant bottleneck entirely. After the patch, the walk cost is dominated by irreducible per-commit work (parsing and object lookup) which scales linearly with commit count regardless of frontier width.

Kristofer Karlsson (3):
  pack-objects: call release_revisions() after cruft traversal
  revision: introduce rev_walk_mode to clarify get_revision_1()
  revision: use priority queue for non-limited streaming walks

 builtin/pack-objects.c |   1 +
 commit.c               |  13 -----
 commit.h               |   2 -
 revision.c             | 113 +++++++++++++++++++++++++++--------------
 revision.h             |  12 ++++-
 5 files changed, 88 insertions(+), 53 deletions(-)

base-commit: c69baaf57ba26cf117c2b6793802877f19738b0d
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2127%2Fspkrka%2Fstreaming-prio-queue-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2127/spkrka/streaming-prio-queue-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2127

-- 
gitgitgadget

Kristofer Karlsson via GitGitGadgetMay 27, 2026, 15:50 UTC in reply to Kristofer Karlsson via GitGitGadget on lore

[PATCH 1/3] pack-objects: call release_revisions() after cruft traversal

From: Kristofer Karlsson <krka@spotify.com>

enumerate_and_traverse_cruft_objects() initializes a rev_info on the stack but never calls release_revisions() afterwards. This is not visible on master but becomes a leak once the revision walking machinery uses dynamically allocated structures.

Add the missing release_revisions() call.

Signed-off-by: Kristofer Karlsson <krka@spotify.com>
---
 builtin/pack-objects.c | 1 +
 1 file changed, 1 insertion(+)

Show changes to builtin/pack-objects.c +1 −0

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 480cc0bd8c..67025e8625 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -4275,6 +4275,7 @@ static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs
 	traverse_commit_list(&revs, show_cruft_commit, show_cruft_object, NULL);
 
 	stop_progress(&progress_state);
+	release_revisions(&revs);
 }
 
 static void read_cruft_objects(void)

-- 
gitgitgadget

Kristofer Karlsson via GitGitGadgetMay 27, 2026, 15:50 UTC in reply to Kristofer Karlsson via GitGitGadget on lore

[PATCH 2/3] revision: introduce rev_walk_mode to clarify get_revision_1()

From: Kristofer Karlsson <krka@spotify.com>

get_revision_1() dispatches to different walk strategies based on a combination of rev_info flags: reflog_info, topo_walk_info, and limited. These conditions are checked in multiple places within the function -- once to select the next commit, and again to decide how to expand parents -- and the two chains must stay in sync.

Extract the mode selection into a rev_walk_mode enum and a small get_walk_mode() helper, resolved once at the top of get_revision_1(). Both dispatch sites now switch on the same mode variable, making it obvious that they agree and easier to verify that all modes are handled.

No functional change.

Signed-off-by: Kristofer Karlsson <krka@spotify.com>
---
 revision.c | 62 ++++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 48 insertions(+), 14 deletions(-)

Show changes to revision.c +48 −14

diff --git a/revision.c b/revision.c
index e1970b9c5d..9d0fc696d0 100644
--- a/revision.c
+++ b/revision.c
@@ -4327,22 +4327,48 @@ static void track_linear(struct rev_info *revs, struct commit *commit)
 	revs->previous_parents = commit_list_copy(commit->parents);
 }
 
+enum rev_walk_mode {
+	REV_WALK_REFLOG,
+	REV_WALK_TOPO,
+	REV_WALK_LIMITED,
+	REV_WALK_STREAMING,
+};
+
+static enum rev_walk_mode get_walk_mode(struct rev_info *revs)
+{
+	if (revs->reflog_info)
+		return REV_WALK_REFLOG;
+	if (revs->topo_walk_info)
+		return REV_WALK_TOPO;
+	if (revs->limited)
+		return REV_WALK_LIMITED;
+	return REV_WALK_STREAMING;
+}
+
 static struct commit *get_revision_1(struct rev_info *revs)
 {
+	enum rev_walk_mode mode = get_walk_mode(revs);
+
 	while (1) {
 		struct commit *commit;
 
-		if (revs->reflog_info)
+		switch (mode) {
+		case REV_WALK_REFLOG:
 			commit = next_reflog_entry(revs->reflog_info);
-		else if (revs->topo_walk_info)
+			break;
+		case REV_WALK_TOPO:
 			commit = next_topo_commit(revs);
-		else
+			break;
+		case REV_WALK_LIMITED:
+		case REV_WALK_STREAMING:
 			commit = pop_commit(&revs->commits);
+			break;
+		}
 
 		if (!commit)
 			return NULL;
 
-		if (revs->reflog_info)
+		if (mode == REV_WALK_REFLOG)
 			commit->object.flags &= ~(ADDED | SEEN | SHOWN);
 
 		/*
@@ -4350,20 +4376,28 @@ static struct commit *get_revision_1(struct rev_info *revs)
 		 * the parents here. We also need to do the date-based limiting
 		 * that we'd otherwise have done in limit_list().
 		 */
-		if (!revs->limited) {
-			if (revs->max_age != -1 &&
-			    comparison_date(revs, commit) < revs->max_age)
-				continue;
+		if (mode != REV_WALK_LIMITED &&
+		    revs->max_age != -1 &&
+		    comparison_date(revs, commit) < revs->max_age)
+			continue;
 
-			if (revs->reflog_info)
-				try_to_simplify_commit(revs, commit);
-			else if (revs->topo_walk_info)
-				expand_topo_walk(revs, commit);
-			else if (process_parents(revs, commit, &revs->commits, NULL) < 0) {
+		switch (mode) {
+		case REV_WALK_REFLOG:
+			try_to_simplify_commit(revs, commit);
+			break;
+		case REV_WALK_TOPO:
+			expand_topo_walk(revs, commit);
+			break;
+		case REV_WALK_STREAMING:
+			if (process_parents(revs, commit,
+					    &revs->commits, NULL) < 0) {
 				if (!revs->ignore_missing_links)
 					die("Failed to traverse parents of commit %s",
-						oid_to_hex(&commit->object.oid));
+					    oid_to_hex(&commit->object.oid));
 			}
+			break;
+		case REV_WALK_LIMITED:
+			break;
 		}
 
 		switch (simplify_commit(revs, commit)) {

-- 
gitgitgadget

Kristofer Karlsson via GitGitGadgetMay 27, 2026, 15:50 UTC in reply to Kristofer Karlsson via GitGitGadget on lore

[PATCH 3/3] revision: use priority queue for non-limited streaming walks

From: Kristofer Karlsson <krka@spotify.com>

The streaming (non-limited) walk in get_revision_1() inserts newly discovered parent commits into a date-sorted queue via commit_list_insert_by_date(), which scans the linked list to find the insertion point -- O(w) per insert, where w is the width of the active walk frontier. Replace this with an O(log w) priority queue.

Add a commit_queue field to rev_info alongside the existing commits linked list. The two representations are mutually exclusive: setup and external callers that need list access use the linked list, then get_revision_1() lazily drains it into the priority queue on first call. Add a REV_WALK_NO_WALK enum value to distinguish the no_walk case (which still uses the commit list) from the streaming case.

The conversion function rev_info_commit_list_to_queue() is public so callers that know they will iterate can convert early.

Combined with the limit_list() priority queue change already in master, this eliminates all O(w) sorted linked-list insertion from the revision walk machinery.

Signed-off-by: Kristofer Karlsson <krka@spotify.com>
---
 commit.c   | 13 -------------
 commit.h   |  2 --
 revision.c | 55 +++++++++++++++++++++++++++++-------------------------
 revision.h | 12 +++++++++++-
 4 files changed, 41 insertions(+), 41 deletions(-)

Show changes to 4 files +41 −41

commit.c, commit.h, revision.c, revision.h

diff --git a/commit.c b/commit.c
index e3e7352e69..5112c7b2af 100644
--- a/commit.c
+++ b/commit.c
@@ -729,19 +729,6 @@ void commit_list_free(struct commit_list *list)
 		pop_commit(&list);
 }
 
-struct commit_list * commit_list_insert_by_date(struct commit *item, struct commit_list **list)
-{
-	struct commit_list **pp = list;
-	struct commit_list *p;
-	while ((p = *pp) != NULL) {
-		if (p->item->date < item->date) {
-			break;
-		}
-		pp = &p->next;
-	}
-	return commit_list_insert(item, pp);
-}
-
 static int commit_list_compare_by_date(const struct commit_list *a,
 				       const struct commit_list *b)
 {
diff --git a/commit.h b/commit.h
index 58150045af..385492fbb1 100644
--- a/commit.h
+++ b/commit.h
@@ -191,8 +191,6 @@ int commit_list_contains(struct commit *item,
 struct commit_list **commit_list_append(struct commit *commit,
 					struct commit_list **next);
 unsigned commit_list_count(const struct commit_list *l);
-struct commit_list *commit_list_insert_by_date(struct commit *item,
-				    struct commit_list **list);
 void commit_list_sort_by_date(struct commit_list **list);
 
 /* Shallow copy of the input list */
diff --git a/revision.c b/revision.c
index 9d0fc696d0..4bb3b16e43 100644
--- a/revision.c
+++ b/revision.c
@@ -1116,7 +1116,7 @@ static void try_to_simplify_commit(struct rev_info *revs, struct commit *commit)
 }
 
 static int process_parents(struct rev_info *revs, struct commit *commit,
-			   struct commit_list **list, struct prio_queue *queue)
+			   struct prio_queue *queue)
 {
 	struct commit_list *parent = commit->parents;
 	unsigned pass_flags;
@@ -1158,8 +1158,6 @@ static int process_parents(struct rev_info *revs, struct commit *commit,
 			if (p->object.flags & SEEN)
 				continue;
 			p->object.flags |= (SEEN | NOT_USER_GIVEN);
-			if (list)
-				commit_list_insert_by_date(p, list);
 			if (queue)
 				prio_queue_put(queue, p);
 			if (revs->exclude_first_parent_only)
@@ -1207,8 +1205,6 @@ static int process_parents(struct rev_info *revs, struct commit *commit,
 		p->object.flags |= pass_flags | CHILD_VISITED;
 		if (!(p->object.flags & SEEN)) {
 			p->object.flags |= (SEEN | NOT_USER_GIVEN);
-			if (list)
-				commit_list_insert_by_date(p, list);
 			if (queue)
 				prio_queue_put(queue, p);
 		}
@@ -1470,7 +1466,7 @@ static int limit_list(struct rev_info *revs)
 
 		if (revs->max_age != -1 && (commit->date < revs->max_age))
 			obj->flags |= UNINTERESTING;
-		if (process_parents(revs, commit, NULL, &queue) < 0) {
+		if (process_parents(revs, commit, &queue) < 0) {
 			clear_prio_queue(&queue);
 			return -1;
 		}
@@ -3257,6 +3253,7 @@ static void free_void_commit_list(void *list)
 void release_revisions(struct rev_info *revs)
 {
 	commit_list_free(revs->commits);
+	clear_prio_queue(&revs->commit_queue);
 	commit_list_free(revs->ancestry_path_bottoms);
 	release_display_notes(&revs->notes_opt);
 	object_array_clear(&revs->pending);
@@ -3726,7 +3723,7 @@ static void explore_walk_step(struct rev_info *revs)
 	if (revs->max_age != -1 && (c->date < revs->max_age))
 		c->object.flags |= UNINTERESTING;
 
-	if (process_parents(revs, c, NULL, NULL) < 0)
+	if (process_parents(revs, c, NULL) < 0)
 		return;
 
 	if (c->object.flags & UNINTERESTING)
@@ -3902,7 +3899,7 @@ static void expand_topo_walk(struct rev_info *revs, struct commit *commit)
 {
 	struct commit_list *p;
 	struct topo_walk_info *info = revs->topo_walk_info;
-	if (process_parents(revs, commit, NULL, NULL) < 0) {
+	if (process_parents(revs, commit, NULL) < 0) {
 		if (!revs->ignore_missing_links)
 			die("Failed to traverse parents of commit %s",
 			    oid_to_hex(&commit->object.oid));
@@ -3938,6 +3935,13 @@ static void expand_topo_walk(struct rev_info *revs, struct commit *commit)
 	}
 }
 
+void rev_info_commit_list_to_queue(struct rev_info *revs)
+{
+	while (revs->commits)
+		prio_queue_put(&revs->commit_queue, pop_commit(&revs->commits));
+}
+
+
 int prepare_revision_walk(struct rev_info *revs)
 {
 	int i;
@@ -4006,7 +4010,7 @@ static enum rewrite_result rewrite_one_1(struct rev_info *revs,
 	for (;;) {
 		struct commit *p = *pp;
 		if (!revs->limited)
-			if (process_parents(revs, p, NULL, queue) < 0)
+			if (process_parents(revs, p, queue) < 0)
 				return rewrite_one_error;
 		if (p->object.flags & UNINTERESTING)
 			return rewrite_one_ok;
@@ -4020,27 +4024,18 @@ static enum rewrite_result rewrite_one_1(struct rev_info *revs,
 	}
 }
 
-static void merge_queue_into_list(struct prio_queue *q, struct commit_list **list)
+static void merge_queue_into_prio_queue(struct prio_queue *from,
+					struct prio_queue *to)
 {
-	while (q->nr) {
-		struct commit *item = prio_queue_peek(q);
-		struct commit_list *p = *list;
-
-		if (p && p->item->date >= item->date)
-			list = &p->next;
-		else {
-			p = commit_list_insert(item, list);
-			list = &p->next; /* skip newly added item */
-			prio_queue_get(q); /* pop item */
-		}
-	}
+	while (from->nr)
+		prio_queue_put(to, prio_queue_get(from));
 }
 
 static enum rewrite_result rewrite_one(struct rev_info *revs, struct commit **pp)
 {
 	struct prio_queue queue = { compare_commits_by_commit_date };
 	enum rewrite_result ret = rewrite_one_1(revs, pp, &queue);
-	merge_queue_into_list(&queue, &revs->commits);
+	merge_queue_into_prio_queue(&queue, &revs->commit_queue);
 	clear_prio_queue(&queue);
 	return ret;
 }
@@ -4331,6 +4326,7 @@ enum rev_walk_mode {
 	REV_WALK_REFLOG,
 	REV_WALK_TOPO,
 	REV_WALK_LIMITED,
+	REV_WALK_NO_WALK,
 	REV_WALK_STREAMING,
 };
 
@@ -4342,6 +4338,8 @@ static enum rev_walk_mode get_walk_mode(struct rev_info *revs)
 		return REV_WALK_TOPO;
 	if (revs->limited)
 		return REV_WALK_LIMITED;
+	if (revs->no_walk)
+		return REV_WALK_NO_WALK;
 	return REV_WALK_STREAMING;
 }
 
@@ -4349,6 +4347,9 @@ static struct commit *get_revision_1(struct rev_info *revs)
 {
 	enum rev_walk_mode mode = get_walk_mode(revs);
 
+	if (mode == REV_WALK_STREAMING && revs->commits)
+		rev_info_commit_list_to_queue(revs);
+
 	while (1) {
 		struct commit *commit;
 
@@ -4360,9 +4361,12 @@ static struct commit *get_revision_1(struct rev_info *revs)
 			commit = next_topo_commit(revs);
 			break;
 		case REV_WALK_LIMITED:
-		case REV_WALK_STREAMING:
+		case REV_WALK_NO_WALK:
 			commit = pop_commit(&revs->commits);
 			break;
+		case REV_WALK_STREAMING:
+			commit = prio_queue_get(&revs->commit_queue);
+			break;
 		}
 
 		if (!commit)
@@ -4390,12 +4394,13 @@ static struct commit *get_revision_1(struct rev_info *revs)
 			break;
 		case REV_WALK_STREAMING:
 			if (process_parents(revs, commit,
-					    &revs->commits, NULL) < 0) {
+					    &revs->commit_queue) < 0) {
 				if (!revs->ignore_missing_links)
 					die("Failed to traverse parents of commit %s",
 					    oid_to_hex(&commit->object.oid));
 			}
 			break;
+		case REV_WALK_NO_WALK:
 		case REV_WALK_LIMITED:
 			break;
 		}
diff --git a/revision.h b/revision.h
index 584f1338b5..04982a3d47 100644
--- a/revision.h
+++ b/revision.h
@@ -12,6 +12,7 @@
 #include "decorate.h"
 #include "ident.h"
 #include "list-objects-filter-options.h"
+#include "prio-queue.h"
 #include "strvec.h"
 
 /**
@@ -122,8 +123,14 @@ struct oidset;
 struct topo_walk_info;
 
 struct rev_info {
-	/* Starting list */
+	/*
+	 * Work queue of commits, stored as either a linked list or a
+	 * priority queue, but never both at the same time.
+	 * rev_info_commit_list_to_queue() converts list to queue.
+	 */
 	struct commit_list *commits;
+	struct prio_queue commit_queue;
+
 	struct object_array pending;
 	struct repository *repo;
 
@@ -400,6 +407,7 @@ struct rev_info {
  * uninitialized.
  */
 #define REV_INFO_INIT { \
+	.commit_queue = { .compare = compare_commits_by_commit_date }, \
 	.abbrev = DEFAULT_ABBREV, \
 	.simplify_history = 1, \
 	.pruning.flags.recursive = 1, \
@@ -478,6 +486,8 @@ void reset_revision_walk(void);
  */
 int prepare_revision_walk(struct rev_info *revs);
 
+/* Drain the commits linked list into the priority queue. */
+void rev_info_commit_list_to_queue(struct rev_info *revs);
 /**
  * Takes a pointer to a `rev_info` structure and iterates over it, returning a
  * `struct commit *` each time you call it. The end of the revision list is

-- 
gitgitgadget