git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v5 07/13] path-walk: support blob size limit filter

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
May 22, 2026, 18:24 UTC
Message-ID
<3884d4737feb2c81d734aea979317e4cd5d64490.1779474277.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.2101.v5.git.1779474277.gitgitgadget@gmail.com>
From: Derrick Stolee <stolee@gmail.com>

Extend the path-walk API to handle the 'blob:limit=<size>' object filter natively. This filter omits blobs whose size is equal to or greater than the given limit, matching the semantics used by the list-objects-filter machinery.

When revs->filter.choice is LOFC_BLOB_LIMIT, the prepare_filters() method stores the limit value in info->blob_limit and clears the filter from revs. If the limit is zero, this degenerates to blob:none (all blobs excluded), so info->blobs is set to 0 instead.

During walk_path(), blob batches are filtered before being delivered to the callback: each blob's size is checked via odb_read_object_info(), and only blobs strictly smaller than the limit are included. Blobs whose size cannot be determined (e.g. missing in a partial clone) are conservatively included, matching the existing filter behavior. Empty batches after filtering are skipped entirely.

The check for inclusion in the path batch looks a little strange at first glance. We use odb_read_object_info() to read the object's size. Based on all of the assumptions to this point, this _should_ return OBJ_BLOB. Since we are focused on the size filter, we use a short-circuited OR (||) to skip the size check if that method returns a different object type.

Notice that this inspection of object sizes requires the content to be present in the repository. The odb_read_object_info() call will download a missing blob on-demand. This means that the use of the path-walk API within 'git backfill' would not operate nicely with this filter type. The intention of that command is to download missing blobs in batches. Downloading objects one-by-one would go against the point. Update the validation in 'git backfill' to add its own compatibility check on top of path_walk_filter_compatible().

Add tests for blob:limit=0 (equivalent to blob:none) and blob:limit=3 (which exercises partial filtering within a batch where some blobs are kept and others are excluded).

Co-authored-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Derrick Stolee <stolee@gmail.com>
---
 Documentation/git-pack-objects.adoc |  2 +-
 builtin/backfill.c                  |  2 +
 path-walk.c                         | 41 +++++++++++++--
 path-walk.h                         |  7 +++
 t/t5620-backfill.sh                 |  2 +-
 t/t6601-path-walk.sh                | 82 +++++++++++++++++++++++++++++
 6 files changed, 130 insertions(+), 6 deletions(-)
diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc
index 2994faf988..85ae48b699 100644
--- a/Documentation/git-pack-objects.adoc
+++ b/Documentation/git-pack-objects.adoc
@@ -404,7 +404,7 @@ will be automatically changed to version `1`.
 +
 Incompatible with `--delta-islands`. The `--use-bitmap-index` option is
 ignored in the presence of `--path-walk`. The `--path-walk` option
-supports the `--filter=<spec>` form `blob:none`.
+supports the `--filter=<spec>` forms `blob:none` and `blob:limit=<n>`.
 
 
 DELTA ISLANDS
diff --git a/builtin/backfill.c b/builtin/backfill.c
index b80f9ebe69..5254a42711 100644
--- a/builtin/backfill.c
+++ b/builtin/backfill.c
@@ -98,6 +98,8 @@ static void reject_unsupported_rev_list_options(struct rev_info *revs)
 		    "--diff-merges");
 	if (!path_walk_filter_compatible(&revs->filter))
 		die(_("cannot backfill with these filter options"));
+	if (revs->filter.blob_limit_value)
+		die(_("cannot backfill with blob size limits"));
 }
 
 static int do_backfill(struct backfill_context *ctx)
diff --git a/path-walk.c b/path-walk.c
index bd81508163..04b924d4de 100644
--- a/path-walk.c
+++ b/path-walk.c
@@ -10,6 +10,7 @@
 #include "hex.h"
 #include "list-objects.h"
 #include "list-objects-filter-options.h"
+#include "odb.h"
 #include "object.h"
 #include "oid-array.h"
 #include "path.h"
@@ -327,13 +328,35 @@ static int walk_path(struct path_walk_context *ctx,
 	/*
 	 * Evaluate function pointer on this data, if requested.
 	 * Ignore object type filters for tagged objects (path starts
-	 * with `/`).
+	 * with `/`), first for blobs and then other types.
 	 */
-	if ((list->type == OBJ_TREE && (ctx->info->trees || path_is_for_direct_objects(path))) ||
-	    (list->type == OBJ_BLOB && (ctx->info->blobs || path_is_for_direct_objects(path))) ||
-	    (list->type == OBJ_TAG && ctx->info->tags))
+	if (list->type == OBJ_BLOB &&
+	    ctx->info->blob_limit &&
+	    !path_is_for_direct_objects(path)) {
+		struct oid_array filtered = OID_ARRAY_INIT;
+
+		for (size_t i = 0; i < list->oids.nr; i++) {
+			unsigned long size;
+
+			if (odb_read_object_info(ctx->repo->objects,
+						 &list->oids.oid[i],
+						 &size) != OBJ_BLOB ||
+				size < ctx->info->blob_limit)
+				oid_array_append(&filtered,
+						 &list->oids.oid[i]);
+		}
+
+		if (filtered.nr)
+			ret = ctx->info->path_fn(path, &filtered, list->type,
+						 ctx->info->path_fn_data);
+		oid_array_clear(&filtered);
+	} else if (path_is_for_direct_objects(path) ||
+		   (list->type == OBJ_TREE && ctx->info->trees) ||
+		   (list->type == OBJ_BLOB && ctx->info->blobs) ||
+		   (list->type == OBJ_TAG && ctx->info->tags)) {
 		ret = ctx->info->path_fn(path, &list->oids, list->type,
 					ctx->info->path_fn_data);
+	}
 
 	/* Expand data for children. */
 	if (list->type == OBJ_TREE) {
@@ -510,6 +533,16 @@ static int prepare_filters(struct path_walk_info *info,
 		}
 		return 1;
 
+	case LOFC_BLOB_LIMIT:
+		if (info) {
+			if (!options->blob_limit_value)
+				info->blobs = 0;
+			else
+				info->blob_limit = options->blob_limit_value;
+			list_objects_filter_release(options);
+		}
+		return 1;
+
 	default:
 		error(_("object filter '%s' not supported by the path-walk API"),
 		      list_objects_filter_spec(options));
diff --git a/path-walk.h b/path-walk.h
index a1736ecb2b..60ceb65433 100644
--- a/path-walk.h
+++ b/path-walk.h
@@ -47,6 +47,13 @@ struct path_walk_info {
 	int blobs;
 	int tags;
 
+	/**
+	 * If non-zero, specifies a maximum blob size. Blobs with a
+	 * size equal to or greater than this limit will not be
+	 * emitted unless included in 'pending'.
+	 */
+	unsigned long blob_limit;
+
 	/**
 	 * When 'prune_all_uninteresting' is set and a path has all objects
 	 * marked as UNINTERESTING, then the path-walk will not visit those
diff --git a/t/t5620-backfill.sh b/t/t5620-backfill.sh
index ede89f8c33..d2ea68e065 100755
--- a/t/t5620-backfill.sh
+++ b/t/t5620-backfill.sh
@@ -20,7 +20,7 @@ test_expect_success 'backfill rejects incompatible filter options' '
 	test_grep "cannot backfill with these filter options" err &&
 
 	test_must_fail git backfill --objects --filter=blob:limit=10m 2>err &&
-	test_grep "cannot backfill with these filter options" err
+	test_grep "cannot backfill with blob size limits" err
 '
 
 # We create objects in the 'src' repo.
diff --git a/t/t6601-path-walk.sh b/t/t6601-path-walk.sh
index b0ee31ee2d..45f366d738 100755
--- a/t/t6601-path-walk.sh
+++ b/t/t6601-path-walk.sh
@@ -477,4 +477,86 @@ test_expect_success 'topic only, blob:none filter' '
 	test_cmp_sorted expect out
 '
 
+test_expect_success 'all, blob:limit=0 filter' '
+	test-tool path-walk --filter=blob:limit=0 -- --all >out &&
+
+	cat >expect <<-EOF &&
+	0:commit::$(git rev-parse topic)
+	0:commit::$(git rev-parse base)
+	0:commit::$(git rev-parse base~1)
+	0:commit::$(git rev-parse base~2)
+	1:tag:/tags:$(git rev-parse refs/tags/first)
+	1:tag:/tags:$(git rev-parse refs/tags/second.1)
+	1:tag:/tags:$(git rev-parse refs/tags/second.2)
+	1:tag:/tags:$(git rev-parse refs/tags/third)
+	1:tag:/tags:$(git rev-parse refs/tags/fourth)
+	1:tag:/tags:$(git rev-parse refs/tags/tree-tag)
+	1:tag:/tags:$(git rev-parse refs/tags/blob-tag)
+	2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})
+	2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})
+	3:tree::$(git rev-parse topic^{tree})
+	3:tree::$(git rev-parse base^{tree})
+	3:tree::$(git rev-parse base~1^{tree})
+	3:tree::$(git rev-parse base~2^{tree})
+	3:tree::$(git rev-parse refs/tags/tree-tag^{})
+	3:tree::$(git rev-parse refs/tags/tree-tag2^{})
+	4:tree:a/:$(git rev-parse base:a)
+	5:tree:child/:$(git rev-parse refs/tags/tree-tag:child)
+	6:tree:left/:$(git rev-parse base:left)
+	6:tree:left/:$(git rev-parse base~2:left)
+	7:tree:right/:$(git rev-parse topic:right)
+	7:tree:right/:$(git rev-parse base~1:right)
+	7:tree:right/:$(git rev-parse base~2:right)
+	blobs:2
+	commits:4
+	tags:7
+	trees:13
+	EOF
+
+	test_cmp_sorted expect out
+'
+
+test_expect_success 'all, blob:limit=3 filter' '
+	test-tool path-walk --filter=blob:limit=3 -- --all >out &&
+
+	cat >expect <<-EOF &&
+	0:commit::$(git rev-parse topic)
+	0:commit::$(git rev-parse base)
+	0:commit::$(git rev-parse base~1)
+	0:commit::$(git rev-parse base~2)
+	1:tag:/tags:$(git rev-parse refs/tags/first)
+	1:tag:/tags:$(git rev-parse refs/tags/second.1)
+	1:tag:/tags:$(git rev-parse refs/tags/second.2)
+	1:tag:/tags:$(git rev-parse refs/tags/third)
+	1:tag:/tags:$(git rev-parse refs/tags/fourth)
+	1:tag:/tags:$(git rev-parse refs/tags/tree-tag)
+	1:tag:/tags:$(git rev-parse refs/tags/blob-tag)
+	2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag^{})
+	2:blob:/tagged-blobs:$(git rev-parse refs/tags/blob-tag2^{})
+	3:tree::$(git rev-parse topic^{tree})
+	3:tree::$(git rev-parse base^{tree})
+	3:tree::$(git rev-parse base~1^{tree})
+	3:tree::$(git rev-parse base~2^{tree})
+	3:tree::$(git rev-parse refs/tags/tree-tag^{})
+	3:tree::$(git rev-parse refs/tags/tree-tag2^{})
+	4:blob:a:$(git rev-parse base~2:a)
+	5:tree:a/:$(git rev-parse base:a)
+	6:tree:child/:$(git rev-parse refs/tags/tree-tag:child)
+	7:tree:left/:$(git rev-parse base:left)
+	7:tree:left/:$(git rev-parse base~2:left)
+	8:blob:left/b:$(git rev-parse base~2:left/b)
+	9:tree:right/:$(git rev-parse topic:right)
+	9:tree:right/:$(git rev-parse base~1:right)
+	9:tree:right/:$(git rev-parse base~2:right)
+	10:blob:right/c:$(git rev-parse base~2:right/c)
+	11:blob:right/d:$(git rev-parse base~1:right/d)
+	blobs:6
+	commits:4
+	tags:7
+	trees:13
+	EOF
+
+	test_cmp_sorted expect out
+'
+
 test_done
-- 
gitgitgadget
Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee via GitGitGadget
Message 88 of 96 in “pack-objects: integrate --path-walk and some --filter options”
  1. 0/7 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 2, 2026
  2. 1/7 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 2, 2026
  3. Junio C HamanoMay 4, 2026
  4. Derrick StoleeMay 4, 2026
  5. 2/7 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 2, 2026
  6. 3/7 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 2, 2026
  7. 4/7 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 2, 2026
  8. Junio C HamanoMay 3, 2026
  9. Derrick StoleeMay 4, 2026
  10. 5/7 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 2, 2026
  11. 6/7 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 2, 2026
  12. 7/7 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 2, 2026
  13. 00/10 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 4, 2026
  14. 01/10 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 4, 2026
  15. 02/10 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 4, 2026
  16. 03/10 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 4, 2026
  17. 04/10 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 4, 2026
  18. 05/10 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 4, 2026
  19. 06/10 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 4, 2026
  20. 07/10 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 4, 2026
  21. 08/10 path-walk: support `tree:0` filterTaylor Blau via GitGitGadget, May 4, 2026
  22. 09/10 path-walk: support `object:type` filterTaylor Blau via GitGitGadget, May 4, 2026
  23. 10/10 path-walk: support `combine` filterTaylor Blau via GitGitGadget, May 4, 2026
  24. Derrick StoleeMay 5, 2026
  25. Taylor BlauMay 5, 2026
  26. Derrick StoleeMay 5, 2026
  27. Taylor BlauMay 5, 2026
  28. Derrick StoleeMay 7, 2026
  29. Junio C HamanoMay 11, 2026
  30. Derrick StoleeMay 11, 2026
  31. 00/12 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 11, 2026
  32. 01/12 t5620: make test work with path-walk varDerrick Stolee via GitGitGadget, May 11, 2026
  33. Taylor BlauMay 12, 2026
  34. 02/12 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 11, 2026
  35. Taylor BlauMay 12, 2026
  36. 03/12 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 11, 2026
  37. Taylor BlauMay 12, 2026
  38. Derrick StoleeMay 13, 2026
  39. 04/12 path-walk: always emit directly-requested objectsDerrick Stolee via GitGitGadget, May 11, 2026
  40. Taylor BlauMay 12, 2026
  41. Derrick StoleeMay 13, 2026
  42. 05/12 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 11, 2026
  43. Taylor BlauMay 11, 2026
  44. Derrick StoleeMay 11, 2026
  45. 06/12 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 11, 2026
  46. Taylor BlauMay 12, 2026
  47. 07/12 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 11, 2026
  48. Taylor BlauMay 12, 2026
  49. Derrick StoleeMay 13, 2026
  50. 08/12 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 11, 2026
  51. 09/12 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 11, 2026
  52. 10/12 path-walk: support `tree:0` filterTaylor Blau via GitGitGadget, May 11, 2026
  53. Taylor BlauMay 12, 2026
  54. Derrick StoleeMay 13, 2026
  55. 11/12 path-walk: support `object:type` filterTaylor Blau via GitGitGadget, May 11, 2026
  56. 12/12 path-walk: support `combine` filterTaylor Blau via GitGitGadget, May 11, 2026
  57. Taylor BlauMay 12, 2026
  58. 00/13 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 13, 2026
  59. 01/13 t5620: make test work with path-walk varDerrick Stolee via GitGitGadget, May 13, 2026
  60. 02/13 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 13, 2026
  61. 03/13 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 13, 2026
  62. Taylor BlauMay 19, 2026
  63. Derrick StoleeMay 21, 2026
  64. 04/13 path-walk: always emit directly-requested objectsDerrick Stolee via GitGitGadget, May 13, 2026
  65. Taylor BlauMay 19, 2026
  66. Derrick StoleeMay 21, 2026
  67. Taylor BlauMay 22, 2026
  68. 05/13 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 13, 2026
  69. 06/13 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 13, 2026
  70. 07/13 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 13, 2026
  71. 08/13 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 13, 2026
  72. 09/13 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 13, 2026
  73. 10/13 t6601: tag otherwise-unreachable treesDerrick Stolee via GitGitGadget, May 13, 2026
  74. 11/13 path-walk: support `tree:0` filterTaylor Blau via GitGitGadget, May 13, 2026
  75. 12/13 path-walk: support `object:type` filterTaylor Blau via GitGitGadget, May 13, 2026
  76. 13/13 path-walk: support `combine` filterTaylor Blau via GitGitGadget, May 13, 2026
  77. Taylor BlauMay 19, 2026
  78. Derrick StoleeMay 21, 2026
  79. Taylor BlauMay 22, 2026
  80. Derrick StoleeMay 22, 2026
  81. 00/13 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 22, 2026
  82. 01/13 t5620: make test work with path-walk varDerrick Stolee via GitGitGadget, May 22, 2026
  83. 02/13 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 22, 2026
  84. 03/13 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 22, 2026
  85. 04/13 path-walk: always emit directly-requested objectsDerrick Stolee via GitGitGadget, May 22, 2026
  86. 05/13 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 22, 2026
  87. 06/13 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 22, 2026
  88. 07/13 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 22, 2026
  89. 08/13 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 22, 2026
  90. 09/13 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 22, 2026
  91. 10/13 t6601: tag otherwise-unreachable treesDerrick Stolee via GitGitGadget, May 22, 2026
  92. 11/13 path-walk: support `tree:0` filterTaylor Blau via GitGitGadget, May 22, 2026
  93. 12/13 path-walk: support `object:type` filterTaylor Blau via GitGitGadget, May 22, 2026
  94. 13/13 path-walk: support `combine` filterTaylor Blau via GitGitGadget, May 22, 2026
  95. Taylor BlauMay 22, 2026
  96. Junio C HamanoMay 24, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.