git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 1/7] odb/source-packed: improve lookup when enumerating objects

From
Justin Tobler <jltobler@gmail.com>
Date
Jul 9, 2026, 19:54 UTC
Message-ID
<ak_uXc0UxB_9Vk9z@denethor>
In-Reply-To
<20260709-pks-odb-for-each-object-filter-v1-1-82fe014b12b3@pks.im>
On 26/07/09 10:35AM, Patrick Steinhardt wrote:
Show 8 quoted lines
> When iterating through packed objects via `odb_for_each_object()` we
> do so via two different mechanisms:
> 
>   - When a multi-pack index is available we use that one to efficiently
>     loop through all objects.
> 
>   - We then loop through all packfiles that aren't covered by a
>     multi-pack index.

To be specific, we are talking only about the for_each_object callback for the packed source `odb_source_packed_for_each_object()` correct? Also, this appears to only matter when we are enumerating OIDs with a specific prefix.

Show 12 quoted lines
> Regardless of which mechanism we use, we then iterate through all the
> objects indexed by the respective data structure. Curiously though,
> while we use the indices for enumerating the objects, we completely
> ignore it for the actual object lookup. Instead, we call into the
> generic `odb_source_read_object_info()` function, which will itself
> consult the indices to figure out where the object in question even
> lives.
> 
> This has two consequences:
> 
>   - It's inefficient, as we basically have to figure out the position of
>     the object a second time.

Since we already have the position from the index, there is no need to start over. Makes sense.

>   - It's subtly wrong, as it may now happen that a specific object will
>     be looked up via a different pack in case it exists multiple times.

Naive question: Is there any real harm in reading the same object, but from a different packfile here?

Regardless I do think it's a good idea to just reuse the same packfile to get the same object here.

Show 35 quoted lines
> Fix the issue by using `packed_object_info()` directly. While at it,
> rename the `store` variable to `source`.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb/source-packed.c | 15 ++++++++-------
>  1 file changed, 8 insertions(+), 7 deletions(-)
> 
> diff --git a/odb/source-packed.c b/odb/source-packed.c
> index 0edea5356d..9cfa02b7a2 100644
> --- a/odb/source-packed.c
> +++ b/odb/source-packed.c
> @@ -143,7 +143,7 @@ static bool should_exclude_pack(struct packed_git *p, enum odb_for_each_object_f
>  }
>  
>  static int for_each_prefixed_object_in_midx(
> -	struct odb_source_packed *store,
> +	struct odb_source_packed *source,
>  	struct multi_pack_index *m,
>  	const struct odb_for_each_object_options *opts,
>  	struct odb_source_packed_for_each_object_wrapper_data *data)
> @@ -170,6 +170,7 @@ static int for_each_prefixed_object_in_midx(
>  		 */
>  		for (i = first; i < num; i++) {
>  			const struct object_id *current = NULL;
> +			struct packed_git *pack;
>  			struct object_id oid;
>  
>  			current = nth_midxed_object_oid(&oid, m, i);
> @@ -177,9 +178,8 @@ static int for_each_prefixed_object_in_midx(
>  			if (!match_hash(len, opts->prefix->hash, current->hash))
>  				break;
>  
> -			if (opts->flags) {
> +			if (opts->flags || data->request) {
I'm not sure I follow why the above condition needed to change.
Show 14 quoted lines
>  				uint32_t pack_id = nth_midxed_pack_int_id(m, i);
> -				struct packed_git *pack;
>  
>  				if (prepare_midx_pack(m, pack_id)) {
>  					pack_errors = true;
> @@ -193,9 +193,9 @@ static int for_each_prefixed_object_in_midx(
>  
>  			if (data->request) {
>  				struct object_info oi = *data->request;
> +				off_t offset = nth_midxed_offset(m, i);
>  
> -				ret = odb_source_read_object_info(&store->base, current,
> -								  &oi, 0);
> +				ret = packed_object_info(source, pack, offset, &oi);

We not longer use the generic function to read object info. This ensures the exact same object is read.

Show 20 quoted lines
>  				if (ret)
>  					goto out;
>  
> @@ -219,7 +219,7 @@ static int for_each_prefixed_object_in_midx(
>  }
>  
>  static int for_each_prefixed_object_in_pack(
> -	struct odb_source_packed *store,
> +	struct odb_source_packed *source,
>  	struct packed_git *p,
>  	const struct odb_for_each_object_options *opts,
>  	struct odb_source_packed_for_each_object_wrapper_data *data)
> @@ -246,8 +246,9 @@ static int for_each_prefixed_object_in_pack(
>  
>  		if (data->request) {
>  			struct object_info oi = *data->request;
> +			off_t offset = nth_packed_object_offset(p, i);
>  
> -			ret = odb_source_read_object_info(&store->base, &oid, &oi, 0);
> +			ret = packed_object_info(source, p, offset, &oi);
And we do the same thing here when reading the object from a packfile.
-Justin
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 3 of 75 in “odb: introduce object filters to `odb_for_each_object()`”
  1. 0/7 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 9, 2026
  2. 1/7 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 9, 2026
  3. Justin ToblerJul 9, 2026
  4. Patrick SteinhardtJul 10, 2026
  5. 2/7 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 9, 2026
  6. 3/7 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 9, 2026
  7. Justin ToblerJul 9, 2026
  8. Patrick SteinhardtJul 10, 2026
  9. Jeff KingJul 11, 2026
  10. 4/7 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 9, 2026
  11. Justin ToblerJul 9, 2026
  12. Patrick SteinhardtJul 10, 2026
  13. 5/7 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 9, 2026
  14. 6/7 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 9, 2026
  15. Justin ToblerJul 9, 2026
  16. Patrick SteinhardtJul 10, 2026
  17. Toon ClaesJul 15, 2026
  18. 7/7 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 9, 2026
  19. Junio C HamanoJul 9, 2026
  20. Patrick SteinhardtJul 10, 2026
  21. 0/8 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 10, 2026
  22. 1/8 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 10, 2026
  23. Taylor BlauJul 10, 2026
  24. Patrick SteinhardtJul 13, 2026
  25. Taylor BlauJul 14, 2026
  26. Patrick SteinhardtJul 14, 2026
  27. 2/8 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 10, 2026
  28. Taylor BlauJul 10, 2026
  29. 3/8 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 10, 2026
  30. Taylor BlauJul 10, 2026
  31. Jeff KingJul 11, 2026
  32. Patrick SteinhardtJul 13, 2026
  33. Taylor BlauJul 14, 2026
  34. Jeff KingJul 14, 2026
  35. 4/8 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 10, 2026
  36. Taylor BlauJul 10, 2026
  37. 5/8 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 10, 2026
  38. Taylor BlauJul 10, 2026
  39. 6/8 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 10, 2026
  40. 7/8 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 10, 2026
  41. Taylor BlauJul 10, 2026
  42. 8/8 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 10, 2026
  43. 9/8 pack-objects: drop unused return value from add_object_entry()Jeff King, Jul 11, 2026
  44. Junio C HamanoJul 11, 2026
  45. 0/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 13, 2026
  46. 1/9 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 13, 2026
  47. 2/9 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 13, 2026
  48. 3/9 pack-objects: drop unused return value from add_object_entry()Patrick Steinhardt, Jul 13, 2026
  49. 4/9 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 13, 2026
  50. 5/9 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 13, 2026
  51. 6/9 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 13, 2026
  52. 7/9 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 13, 2026
  53. Toon ClaesJul 15, 2026
  54. Patrick SteinhardtJul 15, 2026
  55. 8/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 13, 2026
  56. Toon ClaesJul 15, 2026
  57. 9/9 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 13, 2026
  58. Toon ClaesJul 15, 2026
  59. Taylor BlauJul 14, 2026
  60. Patrick SteinhardtJul 14, 2026
  61. Toon ClaesJul 15, 2026
  62. Patrick SteinhardtJul 15, 2026
  63. Jeff KingJul 14, 2026
  64. 0/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 15, 2026
  65. 1/9 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 15, 2026
  66. 2/9 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 15, 2026
  67. 3/9 pack-objects: drop unused return value from add_object_entry()Patrick Steinhardt, Jul 15, 2026
  68. 4/9 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 15, 2026
  69. 5/9 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 15, 2026
  70. 6/9 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 15, 2026
  71. 7/9 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 15, 2026
  72. 8/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 15, 2026
  73. 9/9 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 15, 2026
  74. Toon ClaesJul 15, 2026
  75. Junio C HamanoJul 15, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.