git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 1/6] odb: introduce interface to generate packfiles

From
Karthik Nayak <karthik.188@gmail.com>
Date
Aug 20, 2026, 10:16 UTC
Message-ID
<CAOLa=ZSYyfOCs8Dr0Xdhv-=Q=j0z7vtfYqopDb07XuAm2PU84g@mail.gmail.com>
In-Reply-To
<20260820-b4-pks-odb-generate-pack-v3-1-bc42252f6169@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
Show 37 quoted lines
> Packfiles have two primary use cases:
>
>   - They are used to store objects at rest in a Git repository.
>
>   - They are used on the transport layer to transfer objects between two
>     repositories.
>
> The first class is closely tied to a given object database backend, and
> as such this use is highly specific to how such a backend decides to
> store its data. This shows in git-pack-objects(1), which is used by
> git-repack(1) et al to optimize the object database, which supports lots
> of options that are closely coupled with how data is stored.
>
> But the second class is quite a lot more generic: we don't care about
> specifics of how the object database stores its objects, but to generate
> the packfiles we only care about the object graph itself. Still, this
> use case is also coupled with git-pack-objects(1).
>
> Unfortunately, because git-pack-objects(1) covers both classes, the
> result is that it is very hard to port the whole command to properly
> support pluggable object databases. There are simply way too many
> options that an alternative implementation will have a very hard time to
> support in the first place.
>
> And despite being hard to implement, it's also quite unnecessary to
> implement those backend-specific options. Optimizing the object database
> has already been made pluggable, and an alternative implementation is
> unlikely to care about cruft packs, unpacked objects, keep packs and the
> like. But we still need to make at least _parts_ of the packfile
> generation pluggable so that backends can generate packfiles for the
> transport layer itself.
>
> Introduce a new interface that lets backends generate a new packfile and
> implement that interface for the "files" backend. The options supported
> by the callback are exactly the set of options that are required for the
> transport layer, but nothing more.
>

Okay so we also provide the implementation for the files odb here, which calls 'git-pack-objects(1)' to generate the packfile.

The changes look in order.
[snip]
Show 27 quoted lines
> diff --git a/odb/source.h b/odb/source.h
> index d69f8e2d1c..e2129766fc 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -278,6 +278,23 @@ struct odb_source {
>  	 */
>  	bool (*optimize_required)(struct odb_source *source,
>  				  const struct odb_optimize_options *opts);
> +
> +	/*
> +	 * This callback is expected to start generating a packfile with the
> +	 * given options. The pack shall be generated asynchronously so that
> +	 * the caller can consume the pack data and progress output while the
> +	 * pack is being generated.
> +	 *
> +	 * This callback is optional. Sources that cannot generate packfiles
> +	 * shall leave it unset.
> +	 *
> +	 * The callback is expected to return 0 on success and populate the
> +	 * `out` pointer with the pack generator, a negative error code
> +	 * otherwise.
> +	 */
> +	int (*generate_pack)(struct odb_source *source,
> +			     struct odb_pack_generator **out,
> +			     const struct odb_generate_pack_options *opts);
>  };
>
Nit: I see that `source` is unused anyways, do we need to pass it in? Or
is just for consistency?
Show 25 quoted lines
>  /*
> @@ -520,4 +537,20 @@ static inline bool odb_source_optimize_required(struct odb_source *source,
>  	return source->optimize_required(source, opts);
>  }
>
> +/*
> + * Start generating a packfile from the given source with the given options.
> + * The pack is generated asynchronously; the caller is expected to consume the
> + * file descriptors exposed via the pack generator and to then wait for
> + * completion via `odb_pack_generator_finish()`.
> + *
> + * Returns 0 on success and populates the `out` pointer with the pack
> + * generator, a negative error code otherwise.
> + */
> +static inline int odb_source_generate_pack(struct odb_source *source,
> +					   struct odb_pack_generator **out,
> +					   const struct odb_generate_pack_options *opts)
> +{
> +	return source->generate_pack(source, out, opts);
> +}
> +
>  #endif
>
> --
> 2.55.0.822.g20453c30eb.dirty
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 33 of 58 in “odb: make packfile generation pluggable”
  1. 0/5 odb: make packfile generation pluggablePatrick Steinhardt, Aug 7, 2026
  2. 1/5 odb: introduce interface to generate packfilesPatrick Steinhardt, Aug 7, 2026
  3. 2/5 upload-pack: generate packfiles via the object databasePatrick Steinhardt, Aug 7, 2026
  4. 3/5 send-pack: generate packfiles via the object databasePatrick Steinhardt, Aug 7, 2026
  5. 4/5 builtin/bundle: refactor option handling for progress meterPatrick Steinhardt, Aug 7, 2026
  6. 5/5 bundle: generate packfiles via the object databasePatrick Steinhardt, Aug 7, 2026
  7. Junio C HamanoAug 13, 2026
  8. Patrick SteinhardtAug 14, 2026
  9. Junio C HamanoAug 7, 2026
  10. Patrick SteinhardtAug 10, 2026
  11. Taylor BlauAug 12, 2026
  12. Patrick SteinhardtAug 13, 2026
  13. Junio C HamanoAug 13, 2026
  14. 0/6 odb: make packfile generation pluggablePatrick Steinhardt, Aug 17, 2026
  15. 1/6 odb: introduce interface to generate packfilesPatrick Steinhardt, Aug 17, 2026
  16. Elijah NewrenAug 19, 2026
  17. Patrick SteinhardtAug 20, 2026
  18. Elijah NewrenAug 21, 2026
  19. Patrick SteinhardtAug 21, 2026
  20. 2/6 upload-pack: generate packfiles via the object databasePatrick Steinhardt, Aug 17, 2026
  21. 3/6 send-pack: generate packfiles via the object databasePatrick Steinhardt, Aug 17, 2026
  22. 4/6 builtin/bundle: refactor option handling for progress meterPatrick Steinhardt, Aug 17, 2026
  23. 5/6 bundle: get (mostly) rid of `the_repository`Patrick Steinhardt, Aug 17, 2026
  24. Junio C HamanoAug 17, 2026
  25. Patrick SteinhardtAug 18, 2026
  26. Elijah NewrenAug 21, 2026
  27. Patrick SteinhardtAug 21, 2026
  28. 6/6 bundle: generate packfiles via the object databasePatrick Steinhardt, Aug 17, 2026
  29. Justin ToblerAug 19, 2026
  30. Patrick SteinhardtAug 20, 2026
  31. 0/6 odb: make packfile generation pluggablePatrick Steinhardt, Aug 20, 2026
  32. 1/6 odb: introduce interface to generate packfilesPatrick Steinhardt, Aug 20, 2026
  33. Karthik NayakAug 20, 2026
  34. Patrick SteinhardtAug 20, 2026
  35. Junio C HamanoAug 20, 2026
  36. Patrick SteinhardtAug 21, 2026
  37. 2/6 upload-pack: generate packfiles via the object databasePatrick Steinhardt, Aug 20, 2026
  38. Karthik NayakAug 20, 2026
  39. Patrick SteinhardtAug 20, 2026
  40. Karthik NayakAug 20, 2026
  41. 3/6 send-pack: generate packfiles via the object databasePatrick Steinhardt, Aug 20, 2026
  42. 4/6 builtin/bundle: refactor option handling for progress meterPatrick Steinhardt, Aug 20, 2026
  43. Karthik NayakAug 20, 2026
  44. 5/6 bundle: get (mostly) rid of `the_repository`Patrick Steinhardt, Aug 20, 2026
  45. 6/6 bundle: generate packfiles via the object databasePatrick Steinhardt, Aug 20, 2026
  46. Karthik NayakAug 20, 2026
  47. Patrick SteinhardtAug 20, 2026
  48. Karthik NayakAug 20, 2026
  49. Patrick SteinhardtAug 20, 2026
  50. Elijah NewrenAug 21, 2026
  51. 0/6 odb: make packfile generation pluggablePatrick Steinhardt, Aug 21, 2026
  52. 1/6 odb: introduce interface to generate packfilesPatrick Steinhardt, Aug 21, 2026
  53. 2/6 upload-pack: generate packfiles via the object databasePatrick Steinhardt, Aug 21, 2026
  54. 3/6 send-pack: generate packfiles via the object databasePatrick Steinhardt, Aug 21, 2026
  55. 4/6 builtin/bundle: refactor option handling for progress meterPatrick Steinhardt, Aug 21, 2026
  56. 5/6 bundle: get (mostly) rid of `the_repository`Patrick Steinhardt, Aug 21, 2026
  57. 6/6 bundle: generate packfiles via the object databasePatrick Steinhardt, Aug 21, 2026
  58. Karthik NayakAug 21, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.