git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/3] Implement filtering repacks

From
Taylor Blau <me@ttaylorr.com>
Date
Oct 28, 2022, 19:49 UTC
Message-ID
<Y1wyVpHprWGxEDi/@nand.local>
In-Reply-To
<CAP8UFD2HX6rK4TRP6ynUzWn4eoHa1FrbiFOtxBaxX-ZkBF3FJw@mail.gmail.com>
On Thu, Oct 20, 2022 at 01:23:02PM +0200, Christian Couder wrote:
Show 29 quoted lines
> On Fri, Oct 14, 2022 at 6:46 PM Junio C Hamano <gitster@pobox.com> wrote:
> >
> > Christian Couder <christian.couder@gmail.com> writes:
> >
> > > For example one might want to clone with a filter to avoid too many
> > > space to be taken by some large blobs, and one might realize after
> > > some time that a number of the large blobs have still be downloaded
> > > because some old branches referencing them were checked out. In this
> > > case a filtering repack could remove some of those large blobs.
> > >
> > > Some of the comments on the patch series that John sent were related
> > > to the possible data loss and repo corruption that a filtering repack
> > > could cause. It's indeed true that it could be very dangerous, and we
> > > agree that improvements were needed in this area.
> >
> > The wish is understandable, but I do not think this gives a good UI.
> >
> > This feature is, from an end-user's point of view, very similar to
> > "git prune-packed", in that we prune data that is not necessary due
> > to redundancy.  Nobody runs "prune-packed" directly; most people are
> > even unaware of it being run on their behalf when they run "git gc".
>
> I am Ok with adding the --filter option to `git gc`, or a config
> option with a similar effect. I wonder how `git gc` should implement
> that option though.
>
> If we implement a new command called for example `git filter-packed`,
> similar to `git prune-packed`, then this new command will call `git
> pack-objects --filter=...`.

Conceptually, yes, the two are similar. Though `prune-filtered` is necessarily going to differ in implementation from `prune-packed`, since we will have to write new pack(s), not just delete loose objects which appear in packs already.

So it's really not just a matter of purely deleting redundant loose copies of objects like in the case of prune-packed. Here we really do care about potentially writing a new set of packs to satisfy the new filter constraint.

Presumably that tool would implement creating the new packs according to the given --filter, and would similarly delete existing packs. That is basically what your implementation in repack already does, so I am not sure what the difference would be.

Show 5 quoted lines
> Yeah. So to sum up, it looks like you are Ok with `git gc
> --filter=...`  which is fine for me, even if I wonder if `git repack
> --filter=...` could be a good first step as it is less likely to be
> used automatically (so safer in a way) and it might be better for
> implementation related performance reasons.

If we don't intend to have `git repack --filter` part of our backwards compatibility guarantee, then I would prefer to see the implementation just live in git-gc from start to finish.

Thanks, Taylor

Previous: Christian CouderNext: Junio C Hamano
Message 7 of 46 in “Implement filtering repacks”
  1. 0/3 Implement filtering repacksChristian Couder, Oct 12, 2022
  2. 1/3 pack-objects: allow --filter without --stdoutChristian Couder, Oct 12, 2022
  3. 2/3 repack: add --filter=<filter-spec> optionChristian Couder, Oct 12, 2022
  4. 3/3 repack: introduce --force to force filteringChristian Couder, Oct 12, 2022
  5. Junio C HamanoOct 14, 2022
  6. Christian CouderOct 20, 2022
  7. Taylor BlauOct 28, 2022
  8. Junio C HamanoOct 28, 2022
  9. Christian CouderNov 7, 2022
  10. Christian CouderNov 7, 2022
  11. 0/2 Implement filtering repacksChristian Couder, Oct 25, 2022
  12. 1/2 pack-objects: allow --filter without --stdoutChristian Couder, Oct 25, 2022
  13. 2/2 repack: add --filter=<filter-spec> optionChristian Couder, Oct 25, 2022
  14. Taylor BlauOct 28, 2022
  15. Christian CouderNov 7, 2022
  16. 0/2 Implement filtering repacksChristian Couder, Nov 22, 2022
  17. 1/2 pack-objects: allow --filter without --stdoutChristian Couder, Nov 22, 2022
  18. 2/2 repack: add --filter=<filter-spec> optionChristian Couder, Nov 22, 2022
  19. Junio C HamanoNov 23, 2022
  20. Christian CouderDec 21, 2022
  21. Junio C HamanoNov 23, 2022
  22. 0/3 Implement filtering repacksChristian Couder, Dec 21, 2022
  23. 1/3 pack-objects: allow --filter without --stdoutChristian Couder, Dec 21, 2022
  24. Patrick SteinhardtJan 4, 2023
  25. 2/3 repack: add --filter=<filter-spec> optionChristian Couder, Dec 21, 2022
  26. Patrick SteinhardtJan 4, 2023
  27. Junio C HamanoJan 5, 2023
  28. 3/3 gc: add gc.repackFilter config optionChristian Couder, Dec 21, 2022
  29. Patrick SteinhardtJan 4, 2023
  30. 0/3 upload-pack: support a missing-actionChristian Couder, May 15, 2024
  31. 1/3 rev-list: refactor --missing=<missing-action>Christian Couder, May 15, 2024
  32. Junio C HamanoMay 15, 2024
  33. 2/3 pack-objects: use the missing action APIChristian Couder, May 15, 2024
  34. Junio C HamanoMay 15, 2024
  35. Christian CouderMay 24, 2024
  36. 3/3 upload-pack: allow configuring a missing-actionChristian Couder, May 15, 2024
  37. Junio C HamanoMay 15, 2024
  38. Christian CouderMay 24, 2024
  39. Junio C HamanoMay 24, 2024
  40. Christian CouderMay 28, 2024
  41. Junio C HamanoMay 28, 2024
  42. Christian CouderMay 31, 2024
  43. Junio C HamanoJun 1, 2024
  44. Christian CouderJun 3, 2024
  45. Junio C HamanoJun 3, 2024
  46. Christian CouderMay 15, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.