{"thread":{"id":"52159","subject":"Partial Clone garbage collection?","startedAt":"2019-10-30T17:08:32Z","lastAt":"2019-10-30T20:45:22Z","messageCount":5,"participants":["Simon Holmberg","без имени","Jeff King","Jonathan Tan"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"385163","messageId":"CA+M_GG1SfxGW=p_=418hdR1ypB3v-4GrooK6_75UUNJDb+kk2Q@mail.gmail.com","threadId":"52159","inReplyTo":"CA+M_GG35V3yNCfQ247PSrpP-R_f8bWNcBcmrnTWbrn1Nap_A4A@mail.gmail.com","subject":"Partial Clone garbage collection?","fromName":"Simon Holmberg","fromEmail":"simon.holmberg@avalanchestudios.se","sentAt":"2019-10-30T17:08:18Z","receivedAt":"2019-10-30T17:08:32Z","isPatch":false,"sender":{"key":"simon.holmberg@avalanchestudios.se","avatar":null},"body":"I've been experimenting with the new Partial Clone feature, attempting\nto use it to filter out the otherwise full history of the large binary\nresources in our repos. It works really well on the initial clone. But\nonce you start jumping around in history a lot, the repo will grow out\nof proportion again as promised pack files are fetched.\n\nAre there any plans to add a --filter parameter to git gc as well,\nthat would be able to prune past history of objects and convert them\nback into pack promises? Or am I wrong in assuming that this could\never act as a native replacement for LFS? Without this, a repo would\nonly continue to grow ad infinitum, resulting in the same issues as\nbefore unless you actively chose to delete your entire clone and\nre-clone it from upstream once in a while.\n"},{"id":"385168","messageId":"43646191572459422@myt3-4825096bdc88.qloud-c.yandex.net","threadId":"52159","inReplyTo":"CA+M_GG1SfxGW=p_=418hdR1ypB3v-4GrooK6_75UUNJDb+kk2Q@mail.gmail.com","subject":"Re: Partial Clone garbage collection?","fromName":"без имени","fromEmail":"mykaralw@yandex.ru","sentAt":"2019-10-30T18:17:02Z","receivedAt":"2019-10-30T18:17:13Z","isPatch":false,"sender":{"key":"mykaralw@yandex.ru","avatar":null},"body":"> Are there any plans to add a --filter parameter to git gc as well,\n> that would be able to prune past history of objects and convert them\n> back into pack promises?\n\nThis operation will change the hash, and hence the history. I tried to draw attention to a specific trick:\n\nhttps://public-inbox.org/git/25196441572039199@iva5-58d151f416d2.qloud-c.yandex.net/T/#t\n\nbut he was not supported.\n\n-- \nzvezdochiot\n\n\n"},{"id":"385179","messageId":"20191030203714.GC29013@sigill.intra.peff.net","threadId":"52159","inReplyTo":"CA+M_GG1SfxGW=p_=418hdR1ypB3v-4GrooK6_75UUNJDb+kk2Q@mail.gmail.com","subject":"Re: Partial Clone garbage collection?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-30T20:37:14Z","receivedAt":"2019-10-30T20:37:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 30, 2019 at 06:08:18PM +0100, Simon Holmberg wrote:\n\n> I've been experimenting with the new Partial Clone feature, attempting\n> to use it to filter out the otherwise full history of the large binary\n> resources in our repos. It works really well on the initial clone. But\n> once you start jumping around in history a lot, the repo will grow out\n> of proportion again as promised pack files are fetched.\n> \n> Are there any plans to add a --filter parameter to git gc as well,\n> that would be able to prune past history of objects and convert them\n> back into pack promises? Or am I wrong in assuming that this could\n> ever act as a native replacement for LFS? Without this, a repo would\n> only continue to grow ad infinitum, resulting in the same issues as\n> before unless you actively chose to delete your entire clone and\n> re-clone it from upstream once in a while.\n\nI don't recall seeing anybody actively working on this, but I think it\nwould be a good idea. You'd probably want to be able to specify it in\nyour config somehow, so that subsequent repacks pruned as necessary\nwithout you having to remember to do it each time.\n\nYou could naively just drop everything that matches the filter, and then\nre-fetch it as needed. But for efficiency, you may want to keep some\nother objects:\n\n  - objects mentioned directly in the index, or the tree of HEAD; you'd\n    end up re-fetching these next time you \"git checkout\"\n\n  - perhaps objects fetched recently are more worth keeping (e.g., ones\n    with an mtime less than a day or two). I don't know if that helps,\n    though. What you really care about is how recently they were\n    accessed (assuming there's some locality there), not written. A\n    frequently-accessed object may have been fetched immediately after\n    you cloned, giving it an old mtime.\n\n    Since we can get any of the objects again if we want and we're just\n    optimizing, this is really just a cache-expiration problem. But it\n    may be hard to implement any of the stock algorithms without having\n    logs of which objects were accessed.\n\n-Peff\n"},{"id":"385180","messageId":"20191030203833.GD29013@sigill.intra.peff.net","threadId":"52159","inReplyTo":"43646191572459422@myt3-4825096bdc88.qloud-c.yandex.net","subject":"Re: Partial Clone garbage collection?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-30T20:38:33Z","receivedAt":"2019-10-30T20:38:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 30, 2019 at 09:17:02PM +0300, без имени wrote:\n\n> > Are there any plans to add a --filter parameter to git gc as well,\n> > that would be able to prune past history of objects and convert them\n> > back into pack promises?\n> \n> This operation will change the hash, and hence the history. I tried to\n> draw attention to a specific trick:\n\nI don't think this is quite the same thing. Simon is just talking about\nthe partial-clone system.  The files are still mentioned in history, but\nthe client does not store the matching blobs themselves. Instead, it can\nfetch them on-demand from a remote repository.\n\n-Peff\n"},{"id":"385182","messageId":"20191030204516.53832-1-jonathantanmy@google.com","threadId":"52159","inReplyTo":"20191030203714.GC29013@sigill.intra.peff.net","subject":"Re: Partial Clone garbage collection?","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-10-30T20:45:16Z","receivedAt":"2019-10-30T20:45:22Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> I don't recall seeing anybody actively working on this, but I think it\n> would be a good idea.\n\nI agree with this.\n\n> You could naively just drop everything that matches the filter, and then\n> re-fetch it as needed.\n\nWe should also retain the promisor objects that are not referenced by\nany other promisor object, regardless of whether it matches the filter.\n(If not, the resulting repository will not pass fsck.)\n\nThe place to make this change is most likely in\nrepack_promisor_objects() in builtin/repack.c. Currently, it just\nrepacks all known promisor objects into one - modifying it to only\nrepack the ones we want (and adding a CLI option etc.) should be\nsufficient.\n"}]}