{"thread":{"id":"64087","subject":"Doing blobless clone by default; switching between blobless, treeless and full clones by a command","startedAt":"2025-09-04T09:35:57Z","lastAt":"2025-09-07T19:42:44Z","messageCount":9,"participants":["Дилян Палаузов","Kristoffer Haugsbakk","Derrick Stolee","Junio C Hamano","Patrick Steinhardt","Konstantin Ryabitsev","Ben Knoble"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"525486","messageId":"79ed51fbd94ec2793ab0388b33963b366e48c590.camel@aegee.org","threadId":"64087","inReplyTo":null,"subject":"Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Дилян Палаузов","fromEmail":"dilyan.palauzov@aegee.org","sentAt":"2025-09-04T09:33:08Z","receivedAt":"2025-09-04T09:35:57Z","isPatch":false,"sender":{"key":"dilyan.palauzov@aegee.org","avatar":null},"body":"Hello,\n\nthe most common workflow to see and change files of a repository is to do git clone. With --filter=blob:none this process can be sped up. Blobless and blobfull clones have their trade offs.\n\nI think these additions to git would be useful:\n\n• add a config option to do by default blobless clone, when just git clone URL is invoked.\n• add a git command to download all locally missing history, including for treeless clones and blobless clones\n• add a git command to convert a repository to pure treeless or pure blobless clone (that is delete some commits) - to save disk space.\n\nThe git command to download all locally missing history should show how much per cent it has completed and after Ctrl+C interrupts, when the command is reissued, it should resume downloading the remaining data.\n\nRationale:\n\nThe reason people execute git clone is hardly to issue immediately afterwards git log or git annotate. The reason for git clone is to (try changing something and then) build the software. (Provided that git manages source code.) Doing by default a reduced data download with git clone will sped up the initialization, it will save bytes in transit and reduce server load. In fact I think that by default (without extra configuration) git clone should do a reduced download (blobless) and\ngit should download the other things, when asked to do so. This default download preference should be ideally managed by an option in global gitconfig . When looking at https://git-scm.com/docs/git-config for “filter” I do not recognize anything relevant.\n\nFor the latter https://stackoverflow.com/questions/76770003/is-there-a-way-to-configure-git-to-clone-with-filter-blobnone-by-default suggests adding a variable GITFLAGS='--filter=blob:none' .\n\nThere might be already commands to switch a repository to: full download, blobless clone, treeless clone, but I do not know these. In any case, if it is possible to switch easily between full and blobless repository, in both directions, for me it only makes sense if by default the downloads are blobless.\n\nGreetings\n Дилян\n"},{"id":"525487","messageId":"2ebdff4d-8adf-45d9-8cb5-6d7ee39f218d@app.fastmail.com","threadId":"64087","inReplyTo":"79ed51fbd94ec2793ab0388b33963b366e48c590.camel@aegee.org","subject":"Re: Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2025-09-04T09:41:51Z","receivedAt":"2025-09-04T09:42:55Z","isPatch":false,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Thu, Sep 4, 2025, at 11:33, Дилян Палаузов wrote:\n> • add a git command to download all locally missing history, including \n> for treeless clones and blobless clones\n\nThis sounds like git-backfill(1).\n\nI’ve never used blob/treeless.\n\n-- \nKristoffer Haugsbakk\n\n"},{"id":"525493","messageId":"7713256d-e258-4356-b303-128fdb417972@gmail.com","threadId":"64087","inReplyTo":"2ebdff4d-8adf-45d9-8cb5-6d7ee39f218d@app.fastmail.com","subject":"Re: Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-09-04T12:19:59Z","receivedAt":"2025-09-04T12:20:23Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/4/2025 5:41 AM, Kristoffer Haugsbakk wrote:\n> On Thu, Sep 4, 2025, at 11:33, Дилян Палаузов wrote:\n>> • add a git command to download all locally missing history, including \n>> for treeless clones and blobless clones\n> \n> This sounds like git-backfill(1).\n\nIndeed, 'git backfill' is intended to assist with downloading the blobs\nthat were not selected in a blobless partial clone. \n> I’ve never used blob/treeless.\n\nI don't believe that 'git backfill' is optimized for treeless clones.\nTreeless clones are not intended for \"refilling\" as downloading missing\ntrees is particularly expensive.\n\n...\n\nAnd regarding the original thought for \"we should have an option for\ndoing blobless clones by default\" the current way to do that is to use\n'scalar clone' which is shipped with Git already.\n\nWhen Scalar was originally contributed to Git, it was partly to enable\nusers to opt-in to a version of 'git clone' that changes behavior with\nbest practices and advanced features as they are developed. This is in\ncontrast to 'git clone' which needs to remain backwards compatible, so\nany new features need to be selected with an option or config setting.\n\nBut there was always the possibility that the feedback from having\n'scalar clone' available could lead to a future builtin of the form\n'git big-clone' that adds similar optimizations for large repos. (My\nopinion is that 'scalar clone' _is_ this 'git big-clone' but maybe it\nis not discoverable enough.)\n\nThanks,\n-Stolee\n\n"},{"id":"525549","messageId":"xmqqikhy9nb6.fsf@gitster.g","threadId":"64087","inReplyTo":"7713256d-e258-4356-b303-128fdb417972@gmail.com","subject":"Re: Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-04T16:47:57Z","receivedAt":"2025-09-04T16:47:59Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> But there was always the possibility that the feedback from having\n> 'scalar clone' available could lead to a future builtin of the form\n> 'git big-clone' that adds similar optimizations for large repos. (My\n> opinion is that 'scalar clone' _is_ this 'git big-clone' but maybe it\n> is not discoverable enough.)\n\nA separate command like \"bit-clone\" would not be discoverable\nenough, either. When a new feature matures in the playground, it\nwould be a welcome change for \"git clone\" to borrow it as a new\noption, or even better yet, automatically enable it depending on the\nsize of the thing, with end-user consent.\n\nThanks.\n"},{"id":"525620","messageId":"aLrWUhtCqp1hMcQS@pks.im","threadId":"64087","inReplyTo":"7713256d-e258-4356-b303-128fdb417972@gmail.com","subject":"Re: Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-05T12:23:46Z","receivedAt":"2025-09-05T12:23:52Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Sep 04, 2025 at 08:19:59AM -0400, Derrick Stolee wrote:\n> On 9/4/2025 5:41 AM, Kristoffer Haugsbakk wrote:\n> > On Thu, Sep 4, 2025, at 11:33, Дилян Палаузов wrote:\n> >> • add a git command to download all locally missing history, including \n> >> for treeless clones and blobless clones\n> > \n> > This sounds like git-backfill(1).\n> \n> Indeed, 'git backfill' is intended to assist with downloading the blobs\n> that were not selected in a blobless partial clone. \n> > I’ve never used blob/treeless.\n> \n> I don't believe that 'git backfill' is optimized for treeless clones.\n> Treeless clones are not intended for \"refilling\" as downloading missing\n> trees is particularly expensive.\n\nYeah, indeed. I guess we can tweak the way we backfill trees by batching\nby depth. E.g. we:\n\n 1. Collect all root trees and fetch them in a batch.\n\n 2. For each fetched tree, figure out all missing transitive trees and\n fetch that level.\n\n 3. Repeat for the next-deeper level.\n\nBut that batching is definitely not ideal, and there's going to be cases\nwhere it performs _way_ worse compared to backfilling blobs. That's\nnothing we can really avoid though.\n\nOne idea would be that the remote tells us about all the trees we may\nhave to fetch. But that information alone is not helpful unless we also\nhave the links between trees, and as soon as we have that the info is\nbasically interchangable with having the actual trees in the first\nplace.\n\nSo in general, the recommendation I typically give is to not use\ntreeless clones at all.\n\nPatrick\n"},{"id":"525633","messageId":"a1f72b71-2068-4526-920b-bab82137a509@gmail.com","threadId":"64087","inReplyTo":"aLrWUhtCqp1hMcQS@pks.im","subject":"Re: Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-09-05T13:40:49Z","receivedAt":"2025-09-05T13:40:51Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/5/2025 8:23 AM, Patrick Steinhardt wrote:\n> On Thu, Sep 04, 2025 at 08:19:59AM -0400, Derrick Stolee wrote:\n\n> So in general, the recommendation I typically give is to not use\n> treeless clones at all.\n\nRather, I'd say that treeless clones are useful if you want the\nspeed of a shallow clone with some need to analyze commit history\n(with no path history) for an ephemeral scenario like a CI build.\n\nTreeless clones are not a good approach for doing ongoing work as\na human. They are a tool for a very narrow case, so don't use them\nunless you understand how to avoid their pitfalls.\n\nThanks,\n-Stolee\n"},{"id":"525640","messageId":"aLrz9j8pUmHawNcz@pks.im","threadId":"64087","inReplyTo":"a1f72b71-2068-4526-920b-bab82137a509@gmail.com","subject":"Re: Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-05T14:30:14Z","receivedAt":"2025-09-05T14:30:20Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Sep 05, 2025 at 09:40:49AM -0400, Derrick Stolee wrote:\n> On 9/5/2025 8:23 AM, Patrick Steinhardt wrote:\n> > On Thu, Sep 04, 2025 at 08:19:59AM -0400, Derrick Stolee wrote:\n> \n> > So in general, the recommendation I typically give is to not use\n> > treeless clones at all.\n> \n> Rather, I'd say that treeless clones are useful if you want the\n> speed of a shallow clone with some need to analyze commit history\n> (with no path history) for an ephemeral scenario like a CI build.\n> \n> Treeless clones are not a good approach for doing ongoing work as\n> a human. They are a tool for a very narrow case, so don't use them\n> unless you understand how to avoid their pitfalls.\n\nAh, yes. I should've quantified my statement a bit more carefully.\nThanks for adding in this angle.\n\nPatrick\n"},{"id":"525644","messageId":"20250905-markhor-of-impossible-happiness-df5e58@lemur","threadId":"64087","inReplyTo":"79ed51fbd94ec2793ab0388b33963b366e48c590.camel@aegee.org","subject":"Re: Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2025-09-05T14:49:24Z","receivedAt":"2025-09-05T14:49:26Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Thu, Sep 04, 2025 at 12:33:08PM +0300, Дилян Палаузов wrote:\n> Rationale:\n> \n> The reason people execute git clone is hardly to issue immediately\n> afterwards git log or git annotate. The reason for git clone is to (try\n> changing something and then) build the software. (Provided that git manages\n> source code.) Doing by default a reduced data download with git clone\n> will sped up the initialization, it will save bytes in transit and reduce\n> server load. In fact I think that by default (without extra configuration)\n> git clone should do a reduced download (blobless) and git should download\n> the other things, when asked to do so. This default download preference\n> should be ideally managed by an option in global gitconfig . When looking\n> at https://git-scm.com/docs/git-config for “filter” I do not recognize\n> anything relevant.\n\nAs a counter-rationale, shallow clones generate a lot more load on the server\nside, because there are no packs available for this operation. Making this the\ndefault behaviour will likely result in slower clones for everyone and more\nunavailable servers due to high load.\n\n-K\n\n"},{"id":"525724","messageId":"297E6C80-C998-4D6A-B06D-7B2D1B229269@gmail.com","threadId":"64087","inReplyTo":"79ed51fbd94ec2793ab0388b33963b366e48c590.camel@aegee.org","subject":"Re: Doing blobless clone by default; switching between blobless, treeless and full clones by a command","fromName":"Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2025-09-07T19:42:32Z","receivedAt":"2025-09-07T19:42:44Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"\n> Le 4 sept. 2025 à 05:37, Дилян Палаузов a écrit :\n> \n> Hello,\n> \n> Rationale:\n> \n> The reason people execute git clone is hardly to issue immediately afterwards git log or git annotate.\n\nMaybe for you, and maybe in many contexts, but I also frequently clone things to run various history spelunking searches on them. \n\n> The reason for git clone is to (try changing something and then) build the software. (Provided that git manages source code.) Doing by default a reduced data download with git clone will sped up the initialization, it will save bytes in transit and reduce server load. In fact I think that by default (without extra configuration) git clone should do a reduced download (blobless) and\n> git should download the other things, when asked to do so. \n\nAbsolutely not (in my opinion, :p). Not having the entire repository available except when connected to a network defeats the tremendous advantage of distributed version control. Namely, privileged forks are given status by social agreement, not technical requirement. I want the whole repository available independently. \n\n> This default download preference should be ideally managed by an option in global gitconfig . When looking at https://git-scm.com/docs/git-config for “filter” I do not recognize anything relevant.\n\nThis seems more moderated and achievable. If you would prefer to clone less by default (I would not), go for it. I mostly don’t work with repos where this matters, though, or where git-maintenance doesn’t do most of the job I need after a 1-time setup cost.\n\nI do chastise folks for mis-managing large binary files in history that create large blobs and clone times, though :)"}]}