{"thread":{"id":"61617","subject":"bundles discovery and clones","startedAt":"2024-06-10T18:25:31Z","lastAt":"2024-08-09T12:34:14Z","messageCount":7,"participants":["matthew sporleder","Jeff King","Karthik Nayak","Sitaram Chamarty","Dhruva Krishnamurthy"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"496783","messageId":"CAHKF-AsoF10coLP=+MV-NfkEvWzp2Xbucs7OwtOoCBs3TVMg3A@mail.gmail.com","threadId":"61617","inReplyTo":null,"subject":"bundles discovery and clones","fromName":"matthew sporleder","fromEmail":"msporleder@gmail.com","sentAt":"2024-06-10T18:25:19Z","receivedAt":"2024-06-10T18:25:31Z","isPatch":false,"sender":{"key":"msporleder@gmail.com","avatar":null},"body":"I have recently been playing with git clone --bundle-uri and loving it\nbecause I can clone with almost-*zero* resources being used on the\nserver!\n\nI am a little confused by https://git-scm.com/docs/bundle-uri\nmentioning \"discovery\" and things. Is this something being added to\nthe git cli, a special feature for other clients, or is it still too\nearly-days to talk about much?\n\nI would love to produce bundles of common use cases and have them\nauto-discovered by git clone *without* the --bundle-uri parameter, and\nthen let our CDN do the heavy lifting of satisfying things like:\ngit clone\ngit clone --depth=0\ngit clone --single-branch --branch main\n\nI'm not sure I hold out as much hope for pre-bundling pulls/updates\nbut any movement towards offloading our big-ish repos to CDNs is a win\nfor us.\n\nThanks,\nMatt\n"},{"id":"496836","messageId":"20240611072144.GD3248245@coredump.intra.peff.net","threadId":"61617","inReplyTo":"CAHKF-AsoF10coLP=+MV-NfkEvWzp2Xbucs7OwtOoCBs3TVMg3A@mail.gmail.com","subject":"Re: bundles discovery and clones","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-06-11T07:21:44Z","receivedAt":"2024-06-11T07:21:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 10, 2024 at 02:25:19PM -0400, matthew sporleder wrote:\n\n> I have recently been playing with git clone --bundle-uri and loving it\n> because I can clone with almost-*zero* resources being used on the\n> server!\n> \n> I am a little confused by https://git-scm.com/docs/bundle-uri\n> mentioning \"discovery\" and things. Is this something being added to\n> the git cli, a special feature for other clients, or is it still too\n> early-days to talk about much?\n> \n> I would love to produce bundles of common use cases and have them\n> auto-discovered by git clone *without* the --bundle-uri parameter, and\n> then let our CDN do the heavy lifting of satisfying things like:\n> git clone\n> git clone --depth=0\n> git clone --single-branch --branch main\n> \n> I'm not sure I hold out as much hope for pre-bundling pulls/updates\n> but any movement towards offloading our big-ish repos to CDNs is a win\n> for us.\n\nI don't think the server side is well documented, but peeking at the\ncode, I think you want this on the server:\n\n git config uploadpack.advertiseBundleURIs true\n git config bundle.version 1\n git config bundle.mode any\n git config bundle.foo.uri https://example.com/your.bundle\n\nAnd then the clients need to tell Git that they allow bundle transfers:\n\n git config --global transfer.bundleURI true\n\nI'm not sure if we'd eventually flip the client-side switch to \"true\" by\ndefault (which is what you'd need for this to happen without any user\nparticipation at all).\n\nOne gotcha there is that clients are now accessing an arbitrary URL\nprovided by the server, so there are cross-site security implications.\nIt might make more sense to allow only relative URLs without \"..\" (so if\nI fetched from https://example.com/foo.git, the server could use only\nthe relative \"bundles/bar.bundle\", which would then be found at\nhttps://example.com/foo.git/bundles/bar.bundle\").\n\n-Peff\n"},{"id":"496891","messageId":"CAHKF-AskyrhNYyzZytarKYbEUMz7MzWZhL9jNbk3VQi7s84ceg@mail.gmail.com","threadId":"61617","inReplyTo":"20240611072144.GD3248245@coredump.intra.peff.net","subject":"Re: bundles discovery and clones","fromName":"matthew sporleder","fromEmail":"msporleder@gmail.com","sentAt":"2024-06-11T11:14:48Z","receivedAt":"2024-06-11T11:15:00Z","isPatch":false,"sender":{"key":"msporleder@gmail.com","avatar":null},"body":"On Tue, Jun 11, 2024 at 3:21 AM Jeff King wrote:\n>\n> On Mon, Jun 10, 2024 at 02:25:19PM -0400, matthew sporleder wrote:\n>\n> > I have recently been playing with git clone --bundle-uri and loving it\n> > because I can clone with almost-*zero* resources being used on the\n> > server!\n> >\n> > I am a little confused by https://git-scm.com/docs/bundle-uri\n> > mentioning \"discovery\" and things. Is this something being added to\n> > the git cli, a special feature for other clients, or is it still too\n> > early-days to talk about much?\n> >\n> > I would love to produce bundles of common use cases and have them\n> > auto-discovered by git clone *without* the --bundle-uri parameter, and\n> > then let our CDN do the heavy lifting of satisfying things like:\n> > git clone\n> > git clone --depth=0\n> > git clone --single-branch --branch main\n> >\n> > I'm not sure I hold out as much hope for pre-bundling pulls/updates\n> > but any movement towards offloading our big-ish repos to CDNs is a win\n> > for us.\n>\n> I don't think the server side is well documented, but peeking at the\n> code, I think you want this on the server:\n>\n> git config uploadpack.advertiseBundleURIs true\n> git config bundle.version 1\n> git config bundle.mode any\n> git config bundle.foo.uri https://example.com/your.bundle\n>\n> And then the clients need to tell Git that they allow bundle transfers:\n>\n> git config --global transfer.bundleURI true\n>\n> I'm not sure if we'd eventually flip the client-side switch to \"true\" by\n> default (which is what you'd need for this to happen without any user\n> participation at all).\n>\n> One gotcha there is that clients are now accessing an arbitrary URL\n> provided by the server, so there are cross-site security implications.\n> It might make more sense to allow only relative URLs without \"..\" (so if\n> I fetched from https://example.com/foo.git, the server could use only\n> the relative \"bundles/bar.bundle\", which would then be found at\n> https://example.com/foo.git/bundles/bar.bundle\").\n>\n> -Peff\n\n\nIt wasn't clear to me what the (bundle.foo in your case) referred\nto. Where did 'foo' come from?\n\nAnyway if people are taking suggestions for UX I'll give my $0.02:\ngit clone --try-bundle, with --bundle-uri overriding, to allow the\nclient to ask the server for bundles that satisfy their request.\n"},{"id":"497088","messageId":"20240613102040.GD817573@coredump.intra.peff.net","threadId":"61617","inReplyTo":"CAHKF-AskyrhNYyzZytarKYbEUMz7MzWZhL9jNbk3VQi7s84ceg@mail.gmail.com","subject":"Re: bundles discovery and clones","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-06-13T10:20:40Z","receivedAt":"2024-06-13T10:20:41Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 11, 2024 at 07:14:48AM -0400, matthew sporleder wrote:\n\n> > I don't think the server side is well documented, but peeking at the\n> > code, I think you want this on the server:\n> >\n> > git config uploadpack.advertiseBundleURIs true\n> > git config bundle.version 1\n> > git config bundle.mode any\n> > git config bundle.foo.uri https://example.com/your.bundle\n> >\n> > And then the clients need to tell Git that they allow bundle transfers:\n> >\n> > git config --global transfer.bundleURI true\n> [...]\n> \n> It wasn't clear to me what the (bundle.foo in your case) referred\n> to. Where did 'foo' come from?\n\nIt is not clear to me either. ;) I don't know if is meaningful,\nbeyond grouping related bundle options into a single stanza. In my\nexample, \"foo\" is just a made-up word you can replace with whatever you\nwant. It is visible to clients at the protocol layer, though I don't\nthink Git actually shows it to the user.\n\n> Anyway if people are taking suggestions for UX I'll give my $0.02:\n> git clone --try-bundle, with --bundle-uri overriding, to allow the\n> client to ask the server for bundles that satisfy their request.\n\nYeah, I looked for something similar at first but couldn't find it. You\ncan do:\n\n git -c transfer.bundleURI clone ...\n\n-Peff\n"},{"id":"497210","messageId":"CAOLa=ZRkZb65b1NawPBNOnnxi_gjCU9=85cJuxj0mQxyrPJe0g@mail.gmail.com","threadId":"61617","inReplyTo":"20240611072144.GD3248245@coredump.intra.peff.net","subject":"Re: bundles discovery and clones","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-06-15T13:01:12Z","receivedAt":"2024-06-15T13:01:15Z","isPatch":false,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Jeff King writes:\n\n> On Mon, Jun 10, 2024 at 02:25:19PM -0400, matthew sporleder wrote:\n>\n>> I have recently been playing with git clone --bundle-uri and loving it\n>> because I can clone with almost-*zero* resources being used on the\n>> server!\n>>\n>> I am a little confused by https://git-scm.com/docs/bundle-uri\n>> mentioning \"discovery\" and things. Is this something being added to\n>> the git cli, a special feature for other clients, or is it still too\n>> early-days to talk about much?\n>>\n>> I would love to produce bundles of common use cases and have them\n>> auto-discovered by git clone *without* the --bundle-uri parameter, and\n>> then let our CDN do the heavy lifting of satisfying things like:\n>> git clone\n>> git clone --depth=0\n>> git clone --single-branch --branch main\n>>\n>> I'm not sure I hold out as much hope for pre-bundling pulls/updates\n>> but any movement towards offloading our big-ish repos to CDNs is a win\n>> for us.\n>\n> I don't think the server side is well documented, but peeking at the\n> code, I think you want this on the server:\n>\n> git config uploadpack.advertiseBundleURIs true\n> git config bundle.version 1\n> git config bundle.mode any\n> git config bundle.foo.uri https://example.com/your.bundle\n>\n> And then the clients need to tell Git that they allow bundle transfers:\n>\n> git config --global transfer.bundleURI true\n>\n> I'm not sure if we'd eventually flip the client-side switch to \"true\" by\n> default (which is what you'd need for this to happen without any user\n> participation at all).\n>\n\nThis would indeed be nice. We at GitLab have been experimenting with\nbundle-uri. While it is easy to flip the switch for clients under our\ncontrol (CI pipelines). End users loose out on these benefits, especially\nfor large monorepos where the servers spend a lot of time computing the\npackfile.\n\n> One gotcha there is that clients are now accessing an arbitrary URL\n> provided by the server, so there are cross-site security implications.\n> It might make more sense to allow only relative URLs without \"..\" (so if\n> I fetched from https://example.com/foo.git, the server could use only\n> the relative \"bundles/bar.bundle\", which would then be found at\n> https://example.com/foo.git/bundles/bar.bundle\").\n>\n> -Peff\n\nTrue. But I suspect servers using bundle uri might not always serve them\nfrom the same domain. I know we were experimenting using cloud storage\nand providing the client with a one-time signed URL.\n\nhttps://cloud.google.com/storage/docs/access-control/signed-urls\n"},{"id":"497833","messageId":"Zn9rMyvR6nev3rd9@sita-dell","threadId":"61617","inReplyTo":"20240611072144.GD3248245@coredump.intra.peff.net","subject":"Re: bundles discovery and clones","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2024-06-29T02:02:27Z","receivedAt":"2024-06-29T02:02:33Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Tue, Jun 11, 2024 at 03:21:44AM -0400, Jeff King wrote:\n> On Mon, Jun 10, 2024 at 02:25:19PM -0400, matthew sporleder wrote:\n> \n> > I have recently been playing with git clone --bundle-uri and loving it\n> > because I can clone with almost-*zero* resources being used on the\n> > server!\n> > \n> > I am a little confused by https://git-scm.com/docs/bundle-uri\n> > mentioning \"discovery\" and things. Is this something being added to\n> > the git cli, a special feature for other clients, or is it still too\n> > early-days to talk about much?\n> > \n> > I would love to produce bundles of common use cases and have them\n> > auto-discovered by git clone *without* the --bundle-uri parameter, and\n> > then let our CDN do the heavy lifting of satisfying things like:\n> > git clone\n> > git clone --depth=0\n> > git clone --single-branch --branch main\n> > \n> > I'm not sure I hold out as much hope for pre-bundling pulls/updates\n> > but any movement towards offloading our big-ish repos to CDNs is a win\n> > for us.\n> \n> I don't think the server side is well documented, but peeking at the\n> code, I think you want this on the server:\n> \n> git config uploadpack.advertiseBundleURIs true\n> git config bundle.version 1\n> git config bundle.mode any\n> git config bundle.foo.uri https://example.com/your.bundle\n> \n> And then the clients need to tell Git that they allow bundle transfers:\n> \n> git config --global transfer.bundleURI true\n> \n> I'm not sure if we'd eventually flip the client-side switch to \"true\" by\n> default (which is what you'd need for this to happen without any user\n> participation at all).\n> \n> One gotcha there is that clients are now accessing an arbitrary URL\n> provided by the server, so there are cross-site security implications.\n\nVery sorry for jumping in so late. I just posted in another\nthread related to bundles and then I saw this thread.\n\nGitolite supports this out of the box, and more importantly to\nthe security aspect, it respects gitolite's rules for that repo\nand that user. Link for details is:\n\nhttps://github.com/sitaramc/gitolite/blob/master/src/commands/rsync\n\n> It might make more sense to allow only relative URLs without \"..\" (so if\n> I fetched from https://example.com/foo.git, the server could use only\n> the relative \"bundles/bar.bundle\", which would then be found at\n> https://example.com/foo.git/bundles/bar.bundle\").\n> \n> -Peff\n> \n"},{"id":"500539","messageId":"b76b3a9f-fa4a-41ab-893a-7fafe865ca09@gmail.com","threadId":"61617","inReplyTo":"CAOLa=ZRkZb65b1NawPBNOnnxi_gjCU9=85cJuxj0mQxyrPJe0g@mail.gmail.com","subject":"Re: bundles discovery and clones","fromName":"Dhruva Krishnamurthy","fromEmail":"dhruvakm@gmail.com","sentAt":"2024-08-09T12:34:05Z","receivedAt":"2024-08-09T12:34:14Z","isPatch":false,"sender":{"key":"dhruvakm@gmail.com","avatar":"https://gravatar.com/avatar/96fe022a95b60fd0de7f9f521364d964cdc7c46be5cfef279f8d4379e49e19a6?d=mp&s=160"},"body":"On 6/15/24 6:01 AM, Karthik Nayak wrote:\n> Jeff King writes:\n> \n>> On Mon, Jun 10, 2024 at 02:25:19PM -0400, matthew sporleder wrote:\n>>\n>>> I have recently been playing with git clone --bundle-uri and loving it\n>>> because I can clone with almost-*zero* resources being used on the\n>>> server!\n>>>\n>>> I am a little confused by https://git-scm.com/docs/bundle-uri\n>>> mentioning \"discovery\" and things. Is this something being added to\n>>> the git cli, a special feature for other clients, or is it still too\n>>> early-days to talk about much?\n>>>\n>>> I would love to produce bundles of common use cases and have them\n>>> auto-discovered by git clone *without* the --bundle-uri parameter, and\n>>> then let our CDN do the heavy lifting of satisfying things like:\n>>> git clone\n>>> git clone --depth=0\n>>> git clone --single-branch --branch main\n>>>\n>>> I'm not sure I hold out as much hope for pre-bundling pulls/updates\n>>> but any movement towards offloading our big-ish repos to CDNs is a win\n>>> for us.\n>>\n>> I don't think the server side is well documented, but peeking at the\n>> code, I think you want this on the server:\n>>\n>> git config uploadpack.advertiseBundleURIs true\n>> git config bundle.version 1\n>> git config bundle.mode any\n>> git config bundle.foo.uri https://example.com/your.bundle\n>>\n>> And then the clients need to tell Git that they allow bundle transfers:\n>>\n>> git config --global transfer.bundleURI true\n>>\n>> I'm not sure if we'd eventually flip the client-side switch to \"true\" by\n>> default (which is what you'd need for this to happen without any user\n>> participation at all).\n>>\n> \n> This would indeed be nice. We at GitLab have been experimenting with\n> bundle-uri. While it is easy to flip the switch for clients under our\n> control (CI pipelines). End users loose out on these benefits, especially\n> for large monorepos where the servers spend a lot of time computing the\n> packfile.\n> \n>> One gotcha there is that clients are now accessing an arbitrary URL\n>> provided by the server, so there are cross-site security implications.\n>> It might make more sense to allow only relative URLs without \"..\" (so if\n>> I fetched from https://example.com/foo.git, the server could use only\n>> the relative \"bundles/bar.bundle\", which would then be found at\n>> https://example.com/foo.git/bundles/bar.bundle\").\n>>\n>> -Peff\n> \n> True. But I suspect servers using bundle uri might not always serve them\n> from the same domain. I know we were experimenting using cloud storage\n> and providing the client with a one-time signed URL.\n> \n> https://cloud.google.com/storage/docs/access-control/signed-urls\n\nWe (at Bitbucket) have implemented bundle server for serving bundles\nwith expiring URL from cloud storage. It will be nice to have bundle\nserver discovery based on git v2 protocol based capability exchange.\n\nexample pkt format:\n007bbundle-server=https://cdn-1.bitbucket.org/workspace/repository/bundle,https://bitbucket.org/workspace/repository/bundle\n\n-Dhruva\n"}]}