{"thread":{"id":"54517","subject":"Bundles: Partial Clone & Shallow clone","startedAt":"2020-10-27T22:05:07Z","lastAt":"2020-10-30T08:19:35Z","messageCount":2,"participants":["Robin H. Johnson","Christian Couder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"408537","messageId":"robbat2-20201027T175013-504497035Z@orbis-terrarum.net","threadId":"54517","inReplyTo":null,"subject":"Bundles: Partial Clone & Shallow clone","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2020-10-27T21:55:08Z","receivedAt":"2020-10-27T22:05:07Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"Hi!\n\nI was wondering if anybody has made a start towards easier ways to\ncreate bundles of partial & shallow clones.\n\nI'm looking at potential ways to improve the various snapshot &\nincremental update mechanisms that we have available in Gentoo Linux's\ntree distribution.\n\nRight now, the various mechanisms available are:\n- git tree WITHOUT generated metadata (git-daemon, smart-http)\n\nPlus all of the following WITH generated metadata.\n- git tree (git-daemon OR smart-http)\n- daily full snapshots in several formats/layouts.\n- deltas of full snapshots in some of the formats & layouts [1-day interval, no rollback]\n- rsync:// tree [30-min intervals, no rollback]\n\nOffline/Air-gapped use cases presently are expected to use the\nsnapshots+deltas, but the 1-day cadence is longer than desirable\n\nSeveral of these formats cannot rollback unless they kept the old tree\nor the snapshot it was generated from.\n\nrsync:// performance is heavily impacted by network latency and large\nnumber of small objects being transferred.\n\nWhat I'd like to offer instead, is CDN-replicated bundles generated at\nregular cadences, with the absolute minimal content; taken from the git\ntree WITH generated metadata. The git tree would have tags for every\ncadence point (ideally 30-minutes, with potential pruning of old tags).\nThe bundles would have GPG-signed checksums separately included, to\nprovide verification of the updates.\n\nThese come in two variants:\n1. Daily full snapshots, equivalent to depth=1 clone. \n2. 30-min & daily incremental bundles, using partial clone (needs to\n   include the new blobs that would be present when up to date, and\n   knowledge of which files can be deleted).\n\nThis should let users load some consistent set of daily or 30-min bundle\nonto their gitdir, and hop between those tags [they would not have\ngaps].\n\nIf they did have gaps due to a missing bundle or wanting to go within\nthe cadence points, and were online, they could use the partial\nmechanisms to fill in their tree as needed.\n\nRight now, I can naively generate the snapshots by explicitly making a\nnew detached shallow clone & then generating a bundle of that. \n\nIncremental bundles are already possible, but presently include all\ncommits, trees & blobs between two points.\n\nThe bundles are already generally smaller than our prior snapshots and\ndeltas, but I'm looking to make the process easier and cover the\nremaining gaps [if we have a high-change period, then the bundle winds\nup bigger than other deltas, because of the intermediate blobs].\n\nAsk 1. Ability to generate shallow bundle without the intermediate clone step.\n\nAsk 2. For incremental bundles, ability to exclude blobs not needed by\n       the latest commit (and it's tree).\n\nBoth of these I think would be possible by adding some variant on the\n--filter mechanism to git-bundle.\n\n-- \nRobin Hugh Johnson\nGentoo Linux: Dev, Infra Lead, Foundation Treasurer\nE-Mail   : robbat2@gentoo.org\nGnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85\nGnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136\n"},{"id":"408721","messageId":"CAP8UFD1_h7SBmY8rOgwuXPfN_fygOVhMyjryti+75E+XGO-maA@mail.gmail.com","threadId":"54517","inReplyTo":"robbat2-20201027T175013-504497035Z@orbis-terrarum.net","subject":"Re: Bundles: Partial Clone & Shallow clone","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2020-10-30T08:19:20Z","receivedAt":"2020-10-30T08:19:35Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nOn Wed, Oct 28, 2020 at 10:39 PM Robin H. Johnson <robbat2@gentoo.org> wrote:\n>\n> Hi!\n>\n> I was wondering if anybody has made a start towards easier ways to\n> create bundles of partial & shallow clones.\n\nI don't think anybody has started working on this.\n\n> I'm looking at potential ways to improve the various snapshot &\n> incremental update mechanisms that we have available in Gentoo Linux's\n> tree distribution.\n\nThe next Git Rev News edition\n(https://github.com/git/git.github.io/blob/master/rev_news/drafts/edition-68.md),\nwhich should be published today, has a summary of a discussion thread\n(https://lore.kernel.org/git/CAHpGcMJqmUmCR_u3fxSVppzwBo8_6RJG5-WLrXAWXm+OQ6_3bA@mail.gmail.com/)\nrelated to this that you might find interesting.\n\nBest,\nChristian.\n"}]}