Robin H. JohnsonHi!
I was wondering if anybody has made a start towards easier ways to create bundles of partial & shallow clones.
I'm looking at potential ways to improve the various snapshot & incremental update mechanisms that we have available in Gentoo Linux's tree distribution.
Right now, the various mechanisms available are: - git tree WITHOUT generated metadata (git-daemon, smart-http)
Plus all of the following WITH generated metadata. - git tree (git-daemon OR smart-http) - daily full snapshots in several formats/layouts. - deltas of full snapshots in some of the formats & layouts [1-day interval, no rollback] - rsync:// tree [30-min intervals, no rollback]
Offline/Air-gapped use cases presently are expected to use the snapshots+deltas, but the 1-day cadence is longer than desirable
Several of these formats cannot rollback unless they kept the old tree or the snapshot it was generated from.
rsync:// performance is heavily impacted by network latency and large number of small objects being transferred.
What I'd like to offer instead, is CDN-replicated bundles generated at regular cadences, with the absolute minimal content; taken from the git tree WITH generated metadata. The git tree would have tags for every cadence point (ideally 30-minutes, with potential pruning of old tags). The bundles would have GPG-signed checksums separately included, to provide verification of the updates.
These come in two variants: 1. Daily full snapshots, equivalent to depth=1 clone. 2. 30-min & daily incremental bundles, using partial clone (needs to include the new blobs that would be present when up to date, and knowledge of which files can be deleted).
This should let users load some consistent set of daily or 30-min bundle onto their gitdir, and hop between those tags [they would not have gaps].
If they did have gaps due to a missing bundle or wanting to go within the cadence points, and were online, they could use the partial mechanisms to fill in their tree as needed.
Right now, I can naively generate the snapshots by explicitly making a new detached shallow clone & then generating a bundle of that.
Incremental bundles are already possible, but presently include all commits, trees & blobs between two points.
The bundles are already generally smaller than our prior snapshots and deltas, but I'm looking to make the process easier and cover the remaining gaps [if we have a high-change period, then the bundle winds up bigger than other deltas, because of the intermediate blobs].
Ask 1. Ability to generate shallow bundle without the intermediate clone step.
Ask 2. For incremental bundles, ability to exclude blobs not needed by
the latest commit (and it's tree).Both of these I think would be possible by adding some variant on the --filter mechanism to git-bundle.
-- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
