Re: With big repos and slower connections, git clone can be hard to work with

From: Toon claes <toon@iotcl.com>
Date: Jun 13, 2024, 10:10 UTC
Message-ID: <87msnpjgqc.fsf@iotcl.com>
In-Reply-To: <f5c24dfc-8d35-4418-b8f6-0a03d70c0917@horse64.org>

ellie <el@horse64.org> writes:

> Sorry for again another total newcomer/outsider question:

Don't apologize for asking these questions, you're more than welcome.

> Is a bundle or pack file something any regular git HTTPS instance
> would naturally provide when setup the usual ways?

Yes and no. Bundle and packfile format can used in many places. Packfiles are used to transfer a bunch of objects, or store them locally in Git's object database. A bundle is a packfile, but with a leading header describing refs. You can read about that at https://git-scm.com/docs/gitformat-bundle.

Show 6 quoted lines

> Like, if resume relied on that, would this work when following the
> standard smart HTTP setup procedure
> https://git-scm.com/book/en/v2/Git-on-the-Server-Smart-HTTP (sorry if
> I got the wrong link) and then git cloning from that? That would
> result in the best availability of such a resume feature, if it ever
> came to be.

As mentioned elsewhere in the thread, on clone (and fetch) the client negotiates with the server which objects to download. Because the state of the remote repository can change between clones, so will the result of this negotiation. This means the content of the packfile sent over might differ, which is disruptive for caching these files.

That's why the proposal of bundle URI or packfile URI is suggested. In case of bundle URI, it will tell the client to download a pre-made bundle before starting the negotiation. This bundle can be stored on a CDN or whatever static HTTP(s) server. But it requires the server to create it, store it, and tell the client about it. This is not something that's builtin into Git itself at the moment.

This is not really related to the Smart HTTP protocol, because it can be used over SSH as well. But when such file is stored on a regular HTTP server, we can rely on resumable downloads. Only after that bundle is downloaded, the client will start the negotiation with the server to get missing objects and refs (which should be a small subset when the bundle is recent).

-- Toon