Clones with fetch.bundleURI slower than standard, full clone?
3 messages between Jul 28, 2026 and Jul 29, 2026, from Knop, Ryszard, brian m. carlson.
Plain Markdown or JSON for tools and agents.
Hey all,
I'm working on a Linux kernel-related CI system where we need to perform full clones of the kernel repo in most jobs. Because of some jobs in the pipeline, it usually cannot be a shallow clone :( Since the kernel repo is large and slow to clone, and I don't want to put undue stress on the remote host, I used git bundles, where CI clones the repo over a weekend, packages that as a bundle, then in jobs it gets used like this (weird, but works for <REF> being a branch, tag or a specific commit hash):
git init git remote add origin <REPO-URL> git config set fetch.bundleURI <BUNDLE-URL> git fetch origin <REF> git reset --hard FETCH_HEAD
Cloning a repo this way takes 4-5mins. Unpacking a bundle appears to be super slow. Not even faster than just running a full, normal clone from the remote server, actually (~3-4mins for a single branch).
On one of the build VMs, with Git 2.53 (stock Ubuntu 26.04), GIT_TRACE suggests most of the time is spent in some variation of `/usr/lib/git- core/git index-pack --stdin -v --fix-thin '--keep=fetch-pack 39430 on build-server' --check-self-contained-and-connected`, and indeed that process burns 100% of its single thread for most of that time.
Is it expected that doing it this way is so slow? The alternative is to just package and work with the whole bare repo, but bundles appear to be an elegant way of dealing with exactly this scenario.
Thanks, Ryszard
Re: Clones with fetch.bundleURI slower than standard, full clone?
On 2026-07-28 at 23:42:03, Knop, Ryszard wrote:
Show 20 quoted lines
> Hey all,
>
> I'm working on a Linux kernel-related CI system where we need to
> perform full clones of the kernel repo in most jobs. Because of some
> jobs in the pipeline, it usually cannot be a shallow clone :( Since the
> kernel repo is large and slow to clone, and I don't want to put undue
> stress on the remote host, I used git bundles, where CI clones the repo
> over a weekend, packages that as a bundle, then in jobs it gets used
> like this (weird, but works for <REF> being a branch, tag or a specific
> commit hash):
>
> git init
> git remote add origin <REPO-URL>
> git config set fetch.bundleURI <BUNDLE-URL>
> git fetch origin <REF>
> git reset --hard FETCH_HEAD
>
> Cloning a repo this way takes 4-5mins. Unpacking a bundle appears to be
> super slow. Not even faster than just running a full, normal clone from
> the remote server, actually (~3-4mins for a single branch).
You're comparing apples to oranges here. A clone of a single branch includes only that line of history and only those objects, but when you use a bundle with multiple refs, Git has to handle all of the objects in the bundle's entire pack, not just the ref you've specified. In order to compare adequately, you'd have to compare a bundle containing only that one ref with the single-branch clone or a regular clone of the full repository with your full bundles.
Show 5 quoted lines
> On one of the build VMs, with Git 2.53 (stock Ubuntu 26.04), GIT_TRACE
> suggests most of the time is spent in some variation of `/usr/lib/git-
> core/git index-pack --stdin -v --fix-thin '--keep=fetch-pack 39430 on
> build-server' --check-self-contained-and-connected`, and indeed that
> process burns 100% of its single thread for most of that time.
As far as I can tell, the unbundling code just calls index-pack, so it should honour pack.threads. Setting that value to 0 causes this code to be executed:
/*
* Experiments show that going above 20 threads doesn't help,
* no matter how many cores you have. Below that, we tend to
* max at half the number of online_cpus(), presumably because
* half of those are hyperthreads rather than full cores. We'll
* never reduce the level below "3", though, to match a
* historical value that nobody complained about.
*/
if (nr_threads < 4)
; /* too few cores to consider capping */
else if (nr_threads < 6)
nr_threads = 3; /* historic cap */
else if (nr_threads < 40)
nr_threads /= 2;
else
nr_threads = 20; /* hard cap */
So I would expect this to not be single threaded unless Git was compiled without pthreads, run on a machine with few cores, or configured to use only a single thread. If you can get threading to work here, I expect it will perform better, although I don't have any experience with bundle URIs so I can't really say for certain.
> Is it expected that doing it this way is so slow? The alternative is to
> just package and work with the whole bare repo, but bundles appear to
> be an elegant way of dealing with exactly this scenario.
To be clear, it is insecure to do anything with an untrusted repo except clone or fetch from it, so you will almost certainly not want to distribute bare repos, since that will encourage people to use them as-is (which is insecure). Even if these are internal users who can trust you, it encourages an anti-pattern which has security problems in the general case.
--
brian m. carlson (they/them)
Toronto, Ontario, CA
Re: Clones with fetch.bundleURI slower than standard, full clone?
On Wed, 2026-07-29 at 00:14 +0000, brian m. carlson wrote:
Show 29 quoted lines
> On 2026-07-28 at 23:42:03, Knop, Ryszard wrote:
> > Hey all,
> >
> > I'm working on a Linux kernel-related CI system where we need to
> > perform full clones of the kernel repo in most jobs. Because of some
> > jobs in the pipeline, it usually cannot be a shallow clone :( Since the
> > kernel repo is large and slow to clone, and I don't want to put undue
> > stress on the remote host, I used git bundles, where CI clones the repo
> > over a weekend, packages that as a bundle, then in jobs it gets used
> > like this (weird, but works for <REF> being a branch, tag or a specific
> > commit hash):
> >
> > git init
> > git remote add origin <REPO-URL>
> > git config set fetch.bundleURI <BUNDLE-URL>
> > git fetch origin <REF>
> > git reset --hard FETCH_HEAD
> >
> > Cloning a repo this way takes 4-5mins. Unpacking a bundle appears to be
> > super slow. Not even faster than just running a full, normal clone from
> > the remote server, actually (~3-4mins for a single branch).
>
> You're comparing apples to oranges here. A clone of a single branch
> includes only that line of history and only those objects, but when you
> use a bundle with multiple refs, Git has to handle all of the objects in
> the bundle's entire pack, not just the ref you've specified. In order
> to compare adequately, you'd have to compare a bundle containing only
> that one ref with the single-branch clone or a regular clone of the full
> repository with your full bundles.
I thought that due to changes mentioned in this post, Git 2.50 and newer should just take in all the bundle changes and there should not be much of a difference during index packs:
https://blog.gitbutler.com/going-down-the-rabbit-hole-of-gits-new-bundle-uri
Show 33 quoted lines
>
> > On one of the build VMs, with Git 2.53 (stock Ubuntu 26.04), GIT_TRACE
> > suggests most of the time is spent in some variation of `/usr/lib/git-
> > core/git index-pack --stdin -v --fix-thin '--keep=fetch-pack 39430 on
> > build-server' --check-self-contained-and-connected`, and indeed that
> > process burns 100% of its single thread for most of that time.
>
> As far as I can tell, the unbundling code just calls index-pack, so it
> should honour pack.threads. Setting that value to 0 causes this code to
> be executed:
>
> /*
> * Experiments show that going above 20 threads doesn't help,
> * no matter how many cores you have. Below that, we tend to
> * max at half the number of online_cpus(), presumably because
> * half of those are hyperthreads rather than full cores. We'll
> * never reduce the level below "3", though, to match a
> * historical value that nobody complained about.
> */
> if (nr_threads < 4)
> ; /* too few cores to consider capping */
> else if (nr_threads < 6)
> nr_threads = 3; /* historic cap */
> else if (nr_threads < 40)
> nr_threads /= 2;
> else
> nr_threads = 20; /* hard cap */
>
> So I would expect this to not be single threaded unless Git was compiled
> without pthreads, run on a machine with few cores, or configured to use
> only a single thread. If you can get threading to work here, I expect
> it will perform better, although I don't have any experience with bundle
> URIs so I can't really say for certain.
Ohhh, this is way better, that halved the checkout time. Thank you! It seems like the process still has some single-threaded sections during packing, but it's a problem for another day.
Show 10 quoted lines
> > Is it expected that doing it this way is so slow? The alternative is to
> > just package and work with the whole bare repo, but bundles appear to
> > be an elegant way of dealing with exactly this scenario.
>
> To be clear, it is insecure to do anything with an untrusted repo except
> clone or fetch from it, so you will almost certainly not want to
> distribute bare repos, since that will encourage people to use them
> as-is (which is insecure). Even if these are internal users who can
> trust you, it encourages an anti-pattern which has security problems in
> the general case.
I know it's not ideal, but our reference repos are trusted and the cache files are used in CI only, so it would probably be fine(tm). With that config change above I don't have to fall back to this though.
Thanks again, Ryszard