recursive cloning with --reference-if-able + --dissociate

Rasmus Villemoes· Feb 13, 2020, 14:45 UTC · lore

At $dayjob, most of our projects are organized as a rather small superproject, with a number of git submodules. Usually, one of those submodules is a linux kernel git tree.

For CI testing, we create a fresh docker container which starts by doing a "git clone --recursive $superproject", which takes quite a while due to the linux repo. So we'd like to speed that up by having a reasonable up-to-date linux repo on each build slave, mapping that into the docker container, and then telling git "when you get to cloning the src/linux submodule, look over here for objects", i.e. having some way to have --reference-if-able and --disassociate in effect for submodules.

If this is already possible, I can't find it in the documentation. And if it doesn't exist, I can't really think of a good way to define the UI for giving those options (other than, perhaps, having a top-level --reference-if-able pass down to the submodules with the meaning 'when initializing the submodule at some/path, try to use the submodule, if any, at some/path in the referenced top-level repo'".

Rasmus

Junio C Hamano· Feb 13, 2020, 16:18 UTC · re: Rasmus Villemoes · lore

Re: recursive cloning with --reference-if-able + --dissociate

Rasmus Villemoes <rv@rasmusvillemoes.dk> writes:

Show 18 quoted lines

> At $dayjob, most of our projects are organized as a rather small
> superproject, with a number of git submodules. Usually, one of those
> submodules is a linux kernel git tree.
>
> For CI testing, we create a fresh docker container which starts by doing
> a "git clone --recursive $superproject", which takes quite a while due
> to the linux repo. So we'd like to speed that up by having a reasonable
> up-to-date linux repo on each build slave, mapping that into the docker
> container, and then telling git "when you get to cloning the src/linux
> submodule, look over here for objects", i.e. having some way to have
> --reference-if-able and --disassociate in effect for submodules.
>
> If this is already possible, I can't find it in the documentation. And
> if it doesn't exist, I can't really think of a good way to define the UI
> for giving those options (other than, perhaps, having a top-level
> --reference-if-able pass down to the submodules with the meaning 'when
> initializing the submodule at some/path, try to use the submodule, if
> any, at some/path in the referenced top-level repo'".

Sorry for not giving a full solution but I suspect that discussing this topic will lead to a useful new feature, similar in spirit to the url rewriting (i.e. "url.$LOCAL.insteadOf = $REMOTE") to tell Git to use a local mirror $LOCAL instead of the remote authentic repository when interacting with $REMOTE.

Perhaps "url.$LOCALREF.autoReference = $REMOTE" to specify a local mirror that does not have to be completely up-to-date, that will altomatically be used as if "--reference-if-able $LOCALREF" is passed when running "git clone $REMOTE", or something like that.

The actual design has to get a lot more involved, including deciding:

 - if autoReference is orthogonal to --disssociate (I think it
   should be but others may differ).

 - how the way --dissociate is passed down from the top-level
   superproject to cloning of submodules is controlled (passing
   unconditionally would probably be too coarse, but I dunno).

etc. etc.