threads / discuss / 49127
Subject: git submodule: 3 modules same git repo?
I am trying to create 3 submodules from the same git repo, each pointing to a different branch. Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3 times, I want one clone and then have the 3 submodules to point to different commits.
Is this possible? If not, could it be added?
Jocke
Stefan Beller[PATCH 1/2] store submodule in common dir
Signed-off-by: Stefan Beller <sbeller@google.com> --- path.c | 1 + 1 file changed, 1 insertion(+) On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
> > I am trying to create 3 submodules from the same git repo, each pointing to a different branch. > Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3 > times, I want one clone and then have the 3 submodules to point to different commits. > > Is this possible? If not, could it be added?
yup.
According to recent discussions, it would be just this patch. (plus some unspecified amount of work, TBD).
I thought about proposing something proper later, but here is the WIP patch.
Thanks, Stefan
diff --git a/path.c b/path.c index 34f0f98349a..64c9821b834 100644 --- a/path.c +++ b/path.c @@ -115,6 +115,7 @@ static struct common_dir common_list[] = { { 1, 1, 1, "logs/HEAD" }, { 0, 1, 1, "logs/refs/bisect" }, { 0, 1, 0, "lost-found" }, + { 0, 1, 0, "modules" }, { 0, 1, 0, "objects" }, { 0, 1, 0, "refs" }, { 0, 1, 1, "refs/bisect" },
-- 2.18.0.865.gffc8e1a3cd6-goog
Junio C HamanoRe: [PATCH 1/2] store submodule in common dir
Stefan Beller <sbeller@google.com> writes:
> Signed-off-by: Stefan Beller <sbeller@google.com> > --- > path.c | 1 + > 1 file changed, 1 insertion(+) > > On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote: >> >> I am trying to create 3 submodules from the same git repo, each pointing to a different branch. >> Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3 >> times, I want one clone and then have the 3 submodules to point to different commits. >> >> Is this possible? If not, could it be added? > > yup. > > According to recent discussions, it would be just this patch. > (plus some unspecified amount of work, TBD). > > I thought about proposing something proper later, but here is the WIP patch. > > Thanks, > Stefan
My understanding of what Joakim wants to do is to have a top-level project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 and kernel/v2.6, each of which is a submodule that houses these versions of Linux kernel source, but only clone Linus's repository (as the up-to-late tree has all the necessary history to check out these past development tracks). And that should be doable with just the main checkout, without any additional worktree (it's just the matter of having .git/modules/kernel%2fv2.6/ directory pointed by two symlinks from .git/modules/kernel%2fv2.[24], or something like that).
Isn't "common_dir" stuff used to decide if each of separate "worktree" instance (of the superproject) shares ".git/$stuff" with each other?
Unless I am grossly misinterpreting the original question, I fail to see how changing .git/modules to be shared across worktrees possibly affects anything. I am puzzled...
>
> diff --git a/path.c b/path.c
> index 34f0f98349a..64c9821b834 100644
> --- a/path.c
> +++ b/path.c
> @@ -115,6 +115,7 @@ static struct common_dir common_list[] = {
> { 1, 1, 1, "logs/HEAD" },
> { 0, 1, 1, "logs/refs/bisect" },
> { 0, 1, 0, "lost-found" },
> + { 0, 1, 0, "modules" },
> { 0, 1, 0, "objects" },
> { 0, 1, 0, "refs" },
> { 0, 1, 1, "refs/bisect" },Junio C HamanoRe: [PATCH 1/2] store submodule in common dir
Junio C Hamano <gitster@pobox.com> writes:
> My understanding of what Joakim wants to do is to have a top-level > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > and kernel/v2.6, each of which is a submodule that houses these > versions of Linux kernel source, but only clone Linus's repository > (as the up-to-late tree has all the necessary history to check out > these past development tracks). And that should be doable with > just the main checkout, without any additional worktree (it's just > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > by two symlinks from .git/modules/kernel%2fv2.[24], or something > like that).
Actually I take the last part of that back. When thought naively about, it may appear that it should be doable, but because each of the modules/* directory in the top-level project has to serve as the $GIT_DIR for each submodule checkout, and the desire is to have these three directories to have checkout of three different branches, a single directory under modules/. that is shared among three submodules would *not* work---they must have separate index, HEAD, etc.
Theoretically we should be able to make modules/kernel%2fv2.[24] additional "worktree"s of modules/kernel%2fv2.6, but given that these are all "bare" repositories without an attached working tree, I am not sure how that would supposed to work. Thinking about having multiple worktrees on a single bare repository makes me head spin and ache X-<;-)
Stefan BellerRe: [PATCH 1/2] store submodule in common dir
On Tue, Aug 14, 2018 at 4:20 PM Junio C Hamano <gitster@pobox.com> wrote:
> > Junio C Hamano <gitster@pobox.com> writes: > > > My understanding of what Joakim wants to do is to have a top-level > > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > > and kernel/v2.6, each of which is a submodule that houses these > > versions of Linux kernel source, but only clone Linus's repository > > (as the up-to-late tree has all the necessary history to check out > > these past development tracks). And that should be doable with > > just the main checkout, without any additional worktree (it's just > > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > > by two symlinks from .git/modules/kernel%2fv2.[24], or something > > like that). > > Actually I take the last part of that back. When thought naively > about, it may appear that it should be doable, but because each of > the modules/* directory in the top-level project has to serve as the > $GIT_DIR for each submodule checkout, and the desire is to have > these three directories to have checkout of three different > branches, a single directory under modules/. that is shared among > three submodules would *not* work---they must have separate index, > HEAD, etc. > > Theoretically we should be able to make modules/kernel%2fv2.[24] > additional "worktree"s of modules/kernel%2fv2.6, but given that > these are all "bare" repositories without an attached working tree, > I am not sure how that would supposed to work. Thinking about > having multiple worktrees on a single bare repository makes me head > spin and ache X-<;-)
Well the worktree feature would do all this in a safe manner, but the existing feature of just cloning the submodules with a reference pointer to another repository at least dedupes some of the object store, although warnings need to be read carefully.
Regarding the worktree, I would think we'd want to have
git worktree --recurse-submodules [list, add, etc]
that would do a sensible thing for each action on the worktrees, but you seem to propose to have one of the three submodules the main worktree and the other two would be just worktrees of the first.
I guess you could just
* init/update one of the submodules * add their worktrees manually pointed to where the 2nd and 3rd submodule would go. * make a symbolic link from ln -s .git/modules/<1st>/worktrees/<2nd> .git/modules/<2nd> ln -s .git/modules/<1st>/worktrees/<3rd> .git/modules/<3rd>
as then submodule operations should "just work" and by having the worktrees in-place where the submodules are, we'd also have all the protection against overzealous garbage collection, too.
Stefan
Jonathan NiederRe: [PATCH 1/2] store submodule in common dir
Junio C Hamano wrote:
> Theoretically we should be able to make modules/kernel%2fv2.[24] > additional "worktree"s of modules/kernel%2fv2.6, but given that > these are all "bare" repositories without an attached working tree, > I am not sure how that would supposed to work. Thinking about > having multiple worktrees on a single bare repository makes me head > spin and ache X-<;-)
Agreed about spinning head. This is why I suggested at [1] that anyone intereseted in this start with description of their proposed design, which would have three benefits:
- after implementation, it would document the intent behind whatever we choose to do, hopefully saving people debugging or improving this code some head spinning
- it would allow subject matter experts on-list to suggest refinements and simplifications
- it would avoid the interested contributor going too far down a blind alley in case their initial proposal has a fatal flaw
I also agree with the "theoretically we should be able to make it work". As described in [1], I think most of this is work we're going to have to do eventually, as part of properly supporting multiple worktrees for a superproject. But I don't want to set wrong expectations: this will be hard.
Thanks, Jonathan
[1] https://public-inbox.org/git/20180814231049.GH142615@aiede.svl.corp.google.com/
Re: [PATCH 1/2] store submodule in common dir
On Tue, 2018-08-14 at 16:20 -0700, Junio C Hamano wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > Junio C Hamano <gitster@pobox.com> writes: > > > My understanding of what Joakim wants to do is to have a top-level > > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > > and kernel/v2.6, each of which is a submodule that houses these > > versions of Linux kernel source, but only clone Linus's repository > > (as the up-to-late tree has all the necessary history to check out > > these past development tracks). And that should be doable with > > just the main checkout, without any additional worktree (it's just > > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > > by two symlinks from .git/modules/kernel%2fv2.[24], or something > > like that). > > Actually I take the last part of that back. When thought naively > about, it may appear that it should be doable, but because each of > the modules/* directory in the top-level project has to serve as the > $GIT_DIR for each submodule checkout, and the desire is to have > these three directories to have checkout of three different > branches, a single directory under modules/. that is shared among > three submodules would *not* work---they must have separate index, > HEAD, etc. > > Theoretically we should be able to make modules/kernel%2fv2.[24] > additional "worktree"s of modules/kernel%2fv2.6, but given that > these are all "bare" repositories without an attached working tree, > I am not sure how that would supposed to work. Thinking about > having multiple worktrees on a single bare repository makes me head > spin and ache X-<;-)
You nailed it ! :) My head spins just reading this so I think I got my answer. I can be done but will be tricky to impl. I will keep an eye on how submodules develops, sure would be a welcome feature.
Jocke
Re: [PATCH 1/2] store submodule in common dir
On Wed, 2018-08-15 at 10:28 +0200, Joakim Tjernlund wrote:
> On Tue, 2018-08-14 at 16:20 -0700, Junio C Hamano wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > > > Junio C Hamano <gitster@pobox.com> writes: > > > > > My understanding of what Joakim wants to do is to have a top-level > > > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > > > and kernel/v2.6, each of which is a submodule that houses these > > > versions of Linux kernel source, but only clone Linus's repository > > > (as the up-to-late tree has all the necessary history to check out > > > these past development tracks). And that should be doable with > > > just the main checkout, without any additional worktree (it's just > > > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > > > by two symlinks from .git/modules/kernel%2fv2.[24], or something > > > like that). > > > > Actually I take the last part of that back. When thought naively > > about, it may appear that it should be doable, but because each of > > the modules/* directory in the top-level project has to serve as the > > $GIT_DIR for each submodule checkout, and the desire is to have > > these three directories to have checkout of three different > > branches, a single directory under modules/. that is shared among > > three submodules would *not* work---they must have separate index, > > HEAD, etc. > > > > Theoretically we should be able to make modules/kernel%2fv2.[24] > > additional "worktree"s of modules/kernel%2fv2.6, but given that > > these are all "bare" repositories without an attached working tree, > > I am not sure how that would supposed to work. Thinking about > > having multiple worktrees on a single bare repository makes me head > > spin and ache X-<;-) > > You nailed it ! :) > My head spins just reading this so I think I got my answer. I can be done > but will be tricky to impl. > I will keep an eye on how submodules develops, sure would be a welcome feature. > > Jocke
On a related note, it would be great if git could support sparse checkout/clone for submodules, now one have to manually add .git/info/sparse-checkout. If sparse clone and sparse checkout could be saved in the submodule, then there would be no problem with 3 copies of the same repo in my submodules.
Jocke
Duy NguyenRe: [PATCH 1/2] store submodule in common dir
On Wed, Aug 15, 2018 at 1:20 AM Junio C Hamano <gitster@pobox.com> wrote:
> Theoretically we should be able to make modules/kernel%2fv2.[24] > additional "worktree"s of modules/kernel%2fv2.6, but given that > these are all "bare" repositories without an attached working tree, > I am not sure how that would supposed to work. Thinking about > having multiple worktrees on a single bare repository makes me head > spin and ache X-<;-)
I could only read the mail subject this morning and thought a bit about it on the train, and came to the same conclusion that "git worktree" is the way to go.
The bareness should not affect this at all. If you meant core.bare, it can only affect the main repo. But what I'm thinking is repos with only linked worktrees and no main one. So when you "submodule update" a repo, you "clone -n", then "git worktree add" to put a worktree in place. The repo is strictly speaking not bare, it just does not have a main worktree.
The main HEAD should be detached so that it does not block any worktree from checking out branches. If we could get away without the main HEAD, that would be great, but HEAD is part of the repo signature so it has to stay (and we may end up keeping more objects for this HEAD even after every other heads don't need the same objects anymore).
This also solves the problem with mupltiple worktrees on a repo with submodules, where we have the same problems (if I add a new worktree of the superproject, where do the new submodules go?). In this case ideally all worktrees should have separate ref namespaces to maintain the "separate repo" view that we currently have, but I guess we can live with sharing refs for a while.
And simply sharing $GIT_DIR/modules won't work. For starter, it breaks existing setups. What I've wanted to do is adding a shared "common" directory. Then you could have "common/modules" (among other future common stuff), which is shared across all worktrees, but you don't have to add extra share rules since it's already covered by the "common" rule.
-- Duy
Stefan BellerRe: [PATCH 1/2] store submodule in common dir
On Tue, Aug 14, 2018 at 4:04 PM Junio C Hamano <gitster@pobox.com> wrote:
> > Stefan Beller <sbeller@google.com> writes: > > > Signed-off-by: Stefan Beller <sbeller@google.com> > > --- > > path.c | 1 + > > 1 file changed, 1 insertion(+) > > > > On Tue, Aug 14, 2018 at 3:27 PM Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote: > >> > >> I am trying to create 3 submodules from the same git repo, each pointing to a different branch. > >> Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3 > >> times, I want one clone and then have the 3 submodules to point to different commits. > >> > >> Is this possible? If not, could it be added? > > > > yup. > > > > According to recent discussions, it would be just this patch. > > (plus some unspecified amount of work, TBD). > > > > I thought about proposing something proper later, but here is the WIP patch. > > > > Thanks, > > Stefan > > My understanding of what Joakim wants to do is to have a top-level > project that has three subdirectories, e.g. kernel/v2.2, kernel/v2.4 > and kernel/v2.6, each of which is a submodule that houses these > versions of Linux kernel source, but only clone Linus's repository > (as the up-to-late tree has all the necessary history to check out > these past development tracks). And that should be doable with > just the main checkout, without any additional worktree (it's just > the matter of having .git/modules/kernel%2fv2.6/ directory pointed > by two symlinks from .git/modules/kernel%2fv2.[24], or something > like that).
Ah! I misunderstood due to fast reading.
For that I think you are interested in the feature added in d92a39590d1 (Add --reference option to git submodule., 2009-05-04), i.e. both the update and add command take the --reference flag that can be pointed at another repository such as an outside clone of these three submodules, so some deduplication will be performed.
> Isn't "common_dir" stuff used to decide if each of separate > "worktree" instance (of the superproject) shares ".git/$stuff" > with each other? > > Unless I am grossly misinterpreting the original question, I fail to > see how changing .git/modules to be shared across worktrees possibly > affects anything. I am puzzled...
I did misunderstand grossly.
Stefan
Jonathan NiederRe: git submodule: 3 modules same git repo?
Hi,
Joakim Tjernlund wrote:
> I am trying to create 3 submodules from the same git repo, each pointing to a different branch. > Since the repo is somewhat large, I don't want the 3 submodules to clone the same repo 3 > times, I want one clone and then have the 3 submodules to point to different commits. > > Is this possible? If not, could it be added?
It's an interesting case. You're not the only one that has wanted something like this: for example, "repo" (a similar tool with some differences) included the change [1] (repo: Support multiple branches for the same project, 2014-01-30) for this kind of application.
In practice, it's a bit messy. To allow switching to a branch where the submodule is not present, we store the submodule .git directory in the superproject's .git/modules/<submodule name> directory. This is an ordinary .git directory, with files like
HEAD config packed-refs [etc]
The "git worktree" tool allows having multiple worktrees for a single Git repository. Each worktree has its own HEAD. So the layout would look something like
.git/modules/<submodule name>/ config packed-refs [etc] worktrees/ <branch1>/ HEAD <branch2>/ HEAD
By piggy-backing on the "git worktree" feature, this should work great, but it will take some work to teach Git to set it up.
A side note: as Stefan (cc-ed) mentioned, there is another, related interaction between submodules and "git worktree". Namely: I might want to have multiple worktrees for a single superproject Git repository. In that case, the layout might look something like
.git/ HEAD config packed-refs [etc] modules/ <submodule name>/ HEAD config packed-refs ... worktrees/ <branch1>/ HEAD <branch2>/ HEAD
The patch that Stefan sent heads in this direction, but it has a problem: if the submodule is checked out in both worktrees, then they cannot have the same HEAD. So to support that goal well would also require supporting the goal you've described as a side effect: each submodule would need to have multiple worktrees, at least one per worktree of the superproject.
Sorry, that got a bit contorted after a while. If you'd be interested in pursuing this, I'd be happy to review any thoughts you have (for example if you write a brief design doc).
My experience from interacting with the feature [1] is that it is easy to make mistakes when trying to support this kind of case. (repo had some bugs due to, for example, multiple checkouts of a repository trying to run "git prune" at the same time.) The "git worktree" feature should be a good foundation to build on that avoids some of the problems encountered there.
Thanks, Jonathan
[1] https://gerrit-review.googlesource.com/c/git-repo/+/50715/