threads / discuss / 12915

git-submodule getting submodules from the parent repository

Subject: git-submodule getting submodules from the parent repository

## tl;dr

20 messages between Mar 29, 2008 and Apr 2, 2008.

replies: 19people: 5as markdown or json

Avery Pennarun· Mar 29, 2008, 22:35 UTC · lore
Hi all,

I have a fairly pressing need for git-submodule-like behaviour, but having tried git-submodule, it doesn't really work the way I'd like.

A super-simplified example of what I do:
- Project A (app) includes project B (build environment), which
includes project C (tool library)
- The projects are all open source, but B includes some binary
packages so it's a big download.  If you don't need the binary
packages, people want to just download C (hence the separation). But
everyone using A wants B and C, because they're lazy and bandwidth
isn't a problem.
- We have a local repository at work with mirrors of A, B, and C,
which are also available publicly (but there's no reason for everyone
in our office to be uploading/downloading the same big blobs all the
time).
- We frequently change B and C as part of building A (as well as other
A-like applications).
Here are the main problems, all in a jumble:

It's a pain to check out / mirror / check in / push. git-submodule doesn't even init automatically when you check out A, so you have to run it yourself. The relative paths of A, B, and C on your mirror have to be the same as upstream. You can't make a local mirror of A without mirroring B and C. B and C start out with a disconnected HEAD, so if you check in, it goes nowhere, and then when you push, nothing happens, and if you're unlucky enough to pull someone else's update to A and then "git-submodule update", it forgets your changes entirely. When you check in to C, you then have to check in to B, and then to A, all by hand; and when you git-pull, you'd better to C, then B, then A, or risk having A try to check out a revision from B that you haven't pulled, etc.

...phew.

It would probably be possible to fix each of these problems individually, but it would be a whole series of different fixes. I'd like to propose a rather different way of doing things that I think would solve most of these problems, and get some feedback:

What if *all* the objects for A, B, and C were always in the *same* repository? Almost all the problems would go away. Imagine if it worked like this:

1. git-clone had a way to *not* clone every single object from every
branch in the parent repository; only the ones you were interested in.
So if you git-clone only project C, you don't end up with A and B.  (I
personally would never need to clone A without B and C, but maybe
someone else would.  It's actually a different implementation issue,
since A refers to B, but B doesn't refer to A.)
2. You still check into C, then B, then A, but it doesn't actually
matter if you put B and C on a branch first or not, because 'git push'
will work properly, because it auto-pushes B and C revisions based on
the fact that A refers to them (ie. implicit branches via the
submodule mechanism).
3. You can still upload B and C to their own separate upstream
repository, which is obviously a critical feature.  But you can do
that simply by making a branch in your copy of C and pushing just that
branch.  The objects come from your one local A repository, but you
simply avoid accidentally pushing the wrong branch.
4. You can 'git clone' a local copy of A, and B/C will be cloned
automatically along with it.
5. B and C, when git-submodule checks them out, should have their own
.git directories, but use A as an 'alternatives' entry.
6. git-pull should be modified to auto-download objects referred to by
'submodule' references in trees.
This would really help my workflow a lot.  Am I missing something?
Thoughts?
Thanks,
Avery
Sam Vilain· Mar 29, 2008, 23:22 UTC · re: Avery Pennarun · lore

Re: git-submodule getting submodules from the parent repository

Avery Pennarun wrote:
Show 8 quoted lines
> It would probably be possible to fix each of these problems
> individually, but it would be a whole series of different fixes.  I'd
> like to propose a rather different way of doing things that I think
> would solve most of these problems, and get some feedback:
> 
> What if *all* the objects for A, B, and C were always in the *same*
> repository?  Almost all the problems would go away.  Imagine if it
> worked like this:

Well, that would create a lot of unnecessary work when cloning. Partitioning by project is a natural way to divide the projects up. It's worth noting that the early implementations of submodules were based on this design, of keeping everything together.

However, what you are suggesting should IMHO be allowed to work. In particular, if the submodule path is ".", then I think there's a good case that they should come from within the same project. If it's a relative URL, it should initialize based on the remote URL that was used for the original fetch (or, rather, the remote URL for the current branch).

And, if it happens that after a checkout, that the commit of a submodule is already in the object directory (ie, there's another branch), then maybe that should automatically check out.

> 1. git-clone had a way to *not* clone every single object from every
> branch in the parent repository; only the ones you were interested in.

It could easily, if someone would allow clone to have a --track option like git remote does:

  git init
  git remote add -t branch -f URL
Show 5 quoted lines
> 2. You still check into C, then B, then A, but it doesn't actually
> matter if you put B and C on a branch first or not, because 'git push'
> will work properly, because it auto-pushes B and C revisions based on
> the fact that A refers to them (ie. implicit branches via the
> submodule mechanism).

This push failure thing is regrettable; however it's not clear which branch name the submodules should get. A given commit might exist on several branches, which one do you choose to name it?

> 4. You can 'git clone' a local copy of A, and B/C will be cloned
> automatically along with it.
> 6. git-pull should be modified to auto-download objects referred to by
> 'submodule' references in trees.

I think this could be a switch to git clone/pull, configurable to be the default action.

> 5. B and C, when git-submodule checks them out, should have their own
> .git directories, but use A as an 'alternatives' entry.

There is also a Google Summer of Code project for this - see http://git.or.cz/gitwiki/SoC2008Ideas#head-9215572f23513542a23d3555aa72775bc4b91038

> This would really help my workflow a lot.  Am I missing something?

Well, no, it's true that the current workflow has interface niggles; however it's important to understand why the current implementation is the way it is, and make sure that new designs build on top of the parts which are already designed well, where they can.

Sam
Eyvind Bernhardsen· Mar 30, 2008, 13:32 UTC · re: Sam Vilain · lore

Re: git-submodule getting submodules from the parent repository

On 30. mars. 2008, at 00.22, Sam Vilain <sam@vilain.net> wrote:
Show 12 quoted lines
> Avery Pennarun wrote:
>
>> 2. You still check into C, then B, then A, but it doesn't actually
>> matter if you put B and C on a branch first or not, because 'git  
>> push'
>> will work properly, because it auto-pushes B and C revisions based on
>> the fact that A refers to them (ie. implicit branches via the
>> submodule mechanism).
>
> This push failure thing is regrettable; however it's not clear which
> branch name the submodules should get.  A given commit might exist on
> several branches, which one do you choose to name it?

I solved that by adding a "submodule push" that pushes the detached head of each submodule to its own ref ("refs/submodule-update/commit- $sha1", imaginatively). I also made "submodule update" try to fetch that ref when looking for a sha1.

I ran into trouble trying to avoid pushing every submodule for each "submodule push", and then more or less decided not to use submodules, so it's not quite fit for public consumption. I still think it's a sound idea in principle, so I'll clean it up and send it to the list if there's any interest.

-- 
Eyvind Bernhardsen
Sam Vilain· Mar 30, 2008, 17:48 UTC · re: Eyvind Bernhardsen · lore

Re: git-submodule getting submodules from the parent repository

Eyvind Bernhardsen wrote:
> I solved that by adding a "submodule push" that pushes the detached head
> of each submodule to its own ref ("refs/submodule-update/commit-$sha1",
> imaginatively).  I also made "submodule update" try to fetch that ref
> when looking for a sha1.

Hmm, a reasonable decision, but I think it would be better to force the user to choose which branch they want to push to. Leaving breadcrumbs in the refs/ space like that is best avoided.

Show 5 quoted lines
> I ran into trouble trying to avoid pushing every submodule for each
> "submodule push", and then more or less decided not to use submodules,
> so it's not quite fit for public consumption.  I still think it's a
> sound idea in principle, so I'll clean it up and send it to the list if
> there's any interest.

Indeed - it can only become "fit for public consumption" if people submit their usability enhancements!

Sam.
Eyvind Bernhardsen· Mar 30, 2008, 19:50 UTC · re: Sam Vilain · lore

Re: git-submodule getting submodules from the parent repository

On 30. mars. 2008, at 19.48, Sam Vilain wrote:
Show 12 quoted lines
> Eyvind Bernhardsen wrote:
>> I solved that by adding a "submodule push" that pushes the detached  
>> head
>> of each submodule to its own ref ("refs/submodule-update/commit- 
>> $sha1",
>> imaginatively).  I also made "submodule update" try to fetch that ref
>> when looking for a sha1.
>
> Hmm, a reasonable decision, but I think it would be better to force  
> the
> user to choose which branch they want to push to.  Leaving breadcrumbs
> in the refs/ space like that is best avoided.

Well, the point of "submodule push" was to avoid having to push in each submodule manually; not enforcing the requirement that commits in submodules must be publicly available before pushing from the main module is a recipe for disaster, or at least annoyance. And nobody likes an annoying git.

Pushing to a branch works except that I couldn't figure out what to do if the push doesn't succeed, ie, the branch has advanced on the remote end. That's a problem if more than one module references the submodule or there are multiple branches in the main module.

One solution that occurred to me was to have a branch in each submodule for every main module and branch. A branch name would be provided for each submodule in .gitmodules, used by "submodule push" but not "submodule update". In this case, if the push to the branch fails, the main module branch is probably behind too.

This seemed like a good idea, but it's racy. If two simultaneous "submodule push"es try to push to the same branch on a submodule, one of them will be rejected, but it might already have updated branches on other submodules. Ick.

I briefly toyed with creating tags named after the main module and its branch, with the submodule sha1 included for good measure, but that leaves a _real_ mess in refs/tags. Figuring out that I could use refs/ submodule-push instead seemed like an epiphany at the time.

As an aside, my mental model of what the submodule needs is a fetchable reflog for every main module and branch that uses it, containing the history of commits used by that module/branch.

It's a reflog, not a branch, because a submodule can be changed to a different branch, rewound, etc between commits in the main module; there's no requirement that the old commit is in the new commit's history. You actually don't want to fetch the whole thing, but you have to be able to fetch every sha1 contained in it, by sha1.

...so that's what refs/submodule-push is supposed to look like. You might need to squint a bit.

> Indeed - it can only become "fit for public consumption" if people
> submit their usability enhancements!

True :) I still have no idea how to figure out which submodules are dirty, though. Solving that will enable a safe "submodule update", which I think is more important than "submodule push".

-- 
Eyvind Bernhardsen
Sam Vilain· Mar 30, 2008, 20:19 UTC · re: Eyvind Bernhardsen · lore

Re: git-submodule getting submodules from the parent repository

Eyvind Bernhardsen wrote:
Show 5 quoted lines
> Well, the point of "submodule push" was to avoid having to push in  
> each submodule manually; not enforcing the requirement that commits in  
> submodules must be publicly available before pushing from the main  
> module is a recipe for disaster, or at least annoyance.  And nobody  
> likes an annoying git.
ok.  so, refuse to push without forcing, don't do something dumb.
> Pushing to a branch works except that I couldn't figure out what to do  
> if the push doesn't succeed, ie, the branch has advanced on the remote  
> end.

It's simple. You just fail and tell the user what happened, and let them decide what to do.

> It's a reflog, not a branch, because a submodule can be changed to a  
> different branch, rewound, etc between commits in the main module;  
> there's no requirement that the old commit is in the new commit's  
> history.
If it is a rewind there is no issue, because you don't even need to push.

But again it comes back to - let the user sort it out, don't try to be too clever.

Sam.
Eyvind Bernhardsen· Mar 31, 2008, 10:05 UTC · re: Sam Vilain · lore

Re: git-submodule getting submodules from the parent repository

On 30. mars. 2008, at 22.19, Sam Vilain wrote:
Show 9 quoted lines
> Eyvind Bernhardsen wrote:
>> Well, the point of "submodule push" was to avoid having to push in
>> each submodule manually; not enforcing the requirement that commits  
>> in
>> submodules must be publicly available before pushing from the main
>> module is a recipe for disaster, or at least annoyance.  And nobody
>> likes an annoying git.
>
> ok.  so, refuse to push without forcing, don't do something dumb.

I think you misunderstood: what I'm saying is that submodules' _current_ behaviour is annoying, since you're guaranteed to forget to push a submodule before pushing the main module at least once. My attempt to solve that became too complicated, so I dropped it, and since the current behaviour is annoying, I gave up on submodules entirely.

Show 8 quoted lines
>> Pushing to a branch works except that I couldn't figure out what to  
>> do
>> if the push doesn't succeed, ie, the branch has advanced on the  
>> remote
>> end.
>
> It's simple.  You just fail and tell the user what happened, and let
> them decide what to do.

Sure, that solves the annoyance problem, but I wanted something more automatic.

Show 10 quoted lines
>> It's a reflog, not a branch, because a submodule can be changed to a
>> different branch, rewound, etc between commits in the main module;
>> there's no requirement that the old commit is in the new commit's
>> history.
>
> If it is a rewind there is no issue, because you don't even need to  
> push.
>
> But again it comes back to - let the user sort it out, don't try to be
> too clever.

Yep, my problem was wanting to be cleverer than my limited git skills will allow.

-- 
Eyvind Bernhardsen
Avery Pennarun· Mar 30, 2008, 23:03 UTC · re: Eyvind Bernhardsen · lore

Re: git-submodule getting submodules from the parent repository

On Sun, Mar 30, 2008 at 3:50 PM, Eyvind Bernhardsen <eyvind-git@orakel.ntnu.no> wrote:

>  Pushing to a branch works except that I couldn't figure out what to do
>  if the push doesn't succeed, ie, the branch has advanced on the remote
>  end.  That's a problem if more than one module references the
>  submodule or there are multiple branches in the main module.

That's easy: just error out in that case. If the current system would just error out when I screwed up, I'd at least be able to deal with it. Right now I silently create un-check-outable parent repositories because I failed silently to upload my latest checkins to the child repository.

>  True :)  I still have no idea how to figure out which submodules are
>  dirty, though.  Solving that will enable a safe "submodule update",
>  which I think is more important than "submodule push".
What is unsafe about "submodule update"?
Thanks,
Avery
Eyvind Bernhardsen· Mar 31, 2008, 09:29 UTC · re: Avery Pennarun · lore

Re: git-submodule getting submodules from the parent repository

On 31. mars. 2008, at 01.03, Avery Pennarun wrote:
Show 14 quoted lines
> On Sun, Mar 30, 2008 at 3:50 PM, Eyvind Bernhardsen
> <eyvind-git@orakel.ntnu.no> wrote:
>> Pushing to a branch works except that I couldn't figure out what to  
>> do
>> if the push doesn't succeed, ie, the branch has advanced on the  
>> remote
>> end.  That's a problem if more than one module references the
>> submodule or there are multiple branches in the main module.
>
> That's easy: just error out in that case.  If the current system would
> just error out when I screwed up, I'd at least be able to deal with
> it.  Right now I silently create un-check-outable parent repositories
> because I failed silently to upload my latest checkins to the child
> repository.

As I tried to explain, all the automatic push solutions I could come up with were flawed, so I decided not to use submodules at all and just have the build tool check out every module (that's what we currently do with CVS, so it's the easy way out anyway).

If I understand you correctly, you want to be forced to create a branch and push to that? I don't think that works well with many developers pushing to a shared repository (my situation), and is in any case not the "automagical push" solution that I want. I agree that it would be an improvement, but it doesn't scratch my itch :)

Show 5 quoted lines
>> True :)  I still have no idea how to figure out which submodules are
>> dirty, though.  Solving that will enable a safe "submodule update",
>> which I think is more important than "submodule push".
>
> What is unsafe about "submodule update"?

If you have local changes committed in a submodule that is updated by a pull in the main module, "submodule update" will silently overwrite them. I was wrong, though, because you can fix that just by making "submodule update" error out when a submodule doesn't have its HEAD where the main module thinks it should be.

-- 
Eyvind Bernhardsen
Avery Pennarun· Mar 31, 2008, 21:36 UTC · re: Eyvind Bernhardsen · lore

Re: git-submodule getting submodules from the parent repository

On Mon, Mar 31, 2008 at 5:29 AM, Eyvind Bernhardsen <eyvind-git@orakel.ntnu.no> wrote:

Show 5 quoted lines
> On 31. mars. 2008, at 01.03, Avery Pennarun wrote:
>  As I tried to explain, all the automatic push solutions I could come
>  up with were flawed, so I decided not to use submodules at all and
>  just have the build tool check out every module (that's what we
>  currently do with CVS, so it's the easy way out anyway).

I even *use* git-submodule and had to modify my build scripts because "git submodule init" and "git submodule update" don't seem to kick in automatically for some reason. The ideal situation would be to have git just manage the version control without having to babysit it, of course. That's hard to do in the general case, but should be quite possible in the limited situation that I'm proposing in this thread.

>  If I understand you correctly, you want to be forced to create a
>  branch and push to that?  I don't think that works well with many
>  developers pushing to a shared repository (my situation),

Hmm, this is curious. If you're *not* using submodules, then I don't think you can push successfully without being on a branch, can you? So the suggestion merely extends this behaviour to submodules.

(To be more precise, 'git push' seems only to be able to push branch heads. When you're not using git-submodule, commits are by default attached to branch heads, so this doesn't cause a problem. If you disconnect your HEAD, trying to push will silently do nothing, because it'll push some other branch head that hasn't changed, or maybe no branch at all. But with git-submodule, the *default* is a disconnected HEAD, which is too dangerous. I propose to simply have it fail out in this case.)

If you 'git checkout -b branchname' inside a submodule, then 'git push' will do the right thing, so I'm not sure what you'd want to be more automagical than that.

Show 5 quoted lines
>  If you have local changes committed in a submodule that is updated by
>  a pull in the main module, "submodule update" will silently overwrite
>  them.  I was wrong, though, because you can fix that just by making
>  "submodule update" error out when a submodule doesn't have its HEAD
>  where the main module thinks it should be.

Shouldn't "git merge" get a merge conflict if you've made a checkin that changed the submodule pointer, then try to pull someone else's checking that changes the submodules pointer to something else? It would seem there's no better option than that.

While we're here, it's inconvenient to have to call "git submodule update" at all when there *isn't* a conflict. It should always be safe for git checkout or git merge to do that for you, no?

Thanks,
Avery
Sam Vilain· Apr 1, 2008, 23:05 UTC · re: Avery Pennarun · lore

Re: git-submodule getting submodules from the parent repository

Avery Pennarun wrote:
> I even *use* git-submodule and had to modify my build scripts because
> "git submodule init" and "git submodule update" don't seem to kick in
> automatically for some reason.

The reason is that not everyone wants that by default. Perhaps it is a good idea for it to be default behaviour; but all in good time. It can be a user-selected option to clone first.

> The ideal situation would be to have
> git just manage the version control without having to babysit it, of
> course.

I can understand the motivation to write such disparaging remarks; however it may be more productive to come up with good ideas about how it can be made to work better for you, without getting in the way of other users. patches are even better!

Show 6 quoted lines
>>  If I understand you correctly, you want to be forced to create a
>>  branch and push to that?  I don't think that works well with many
>>  developers pushing to a shared repository (my situation),
> 
> Hmm, this is curious.  If you're *not* using submodules, then I don't
> think you can push successfully without being on a branch, can you?
Sure, you could;
  git push origin HEAD:branchname

However I think the right solution to this is to name the branch appropriately somehow, so that the default push operation works.

> If you 'git checkout -b branchname' inside a submodule, then 'git
> push' will do the right thing, so I'm not sure what you'd want to be
> more automagical than that.

Well, where did you get the branch name from? That's the part that requires user intervention. You could make an educated guess, such as with git name-rev, but it would not necessarily be the right guess - so user confirmation of the choice would be desirable.

Sam.
Avery Pennarun· Apr 1, 2008, 23:56 UTC · re: Sam Vilain · lore

Re: git-submodule getting submodules from the parent repository

On 4/1/08, Sam Vilain <sam@vilain.net> wrote:
Show 8 quoted lines
>  > The ideal situation would be to have
>  > git just manage the version control without having to babysit it, of
>  > course.
>
> I can understand the motivation to write such disparaging remarks;
>  however it may be more productive to come up with good ideas about how
>  it can be made to work better for you, without getting in the way of
>  other users.  patches are even better!
I didn't mean anything disparaging.  I have nothing against babysitters :)

I'll be happy to work on patches once we have some sort of consensus on what would be desirable. I think we're slowly getting there.

Show 10 quoted lines
>  >>  If I understand you correctly, you want to be forced to create a
>  >>  branch and push to that?  I don't think that works well with many
>  >>  developers pushing to a shared repository (my situation),
>  >
>  > Hmm, this is curious.  If you're *not* using submodules, then I don't
>  > think you can push successfully without being on a branch, can you?
>
> Sure, you could;
>
>   git push origin HEAD:branchname

Okay, yes. But that's just arbitrarily avoiding a local branch and creating a remote one instead. I can't imagine a situation where you'd really want the local branch to be anonymous while the remote one is not.

When doing a normal "git clone" without submodules, git automatically creates you a local branch with the same name as the remote's .git/HEAD - which is rather arbitrary, but even an arbitrary local name is better than no name, and when checking out a brand new submodule, there are *no* local branches, so a name conflict is impossible.

Show 8 quoted lines
>  > If you 'git checkout -b branchname' inside a submodule, then 'git
>  > push' will do the right thing, so I'm not sure what you'd want to be
>  > more automagical than that.
>
> Well, where did you get the branch name from?  That's the part that
>  requires user intervention.  You could make an educated guess, such as
>  with git name-rev, but it would not necessarily be the right guess - so
>  user confirmation of the choice would be desirable.

Here's a paraphrase of what I suggested earlier. I don't think it got a response:

Instead of storing only the commitid of each submodule in the parent tree, store the current branch name as well. Use this as a hint to 'submodule update' so that when it checks out commitid, it names the local branch with the same name as it used to have. (This is rather user-friendly since if I check in, push, and clone, my new submodule checkout will have the same branchname as it used to have.)

Note that the newly checked-out submodule branch will probably have the same name as as remote branch. However, the remote branch may refer to a different commitid (for example, if someone has pushed to that branch after the parent repo was last updated). This is exactly right; it means that if I cd into the submodule and "git push", it'll fail because I'm not up to date (I can always switch to a new branch if I want), and if I "git pull", it'll pull from the place where it should.

This way, cloning a project with submodules will work much like cloning the parent project; pushing and pulling the parent and the submodules will do as you expect.

The bad news is that this would require a change to the tree format for submodules (to contain the branch name). Is that a problem? Can it be done in a backwards-compatible way?

Also, I think this is the only time I've seen a branch name in the commit/tree structure, which normally refers only to sha-1 hashes. Is that a problem? Is there a better way?

Thanks,
Avery
Junio C Hamano· Apr 2, 2008, 00:35 UTC · re: Avery Pennarun · lore

Re: git-submodule getting submodules from the parent repository

"Avery Pennarun" <apenwarr@gmail.com> writes:
Show 6 quoted lines
> Instead of storing only the commitid of each submodule in the parent
> tree, store the current branch name as well.
> ...
> This way, cloning a project with submodules will work much like
> cloning the parent project; pushing and pulling the parent and the
> submodules will do as you expect.

That goes quite against the fundamental design of git submodules in that the submodules are by themselves independent entities. An often-cited example is an appliance project, where superproject bundles a clone of Linux kernel and a clone of busybox repositories as its submodules.

Each submodule is an independent project, and as such, must not know anything about the containing superproject (iow, the superproject can know what the submodules are doing, but submodules should not know they are contained within a particular superproject).

If your superproject (i.e. the appliance product) uses two branches to manage two product lines, named "v1" and "v2", these names are local to the superproject. It should not force the projects you borrow your submodules from to have branches with corresponding name.

Also, the submodules and the superproject are meant to be loosely coupled. A single branch in superproject (say "v1") may have many branches in a submodule ("add frotz to v1 product", "improve nitfol in v1 product") that can potentially be merged and bound to.

The work flow for updating a tree would look like:
 - People "git fetch" in superproject and in its submodules. They
   obviously prime their tree with "git clone" of superproject, and the
   submodules they are interested in, and a single fetch will update all
   the remote tracking branches, so it does not really matter which branch
   is checked out.  However, if you employ a central repository model to
   keep them, an invariant must hold: all the necessary commits in
   submodules must be _reachable_ from some branch in them.
 - When not working in a particular submodule, but using it as a component
   to build the superproject, it would be better to leave its HEAD
   detached to the version the superproject points at.  IOW, usually you
   won't have to be on any branch in submodules unless you are working in
   them.
 - Sometimes you need to work in a submodule; e.g. you would want to add
   'frotz' tool to your copy of busybox.  You chdir to the submodule
   directory, and develop as if there is no superproject.
   - Most likely, because the superproject and the submodules are meant to
     be tied loosely, the (detached) HEAD you see in the submodule
     together with the superproject may be _behind_ the tip of any
     branches in the submodule repository.  This is really up to the
     project's policy, but the superproject is meant to point at a commit
     from submodules that is proven to be robust, and it may even be the
     responsibility of people above you to decide when and to what commit
     to advance the commit the superproject points at, and developers are
     given a single "development" branch to work on.  The project policy
     might be "a new feature should be on a new topic branch forked at the
     commit the superproject points at, and let the project leader to know
     what the topic is about so that it can be merged to the integration
     branch".  Follow the project policy and check out the branch to work
     on.
   - Then work on adding that 'frotz' tool.  Make commits, test it in
     isolation and test with superproject.  Push it out as whichever
     remote branch the project policy asks you to.
 - If you are the person who decides which commit in a submodule should be
   pointed at by the tree of the superproject, then check out the
   superproject, and check out the submodule at that commit.  Update the
   gitlink for the submodule and make a commit in the superproject.  This
   step may happen at the same time as the previous step if you worked in
   the submodule to advance its development branch tip and if you are the
   person who decides which submodule commit to point at from the
   superproject.
 - At any step above, a new clone (or a previously cloned trees, updated
   with "git fetch" in appropriate places) will have all necessary
   objects.
Avery Pennarun· Apr 2, 2008, 02:03 UTC · re: Junio C Hamano · lore

Re: git-submodule getting submodules from the parent repository

On 4/1/08, Junio C Hamano <gitster@pobox.com> wrote:
Show 10 quoted lines
> "Avery Pennarun" <apenwarr@gmail.com> writes:
>  > Instead of storing only the commitid of each submodule in the parent
>  > tree, store the current branch name as well.
> > ...
> > This way, cloning a project with submodules will work much like
>  > cloning the parent project; pushing and pulling the parent and the
>  > submodules will do as you expect.
>
> That goes quite against the fundamental design of git submodules in that
>  the submodules are by themselves independent entities.

Not sure what you mean here; the supermodule already stores the commitid of the submodule. All I'm proposing is that it also store the default branchname (ie. the branchname that the submodule was using when its gitlink was checked into the supermodule) along with that commitid. The submodule never knows anything about the supermodule.

>  An often-cited
>  example is an appliance project, where superproject bundles a clone of
>  Linux kernel and a clone of busybox repositories as its submodules.
What a coincidence!  This is almost exactly like my situation :)
>  If your superproject (i.e. the appliance product) uses two branches to
>  manage two product lines, named "v1" and "v2", these names are local to
>  the superproject.  It should not force the projects you borrow your
>  submodules from to have branches with corresponding name.

I meant that we should store the submodule's branch name when committing the superproject, and put it back when checking out the submodule fresh from the superproject.

Show 5 quoted lines
>   - When not working in a particular submodule, but using it as a component
>    to build the superproject, it would be better to leave its HEAD
>    detached to the version the superproject points at.  IOW, usually you
>    won't have to be on any branch in submodules unless you are working in
>    them.

I agree that the submodule should have its HEAD pointing at exactly the superproject-specified commit. However, I believe this commit should have a local branch name (in the subproject) attached to it, or else (as I and my co-workers have frequently experienced) people will accidentally check in to a nameless branch, causing 'git push' to silently not upload anything, and thus lose track of their commits. I have lost work this way.

The idea of naming the local-subproject-branch with the same name as it had on checking is that then "git pull" in the subproject will work exactly as expected: it'll get you the latest version of the branch the superproject developer was on. But if you *don't* explicitly "git pull" in the subproject, I'd expect (of course) the checkout to stick to the commit specified by the superproject - and also to leave its local branch name pointing at exactly that commit.

>   - Sometimes you need to work in a submodule; e.g. you would want to add
>    'frotz' tool to your copy of busybox.  You chdir to the submodule
>    directory, and develop as if there is no superproject.

This is where my workflow is a bit different. One of my subprojects is a library that gets used by several application superprojects. I often add features to my library in the process of editing a particular superproject. I also expect my co-developers to want to do the same. Thus, the difference from your example is that I want to streamline the process of working in a subproject as well as a superproject, and minimize the chances of losing data in this case.

With the current system the way it is, it's too easy to make mistakes, and it requires too many steps to fetch/merge/rebranch each submodule.

>    - Then work on adding that 'frotz' tool.  Make commits, test it in
>      isolation and test with superproject.  Push it out as whichever
>      remote branch the project policy asks you to.

As an orthogonal secondary wish, I'd like to have the subproject and superproject hosted in the same remote repository. This appears to be possible (albeit inefficiently right now) by using "." as the remote repo name in .gitmodules. It would be more efficient if git-submodule-update would use the superproject's checkout as a --alternate when cloning the submodule... I think that would be easy and harmless, right?

The super-summary of all that is I think I'd like to make three git changes here:

1) When checking out a submodule from scratch, use the local
supermodule as a --alternate.  That way if both super and submodule
are hosted in the same remote repo, I don't have to clone them twice.
(And cloning my local repo to another copy doesn't stop git-submodule
from working.)
2) When checking out a submodule, give the submodule's current commit
a useful branch name (ideally, the name it had when the gitlink was
checked into the supermodule).  When updating a submodule with
git-submodule-update, quietly fixup the submodule's local branch ref
if it hasn't been changed; else produce a conflict of some sort.
3) Bonus: make "git push" operate recursively on submodules, and "git
pull" automatically run git-submodule-update.
Does that make sense?
Thanks,
Avery
Sam Vilain· Apr 2, 2008, 20:06 UTC · re: Avery Pennarun · lore

Re: git-submodule getting submodules from the parent repository

Avery Pennarun wrote:
Show 12 quoted lines
>>  > Instead of storing only the commitid of each submodule in the parent
>>  > tree, store the current branch name as well.
>> That goes quite against the fundamental design of git submodules in that
>>  the submodules are by themselves independent entities.
>>     
> Not sure what you mean here; the supermodule already stores the
> commitid of the submodule.  All I'm proposing is that it also store
> the default branchname (ie. the branchname that the submodule was
> using when its gitlink was checked into the supermodule) along with
> that commitid.  The submodule never knows anything about the
> supermodule.
>   
How about this.

This could be an optional disambiguator in .gitmodules in the superproject, to allow you to "store the branch it was made from". Glue to make this automatic/easy optional.

When updating a submodule, with an option set (or configured; which might even later become a default if people like it enough), it will try to figure out a reasonable branch for that commit, using git-name-rev, and check out the branch with that name. It first uses the hint above as an argument to git-name-rev --refs=XX, and if that doesn't provide a reasonable answer then look for any branch.

I think this approach would not get in the way of people who don't want to use it, and provide for making things automatic for those who want that.

Show 8 quoted lines
> I agree that the submodule should have its HEAD pointing at exactly
> the superproject-specified commit.  However, I believe this commit
> should have a local branch name (in the subproject) attached to it, or
> else (as I and my co-workers have frequently experienced) people will
> accidentally check in to a nameless branch, causing 'git push' to
> silently not upload anything, and thus lose track of their commits.  I
> have lost work this way.
>   

I think this is a separate argument against git-push, the default behaviour of which also causes me to tell people not to use the argument-less form of git-push until they understand how to use the two-argument form.

In the context of git-submodule, adding features to it to avoid this if that is the way that the user wants to work would be good IMHO.

> The idea of naming the local-subproject-branch with the same name as
> it had on checking is that then "git pull" in the subproject will work
> exactly as expected:

It's not really the local branch name anyway, it's how the default push gets configured; perhaps it's worth distinguishing which part you are interested in.

Show 9 quoted lines
> The super-summary of all that is I think I'd like to make three git
> changes here:
>
> 1) When checking out a submodule from scratch, use the local
> supermodule as a --alternate.  That way if both super and submodule
> are hosted in the same remote repo, I don't have to clone them twice.
> (And cloning my local repo to another copy doesn't stop git-submodule
> from working.)
>   

Yes - you've already seen the SoC plan for that, although I believe no students applied for that one, and if you think it's minor enough to do, great.

Show 6 quoted lines
> 2) When checking out a submodule, give the submodule's current commit
> a useful branch name (ideally, the name it had when the gitlink was
> checked into the supermodule).  When updating a submodule with
> git-submodule-update, quietly fixup the submodule's local branch ref
> if it hasn't been changed; else produce a conflict of some sort.
>   
See above for how I think that should work.
> 3) Bonus: make "git push" operate recursively on submodules, and "git
> pull" automatically run git-submodule-update.
>   

I'd appreciate that feature - though I'm more interested in making sure that I don't push anything where the submodule commit is not available via the URL listed in .gitmodules.

Presuming such a check, would that check happen at push time, or do you check at a different time, such as when committing, or when adding the submodule to the index?

I think checking that referential integrity is something perhaps easier to bite off and get people to agree on. I think it would solve the overall process problem, by forcing people to push the submodule before the commit of the superproject can succeed without forcing.

Thoughts/comments? Sam

Junio C Hamano· Apr 2, 2008, 21:32 UTC · re: Eyvind Bernhardsen · lore

Re: git-submodule getting submodules from the parent repository

Eyvind Bernhardsen <eyvind-git@orakel.ntnu.no> writes:
Show 7 quoted lines
> One solution that occurred to me was to have a branch in each
> submodule for every main module and branch.  A branch name would be
> provided for each submodule in .gitmodules, used by "submodule push"
> but not "submodule update".  In this case, if the push to the branch
> fails, the main module branch is probably behind too.
>
> This seemed like a good idea, but it's racy.

It's not just racy, but I think it's wrong to limit to _one_ branch in each submodule..

A submodule is an independent project on its own.
Suppose the commit DAG in the submodule looked like this:
                 o---o
                /     \
     --o---o---o---o---o---X---o---Z
            \                 /
             o---o---o---o---o---o
                  \     /
                   o---o

and the superproject points at commit X. You may need to tweak the submodule to make it work better with the change you are making to the superproject.

You have two choices:
 (1) update to some "stable" branch head that is descendant of X first,
     and make sure it works with the superproject.  Then develop on top of
     it, and bind the tip of suc development trail to the superproject:
                 o---o
                /     \
     --o---o---o---o---o---X---o---Z---o---o---Y (your changes are Z..Y)
            \                 /
             o---o---o---o---o---o
                  \     /
                   o---o

I think this is what you are suggesting. But the superproject may not be ready to use the submodule with the history from the lower side branch merged in. You would

 (2) fork off of X and develop; bind the tip of such development trail to
     the superproject.  IOW, you make the submodule DAG like this, and
     then "git add" commit Y in superproject.
                 o---o       o---o---Y (your changes)
                /     \     /
     --o---o---o---o---o---X---o---Z
            \                 /
             o---o---o---o---o---o
                  \     /
                   o---o

Sometimes forked branches need to be maintained until it proves stable (and then your "tip" Y may be merged back to the tip of a public branch Z). So you would at least need to allow a set of topic branches in submodules that corresponds to a single lineage of superproject history.

Then when both Z (with the changes from the lower side branch) and Y (your changes) prove stable, the submodule project may decide to make a merge between Y and Z.

                 o---o       o---o---Y (your changes)
                /     \     /         \
     --o---o---o---o---o---X---o---Z---W
            \                 /
             o---o---o---o---o---o
                  \     /
                   o---o

The superproject may decide to "git add" the result of such a merge, but that decision is done separately (and obviously after such a merge is made).

If everybody involved in the superproject forked from whatever happened to be bound to the superproject, however, the submodule will have uncontrolled number of unmerged "tips". For the submodule to stay viable as an independent project, some management has to be done to clean up this mess. To manage the forked development inside submodule properly, I do not think you can autocruise from the toplevel superproject and leave the random branches or unnamed commits unmerged.

That is why I suggested to:
 * Leave the HEAD in submodule detached, if you are not working in it;
 * Have a project policy regarding the use of branches in the submodule.
   When you need to work on submodule, first switch to the branch (the
   policy may allow/encourage you to create a new topic branch here), and
   commit to it.
 * The policy should also say when these forked branches should be merged;
   keep them tidy by following that policy.

And by pushing from submodule and then in toplevel, you will never have "superproject names a commit unreachable from any of the branch tips of submodules" problem. Nor there is any raciness issue -- only after you push out the submodule successfully, you push out the toplevel (and if the former fails, you may need to redo the toplevel commit, but that happens before you push it out so you can afford to rebase or amend).

Avery Pennarun· Mar 30, 2008, 23:00 UTC · re: Sam Vilain · lore

Re: git-submodule getting submodules from the parent repository

On Sat, Mar 29, 2008 at 7:22 PM, Sam Vilain <sam@vilain.net> wrote:
Show 7 quoted lines
> Avery Pennarun wrote:
>  > What if *all* the objects for A, B, and C were always in the *same*
>  > repository?  Almost all the problems would go away.  Imagine if it
>  > worked like this:
>
>  Well, that would create a lot of unnecessary work when cloning.
>  Partitioning by project is a natural way to divide the projects up.

What unnecessary work do you mean? Certainly fetching only a particular set of refs from a remote repository is possible, as that's what 'git pull' does.

I agree that partitioning by project makes sense... but it also seems to me that throwing extra objects into a repository that requires them anyhow shouldn't have any major negative results. After all, if you can't build A without B, then downloading A might as well download the objects from B too. Which is not to say that B shouldn't *also* have its own repository.

>  It's worth noting that the early implementations of submodules were
>  based on this design, of keeping everything together.

I'd like to read about the rationale behind this change. Is there a thread you can point to?

Show 5 quoted lines
>  However, what you are suggesting should IMHO be allowed to work.  In
>  particular, if the submodule path is ".", then I think there's a good
>  case that they should come from within the same project.  If it's a
>  relative URL, it should initialize based on the remote URL that was used
>  for the original fetch (or, rather, the remote URL for the current branch).

I agree, there's no reason to take away the existing functionality of allowing split repos. I was more suggesting a new functionality so that splitting isn't *required*.

Show 9 quoted lines
>  > 2. You still check into C, then B, then A, but it doesn't actually
>  > matter if you put B and C on a branch first or not, because 'git push'
>  > will work properly, because it auto-pushes B and C revisions based on
>  > the fact that A refers to them (ie. implicit branches via the
>  > submodule mechanism).
>
>  This push failure thing is regrettable; however it's not clear which
>  branch name the submodules should get.  A given commit might exist on
>  several branches, which one do you choose to name it?

One option is to make a simple "git push origin" operation fail if you're not on any branch; iirc, if you try that now, it just silently *succeeds* without uploading anything at all, which is one reason I so frequently screw it up. Alternatively, is there a reason I can't upload an object *without* giving it a branch name? I guess that would cause problems with garbage collection.

Now, the fail-on-branchless-push option still isn't really perfect,
because then I'll screw up like this:
- make change
- check in
- try to push: fails
- switch to branch
- realize I've lost my checkin(s) and have to go scrounge in the
reflog to try to find it

If we could disallow checkins to disconnected heads, then I'd get an error at step 1, before I had a chance to screw up. I think that would be a usability improvement to git in general. For example, if I screw up a git-rebase and forget to abort, my HEAD ends up disconnected and I occasionally check things in by accident and then lose them (only to be saved by the reflog). Perhaps an extra option to git-commit that must be used if you want to check into a non-branch? Is that too harsh?

Another option would be to simply *always* create/update a branch tag when doing "git submodule update". But then the question is which branch tag. One thing that would give pretty useful semantics would be to create a local branch tag with the same name as it had when the submodule ref was checked in in the first place.

That is, if I make a change to B on branch "master," and then check that into A, then the next time someone checks out A, it would be great if it retrieved B with the same commitid, then named that commitid "master". This is *despite* the fact that the "remotes/master" branch might be *newer* than the newly created local "master". Why? Because it would give standard git semantics to my new checkout: I can "git pull" to pull in the latest changes from remote/master, or I can checkin and try to "git push" and it'll fail, just like it should, which would encourage me to either "git pull" or create a *new* topic branch, just as it should.

I think that solution would be great for me, but it would require changes to the tree format in order to store the branch name, which is unfortunately since trees currently don't know *anything* about branch names afaict. Is there a better way to achieve the same result? (Note that in this case, "correct" operation doesn't *require* the new branch name information; we just use it as a hint.)

Show 8 quoted lines
>  > 4. You can 'git clone' a local copy of A, and B/C will be cloned
>  > automatically along with it.
>
> > 6. git-pull should be modified to auto-download objects referred to by
>  > 'submodule' references in trees.
>
>  I think this could be a switch to git clone/pull, configurable to be the
>  default action.

Sure. Or it could try by default, but not error out if it turned out not to be able to find them.

Show 5 quoted lines
>  > 5. B and C, when git-submodule checks them out, should have their own
>  > .git directories, but use A as an 'alternatives' entry.
>
>  There is also a Google Summer of Code project for this - see
>  http://git.or.cz/gitwiki/SoC2008Ideas#head-9215572f23513542a23d3555aa72775bc4b91038

ok. I was hoping it wouldn't be so hard as to require an entire SoC project, since using --alternate when checking out the child repo shouldn't be too hard.

Have fun,
Avery
Sam Vilain· Apr 1, 2008, 23:10 UTC · re: Avery Pennarun · lore

Re: git-submodule getting submodules from the parent repository

Avery Pennarun wrote:
>>  Well, that would create a lot of unnecessary work when cloning.
>>  Partitioning by project is a natural way to divide the projects up.
> 
> What unnecessary work do you mean?
A full clone takes a few shortcuts, especially over dumb transports like
HTTP.  I think there might be shortcuts in the git-daemon code as well.
 Forcing these to be partial might make these full fetches involve more
time.
Show 8 quoted lines
>>  However, what you are suggesting should IMHO be allowed to work.  In
>>  particular, if the submodule path is ".", then I think there's a good
>>  case that they should come from within the same project.  If it's a
>>  relative URL, it should initialize based on the remote URL that was used
>>  for the original fetch (or, rather, the remote URL for the current branch).
> I agree, there's no reason to take away the existing functionality of
> allowing split repos.  I was more suggesting a new functionality so
> that splitting isn't *required*.
Yes - I look forward to a patch.
Show 7 quoted lines
>>  This push failure thing is regrettable; however it's not clear which
>>  branch name the submodules should get.  A given commit might exist on
>>  several branches, which one do you choose to name it?
> One option is to make a simple "git push origin" operation fail if
> you're not on any branch; iirc, if you try that now, it just silently
> *succeeds* without uploading anything at all, which is one reason I so
> frequently screw it up.
Sounds workable.
> Alternatively, is there a reason I can't
> upload an object *without* giving it a branch name?  I guess that
> would cause problems with garbage collection.
You've answered your own question there.
Show 6 quoted lines
>>  There is also a Google Summer of Code project for this - see
>>  http://git.or.cz/gitwiki/SoC2008Ideas#head-9215572f23513542a23d3555aa72775bc4b91038
> 
> ok.  I was hoping it wouldn't be so hard as to require an entire SoC
> project, since using --alternate when checking out the child repo
> shouldn't be too hard.

If you think it is simpler, then I'm sure that submodules users would appreciate you sharing your ideas as a patch. Sorry if I am starting to sound like a parrot ;-).

Sam.
Johannes Sixt· Mar 31, 2008, 06:22 UTC · re: Avery Pennarun · lore

Re: git-submodule getting submodules from the parent repository

Avery Pennarun schrieb:
Show 12 quoted lines
> It's a  pain to check out / mirror / check in / push.  git-submodule
> doesn't even init automatically when you check out A, so you have to
> run it yourself.  The relative paths of A, B, and C on your mirror
> have to be the same as upstream.  You can't make a local mirror of A
> without mirroring B and C.  B and C start out with a disconnected
> HEAD, so if you check in, it goes nowhere, and then when you push,
> nothing happens, and if you're unlucky enough to pull someone else's
> update to A and then "git-submodule update", it forgets your changes
> entirely.  When you check in to C, you then have to check in to B, and
> then to A, all by hand; and when you git-pull, you'd better to C, then
> B, then A, or risk having A try to check out a revision from B that
> you haven't pulled, etc.
Would a "recurse" sub-command help your workflow?
http://thread.gmane.org/gmane.comp.version-control.git/69834
-- Hannes
Avery Pennarun· Mar 31, 2008, 21:24 UTC · re: Johannes Sixt · lore

Re: git-submodule getting submodules from the parent repository

On Mon, Mar 31, 2008 at 2:22 AM, Johannes Sixt <j.sixt@viscovery.net> wrote:
Show 17 quoted lines
> Avery Pennarun schrieb:
> > It's a  pain to check out / mirror / check in / push.  git-submodule
>  > doesn't even init automatically when you check out A, so you have to
>  > run it yourself.  The relative paths of A, B, and C on your mirror
>  > have to be the same as upstream.  You can't make a local mirror of A
>  > without mirroring B and C.  B and C start out with a disconnected
>  > HEAD, so if you check in, it goes nowhere, and then when you push,
>  > nothing happens, and if you're unlucky enough to pull someone else's
>  > update to A and then "git-submodule update", it forgets your changes
>  > entirely.  When you check in to C, you then have to check in to B, and
>  > then to A, all by hand; and when you git-pull, you'd better to C, then
>  > B, then A, or risk having A try to check out a revision from B that
>  > you haven't pulled, etc.
>
>  Would a "recurse" sub-command help your workflow?
>
>  http://thread.gmane.org/gmane.comp.version-control.git/69834

Well, typing "git submodule recurse push" or something would allow me to lose the same data without typing quite as much, so strictly speaking I guess it would be an improvement :)

I'd like it even more if "git push" actually somehow refused to push at all if I forgot to push in the submodules.

Have fun,
Avery

← back to recent threads