patch, 16 partsgit-config: read remote config files over HTTP

From: Sven Verdoolaege <skimo@kotnet.org>

This patch series implements a mechanism for cloning submodules. Each submodule is specified by a 'submodule.<submodule>.url' configuration option, e.g.,

bash-3.00$ ./git-config --remote=http://www.liacs.nl/~sverdool/isa.git --get-regexp 'submodule\..*\.url' submodule.cloog.url /home/sverdool/public_html/cloog.git submodule.cloog.url http://www.liacs.nl/~sverdool/cloog.git

git-clone will use the first url that works. E.g., a

git clone --submodules ssh://liacs/~/public_html/isa.git

(which only works for me), will use the first url, while a

git clone --submodules http://www.liacs.nl/~sverdool/isa.git

will use the second.

The cloning of submodules is now handled inside git-fetch.

skimo

From: Sven Verdoolaege <skimo@kotnet.org>

This command dumps the config of a repository and will be used to read config options from a remote site.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 .gitignore                        |    1 +
 Documentation/cmd-list.perl       |    1 +
 Documentation/git-dump-config.txt |   37 +++++++++++++++++++++++++++++++++++++
 Makefile                          |    1 +
 daemon.c                          |    7 +++++++
 dump-config.c                     |   29 +++++++++++++++++++++++++++++
 6 files changed, 76 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/git-dump-config.txt
 create mode 100644 dump-config.c

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 Documentation/git-config.txt |   33 +++++++++++++++++++++---------
 builtin-config.c             |   44 ++++++++++++++++++++++++++++++++---------
 cache.h                      |    1 +
 config.c                     |   26 ++++++++++++++++++++++++
 4 files changed, 84 insertions(+), 20 deletions(-)

From: Sven Verdoolaege <skimo@kotnet.org>

This allows us to use the methods provided by http.c from within libgit, in particular config.c.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 http-fetch.c |    5 ++++-
 http-push.c  |    5 ++++-
 http.h       |    2 +-
 3 files changed, 9 insertions(+), 3 deletions(-)

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 Makefile           |    7 ++++++-
 builtin-config.c   |    8 ++++++--
 config.c           |   16 +++++++++++++++-
 http.c             |   10 ++++++----
 http.h             |    2 +-
 http_config.h      |    1 +
 http_config_curl.c |   52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 http_config_none.c |    6 ++++++
 8 files changed, 93 insertions(+), 9 deletions(-)
 create mode 100644 http_config.h
 create mode 100644 http_config_curl.c
 create mode 100644 http_config_none.c

From: Sven Verdoolaege <skimo@kotnet.org>

This code was killed by commit fcc387db9bc453dc7e07a262873481af2ee9e5c8.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 unpack-trees.c |    4 ----
 1 files changed, 0 insertions(+), 4 deletions(-)

skimo@liacs.nl writes:

Hmmm.

I am not absolutely sure if the fcc387db change was correct anymore, but in any case, this removal of dead code should not break anything.

But this does not belong to your series either.

Perhaps I should apply this to 'master' regardless of the rest of the series.

From: Sven Verdoolaege <skimo@kotnet.org>

We will need the full cache_entry later to figure out if we are dealing with a submodule.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 unpack-trees.c |   32 ++++++++++++++++----------------
 1 files changed, 16 insertions(+), 16 deletions(-)

From: Sven Verdoolaege <skimo@kotnet.org>

This option currently has no effect.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 builtin-read-tree.c |   25 ++++++++++++++++++++++---
 cache.h             |    3 ++-
 unpack-trees.c      |    1 +
 unpack-trees.h      |    1 +
 4 files changed, 26 insertions(+), 4 deletions(-)

skimo@liacs.nl, Fri, May 18, 2007 21:24:56 +0200:

Can we have this option (and corresponding support in the following patches, of course) first? It is enough to have subprojects working locally, and people can start using them immediately: anyone can clone the subprojects manually if he wishes so.

Cloning of subprojects is still unclear, and frankly I'm not sure it should be done at all. Not even with an option which is off by default.

I noticed there's a whole thread about subprojects that I haven't read yet, so this may have been addressed already ....

On Fri, May 18, 2007 at 11:53:12PM +0200, Alex Riesen wrote:

That's why the clone thing comes last.

Anyone can run git-write-tree and git-commit-tree is she wishes so...

Then don't use it.

The reason for not putting this in shouldn't be that someone doesn't think it is useful; the reason should be that my code is crap.

skimo

Sven Verdoolaege, Sat, May 19, 2007 00:08:26 +0200:

Not the checkout, which is strange. It's mostly about cloning.

It is much more tedious. It have to be done recursively, and with right SHA and you have to cd into right direcotry first and it is git-read-tree and git-checkout-index, BTW.

IOW, it is hard.

The code is not a problem. It can be also discarded because you implemented something no one wants.

I just meant to say, that even if no one wants your subproject cloning code, _I_ support your checkout effort and I am asking for it to be put in.

"First", as the cloning discussion does not seem to be finished (and, as I said, I am not interested in cloning anyway).

Alex Riesen <raa.lkml@gmail.com> writes:

More specifically, at the very high level, what you do and what people want to happen might share the description (e.g. "this allows checkout of subprojects", or "this implements clone of superproject to recurse") but with semantics that may be different from what people would want (I am not saying that is the case, as I do not think the current discussion concluded yet). The _first_ implementation that goes in the mainline pretty much sets the _semantics_ so we would need to be extra careful, all the more so as this is a feature many people seem to want.

It was partly my fault that I mentioned "clone" example in the original message, and then let the discussion drifted to a tangent of the clone topic, namely, how the URL would be determined to clone the subproject from.

I would agree that checkout is a more fundamental operation, and I wanted to make that clear in my message, but checkout and clone has certain chicken-and-egg factor between them. After the clone of superproject, checking it out recursively would need cloning the subprojects. Also after a clone of superproject without the recursive behaviour, when the user explicitly asks a subproject to be checked out, somebody needs to do a clone before the subject can be checked out.

Having said that, let's throw out an strawman for checkout proper and then merge.

The user may or may not want to deal with subprojects, and for something truly large like the KDE case, which is where the superproject support is really needed, a large On/Off switch where an option --subproject makes everything checked out and no subproject is checked out without it is not a usable option. I've already outlined how the .git/config file can be used to define which subprojects are of interested so that the Porcelain layer can decide which ones to recurse into and which ones to leave alone. The design is NOT the only possible/sensible one, and I am sure other people will come up with much nicer organization, but I would consider that is just the matter of details.

Now, suppose "git checkout" needs to recurse into one subdirectory that is to have a subproject. There are three cases:

 (1) There is no git repository yet (the plumbing layer already
     makes sure there is a directory, but does not do anything
     else).

 (2) There already is a git repository there, which is the
     correct repository (perhaps determined by .gitmodules and
     .git/config in the superproject, or presense of the commit
     that is recorded in the superproject's index).

 (3) There is a git repository but it is not the correct one.

We've discussed in the other thread about what to do in case (1) to some degree.

For case (2), I think what should happen there is an equivalent of this:

	$ commit=$(git-rev-parse :subproject)
        $ cd subproject
	$ git-rev-parse --verify $commit || git fetch || barf
        $ git checkout $commit

That is,

 - figure out what commit should be checked out from
   superproject index;

 - make sure the named commit exists, or fetch to make it exist.

 - go there and check out that commit; this implies two things:

   1. if there are local changes, it will be carried along and we
      checkout the named commit;

   2. the repository's HEAD becomes detached;

It is entirely possible that the repository is the _correct_ one but not quite up to date, and you haven't fetched $commit. This is really a variant of (1) -- before being able to check out, somebody has to clone the subproject. Before being able to check out to update the latest, somebody has to fetch in the subproject.

If there are local changes, we would not at least lose them. If you want to get to a clean slate, you can cd there and perform "git reset --hard". If you want to mark that commit in the subproject, you may want to do "git checkout -b branch" after the recursive checkout from the superproject detached the HEAD to the commit.

There is another variant that has already been suggested. The superproject tree and index could record 0{40} object name for the subproject, and say "whatever commit happens to be at the tip of the branch of subproject" (and most likely that URL and branch information would come from .gitmodules and confirmed in the .git/config file). In such a case, the above outline would be adjusted _BUT_ I think what would be checked out will not be the named branch (e.g. refs/heads/master) itself, but the remote branch that tracks it (e.g. refs/remotes/origin/master).

While I am at it, let me think aloud as to what I _think_ should happen in a superproject merge.

 - Carry out the tree-level 3-way merge.  If it trivially
   resolves at the tree-level, we are happy.

 - There could be a case where the commit fetched/merged branch
   has and what the current branch has are different.  If one is
   a fast forward of the other, take it.

 - All other cases will leave the superproject index unmerged.

When the merge is cleanly done, the resulting commit is what we should check out in the subproject directory (if we are recursing into it, of course).

It is likely that in some cases you would want go to the subproject directory and merge the commits at the subproject from our branch and their branch in the superproject's index, and make the resulting commit as the result of the merge for that subproject path in the superproject, but I do not think it is the only valid solution.

Junio C Hamano <junkio@cox.net> wrote:

So I've implemented my own Git subproject support in a Java based tool we use internally. Its actually driving the Git plumbing (as JGit isn't complete enough to do the job) but applies quite a bit to this discussion as it is a working system that implements this "checkout in superproject and recurse into subproject".

First I don't use the subproject support in the core plumbing, because that came along from Linus about 2 days after I wrote this implementation. Our data file looks like:

	use-component: subproject1 >=df8cfac815...
	use-component: subproject2 >=af9b543820...

Or really anything that is a valid commit-ish, and often these are actually just annotated tag names.

During our build process we scan the root project's data file, and clone by the relative URL anything we cannot find locally:

	$(git config remote.origin.url)/../component-links/subproject1.git

to get the subproject repository. We don't require that the subproject1 directory actually be called subproject1 in the superproject, its just a recommendation. That data file is also our build-system driver and the build system driver is pretty darn smart about guessing what is going on. ;-)

You'll notice however that we (more or less) have a very flat structure. The component-links directory is really just a set of symlinks pointing back up a level, as sometimes a component is not stored in a repository named the component name, but the component name matters to the build system.

Yes.  Except we do a few things differently:

 - Only update the subproject if its a strict fast-forward.

 - Abort on a dirty working directory in the subproject or if a merge
 would be required to keep the current commit and the new commit.
 Yes, we don't really support going "backwards".

 - The merge aborting thing is probably wrong for some users,
 but blindly switching to the target commit feels somewhat wrong
 in our own uses.  Sometimes you need the current version of the
 subproject to help you debug an older version of the superproject,
 or sibling subproject.

 - You can't just checkout $commit if you can rev-parse it.
 You need to verify it and its entire reachable object set exists.
 See the local fetch fast-path thing you did recently in e3c6f240fd.

 - We update the user's current branch.  Because we are doing
 a strict fast-forward we're also assuming the user wants the
 current branch to stay correlated to the superproject branch.  Why?
 Most of our users keep the same branch name in all repositories.

Junio C Hamano, Sat, May 19, 2007 05:59:48 +0200:

What if the fetch is not possible? You can't checkout? What about the other subprojects, where the checkout succeeded? Will they be reset to the previuos state?

To me, the fetch sounds pretty dangerous. Maybe the checkout should be two stage: first - we check all subprojects to be checked out if it is possible, second - either fail (default) or checkout what possible, warn the user, leave the incomplete subprojects changed (but not update the index with them, so that they wont be accidentally committed).

Shouldn't that depend on "-m" option given to git-checkout in superproject? Sometime the user have to be sure he can checkout everything as it were, but without breaking the local state (like what current git-checkout without "-m" does).

Universally agreed upon

On Fri, May 18, 2007 at 08:59:48PM -0700, Junio C Hamano wrote:

Does everyone agree that we should fetch (possibly after asking for confirmation from the use) _during_ the checkout ? I now only fetch submodules during a fetch of the supermodule (actually, in my current patch set, I only fetch a submodule the first time I see it, but that's a bug), but if there is a consensus on this, I can switch to fetching during checkout.

As to the key to use to lookup the URL in the config, right now I simply use the directory name where it is attached (which seems like a useful default to me). I'm not all that convinced that we should store a default URL in history, so AFAICS, the only thing we need to store is a mapping between directory names and subproject names. It has been suggested to do that in .gitattributes. Is that OK for everyone, or do we really need a separate .gitmodules ?

skimo

Sven Verdoolaege <skimo@kotnet.org> writes:

I think fetching of subproject during fetch or clone of superproject would not make much sense. Making it part of superproject checkout would probably be the way we will end up going. The detail of "which part of the checkout" would need to be defined, and I tend to agree with Alex that checkout itself would need to be multi-phased, but I think that is a minor implementation detail we can discuss after how the overall flows should look like.

If your (super)repository is _the_ only repository that knows about the aggregation it is doing, I do not think you need anything, as .git/config in the subproject would know where to get updates from. Otherwise, Project wide suggestions need to be there in some machine readable form if you were to allow such superproject distributed, be it in .gitmodules or .gitattributes, don't they?

And frankly, I think .gitattributes is a wrong place to store it, as its semantics is to give attributes to paths that MATCH the glob. You can argue that a pattern in .gitattributes can be written to match only one path, but it still feels conceptually wrong. Something more concrete and exact, like the second level key of .git/config and .gitmodules file format, is preferable.

On Sat, May 19, 2007 at 11:20:12 -0700, Junio C Hamano wrote:

IMHO it makes more sense to fetch during fetch of superproject:

 - If you don't fetch the superproject, it won't start refering to
   unavailable commit of subproject. So should only need to fetch subproject
   after fetching superproject.

 - If you fetch from more than one location, you want to fetch subproject
   from location corresponding to where you fetch superproject from.
   
   Let's have a repository of project P with remotes PA and PB. Let it have
   a subproject S with remotes SA and SB.

   Whenever I pull P from PA, it might refer to commit of S, that is only
   available from SA (because that's what PA owner uses). Whenever I pull
   P from PB, it might refer to commit of S, that is only available from SB
   (again because that's what PB owner uses).

   Now checkout does not know, whether I pulled the target revision from PA
   or PB, so:
    - Either it has to fetch both. But say the commit I want is in SB and SA
      contains a lot of new stuff, which will slow the thing down, though
      I don't need it.
    - Or it has to guess by looking whether any heads in remotes/PA or
      remotes/PB are descendants of the commit being checked out. But that
      feels rather hacky.

   I see several options:
    - Fetch will recurse. This should work ok and is IMHO least magic. We can
      also add some way to specify refspecs for the subproject, giving user
      control over what is fetched.
    - Fetch will store a "pending fetch from" note in the subproject and
      checkout, if it does not find the revision, will try fetching from all
      sources pointed to by those notes. There is still a problem with what
      exactly to fetch (user can specify in config).
    - Checkout will ask all subproject repositories whether they have given
      commit and pull the first one that does. This would get the needed
      commit most certainly. It would be slower though, because it would need
      to ask all the repositories whether they have the particular object.
      It also leaves the tracking branches in subproject in somewhat random
      state (maybe both repositories had the commit, so it pulled from the
      other one that user would etc.).

The extra level of indirection has the advantage, that you can describe moving the same subproject to a different directory.

Jan Hudec <bulb@ucw.cz> writes:

Eh, I was suggesting that the subproject fetch would come after checkout in "fetch and then checkout" sequence of the superproject, and if you are arguing against it, you should justify why it should not happen before checkout, as we both agree it should come after fetch of superproject. Your argument is like saying you have to git-init before doing anything so you should fetch when you git-init. That's not a justification.

Not at all. There is no reason to believe that the case that superproject and subproject come from related URLs is more common. One of the reasons to do a separated project organization is to allow looser bindings of the project from project administrative viewpoint. The integrator may not necessarily have any control over what the subproject guys do, and more importantly, the subproject people do not even care nor be aware of the fact that their project is sometimes bound inside other peoples' superprojects. Think of the embedded appliance vendor binding the kernel, libc and busybox in their superproject that holds them together with the build infrastructure. The kernel folks certainly do not particularly care about the vendor.

On Sun, May 20, 2007 at 11:33:17AM -0700, Junio C Hamano wrote:

I think what Jan means is that if there are two (or more) copies of the superproject then it is more likely that the subproject commit can be found in the subproject repo "pointed to" (e.g., through my submodule.*.url) by the superproject repo you fetched from.

skimo

Hello,

I gave the problem some more thought, and though I follow up on my previous comment below, I can now see this:

So far it was discussed what should happen in fetch (+ checkout). But I think the following are the interesting cases. Please read ALL before responding, they are in somewhat random order:

 For following, assume there is a repo of project super has two branches,
 master and next. The next branch adds subproject sublib. In that state, I fetch
 refs/heads/*:refs/remotes/origin/* and check out master. So I have
 a repository, that contains revision with submodule, but I did not check it
 out yet.

 - Some time later, *without* fetching again, I simply
      git checkout --submodules -b feature1 remotes/origin/next
   Obviously it needs to give me the module.

 - Should that checkout work without network access?

 - Ok, now I start hacking on feature1 and find a bug in sublib, that I need
   to fix for it to work. Therefore I change something within sublib.

   However few days later I am asked to fix a bug in stable release of super.
   Therefore I: git checkout master Now, where does sublib go? It contains
   precious data!

   For the worst case situation assume, that the master branch also has
   directory sublib, so it can't stay where it was as unversioned.

 - The fix in master is done, back to our feature1, right?  git checkout
   --submodules feature1 Obviously re-fetching from upstram won't work. The
   head feature1 now refers to a commit that I made and only exists localy.

 - Now the maintainer of super wants to test the feature1. However sublib
   upstream did not accept the bugfix yet (and is perhaps waiting for
   confirmation, that the fixed version really works well for super, so we
   have to test).

   Therefore I push feature1 to my public repo, set up a public repo with my
   fixes to sublib and configure my public super repo to know about it.

   The maintainer already has a repo of super including sublib submodule. But
   when he pulls from me, he does not have the repo with my fixes.

 - The maintainer reviewed my feature1 and now needs to work on feature2.
   That however requires new upstream version of super. Therefore he needs to
   pull alternatively from both upstream and my repo with super, depending on
   what he works on.

   For the most complex case, assume here that I add more fixes to sublib
   while author of feature2 uses more and more bleeding edge stuff, so the
   maintainer really needs futher changes in sublib from both repos.

 - Also git has to fail safe if I forget to push the sublib, so when the
   maintainer tries to pull super, the refered revision of sublib simply
   won't be found.

I am not sure how to handle these cases. But they are cases that can happen in real life and should be handled somehow. Even if some of them just require some manual configuration.

Here is one possible idea:

We could store the GIT_DIR of submodule within the GIT_DIR of the
superproject instead of the submodule directory itself. So instead of:
 /
 /.git
 /subdir
 /subdir/.git

There would be:
 /
 /.git
 /subdir
 /.git/submodules/submodule-name.git

This would require changes to the logic how git finds GIT_DIR (which would be really deep change), but it would provide place to store the submodule data while the submodule is not being checked out.

This does not address the last two cases above with mutliple sources, each containing some revisions. There I see two options:

 - The submodules are fetched during superproject fetch (based on them being
   configured, even if they are not checked out) and the URL might depend on
   url configured for superproject. That is:
      git fetch --submodules foobar
   would do roughty:
      for GIT_DIR in $GIT_DIR/submodules/*.git; do
         git fetch foobar || git fetch
      done
   So if you configured source of the same name for the subproject, it would
   be pulled, otherwise the default one would.

   Checkout would then be local-only operation, because subprojects are
   up-to-date.

 - The superproject checkout would try fetching all sources of the
   subproject, until the requested revision is found.

   This could be extended to normal checkout doing it as well --
   "git checkout sha1" would try fetching all configured sources if the
   revision was not found.

Perhaps we could actually do both. That is, "git fetch --subprojects" to also fetch all of "$GIT_DIR/submodules/*.git" and checkout to try fetching if it can't find the desired revision.

On Sun, May 20, 2007 at 11:33:17 -0700, Junio C Hamano wrote:

It definitely has to come after fetch on superproject. My original thought was, that it would be weird if it was part of the checkout itself, meaning even checkout that does not follow a fetch. However I thought about it some more and that might conflict with other requirements.

I definitely don't think it's more common. But it's the harder case and it might happen. Generally it will happen if some people work on both the superproject and the subproject. Of course the argument is that than it should not be separate projects, but maybe the teams just partly overlap.

Example of this situation is given above. IMHO it needs to be handled somehow (probably git would have to check all potential sources whether they have the revision in question).

On Mon, May 21, 2007 at 06:59:38PM +0200, Jan Hudec wrote:

I have this already, except that I use /.git/submodules/submodule-name/.git because I was too lazy to figure out how to get clone to use the above without using --bare, because --bare disables separate-remotes.

Is there any good reason, btw for --bare not to do separate-remotes ? We could throw out a lot of speical cases, especially the --bare http fetch if we would simply always do a separate-remotes.

Euhm.... I just add a symlink...

skimo

Sven Verdoolaege <skimo@kotnet.org> writes:

Traditinoally, the purpose of --bare is to set up distribution points, either CVS style "shared public repository that everybody pushes into and fetches from", or "owned by me and I push my change into it to publish". In either case, there is no point of even having an upstream in such a repository, let alone refs/remotes/origin/ hierarchy.

So that's the "good reason" part.

Having said that, last night I was thinking about making the layout after a clone a bit more flexible and orthogonal to existence of working tree. What got me thinking about it was different from your motivation, though.

If you have an ordinary working tree, and if you want to have a clone of that working tree that you can work in, independently (i.e. not treating the repository you cloned from any specially from other repositories by marking it "origin"), currently there is no straightforward way to do so, other than doing something like:

 (1) do a --bare clone, which would create "project.git";

 (2) mkdir "project" && mv "project.git" "project/.git";

 (3) edit "project/.git/config" and mark it as a non-bare
     repository.

 (4) "git checkout -f HEAD".

I'd call this a "pure" clone. It is as close as the original repository you would get without copying other per-repository data such as .git/info, .git/config, so that you can start using it as an _independent _repository. It is like a --bare clone but with a working tree associated with it.

What you want is probably the opposite. A bare, but non-pure clone. By non-pure, what I mean is:

 (1) it has an upstream ("origin"), hence
 (2) it has refs/remotes/origin, and "remotes.origin.*"

If the word "non-pure" has a negative connotation, you could call it "a follower repository".

In short, the current behaviour of --bare is "bare and pure", and clone without --bare is "not bare, and a follower". What I wanted was "not bare, but pure", and I think what you want is "bare, but a follower".

On Mon, May 21, 2007 at 20:05:06 +0200, Sven Verdoolaege wrote:

Glad to hear it. I didn't really have time to read through all the patches.

Yes. Except I am not sure mingw supports that.

hoi :)

On Mon, May 21, 2007 at 06:59:38PM +0200, Jan Hudec wrote:

I agree that we need something like that.

We don't have to move the entire subproject.git into the superproject, but we need to have all _referenced_ objects in the .git dir of the superproject.

There are several possibilities to do so:

 * move the entire .git dir
 * move .git/objects
 * explicitly copy all referenced objects

I have some experimental code to configure a per-subproject directory in the superproject/.git as alternate object store for the submodule to make the last two solutions possible. Perhaps I should dig it out again and adapt it to current git.

If there is a 1:1 relationship between subproject and object store then even efficient fsck and repack/prune are possible for the submodule without loosing objects. But such a 1:1 relationship is bad when you move subprojects to another location (or include the same subproject several times in different locations of the tree). Perhaps the user should be able to choose which one he wants.

I think it will be _very_ common to store super and subprojects in related locations. First to be independent from third-party servers while working on the superproject. Second (and I think more important) because many times there will be superproject related adaptations in the subproject. Yes they are independent, and exactly for that reason the subproject upstream maintainers may not take every change which is needed to satisfy the superproject. We _now_ see that in all Linux distributions already. So when you use superprojects to integrate several independent projects, then the superproject maintainer/administrator should really keep a clone of all subprojects handy on his site.

On Mon, May 21, 2007 at 23:11:34 +0200, Martin Waitz wrote:

I believe we really need entire .git dir. When the superporject checks out revision which does not reference that subproject, we still need to preserve not only the objects of subproject, but also the refs and config.

That's why there should be the extra level of indirection using .gitmodules. It should map the directory name to the object store name, so you can relocate the subproject.

Including the same project several times is indeed interesting. Maybe the subprojects should be "light checkouts" (I believe something like this was already discussed on the list sometime). Those would be .git dirs, that would only have HEAD and pointer to another .git dir with everything else.

Yes, repositories with distribution-specific patches will add a large class of cases requiring multiple sources support.

On Tue, May 22, 2007 at 09:37:06PM +0200, Jan Hudec wrote:

but all the other refs do not belong to the superproject. For those who are working on the subproject there are of course a lot of refs which they have to work with, but that can be dealt with outside of the superproject scope. The subproject is still a normal Git repository, after all. That is, you can have remote entries, branches and what not. But all that is not interesting in the superproject scope.

So I thing moving the entire subproject.git into the superproject.git is too much. The superproject is only interested in the objects and in one ref -- the one stored inside its tree.

Well, even if they might share a lot of objects they might be included for completely different reasons and so might need to work with different communities (remote entries, branches, etc.).

So sharing objects makes sense, sharing the rest of .git is not neccessary.

You don't really need multiple sources for it. The subproject contains both upstream and local changes, but I think it makes sense to keep the entire object store local (the same way to keep all the entire history local even if you only want to add to it in a normal repository). Those people who work on the subproject and communicate with its upstream developers of course need remote entries and have to synchronize the subproject with upstream. But that is not related to the superproject at all.

So yes, you have different sources but you don't need extra support in the subproject implementation for it.

Martin Waitz wrote:

One of the final ideas for "lightweight checkout" was having in .git/config the location of "true" $GIT_DIR (or parts of it: GIT_INDEX_FILE, GIT_OBJECT_DIRECTORY), and "shadowing" the rest of "true $GIT_DIR" with what is present in .git. It means that you can have .git/index and .git/HEAD, and if you don't find appropriate .git/refs/heads/master file you look to "true $GIT_DIR".

On Thu, May 24, 2007 at 17:48:33 +0200, Martin Waitz wrote:

No, the way I mean it the subproject and superproject don't share a single bit. The subproject.git dir is subdirectory of superproject.git, but has no thing in common with it.

Ah. My bad. Didn't notice this. I do NOT want to share any objects between subproject and superproject. At least not unless the user explicitely asks for that, which might make sense if the subproject was carved out of the superproject.

No, I didn't mean the subproject and superproject would share anything.
The case I talk about is if project foo has subdirs A and B and they both
contain (different revisions of) the same subproject. The .gitmodules
definition is:
    [submodule "A"]
    name=bar
    [submodule "B"]
    name=bar

In such case A/.git and B/.git can't both be symlinks to .git/subprojects/bar.git, because they have different HEAD, but everything else should be defined by .git/subprojects/bar.git

Upstream + local is not the interesting case. Multiple upstreams is.

Well, if the subproject is not auto-fetched, there's no need for extra support. But if there is auto-fetch, it should be aware of possibility to have multiple upstreams.

Martin Waitz <tali@admingilde.org> writes:

I was hoping that we can start from an initial cut that supports only a superproject that had its subprojects in their places from its initial commit, and did not have to worry about this from day one, and deal with this kind of "more advanced" stuff incrementally. Unfortunately it's more fun to talk about more advanced stuff than starting with small but solid stuff.

And we would need to make sure whatever we do as the "small but solid" initial round can later support more advanced arrangements later, so we would need to think about the issues now anyway to a certain degree.

How about doing something like this, instead?

 (1) superproject .gitmodules (in-tree) and .git/config (local
     repository) use the three-level naming in $gmane/47567.
     Namely, (1a) .gitmodules says which subdirectory has a
     checkout of what project, and names the project in
     logical/abstract terms, not with a URL (e.g. "kernel26");
     (1b) .gitmodules also associates a set of suggested URLs
     for each of the logical/abstract project name; (1c)
     .git/config records which project are of interest.

 (2) In superproject .git/, we would have a bare repository for
     each project used by the superproject.

	.git/subproject/kernel26/{objects,refs,...}

     This is created by making a bare clone from the upstream
     URL, decided by the user with the help from suggested URL
     described in the superproject .gitmodules.

     The idea is to use this repository as a long-term
     subproject state across branch switching.

 (3) When we need to check out a revision of superproject whose
     .gitmodules has "kernel-src/ -> kernel26", and when we
     haven't done so (perhaps we are doing an initial checkout,
     perhaps we are switching from a different revision of the
     superproject that did not have "kernel26" project at
     kernel-src/ directory), we rm -f kernel-src/ and then
     "git-clone -l -s" from the repository we keep in (2) to
     populate kernel-src/ directory.

 (4) Before performing the above step (3), we need to make sure
     we are not losing anything in kernel-src/ if exists.  Three
     cases plus one:

     (4a) The path kernel-src/ in the old checkout was not a
     subproject (either it did not exist, it was a blob, or it
     was a directory with files that are tracked as part of the
     superproject).  The usual "don't lose local modification"
     rule we use try to carry local changes forward across
     branch switching, but in this case we shouldn't do so.

     (4b) It has the same logical/abstract project checked out;
     the commit recorded in the superproject tree may or may not
     be the same as what its HEAD points at.  In this case we do
     not have to worry about swapping the git repository at
     kernel-src/ directory, although we would need to check out
     the correct revision, and worry about what to do with any
     local modification (I think the usual "don't lose local
     modification but carry them forward" rule would be Ok in
     this case).

     (4c) It has a different project checked out; we need to be
     careful to keep local changes, and also we need to make
     sure the local changes in this subproject repository are
     pushed back to (2).  It could be that automated "git push"
     after making sure everything is committed is sufficient and
     have the user handle failure cases.

     (4d) This applies not just "before step (3)", but in cases
     where we need to replace a checked out subproject directory
     with something else (e.g. blob or directory that belong to
     the superproject, or noneness).  We would need to make sure
     no local change is lost, and the repository is synched up
     with (2).

I think an arrangement like this would solve "symlink is a bitch for MinGW" problem Johannes Sixt brought up today with Sven's RFC as well.

On Thu, May 24, 2007 at 11:26:01AM -0700, Junio C Hamano wrote:

What about the idea of considering any project that is already present to be of interest by default ?

Do you mean a "pure" clone, i.e., without a working tree, but with separate-remotes?

We may also need to fetch from the remote subproject. Should I do this with a "git fetch" during the checkout as I do now?

skimo

Sven Verdoolaege <skimo@kotnet.org> writes:

You could do that -- I consider that a minor detail in the implementation.

I meant a bare clone without separate remotes.

The counter-proposal outline essentially says, for the sake of simplicity, "nuke existing subproject directory whenever we need to replace it with something else, and reclone a new/replacement subproject directory every time we need to check it out, after making sure nothing is lost".

Except that having this intermediate repository (a) helps making such a re-clone extremely cheap "git clone -l -s -n", and (b) provides with a place to hold locally committed changes that are not pushed back to the true upstream (you may _never_ push it back in the first place).

If you are disconnected, you obviously cannot afford to update (2) every time you switch branches in the superproject, and even if you are connected, updating (2) would not be needed most of the time if what you are doing is switching superproject branches, as long as your last superproject fetch is not very much more recent than your last subproject fetch.

If we were to follow the outline in the counter-proposal, I'd imagine that update of (2) can happen at any time. It could be part of "git fetch" in superprojects, of lazily done when we need to checkout a new revision for a particular subproject, but only if the last time you fetched superproject is more recent than the time you updated (2) for the subproject last time.

Or something like that. I consider that also a minor detail in the implementation.

On Thu, May 24, 2007 at 11:58:26AM -0700, Junio C Hamano wrote:

Why without separate remotes? It has been argued before that changes in the subproject may come from different remotes, so the user may want to configure extra remotes from which to fetch.

And she can't do it in the clone in his working tree if that's going to get nuked from time to time.

But you still need figure out _what_ to fetch. Before you suggested to just use the default set up by clone with separate remotes, but you no longer have that in your new proposal.

skimo

Sven Verdoolaege <skimo@kotnet.org> writes:

[side note: I am ignoring your reply-to: liacs.nl as its MTA seems to use sorbs that has my ISP's outgoing sender identified as spam source; I'll send the bounce to you privately in a separate message.]

By different remotes, which do you mean?

 (1) .git/subprojects/kernel26/ repository has 'origin'
     different from any of the suggested URL in .gitmodules, but
     as far as it is concerned there is one 'origin';

or

 (2) it has 'origin' that is what the superproject suggests, but
     the user locally uses additional repositories to pull and
     merge from;

If the former that is not an argument, so I'd assume the latter. I would say in such a case, you are better off having a usual repository to manage the development of subproject part, not grafted to any superproject repository, and handle such merges there (after all, a "subproject" should stand on its own without having any of the superproject stuff). And treat THAT repository as the 'origin' used in (1) above. It might be easier to use non separate-remote layout in the standalone repository for the subproject, but that is a separate issue.

And she does not have to. She can do the development/fixes in (temporarily) checked out subproject tree, and push it back to the .git/subproject/kernel26/ repository in the superproject before she leaves (i.e. before branch switching at superproject level needs to obliterate it). The change stored in the .git/subproject/kernel26/ repository in the superproject can further be pushed back to its 'origin', be it the true "upstream", or "the standalone repository for the subproject" I mentioned above.

I do remember saying the "default set up by clone" but I did not mean separate remotes.

What is fetched by a bare and non-separate-remote repository vs a repository that uses separate-remote layout from 'origin' is exactly the same -- the difference is only 'pure/bare' layout would use

	fetch = refs/heads/*:refs/heads/*

while separate-remotes would use

	fetch = refs/heads/*:/refs/remotes/origin/*

So I do not think the difference matters for our purpose of being able to check out commits that are referenced in superproject trees. As long as we require that the 'origin' for "longer term repository to keep track of the subproject in superproject" (aka repository (2) in my message you are responding to) always contain the commit referenced by tree objects in the superproject, which I think is a sensible thing to require (otherwise you cannot even clone and checkout the whole superproject), both layout would work equally well. It's just bare/pure layout is easier to understand because it is essentially a "mirror" of the upstream.

On Thu, May 24, 2007 at 10:32:32PM CEST, Junio C Hamano wrote:

..snip..

I OTOH think that it's less confusing to just keep it the same for all the normal repositories - and you yourself did not rule out the possibility to have local changes in the repository, in which case we certainly should use the separate-remotes layout or we go back to last year's refs mess.

Petr Baudis <pasky@suse.cz> writes:

That "local changes in the repository" will happen in the clone created in kernel-src/ directory and I do not have any problem using separate remotes there. I was talking about the refs layout used in .git/subproject/kernel26/, which is a mirror of the upstream and the clone source for that (temporarily checked out) repository at kernel-src/ directory. After all, the "mirror" is where you would push your local changes back into, and it is more common to use non-separate-remote layout (without even 'master'->'origin' mapping, hence I had to come up with a new name 'pure', but I realize that what I meant really is a 'mirror') there.

Junio C Hamano <junkio@cox.net> writes:

Sorry, I hate to send "amend" message like this, but there are some places I was not clear enough...

... are of interest, and perhaps which URL to use but that is not necessary, as we can record that in repositories we create in (2).

This bare clone probably have its own config that records remote.origin.url, which would probably be the same as one of the URL suggested in .gitmodules for many people, but could be different. As to what should happen when the .gitmodules that comes from the upstream changes the set of suggested URLs recorded in there, see $gmane/47621,

The last part of the above should have been:

       we "rm -fr kernel-src/" and then "git-clone -l -s -n"
       from the repository we keep in (2).  And then check out
       the commit that is pointed at by the superproject tree
       (rather, "index").

The last part of the above should have been:

       ... but in this case we shouldn't even try to carry local
       changes forward, as there is nowhere to carry them to.
       Instead we should error out and have the user clean-up
       the subproject repository (either "git reset", or "git
       commit && git push" to stash the change back to
       repository we prepared in (2)).  We might also want to
       allow the user to say "local changes in this subproject
       checkout does not matter" by saying "git checkout -f
       $commit" at the superproject level, but I think that is
       rather dangerous.

... which is the same check and action we would do in case (4c).

On Thu, May 24, 2007 at 08:26:01PM CEST, Junio C Hamano wrote:

These discussions are spread over so many posts (and especially threads) that it's far beyond me to track it all - I hope I won't repeat something already debunked.

How do you deal with clashes in subproject names? Until now, several independent projects might live happily in a single repository, this breaks that. When merging disparate projects, you can resolve name clashes in .gitmodules, but going back to one of the merged trunks just won't work. It's ugly.

Now, we can just declare that we don't care about this case. This stance wouldn't make me comfortable at all, but at least we should make it clear that we know about this problem and the users are on their own when it happens.

When not using subproject aliases, you could just name this after some normalized form of the URL (anything suitable up to an sha1sum of the URL :).

On Thu, May 24, 2007 at 11:26:01 -0700, Junio C Hamano wrote:

It's almost exactly what I had in mind, except much better described, so I definitely support this. It seems that it can work.

If the "clone" could also share the refs, config and everything except HEAD, it would make it completely (or almost so) transparent to the user. Making it non-transparent will work well enough though and should not require any new changes.

The problem is, that than the HEAD could get out of sync with the refs (because they are linked from other repo), so it would have to remember both the commit name and the symbolic ref. I would behave as symref only if the commit name in it and the target are the same and as hard ref otherwise.

It would also solve (rare) case when for some reason the same subproject should be checked out twice (different revisions).

Jan Hudec <bulb@ucw.cz> writes:

I have a feeling that you are equating two subprojects that happens to have the same upstream URL the same, and if that is the case I think that is a mistake.

If you were doing an efficient cgi script that renders history of git managed projects, binding git as its subproject, and that system can be built with either 'maint' (i.e. 1.5.2 series) or 'master' (i.e. ultrastable WIP towards 1.5.3), even though they both might come from git://git.kernel.org/pub/scm/git/git.git/, I think they should be registered as two separate logical subprojects.

The .gitmodules file might have:

	[module "git-maint"]
        	path = git-stale/
        	url = git://git.kernel.org/pub/scm/git/git.git/
                ;; branch = maint

	[module "git-master"]
        	path = git-stable/
        	url = git://git.kernel.org/pub/scm/git/git.git/
                ;; branch = master

and two paths (git-stale/ and git-stable/) in the superproject tree would have commit object names from the named branches.

Junio C Hamano wrote:

I agree strongly with this, and it's another good reason that we have to be able to use something other than the URL as the key to look up a subproject's repository location. If you use the URL it is impossible to differentiate the two subprojects in this case.

-Steve

Steven Grimm <koreth@midwinter.com> writes:

Heh, you do not have to rub it in ;-) I already agreed that your three-level arrangement is easier to work with.

On Fri, May 18, 2007 at 09:24:56PM CEST, skimo@liacs.nl wrote:

Nacked-by: Petr Baudis <pasky@suse.cz>

Please do not add more undocumented parameters - include documentation in the patch adding the parameter.

From: Sven Verdoolaege <skimo@kotnet.org>

If the submodules are not clean, then we will get an error when we actally do the checkout.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 unpack-trees.c |   43 ++++++++++++++++++++++++++++++++++---------
 1 files changed, 34 insertions(+), 9 deletions(-)

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 entry.c |   42 ++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 40 insertions(+), 2 deletions(-)

skimo@liacs.nl, Fri, May 18, 2007 21:24:58 +0200:

We may need an option to ignore these failures. Maybe even active by default. Imagine a superproject with _optional_ submodules, where it is just nice to know that some submodules weren't checked out. BTW, doesn't git-checkout already prints an error?

On Fri, May 18, 2007 at 11:56:42PM +0200, Alex Riesen wrote:

Probably.  You probably noticed that I haven't written any tests yet...

Still, the error that git-checkout prints may not give enough of a clue that something was wrong with a submodule.

skimo

Sven Verdoolaege, Sat, May 19, 2007 00:03:23 +0200:

I see. It was a very ... provocative patch series :)

Like, for example, it failed because the directory is not a git repo yet, because the previous git-checkout was called _without_ --submodule and the directories created are just empty.

Anyway, just a "failed to run git-checkout" is not very helpful either. Come to think about it, there is not very much you can tell out of super-project context. git-checkout will always know better.

skimo@liacs.nl, Fri, May 18, 2007 21:24:58 +0200:

How about modifying run_command to chdir after fork?

You'd save the hassle of save/restoring cwd and don't mess up process' context (which is always a good idea to preserve). The code'd be simplier, too.

It can make code simplier (no need to preserve cwd) and safer (no chance the cwd of the current process is accidentally forgotten).

Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
---
Alex Riesen, Sat, May 19, 2007 00:00:14 +0200:

something like this

 run-command.c |   27 ++++++++++++++++++++++-----
 run-command.h |    2 ++
 2 files changed, 24 insertions(+), 5 deletions(-)

Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
---
 entry.c |   12 +-----------
 1 files changed, 1 insertions(+), 11 deletions(-)

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 git-checkout.sh |   20 +++++++++++++++-----
 1 files changed, 15 insertions(+), 5 deletions(-)

On Fri, May 18, 2007 at 09:24:59PM CEST, skimo@liacs.nl wrote:

Thus Documentation/git-checkout.txt needs to be updated as well.

From: Sven Verdoolaege <skimo@kotnet.org>

This makes it easier for scripts to call git-fetch with options that may or may not be set.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 git-fetch.sh |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

skimo@liacs.nl writes:

For git-fetch it does not matter as I do not think there is any valid case to pass an empty string as a parameter to it (even "fetch from our own repository" requires a single dot). But from discipline point of view, I am not happy about this.

If you are talking about shell scripts, the standard way to do that is to say ${1+"$1"}.

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 builtin-fetch--tool.c |   48 ++++++++++++++++++++++++++++++++++++++++++++----
 1 files changed, 44 insertions(+), 4 deletions(-)

skimo@liacs.nl, Fri, May 18, 2007 21:25:01 +0200:

This code produces warning about possible uninitialized used of single_force and not_for_merge. I used the patch below, but didn't look into what the "all" does.

---
 builtin-fetch--tool.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

On Sat, May 19, 2007 at 12:52:34AM +0200, Alex Riesen wrote:

Sorry for being sloppy. It is assumed that "all" is only used for cloning and then the other two flags don't really matter, but I should've set some defaults. Yours look fine, thanks.

A fixed-up version is available in the "submodules" branch of http://www.liacs.nl/~sverdool/git.git or www.liacs.nl/~sverdool/gitweb.cgi?p=git.git;a=shortlog;h=submodules

skimo

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 git-clone.sh |   20 ++++++++++++--------
 git-fetch.sh |   28 ++++++++++++++++++++++------
 2 files changed, 34 insertions(+), 14 deletions(-)

From: Sven Verdoolaege <skimo@kotnet.org>

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 git-clone.sh |    6 +++---
 git-fetch.sh |   20 ++++++++++++++++++++
 2 files changed, 23 insertions(+), 3 deletions(-)

From: Sven Verdoolaege <skimo@kotnet.org>

---
 builtin-read-tree.c |    9 ++++++---
 unpack-trees.c      |    3 +++
 2 files changed, 9 insertions(+), 3 deletions(-)

From: Sven Verdoolaege <skimo@kotnet.org>

When the --submodules option is specified, git-clone will search for submodule.<submodule>.url options in the remote configuration and clone each submodule using the first url that it can use from the local site.

Signed-off-by: Sven Verdoolaege <skimo@kotnet.org>
---
 Documentation/config.txt    |    7 +++
 Documentation/git-clone.txt |    6 ++-
 git-clone.sh                |   18 +++++++-
 git-fetch.sh                |   90 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 117 insertions(+), 4 deletions(-)

I forgot to mention that these changes go on top of next (v1.5.2-rc3-762-g77e153b).

skimo