RFC patch, 3 partsEnable in-process submodule traversal

This patch series implements basic support for traversing the tree objects in submodules when the linked commit object is reachable. Normally such linked commit objects will not be reachable in the containing repository, but adding local copies of submodule repositories as alternate object databases for the containing repo solves this issue.

The first patch in the series does all the 'hard work' required for the traversal to work, while the next two patches adds a '--submodules' flag to git-archive and git-ls-tree as proof of concept.

Lars Hjemli (3):
  tree.c: add support for traversal of submodules
  archive.c: enable traversal of submodules
  builtin-ls-tree: enable traversal of submodules

 archive.c         |    2 ++
 builtin-ls-tree.c |   23 ++++++++---------------
 tree.c            |   20 +++++++++++++++++---
 tree.h            |    1 +
 4 files changed, 28 insertions(+), 18 deletions(-)

If the commit referenced by a gitlink is available in the (possibly alternate) object database, read_tree_recursive() is now able to descend into the tree of the linked commit if the flag 'traverse_gitlinks' is turned on.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 tree.c |   20 +++++++++++++++++---
 tree.h |    1 +
 2 files changed, 18 insertions(+), 3 deletions(-)

The option --submodules can be used to activate traversal of sub- modules when creating archives.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 archive.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

The option '--submodules', which implies '-r', activates the traversal of all submodules for which the linked commit is reachable.

Signed-off-by: Lars Hjemli <hjemli@gmail.com>
---
 builtin-ls-tree.c |   23 ++++++++---------------
 1 files changed, 8 insertions(+), 15 deletions(-)

Lars Hjemli <hjemli@gmail.com> writes:

I think we tend to put these global settings that will affect everybody in environment.c. You do not have to initialize variable to zero; BSS will take care of it.

When the user explicitly asks you to traverse into submodules and the necessary commit is not available in a submodule, the code goes on without complaining. I am not saying it is bad, but I wonder if we would want to distinguish these three cases:

 (1) the submodule is initialized and the necessary commit is there.

 (2) the submodule is initialized, but the necessary commit is missing.

 (3) the submodule is not even initialized (aka "the user is not
     interested in it"); there is only an empty directory.

I think it is perfectly fine not to say anything for (3) but I am unsure about the second case.

On Mon, Jan 12, 2009 at 04:15, Junio C Hamano <gitster@pobox.com> wrote:

Ok, I'll add a proper interface in environment.c for this setting.

Do we want to impose the porcelainish rules of git-submodule (.gitmodules, .git/config) in read_tree_recursive()?

If so, I guess a new submodule.h might provide something like this (disclaimer: coded in gmail):

	struct submodule {
		int interesting:1;
		char *name;
		char *url;
		char **objectdirs;
		char **paths;
	}

	typedef int (*submodule_cb)(struct submodule *submodule, void *data);

	int load_submodule_config();
	struct submodule *get_submodule_from_path(char *path);
	int add_submodule_objectdb(struct submodule *item);
	int for_each_submodule(submodule_cb cb, void *data);

Then, in read_tree_recursive(), we could check submodule->interesting to decide if we should follow the gitlink. Also, for bare repositories, we'd need to support something like 'submodule.<name>.objectdir' in the config file (and .gitmodules must obviously be read from the objectdb).

If we don't want to impose these rules, the current patch should be minimally sufficient (the user has to edit .git/objects/info/alternates by hand and missing submodule commits are treated as if the submodule is not interesting).

Lars Hjemli <hjemli@gmail.com> writes:

I do not see why you would need anything more than we already have to tell (3) from (1) and (2). And I do not see why you need to have the Porcelain policy in the picture for telling these three cases apart, either.

For example, there is this code in read-cache.c::ce_compare_gitlink():

        static int ce_compare_gitlink(struct cache_entry *ce)
        {
                unsigned char sha1[20];

                /*
                 * We don't actually require that the .git directory
                 * under GITLINK directory be a valid git directory. It
                 * might even be missing (in case nobody populated that
                 * sub-project).
                 *
                 * If so, we consider it always to match.
                 */
                if (resolve_gitlink_ref(ce->name, "HEAD", sha1) < 0)
                        return 0;
                return hashcmp(sha1, ce->sha1);
        }

It asks resolve_gitlink_ref() to see if the directory (where the submodule checkout _might_ be present if the user is interested in it) has .git/HEAD that resolves. If so, the user has a checkout and is interested in it. Otherwise, there is no checkout, in other words, we have case (3) above.

Whether you force the user to link the submodule object store to the primary one as alternates, or do that for the user temporarily inside the process [*1*], you would then be able to tell (1) and (2) apart by asking has_sha1_file() if you can see the commit.

One thing that is unclear is to me is for whom the commit is missing (or present). I think the outline I gave above follows the design of your patch to assume that the commit may (or may not) be available to the superproject and traverse into the commit when that is the case. It does not mean the commit is available to the submodule itself (the commit may have found in the primary project itself, not via the alternates), but such an arrangement makes it somewhat useless.

What's the typical direction of using alternates in a setting with superproject with a submodule? Do people have alternates in the submodule repository that borrows from the superproject repository? Or the other way around? What's the rationale for having such alternates for normal use case? I am suspecting that there is no reason (other than this "recursive tree traversal") to have an alternates file in either direction, but I also strongly suspect that I am missing some unwritten assumption you are making.

[Footnote]

*1* If you want to recurse seemlessly, it might make sense to add (during the course of this "recursive tree traversal") the object store of the submodule repository to the process'es list of alternate object databases yourself, instead of forcing the user to do so permanently by mucking with the alternates list of the primary project repository. But that is an independent issue.

On Mon, Jan 12, 2009 at 11:07, Junio C Hamano <gitster@pobox.com> wrote:

Ah, yes, this makes sense. Thanks.

If resolve_gitlink_ref() returns 0, I think we should automatically insert the objectdir of the submodule as a tempory alternate.

Yes (I've also got a use-case for this with bare repositories [*1*], but in that setting I guess it's ok to force the user to link the alternates manually).

I think we can ignore this issue; if someone has added the superproject as an alternate for the submodule and then done a checkout of a superproject commit in the submodule followed by committing this gitlink in the superproject, we can only hope the user really knew what he was doing...

Likewise.

I'm only assuming that we want to support traversal into the submodules from core git.

*1* This will be (ab)used by cgit to support downloading of 'complete' tarballs, as outlined in http://thread.gmane.org/gmane.comp.version-control.git/102827.

-- larsh

Sounds interesting except 1/3 didn't seem to reach the list...