Re: [RFC] Light-weight checkouts via ".gitlink"

Hi,

when I recently thought about submodule support, I had the idea that it is easier to get it by going in small, incremental steps, introducing usefull subfeatures on their on while on it.

The following is one outcome of this, a proposal for light-weight checkouts of git branches, without the need to have to full repository in a .git subdirectory, but the just have a file .gitlink is simple as possible, which manages the link to the real repository.

Of course, this feature is tailored to support checkouts of submodules as being such light-weight checkouts themself. The current proposal just says that light-weight checkouts should be ignored when existing inside of another checkout. For real submodule support, we would want to be able the do a "git add" on these light-weight checkouts (which of course are bound to some commit), and on "git commit", this would add a "submodule object" at this place into the tree of the outer repository.

Comments?

Josef

============================================================

Support for multiple external light-weight checkouts aka ".gitlink" proposal

Main ideas behind light-weight checkouts ----------------------------------------

Make submodules easier to implement by separating part of the needed infrastructure into a independent, yet useful feature

(1) Allow to separate a branch checkout from its repository location on the local filesystem. This minimally needs to be only _one_ file, called ".gitlink" here. One should be able to move the checkout directory around (within some limits) without breaking the link to its "base" repository.

We want this later for submodule checkouts inside of a supermodule checkout: (a) to not loose the objects+index+HEAD if the user removes the whole checkout to remove the submodule in next supermodule commit; (b) to allow for moving submodules around between supermodule commits.

This can be implemented with a small script reading .gitlink and setting up $GITDIR and $GIT_INDEX_FILE accordingly. However, there should be a further environment variable to use a file for HEAD ($GIT_HEAD_FILE?). Interpretation of the link to the base directory has to be a little smart, ie. by prefixing a relative path with as many ".." as needed to find a git repository.

(2) Light-weight checkouts should work inside of another checkout, be it a normal or a light-weight checkout itself. The subdirectory of the light-weight checkout has to be ignored in the outer checkout (at least by default).

This is needed to be able to use light-weight checkouts for submodels, as these should be checked out inside of another checkout (even inside of another submodule checkout for hierarchical submodules).

This can be implemented by enhancing git to ignore any subdirectory which has a file .gitlink in it.

Example usage -------------

Keep a checkout of the todo branch of git.git inside of the checkout of any other branch (or fully outside). You should be able to use all the normal git commands inside of this directory (commit etc). Something like inside of e.g. a master checkout of git.git:

 make todo
 cd todo
 touch .gitlink
 git checkout todo

The .gitlink file can be empty in this case when we make the smart lookup of the base repository do the right thing with a default of an emtpy relative path.

Another usage is to keep all your git repositories (looking like bare ones) in one place, e.g. below $HOME/git-repositories/, and have the the checkouts you are working in at another place, without the need for setting $GITDIR etc for this.

Some thoughts about implementation ----------------------------------

As we need our own index and HEAD for any light-weight checkout, we can choose a subdirectory of .git of the base repository to store a full, independent working GITDIR on its own. It would default to linking the objects and refs namespace to the base repository, but for submodules, it *can* have its own object database and refs namespace; it just would be locateable via the GITDIR of the base repository.

"Locateable" means that we need a name for different light-weight checkouts. The name can be determined from the relative position of the light-weight checkout to its base repository, or could be fixed and specified in the .gitlink file.

Example for an light-weight checkout with name "mywork", linking to the (bare) base repository "base":

 base.git/external/mywork/HEAD
 base.git/external/mywork/index
 base.git/external/mywork/refs -> ../../../refs
 base.git/external/mywork/objects -> ../../../objects

A light-weight checkout in a sibling directory to base.git needs the relative path to its own .git directory: "../base.git/external/mywork".

This can be split up in two things:
- relative path to base repository
- a name for this light-weight checkout
One has to be able to get this out of the content of the .gitlink file.

Proposal for .gitlink entries (one line per entry) --------------------------------------------------

* Gitdir = "<Path to base git repository>"

Optional. An absolute or relative path to the base git repository. With relative path, a heuristic is used to find the git directory: the path will be prefixed by as many "../" as needed, and ".git" or .gitlink appended. The value defaults to an empty relative path, which will check all parent directories for a .git subdirectory.

* Name: <explicit name for this checkout>

Optional. Git uses this name to find its own GITDIR in the GITDIR of the base repository. If not specified, the name defaults to the relative path of the light-weight checkout from the base directory, stripping any ".." in front.

For the "mywork" example above, we could have a checkout directory, sibling to "base.git" and called "mywork". For the .gitlink file inside, it is enough to specify

 Gitdir = base

Because the search heuristic will find the base repository at real relative path "../base.git". Further, relative path of the checkout from the base repository is "../mywork", giving the name "mywork".

When moving the checkout into another directory, it's name would change if not explicitly specified in the .gitlink file. For submodules, this could be the wanted semantic. However, with changing name, we have possibly lost our index and HEAD. A solution for this could be to always store a copy of HEAD into .gitlink, or even have a "Head:" .gitlink entry itself as HEAD.

============================================================

A few (very few) comments:

Josef Weidendorfer wrote:

If I remember correctly, while git ignores .git, it does not ignore by default (i.e. without entry in either GIT_DIR/info/excludes, or .gitignore) the directory which has .git directory in it.

And that should not change for .gitlink. You can always add
.gitignore file with * .* patterns in it (ignore all).
 

[...]

Why use once "key = value", once "key: value" form? Better to stick with one. I Would prefer "key = value" one.

GIT_DIR = path to base git repository it is equivalent to setting the following:

GIT_INDEX_FILE = path to index file GIT_OBJECT_DIRECTORY = path to object directory GIT_HEAD_FILE = path to HEAD file GIT_REFS_DIRECTORY = path to refs directory

NAME = name should match "name subdirectory" entry in modules file in superproject.

Perhaps instead of adding arbitrary number of .. in front of relative path, we better use some magic, like ... for finding somewhere up?

On Friday 08 December 2006 23:18, Jakub Narebski wrote:

I know. But this is essential. We _have_ to ignore all the files and subdirectories in the directory which contains the .gitlink file, as these files/subdirectories belong to the submodule.

There is no other way. You could try to use a special name for the whole directory with the light-weight checkout, e.g. ".checkout".

But then, this is useless for submodules, as for submodules, we want to be able to specify the root directory name of the submodule, as that is the name which will end up in the tree object of the supermodule.

That is not possible: .gitignore file has its own meaning inside of the light-weight checkout aka submodule, as this directory is the root directory of a git checkout.

AFAIK, Martin's submodule support does it the same, only for directories with .git, as he stores the GITDIR directly in the submodule checkout.

Sorry. Typo ;-)

AFAIK the latter two do not exist yet, or do they?

I would also be fine with .gitlink looking like some shell script, defining these variables. However, we need the smart directory lookup. And IMHO the keys can be case insensitive as in .git/config.

I am not sure we want to allow the freedom of being able to put any of GIT_INDEX_FILE, GIT_OBJECT_DIRECTORY, GIT_HEAD_FILE, GIT_REFS_DIRECTORY in the .gitlink file.

It is enough if GITDIR and NAME is given. With GITDIR_REAL after the smart lookup, e.g. GIT_INDEX_FILE would default to $GITDIR_REAL/external/$NAME and so on.

However, for submodules we really _want_ to have fully independent GITDIRs for each submodule somewhere, and we would have to warn:

 # Warning: if you change one GIT_INDEX, ... in this file, you
 # will screw up the possibility to clone from the GITDIR directory

Yes. This would be in my next proposal about how to build the submodule support on light-checkouts ;-)

I thought about it. But why whould you need it? If the value of GITDIR in .gitlink begins with "/", it is an absolute path. If not, I think you always want the smart lookup the go upwards, i.e. looking for

  ../<relpath>.git
  ../../<relpath>.git
  ../../../<relpath>.git

So there is no need to add "..." in front of the relative path.
Or do you see a usecase for
 rel/path/start/.../rel/path/end

Ah, yes, I see. Perhaps this makes sense with absolute paths:

	/home/user/repos/.../linux

Josef

Dnia piątek 8. grudnia 2006 23:54, Josef Weidendorfer napisał:

I have forgot about that. Right.

The only possibility would be to use GIT_DIR/info/excludes with path to submodule, and this conflict with the ability to rename and move submodules.

Ah. 

[...]

They do not exist; perhaps they should for completeness.

[...] 

Not $GITDIR_REAL/submodules/<name>/index (or modules instead of submodules)?

I have thought that with "each submodule as separate repository" approach to submodules the modules file would have module name and either subdirectory in which submodule resides, or GIT_DIR of submodule. And this file could be generated on checkout... which doesn't survive closer scrutiny.

But this would work well with submodules, that's a fact.
  

You mean that the above means to check the following paths:

  /home/user/repos/linux
  /home/user/linux
  /home/linux
  /linux

not the searching subdirectories of /home/user/repos for linux directory (there can be many)? BTW web2c implementation of TeX, namely kpathsea(rch) uses // for that, i.e. a//b means b which is somwehere in subdirectories of a.

On Saturday 09 December 2006 00:24, Jakub Narebski wrote:

Yes. This whole .gitlink thing more or less is about trying to avoid as far as possible any path configuration in the supermodule which would have to be changed when the user moves or even deletes the submodule. Exactly for the latter, we want the GITDIR for submodules better be separate.

Actually, I am fine with allowing them in .gitlink. This makes the whole thing much more flexible.

Ooops, yes. I am not actually sure what's the best name here: "external", "submodule", ... ? I thought the the SVN name also fits for the submodule case. The submodule is independent, and possibliy comes from an external git repository.

Of course, that is a more simple approach. But I think the .gitlink thing really is more flexible without being more complex.

No.

Yes. But you can scratch this.

On Friday 08 December 2006 23:54, Josef Weidendorfer wrote:

Forgot to mention in the proposal: If you recursively have light-weight checkouts inside each other, the real "name" (for .git/external/<name/ and for further submodule configuration e.g. in .git/modules of the base repository) should of course be the concatenation of the names in the .gitlink files while going up to the base repository.

No need. Something like

is crazy. Do you want to scan all of your home directory everytime this lookup is needed? So "..." really only makes sense in front of the relative path, but there, you also can leave it out.

Josef Weidendorfer wrote:

Why concatenation? I thought the name would be ID of submodule, and should be just somehow unique.

And if concatenation, pehaps some forbidden character inserted between
them? Like '/' for example ;-)
 

No. I meant /home/user/.../linux to mean searching for
  /home/user/linux
  /home/linux
  /linux
but I don't think it is useful. As to relative path matching in any
parent directory... well, that differs only in direction (up instead of 
down) in matching filename in .gitignore when path does not contain /
(well, actually it is taken as fileglob).

On Saturday 09 December 2006 00:53, Jakub Narebski wrote:

Yes, you are right. Nesting of submodules really is an important issue. The .gitlink file allows us to put the submodule GITDIR somewhere in the supermodul's GITDIR. The idea is that you can clone the submodule GITDIR if you want. With submodule "inner" nesting inside of submodule "outer", the GITDIR of "outer" should have the GITDIR of inner inside to allow for cloning "outer" together with its submodule "inner".

So it is not enough to have a submodule "outer" and a submodule "outer/inner" in the supermodule. We want

	super.git/ext/outer.git/

to be the GITDIR for submodule "outer", and inside 

	super.git/ext/outer.git/ext/inner.git
.

Actually, it would be nice if this .gitlink proposal did not have to deal with it. Instead, a .gitlink should be so flexible to allow such nesting if needed.