What is in git.git

The "bind commit" experiments for subproject support is coming along rather nicely. Near the tip of the "pu" branch, there are:

 - read-tree --prefix;
 - write-tree --prefix and write-tree --exclude;
 - commit-tree --bind;
 - fsck-objects and convert-objects that understand "bind" lines
   in commit objects;
 - rev-list --objects that understand "bind" lines
   in commit objects;

I think the first four are more-or-less well debugged.

I am reasonably confident that I did not break rev-list for repositories without "bind" commits, but I have no clue how correct it is when dealing with commits with "bind" lines. This is the last major remaining piece of the puzzle, and the rest is just the matter of scripting. I'd be sending out a request for help on the rev-list in a separate message.

There still is no barebone Porcelainish work done using these changes. The attached script demonstrates a superproject that binds two subprojects with their own development histories.

-- >8 --
#!/bin/sh
rm -fr .git
git init-db
: >main
for i in 1 2
do
	echo "Version #$i of primary subproject" >main
	git add main
	git commit -a -m "`cat main`"
	git tag main-$i
done
git branch main
rm -f .git/refs/heads/master .git/index

mkdir dir
: >sub1
: >dir/sub2
for i in A B C
do
	echo "subproject sub1 ($i)" >sub1
	echo "subproject dir/sub2 ($i)" >dir/sub2
	git add sub1 dir/sub2
	git commit -a -m "subproject #$i"
	git tag subpro-$i
done
git branch subpro
rm -fr dir sub1 main
rm -f .git/refs/heads/master .git/index

git read-tree main-1
git read-tree --prefix=sub/ subpro-A
toptree=`git write-tree`
topcommit=$(echo "Initial top commit" |
git commit-tree $toptree \
	--bind `git rev-parse --verify main-1` / \
	--bind  `git rev-parse --verify subpro-A` sub/)
echo $topcommit >.git/refs/heads/master
git tag top-1A

git read-tree main-1
git read-tree --prefix=sub/ subpro-B
toptree=`git write-tree`
topcommit=$(echo "Second top commit" |
git commit-tree $toptree \
	-p `git rev-parse --verify top-1A` \
	--bind `git rev-parse --verify main-1` / \
	--bind  `git rev-parse --verify subpro-B` sub/)
echo $topcommit >.git/refs/heads/master
git tag top-1B

git read-tree main-2
git read-tree --prefix=sub/ subpro-C
toptree=`git write-tree`
topcommit=$(echo "Third top commit" |
git commit-tree $toptree \
	-p `git rev-parse --verify top-1B` \
	--bind `git rev-parse --verify main-2` / \
	--bind  `git rev-parse --verify subpro-C` sub/)
echo $topcommit >.git/refs/heads/master
git tag top-2C

git log

git rev-list --objects top-1A..top-2C | git name-rev --tags --stdin

Somehow first mail was not reached the list, resending it.

...

I tested this with your script. It works well. But I have found some downsides.

subpro and main are separate projects and master is the join of them. If I want to modify subpro I have to checkout subpro branch, edit files. When I have to got to master and bind new version of subpro to it. Worse, if I will edit subpro's files bined to master branch changes will go to master branch instead of subpro's history. As a result all other project (imagine master-2) that use subpro will lose this change.

1. Can I bind some branch instead of tag (commit) ?
2. Is it possible to commit changes of subpro's file in master branch into 
subpro branch to make this changes visible to master-2 ?

Thanks for attention. Alexander Litvinov.

I suspect there is an misunderstanding or two, because I did not repeat what the other parts (Porcelain-ish) would do and everything fits together. For that, you need to refer to a couple of earlier messages from me on the "RFC: Subprojects" thread [*1*], and perhas use some imagination, because the outline I did back then was done without having much of what are in "pu" today and I may have got some details wrong. The message you are responding to was to give a demonstation of the current status of how the core part works.

Alexander Litvinov <lan@ac-sw.com> writes:

The "bind" line describes relationship among specific points in development histories of subprojects and superprojects so it needs to use commit object names.

If you mean by "binding a branch", to record how each subproject relates to the toplevel project (i.e. "the subproject bound to X/ subdirectory of the toplevel project comes from branch Y"), that information needs to be somewhere, but recording it in the commit object goes against the whole git philosophy.

Branch naming is a local matter. "master" in my repository represents a different development history of the project from what your "master" does. The subproject I happen to use the "subpro" branch to keep track of might be called "sub" in yours. Recording the name "subpro" on a "bind" line in a commit object makes that commit object useless when you fetch such a commit from my repository.

You are way ahead of what I am putting in "pu". That is all responsibility of the scripting part that use the core part; refer to the outline in earlier messages on the subprojects thread.

I think the above relates to this part of what you said:

I do not see any problem with this. The core level tools in "pu" supports that mode of operation. They also support another mode of operation, checking out the whole thing and making a commit for subprojects.

Simply untrue.

Even though I will use present tense (e.g. "the checkout command notices") in the following, you should remember that the current set of barebone Porcelainish scripts have not been taught about any of this.

You need to keep a file that describes how your repository is tracking the development histories of each subproject in $GIT_DIR/bind, that would look like:

	master main=/ subpro=sub/

meaning:

	In this repository, the project whose history is kept
	track of by "master" branch binds the projects whose
	history is kept track of by "subpro" branch at sub/ and
	another project that holds the rest whose history is
	kept track by "main" branch.

You can have more than one branches for subprojects and the combined project. Just have more lines in $GIT_DIR/bind, like so, to achieve that [*2*]:

     master main=/ subpro=sub/
     master20 main2=/ subpro=sub/
     master02 main2=/ subpro2=sub/
	...

To work on the example project, there are two alternative workflows that can be supported:

1. You can be on "master" branch (i.e. checking out everything
   at the right place), and make necessary changes, be they at
   toplevel or sub/ directory.  

   The checkout command notices you are on "master" branch, and
   you are checking out a bound commit which has "bind" lines
   for / and sub/ directories.  When committing, a single "git
   commit" may make potentially three commits.  (1) If you want
   to commit changes you made to sub/ part, that is committed to
   "subpro" branch; (2) If you want to commit changes to the
   rest, that is committed to "main" branch; (3) and a commit to
   the "master" branch is made to record the state of the whole
   thing.

2. You can work on an individual subproject without bothering
   the combined project.  You can have another repository to do
   developments of the project this repository uses "subpro"
   branch to keep track of.  When that work in the other
   repository on a subproject is ready to be integrated into the
   whole, you would fetch/merge the subproject part into this
   repository from there, advancing "subpro" branch head in this
   repository.

   When this happens, you can notice that the commit on the
   "bind" line for sub/ part does not match the branch head that
   keeps track of that part of the tree (i.e. "subpro")
   anymore, and can update sub/ part by merging.

This is precisely why I said "the branch naming is a local issue", and the commit object does not record branch name. In the above workflow #2, there is no reason for the other repository that develops the sub/ subproject part to name its primary branch "subpro". Most likely it is named "master" there, and the combined project would fetch its advancement by issuing:

	$ git fetch ../subprorepo master:subpro

If you have more than one branches in the other repository and would want to use that instead, you would fetch from that branch not from master there.

You can also keep more than one branch for a subproject inside the combined repository and have more than one $GIT_DIR/bind lines that describe different superprojects that bind different branches of the same subproject at the same location in the corresponding superproject branch..

[Footnote]

*1* Here are a couple of key messages in the thread, that attempt to describe how the things would fit together:

    http://article.gmane.org/gmane.comp.version-control.git/14781
    http://article.gmane.org/gmane.comp.version-control.git/14809

*2* Theoretically you could have NxMxOx... master project branches for subprojects with N, M, O,... branches, but in practice, the combined project is an integration field, and most of the combinations are not something you are interested in.

Not all of the N branches in a subproject need to be directly integrated into the whole --- most of them are used only while coming up with the version of the subproject that is suitable for the integration.

Junio C Hamano <junkio@cox.net> writes:

Sorry, these came out somewhat in a wrong way, so let me clarify.

What I meant was that there isn't anything coded so far that makes your worries real issues yet, and I do not intend to code Porcelainish scripts that are broken in the ways you see as problems in your message.

The point you raised are valid concerns. You need to keep them in mind when you start writing subproject aware version of git-checkout, git-commit and git-merge commands (among other things I might have forgotten, but I think these three covers pretty much everything). You are welcome to beat me to it, since I am not planning to do them right away.

On Saturday 21 January 2006 20:37, you wrote:

The original gitlink proposal did exactly this: it recorded the place where a subproject is bound by putting a gitlink into a tree. This way, the binding point can be changed, and is subject to versioning itself.

I just realized that this is not currently possible with the bind lines.
What about the following usage szenario:
- in a superproject, I use a subproject X implementing some lib by 
  binding it at X/. My Makefile recurses into X/ for this.
  This is recorded at commit point (A)
- later on, I realize I need another lib from a probject Y; I want
  to put the libs X and Y into subdirectory lib/ of my superproject;
  i.e. I bind Y at lib/Y/ and move the binding point of X to lib/X/.
  The Makefile is changed accordingly to build the subprojects.
  This is recorded at commit point (B)

A $GITDIR/bind alone will no work, as moving back to (A) would keep the binding point of subproject, and make is broken.

I understand that "moving binding point of X from X/ to lib/X/" is not representable within the index as a simple change. Is this the main issue for your "against the whole git philosophy"?

I think it still is quite useful to put the binding point into bind lines of the commit. Of course, moving a binding point has to go together with a new commit.

What about putting $GITDIR/bind information directly into reference files?

 $HOME/gitproj> cat .git/refs/heads/master
 92347432598...
 bind main=/
 bind subpro=sub/

This way, you can rename/copy heads, and the binding info will stay with the commit. This also is nice for subprojects that do not need the bind lines, but similar to current cogito subproject proposal:

  92347432598...
  main=/
  subpro=sub/

would be the same superproject/subproject configuration without commiting bind lines.

Josef

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> writes:

The original gitlink proposal records commit object name in the link object itself, so do bind lines in the commit object in bound commit proposal. In either way, you need to deal with the subproject relocation at the Porcelain level.

I was hoping that, upon seeing these two commits (let's say we are dealing with two-way merge aka "checkout"):

	In commit 1:
		bind xxxxx... X/

	In commit 2:
		bind yyyyy... lib/X/
                bind zzzzz... lib/Y/

the tool could notice that xxxxx... and yyyyy... are related in their ancestry chain, detect the relocation of subprojects, and update the $GIT_DIR/bind file (maybe with some help from the end user). We can do something similar in gitlink approach as well.

I do not see why. $GIT_DIR/bind can be adjusted by the tool upon checkout to reflect the reorganized tree.

I think that would also work. Although I do not immediately see major difference in expressiveness either way, that may be a cleaner way to achieve what we want to do.

On Sunday 22 January 2006 03:44, you wrote:

Ah, OK, sorry. This was a misunderstanding from my side; please forget the other suggestion about change reference files, too ;-)

Josef

Just-in-case panic response...

Dear diary, on Sun, Jan 22, 2006 at 12:33:25AM CET, I got a letter where Josef Weidendorfer <Josef.Weidendorfer@gmx.de> said that...

Please don't. I can see no real advantage to separate files (except for the very rare case of rename/copy/removal of heads) except for massive porcelain breakage and significant clutter-up of all the code dealing with refs.

Please leave our poor refs alone. :) They are fine as they are - simple one-value thing.

On Sun, 22 Jan 2006, Josef Weidendorfer wrote:

I think you're misunderstanding the use of the "bind" file or equivalent. It isn't a configuration file that specifies where the subprojects go; it's a state file that tracks where the subprojects currently are. The commits by themselves are sufficient to indentify the locations of the subprojects, and the bind file would be written by "git checkout" reading a commit with subprojects. It's used to create the next commit, in much the same way that MERGE_HEAD is used to create the next commit by storing information between the git call that starts a merge that needs user interaction and the git call to commit the merge.

So moving back to (A) wouldn't keep the binding point of subproject, because it would rewrite bind to what it had been.

I'm going to suggest again keeping this information in the index file (but not in the index data structure, so the changes to the code are only in the library routines to read and write the file, and, of course, anything that's actually trying to manipulate the binding locations). I started working on a patch to pu to skip S_IFDIR entries from the index file when building the table in memory, and that was straightforward, but I got into sysadmin issues when I was going to test giving it something to skip.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow <barkalow@iabervon.org> writes:

A lot better said than my version of the response.  Thanks.

I have been thinking about this one, and having read that read-cache code I think the coding is not too involved.

My current inclination is to use the same version number (2) by default and promote it to a new version number (3) once you add subproject-binding information to the index file. Then current tools would keep working on repositories created or operated upon with the new tools, as long as the project does not use the new feature.

On Sun, 22 Jan 2006, Junio C Hamano wrote:

I think bumping the index file version number shouldn't be too big a deal; index files are rarely used by different versions, except for the case where you've installed a new version of git, and that's generally a later version, not an earlier one.

It might make sense to add logic to make the stable series accept version 3 files without special entries, though. And maybe future-proof by having it look for unexpected flags and give a "too new version" error for them, even if the version number is within range. (I.e., if we add more special entry types later, it would tell you if your index file uses a feature that the version of git you're using doesn't support, even if the version number hasn't changed, and we reserve changing the version number for things where the file wouldn't read right at all or something previously legal changes meaning.)

On a side topic, is there any reason not to convert ls-tree to use struct tree, and kill the other tree-object-parsing code, which doesn't seem to be used by anything else?

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow <barkalow@iabervon.org> writes:

ls-tree _might_ be doing something struct tree interface does not support, but I do not know.

If you are inclined to, please go wild ;-).

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> writes:

No. I meant an exposure of local branch names by recording them in the commit object, and nothing else, by that comment.

Instead of using an extra $GIT_DIR/bind file extending what we record in the index file is OK. $GIT_INDEX_FILE, $GIT_DIR/HEAD, and $GIT_DIR/bind pretty much go hand-in-hand anyway and they _are_ local to the repository, just like branch names are, so I do not have any problem with using local branch names in these places, but not in a commit object (or gitlink object for that matter).

One more comment: it seems to me it is not possible to make two branches on separate subprojects with the same name.