Feature Enhancement Idea.

Hi,

I'm not sure if this is the right place, but I thought I'd post my idea and maybe somebody will either redirect me to the right place, or give me that "that wont happen".

Im fairly new to GIT (wish I had discovered it long ago), and I really like using it - great work guys/garls :)

My idea is to enhance GIT to support (I'll call it) development "layers". The current design of GIT is that the working repository and working directory assume that all files belong together in the same project. I would like to see GIT go 3D and support layers, so that files (and/or file content) can belong to multiple repositories (or considered unique projects), even though the working tree presents all files as if they were one.

To explain further...

Lets say, I am not the primary developer of a project, however I am a "module"/"plugin"/"addon" contributor to a project - ie: my primary involvement is to write additions to existing projects. (EG: I write/support a driver in the kernel (that is not included in the base tree), or I write an "addon" to an existing application, using that existing applications "module" capability).

As part of me developing my "module" (or modules if I develop more than one), I need the working tree to have the "base" and "my bits". I want to manage both the changes I make to "my bits", and also record any changes that need to be made to the "base" (so that "my module(s)" will work). Normally, the changes to the "base" would be submitted upstream (and hopefully accepted), while I would normally be responsible for the packaging and change control of "my bits".

If the upstream chooses not to accept my contributions (ie: my changes never appear in base), then whenever I package "my bits", GIT would also include the base components as well.

I know I can achieve some of this by using GIT branches (I've been doing that so far), however, GIT branches has a few limitations that I am sure that GIT "layers" would overcome... Ultimately, I believe GIT can handle my layers idea - and it should be possible to have multiple layers (where multiple components could come from different GIT repositories), however, they are all "checked out" into the one working tree.

Thinking about this further, there would be two possible layer
scenarios (I think GIT can handle both).
* File autonomy (most cases)
This is where files (by filename) belong to different projects. EG: If
my working tree had files "a, b & c", "1, 2 & 3", and "X Y & Z". Files
"abc" could be layer one (the base), files "123" could be layer two
(dependant on base) and files XYZ could be layer three (which could be
dependant on base OR dependant on layer two).

Whenever modifications were done to any file, GIT would know which layer owns the file and GIT processing is done as normal. Upstream pulls from any layer should not normally generated any conflicts, unless there are filename clashes between layers (and in this situation the layer hierarchy should be considered authoritative, with the conflict needed to be resolved in a lower layer.)

* Content autonomy
This is were some content in files belongs to "my work" (eg: Modifying
a Makefile to compile my work when the base is compiled). In this
situation, I may have two outcomes - I either want the changes to
flagged for upstream (to hopefully be included), or I may want to keep
the changes with my work, because I know it upstream would never
accept them (or it isnt appropriate).

In either cases, upstream pulls should be considered authoritative and any conflicts I would need to resolve as normal commit (either wanting them to be resubmittable for upstream, or commiting them as part of my work.)

Like I mentioned, I can achieve some of this by the use of branches
already, however, where it comes complicated, is when:
* I commit a change to the wrong branch (and thus upstream will never
see my enhancements),
* I want to identify changes to one layer (that I went to send to
upstream for review), without including the other layers (because it
probably isnt relevant)
* I want to work on more than one layer (I need to be diligent about
pulling and merging)

An example of usage might be:
* git clone ... (or git init) -layer "A"
* git checkout -b mywork -layer "A"
* git clone ... (or git init) -layer "B" -dependson "A"
* git checkout -b mymodule -layer "B"
* add/remove/edit files
* git add file x -layer "A"
* git add file y -layer "B"
* git rm file z -layer "A"
* git commit (as usual)
* git tag "V2.8" -layer "A"
* git tag "mymodule V1.0" -layer "B"

git diff -layer "A" mywork.. would show my changes that I would want sent upstream (without mymodule commits) git archive -layer "B" would package up my module for distribution (without layer "A") git archive -layer "A" would package up my version of layer A (without layer "B")

Could this be included as part of GITs functionality (or is it possible already) ?

...deon

Hi,

On Wed, Sep 23, 2009 at 8:17 AM, Deon George <deon.george@gmail.com> wrote:

Perhaps you could have a look at "git replace" that is now in the master branch. It could be improved to provide different "views" of a single repository. I don't think that alone it would provide everything you want though.

Best regards, Christian.

On Wednesday 23 September 2009, Deon George wrote:

Have a look at git submodules ('git help submodule'), or the git-subtree script that has been discussed on this list a couple of times [1].

[1] http://alumnit.ca/~apenwarr/log/?m=200904#30 and http://github.com/apenwarr/git-subtree

...Johan

2009/9/23 Johan Herland <johan@herland.net>:

git submodule looks like it will do a little of what I want - I'll do some more reading on it to see exactly how it works. Thanks for the tip.

My initial look at it seems to miss an important feature that my layer idea would provide. It looks like git submodule assumes that everything in a subdirectory belongs to a repository - with my layer idea, I would want any layer to share the same directory structure and files (or content) belonging to a distinct layer's repository.

IE: the base might provide plugins plugins/README modules modules/README lib/common.php

and a layer might provide plugins/a.php modules/a.php lib/a.php

another layer might provide plugins/b.php modules/b.php lib/b.php

If I was a C developer, I'd have a go at creating it - but I'm just a php developer :) ...deon

On Wed, Sep 23, 2009 at 2:12 PM, Deon George <deon.george@gmail.com> wrote:

    Practically you want something like unionfs [1] but for git. Right?

    But probably you could settle for something like Hg Queues [2]. Is
there similar for Git?

    Ciprian Craciun.

    [1] http://en.wikipedia.org/wiki/UnionFS
    [2] http://mercurial.selenic.com/wiki/MqExtension

On Wednesday 23 September 2009, Ciprian Dorin, Craciun wrote:

TopGit, StGit, Guilt?

Not sure they (or Hg Queues) solve Deon's problem though...

...Johan

Deon George <deon.george@gmail.com> writes:

I think you are getting ahead of yourself by coming up with "layers" as a solution, without really spelling out and then thinking through what problem you are trying to solve.

I think the primary scenario your itch comes from is (I am writing this down to make sure I understand where you are trying to go):

 - you would want to build your changes on top of somebody else's code;

 - while doing so it is often more convenient if you do not have to think
   about which pieces of changes should go to upstream, and which other
   pieces of changes should stay local;

 - git allows you to do this with topic branch workflow, with selective
   adding with "add -p", stashing and branch switching.

 - You can however misidentify which bits should go to upstream and commit
   to a wrong branch.  You want to reduce the chance of this mistake.

Am I following you Ok so far?

Now, how does "layers" (I understand by this word you mean the concept of "layering a worktree from one branch on top of layering a worktree from another") solve that problem? If you come up with a nice way to identify, and have the tool help you identify, "these bits belong to that layer", wouldn't that approach also apply to existing topic branch workflow by having the same logic of that magic tool to help you say "these bits should go to that topic branch"?

When the granularity of the change is per-file (your "autonomous" case), "git checkout" to switch back to your topic would already solve half of that issue.

Suppose you have mainline (origin/master) and your customization topic (custom), and file F is something that does not and will never exist in the mainline. You use your master branch as your integration testing branch. With traditional topic branch workflow, you would

    $ git checkout custom
    ... work on F and other files, test the changes to your satisfaction
    $ git commit -m 'I am satisfied on my custom work'
    $ git checkout master
    $ git rebase origin/master ;# keep up with the upstream
    $ git cherry-pick ... ;# some commits in custom that should go to upstream
    $ git merge custom ;# test merge to be thrown away!

    ... now you are on the integrated result in 'master'.
    ... test, notice breakages in F, and edit it to improve it
    ... the changes to F belongs to 'custom', not for upstream!
    $ git checkout custom ;# this will take the changes to F with you
    $ git commit -m 'I should have done this to F instead' F

    ... go back to do more work
    $ git checkout master
    ... loop forever ;-)

and after you are satisfied, you would reset the merge from custom away and offer the master that is combination of what was in the origin/master plus the "upstream worthy" bits you cherry-picked ones from your custom branch.

When the granularity is sub-file, "checkout -m custom" or "stash, checkout custom, then unstash" would help carrying the changes across branches. You would need to identify which changes you would want to commit to "custom" and which changes you would want to give to upstream by making a commit on "master" when you go back there again. The middle part of the above workflow will be slightly modified for a file G that is shared between origin/master and custom branches, perhaps something like:

    ... now you are on the integrated result in 'master'.
    ... test, notice breakages in G, and edit it to improve it
    ... the changes to G belongs to 'custom', not for upstream!
    $ git checkout -m custom ;# this will take the changes to G with you
    $ git add -p G ;# pick changes that only belong to custom
    $ git commit -m 'I should have done this to G instead'

If the "layers" logic somehow can help you automate the process of sifting your changes into these two categories (perhaps it may scan the file and knows which function belongs to "custom" alone, I dunno and care about the details at this level of discussion), wouldn't that same logic help you the same way?

Past attempts that made into the toolchest are "add -p" (and "stash -p" that uses the same mechanism) which is interactive and not automated. Maybe you can build some logic to automate it further (e.g. "changes to this function should automatically be excluded from "git add" while on 'master', but should automatically be included in "git add" while on 'custom'), and it may turn out be a good addition to the toolchest.

It also may make a good addition to the toolchest to have a tool to further simplify the branch switching explicitly initiated by the end user in the above illustrations. In some cases, such as the "autonomous" case above, you do not _have_ to go back to custom branch from the work-flow point of view (you would be committing a totally untested work to custom branch, but as long as you will revisit and retest the changes in its context, it is not a major crime at all). It would be useful if you can say, while still on 'master' branch, "I just made this fix to file F that belongs to 'custom'; commit that change alone to 'custom' branch". Written as a shell script, roughly, it would look something like:

    #!/bin/sh
    target_branch="$1"
    base=$(git rev-parse --verify "$target_branch") || exit
    shift ;# all others are paths
    GIT_INDEX_FILE=.git/temp
    export GIT_INDEX_FILE
    git read-tree -m "$1"
    git add "$@"
    c=$(echo "side commit" | git commit-tree $(git write-tree) -p "$base")
    git update-ref "refs/heads/$target_branch" "$c"

But I do not see how the concept of "layering a worktree from one branch on top of layering a worktree from another", which is the implication I get from the word "git layers", helps any of that. It seems an orthogonal concept that we can do without to solve the problem you are describing.

Junio C Hamano wrote:

Yes - more correctly, I want to enhance an application, where I am (or different person - it doesnt matter) is responsible for that code. Thus I want to source code manage my enhancement, and (for whatever reason), the other components of the application dont want my part in their "base".

Not strictly true, but yes - I am aware that my modules might be completely autonomous to the base code (thus I source control manage everything that my module requires), or I discover bugs in the base code, that I believe upstream should have. There may also be a case, where I want to modify some code in the base, but I know that it shouldnt never be included in the base code (and thus I would need my commit to be stored in my SCM, not the base's SCM) - eg: Modifying a Makefile...

So while I would commit to a layer, I should make a conscious decision of where the commit should go (I own the commit work, or I want to send the commit upstream as a contribution enhancement/fix) - because the default assumption may be wrong.

I dont know about topic branches "git add -p"? (I havent got that advanced yet, looks like I need to do some more research on the features of GIT :)

Yes, but one more important criteria - my working tree is made up of files (or content) managed in more than once source control management repository - I use an SCM to manage and track changes, and I use multiple repository because any one owner of one repository may not want the code from another as part of their "base". IE: It could be I write a custom driver for a widget device, that Linus doesnt want that code in the base kernel (for whatever reason) and I maintain that part of the code, or (in my case), I want to maintain (and encourage others to contribute) to a module (or set of modules) for an existing web application framework.

Additionally, each layer should be able to switch between its own branches (ie: My driver 1.0 works well with kernel "N", now Linus publishes "N+1", I want to switch that (base) layer to "N+1" and test/fix my module to work with that release - and then commit/tag it as such - ie: my driver (now 1.1) works with "N+1"). Further a bug is now found in my 1.0 driver, so I want to switch back the kernel layer to "N", and my driver layer to 1.0, make the fix, and release 1.0.1)... Kinda get the idea Im working towards...?

Additionally, it should not necessarily be "one base" and "my layer", it may be multiple layers (and hence multiple repositories), but all layers make up an application with multiple features, each individually source control managed. Each layer would have a dependency to another (otherwise they could be developed in their own repository entirely - and the dependency may be needed to be know for merge conflict resolution (where do the conflicts need to be commited), when pulling from the upstream owners of that layer)

Thus, when I package up my module - it shouldnt include the base (since the base may already be installed), and the base developers want to keep their application more like a framework - so they are responsible for the core workings of the code, and the module developers are responsible for their components.

To achieve what I want today, I think I would need to have:

master master-mycontribs (branched from master, and merged from master) mymodule (branched from master-mycontribs and merge from there - this is pushed to my SCM repository) mymodule-working (where I test and commit - and merge back to my module)

(this comes stuck, when I want to switch master to a previous version...)

And thus, the diff between "master" and "master-mycontribs", are my contributions to go upstream. The diff between "master" and "mymodule" is what I would package to make my module as an enhancement to anyone already using "master" (at release "N").

If upstream accepts my enhancements, then "master" = "master-mycontribs" (and master is now release "N+1"), and the next release of my module (1.2) would still be the diff between "master-mycontribs" and "mymodule" (but would be excluding my code that I added to master-mycontribs (since master now equals that), because the end user who is already running N+1 has those enhancements.

I'll research git "topics" and see if that does what I am dreaming of :)

...deon

Deon George <deon.george@gmail.com> writes:

[cut very long description]

First, I assume there that you do not allow for the same file to belong to different repositories.

Second, if all parts that you want to belong to other repository are in separate subdirectories, and all files in those subdirectories belong to this other repository, you can try either submodules (git-submodule), or subtree (subtree merge, or third-party git-subtree helper). Note also that this assume that you want to have 'master' repository which indirectly or directly has al the files.

Third, if the above isn't what you want, then you can manually intermingle working directories of different git repositories (probably requiring decouplig of bare git repository (git-dir) from working area (work-tree)). Git repository know what files it tracks, so you would only need to take care to ignore files that belong to other repositories.

If it is to manual for you, and to error prone, you are welcome to come up with set of scripts implementing "layers" feature you want. That is how initial version of submodule feature was done...

Jakub Narebski wrote:

Well, no, I cant see why a file cannot belong to both repositories. I'm sure it would be easier if a file did belong to a unique repository, but given that GIT operates at a content layer (from what I've read anyway), I cant see why it couldnt belong to more than one.

I had a quick look at submodule - I dont think it helps me anyway, since I want the flexibility for files to belong to many sub-directories - not all under 1 sub directory hierarchy.

Ahh, now this sounds like it might be what I want to do - I think I'll explore this. I can see that it would provide file level autonomy only, but as a starting point I think it will help heaps...

I'll work on it as time permits and come up with some scripts to provide what I am after (need to learn more on the workings of GIT first - everyday I learn a new feature :)

Thanks for your idea...

...deon

Deon George <deon.george <at> gmail.com> writes:

Proof of concept:

export GIT_WORK_TREE=.

git --git-dir=.git1 init git --git-dir=.git1 add view1-file git --git-dir=.git1 commit -m"view1 initial"

git --git-dir=.git2 init git --git-dir=.git2 add view1-file git --git-dir=.git2 commit -m"view2 initial"

git --git-dir=.git1 log git --git-dir=.git2 log