threads / rfc / 1576

[RFC] Stgit - patch history / add extra parents

Subject: [RFC] Stgit - patch history / add extra parents

## tl;dr

22 messages between Aug 18, 2005 and Aug 31, 2005.

replies: 21people: 5as markdown or json

Jan Veldeman· Aug 18, 2005, 19:57 UTC · lore
Hi,

I like stgit very much, but I feel there is still something missing: stgit is very handy when you use it for patches which should be pushed to mainline rather quickly. But for pacthes which won't be pushed immediately to mainline, it would be usefull to have a history of the patches itself.

The patch below, together with the following script could be used to make snapshots of the patch stack (I call it freeze, as I thought snapshot was already going to be used for something else):

#!/bin/bash # stg-freeze set -ex top=$(stg top)

# Freeze the current stack
branchname=$(basename $(readlink .git/HEAD))
while read patch
do
	cp .git/patches/$branchname/$patch/top .git/patches/$branchname/$patch/parent
done < .git/patches/$branchname/applied

# Refresh the patches stg pop -a stg push -t $top # end stg-freeze

Note: this is a proof of concept, when/if it would be incorporated, it
should be implemented in stgit itself, and a bit more efficient, especially
the refreshing of the patches ;-)

This is how it works: when not doing anything, stgit works as normal. Once stg-freeze is called, it creates .git/patches/$branchname/$patch/parent which contains the SHA1 id of the "frozen" patch. By refreshing the stack, all patches now include the corresponding frozen patch as a parent.

The following script is a test I use. Add stg-freeze to the path and source/run it in an empty directory, view with gitk afterwards:

### echo "Initial commit" | cg-init stg init

stg new a -mpatch-1 echo a > a stg add a stg refresh -mpatch-1 stg new b -mpatch-2 echo b > b stg add b stg refresh -mpatch-2 stg-freeze stg pop echo a2 >> a stg refresh -mpatch-1-update stg push stg-freeze echo b2 >> b stg refresh -mpatch-2-update

stg pop -a echo c > c cg-add c cg-commit -m"Mainline advance" stg push -a

stg-freeze

stg pop -a echo d > d cg-add d cg-commit -m"Mainline advance 2" stg push -a

stg pop -a echo e > e cg-add e cg-commit -m"Mainline advance 3" stg push -a

###
Comments/remarks are very welcome.
---
 stgit/stack.py |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)
diff --git a/stgit/stack.py b/stgit/stack.py
--- a/stgit/stack.py
+++ b/stgit/stack.py
@@ -140,6 +140,13 @@ class Patch:
         elif os.path.isfile(fname):
             os.remove(fname)
 
+    def get_parents(self):
+        parents=[]
+        if (self.__get_field('parent') != None):
+                parents = parents + [self.__get_field('parent')]
+        parents = parents + [ self.get_bottom() ]
+        return parents
+
     def get_bottom(self):
         return self.__get_field('bottom')
 
@@ -362,7 +369,7 @@ class Series:
         if not committer_email:
             committer_email = patch.get_commemail()
 
-        commit_id = git.commit(message = descr, parents = [patch.get_bottom()],
+        commit_id = git.commit(message = descr, parents = patch.get_parents(),
                                allowempty = True,
                                author_name = author_name,
                                author_email = author_email,
Catalin Marinas· Aug 19, 2005, 09:53 UTC · re: Jan Veldeman · lore

Re: [RFC] Stgit - patch history / add extra parents

Jan Veldeman <jan.veldeman@gmail.com> wrote:
Show 5 quoted lines
> I like stgit very much, but I feel there is still something missing:
> stgit is very handy when you use it for patches which should be pushed to
> mainline rather quickly. But for pacthes which won't be pushed immediately
> to mainline, it would be usefull to have a history of the patches
> itself.

The patch history feature was available in StGIT 0.1/0.2 releases where you should have run a 'stg commit' before 'stg refresh'. The commit was handling all the history changes, with separate commit messages and refresh was updating the main commit with 2 parents. I removed it in 0.3 after some people (I think it was Paul Jackson and Daniel Barkalow) convinced me that this would make it yet another SCM interface on top of GIT, which wasn't really my intention.

The main problem with having multiple parents for a commit object corresponding to a patch is the upstream merging via 'git pull'. In general you don't want a gatekeeper to pull the history of your patch but the patch only.

> The patch below, together with the following script could be used to
> make snapshots of the patch stack (I call it freeze, as I thought snapshot
> was already going to be used for something else):

'snapshot' is not yet used for anything and I'm not sure how it is best to be implemented. I thought about simply saving the current HEAD into some .git/refs/heads/<file>, without preserving any history for the patch. A gitk on this file would show the patches as they were on the time of the snapshot creation. A new snapshot would remove this.

It might be best for a per-patch history to have a separate file in <branch>/<patch>/, maybe called freeze, which keeps this history information. The top one should remain unchanged. Its hash could be accessed with the 'stg id /freeze' command (implemented yesterday). This file would only be updated via the 'freeze' command and its parent would be the previous freeze value.

Would this be close to what you need?
-- 
Catalin
Jan Veldeman· Aug 19, 2005, 18:27 UTC · re: Catalin Marinas · lore

Re: [RFC] Stgit - patch history / add extra parents

Catalin Marinas wrote:
Show 8 quoted lines
> 
> The patch history feature was available in StGIT 0.1/0.2 releases
> where you should have run a 'stg commit' before 'stg refresh'. The
> commit was handling all the history changes, with separate commit
> messages and refresh was updating the main commit with 2 parents. I
> removed it in 0.3 after some people (I think it was Paul Jackson and
> Daniel Barkalow) convinced me that this would make it yet another SCM
> interface on top of GIT, which wasn't really my intention.
hmm, I must have misted those threads, I'll try to find and read them.
Show 5 quoted lines
> 
> The main problem with having multiple parents for a commit object
> corresponding to a patch is the upstream merging via 'git pull'. In
> general you don't want a gatekeeper to pull the history of your patch
> but the patch only.

I agree that such history should not be imported into the mainline, but such history would still be very usefull when these patches won't be pushed to mailine immediately. Also, when pushing to mainline, this history can easily be removed by removing the branch/patch/parent files and refreshing (this should off course be automated)

Show 20 quoted lines
> 
> > The patch below, together with the following script could be used to
> > make snapshots of the patch stack (I call it freeze, as I thought snapshot
> > was already going to be used for something else):
> 
> 'snapshot' is not yet used for anything and I'm not sure how it is
> best to be implemented. I thought about simply saving the current HEAD
> into some .git/refs/heads/<file>, without preserving any history for
> the patch. A gitk on this file would show the patches as they were on
> the time of the snapshot creation. A new snapshot would remove this.
> 
> It might be best for a per-patch history to have a separate file in
> <branch>/<patch>/, maybe called freeze, which keeps this history
> information. The top one should remain unchanged. Its hash could be
> accessed with the 'stg id /freeze' command (implemented
> yesterday). This file would only be updated via the 'freeze' command
> and its parent would be the previous freeze value.
> 
> Would this be close to what you need?
>

hmm, not exactly, for example, when reordering the patches (including the top one), I would like to see this in gitk. Or when a patch has been dropped (amongst a lot of patches), it should be easily spotted.

But even if the "stg-freeze" would not be incorporated into stgit, would it still be possible to include some sort of extra parents directory. So that the freeze can be implemented on top of stgit?

TIA

Best regards, Jan

Catalin Marinas· Aug 19, 2005, 22:15 UTC · re: Jan Veldeman · lore

Re: [RFC] Stgit - patch history / add extra parents

On Fri, 2005-08-19 at 20:27 +0200, Jan Veldeman wrote:
> hmm, not exactly, for example, when reordering the patches (including the
> top one), I would like to see this in gitk.
> Or when a patch has been dropped (amongst a lot of patches), it should be
> easily spotted.

I tried your patch but the gitk image confused me. I will look again at it tomorrow (it's quite late in the UK now).

One immediate thing I noticed was that the commits directly accessible via .git/HEAD are shown as empty by gitk and you would need to follow the parents to see what they contain. For every freeze, the patches expand to the right in gitk and the graph could get very complex after several freeze commands.

-- 
Catalin
Jan Veldeman· Aug 19, 2005, 19:48 UTC · re: Catalin Marinas · lore

Re: [RFC] Stgit - patch history / add extra parents

Catalin Marinas wrote:
Show 8 quoted lines
> 
> The patch history feature was available in StGIT 0.1/0.2 releases
> where you should have run a 'stg commit' before 'stg refresh'. The
> commit was handling all the history changes, with separate commit
> messages and refresh was updating the main commit with 2 parents. I
> removed it in 0.3 after some people (I think it was Paul Jackson and
> Daniel Barkalow) convinced me that this would make it yet another SCM
> interface on top of GIT, which wasn't really my intention.

I've quickly reread the threads about stg commit. Am I right to assume that _all_ history was being recorded? Because this is not what I want. The person controlling the archive should specify when to record the history.

So for example, you only tag (freeze) the history when exporting the patches. When an error is being reported on that version, it's easy to view it and also view the progress that was already been made on those patches.

Best regards, Jan

Catalin Marinas· Aug 20, 2005, 21:12 UTC · re: Jan Veldeman · lore

Re: [RFC] Stgit - patch history / add extra parents

On Fri, 2005-08-19 at 21:48 +0200, Jan Veldeman wrote:
> I've quickly reread the threads about stg commit. Am I right to assume that
> _all_ history was being recorded? Because this is not what I want. The
> person controlling the archive should specify when to record the history.

True, I was talking about the full history, but I don't find the idea much different. There was another mail on this issue - http://article.gmane.org/gmane.comp.version-control.git/5191

> So for example, you only tag (freeze) the history when exporting the
> patches.  When an error is being reported on that version, it's easy to view
> it and also view the progress that was already been made on those patches.

I agree that it is a useful feature to be able to individually tag the patches. The problem is how to do this best. Your approach looks to me like it's not following the GIT DAG structure recommendation. Maybe the GIT designers could further comment on this but a commit object with multiple parents should be a result of a merge operation. A commit with a single parent should represent a transition of the tree from one state to another. With the freeze command you proposed, a commit with multiple parents is no longer a result of a merge operation, but just a convenience for tracking the patch history with gitk.

I tried your patch and test script but, if you try to reorder the patches, the DAG structure become quite hard to understand and that's only for a small set of frozen states.

I think the best approach would be to keep a list of tags for each patch. They won't be browsable with gitk but the DAG structure would be clean. We might get another tool like gquilt to be able to show all the saved states of a patch.

Sorry for not including your patch but, while I agree with the general idea of frozen patch states, I disagree with its implementation.

-- 
Catalin
Jan Veldeman· Aug 21, 2005, 09:40 UTC · re: Catalin Marinas · lore

Re: [RFC] Stgit - patch history / add extra parents

Catalin Marinas wrote:
Show 13 quoted lines
> > So for example, you only tag (freeze) the history when exporting the
> > patches.  When an error is being reported on that version, it's easy to view
> > it and also view the progress that was already been made on those patches.
> 
> I agree that it is a useful feature to be able to individually tag the
> patches. The problem is how to do this best. Your approach looks to me
> like it's not following the GIT DAG structure recommendation. Maybe the
> GIT designers could further comment on this but a commit object with
> multiple parents should be a result of a merge operation. A commit with
> a single parent should represent a transition of the tree from one state
> to another. With the freeze command you proposed, a commit with multiple
> parents is no longer a result of a merge operation, but just a
> convenience for tracking the patch history with gitk.

My interpretation of parents is broader than only merges, and reading the README file, I believe it also the intension to do so (snippet from README file):

A "commit" object ties such directory hierarchies together into a DAG of revisions - each "commit" is associated with exactly one tree (the directory hierarchy at the time of the commit). In addition, a "commit" refers to one or more "parent" commit objects that describe the history of how we arrived at that directory hierarchy.

Another advantage of this approach is when pushing the top of your repository to others, less rebasing would be noticed: When pushing the top of your stack out to others, the new one won't be a (possibly indirect) child of the previous push. This makes updating such a head more dificult. With this approach, the new head will be a (possibly indirect) child of the previous one (except when the top patch has been removed, but this corner case could also be solved).

Because of this less rebasing, this aproach also facilitates co-development on stgit patch stacks: by using a base:top id pair, the stgit patch stack could be initialized, including its exported history. By following the parent/child relations, 2 patch stacks could be compared and merged.

> 
> I tried your patch and test script but, if you try to reorder the
> patches, the DAG structure become quite hard to understand and that's
> only for a small set of frozen states.

I acknowledge this problem, but the solution for this would be to give gitk another way of viewing such history: only the latest description of a patch should be visible and the history should become more horizontal (because the older descriptions wouldn't be shown, the history of those patches could be viewed on the line with the latest description)

But changing gitk would only be usefull when this approach would be taken.
Show 9 quoted lines
> 
> I think the best approach would be to keep a list of tags for each
> patch. They won't be browsable with gitk but the DAG structure would be
> clean. We might get another tool like gquilt to be able to show all the
> saved states of a patch.
> 
> Sorry for not including your patch but, while I agree with the general
> idea of frozen patch states, I disagree with its implementation.
>

I understand your objections when limiting parents only to merges, but I see them broader. So as you said: we indeed need a git designer to clear this up for us.

Best regards, Jan

Daniel Barkalow· Aug 22, 2005, 22:10 UTC · re: Jan Veldeman · lore

Re: [RFC] Stgit - patch history / add extra parents

On Sun, 21 Aug 2005, Jan Veldeman wrote:
Show 25 quoted lines
> Catalin Marinas wrote:
>
> > > So for example, you only tag (freeze) the history when exporting the
> > > patches.  When an error is being reported on that version, it's easy to view
> > > it and also view the progress that was already been made on those patches.
> >
> > I agree that it is a useful feature to be able to individually tag the
> > patches. The problem is how to do this best. Your approach looks to me
> > like it's not following the GIT DAG structure recommendation. Maybe the
> > GIT designers could further comment on this but a commit object with
> > multiple parents should be a result of a merge operation. A commit with
> > a single parent should represent a transition of the tree from one state
> > to another. With the freeze command you proposed, a commit with multiple
> > parents is no longer a result of a merge operation, but just a
> > convenience for tracking the patch history with gitk.
>
> My interpretation of parents is broader than only merges, and reading the
> README file, I believe it also the intension to do so (snippet from README
> file):
>
> A "commit" object ties such directory hierarchies together into
> a DAG of revisions - each "commit" is associated with exactly one tree
> (the directory hierarchy at the time of the commit). In addition, a
> "commit" refers to one or more "parent" commit objects that describe the
> history of how we arrived at that directory hierarchy.

One factor not mentioned there is that, as things move upstream, we often want to discard a lot of history; if someone commits constantly to deal with editor malfunction or something, we don't really want to take all of this junk into the project history when it is cleaned up and accepted.

So the point is that there are things which are, in fact, parents, but we don't want to list them, because it's not desired information.

Probably the right thing is to have two views of the stack: the internal view, showing what actually happened, and the external view, showing what would have happened if the developers had done everything right the first time. When you make changes to the series, this adds to the internal view and entirely replaces the external view.

I think that users will also want to discard the commits from the stack before rebasing in favor of the commits after, because (a) rebasing isn't all that interesting, especially if there's minimal merging, and (b) otherwise you'd get a ton of boring commits that obscure the interesting ones.

I think that the best rule would be that, when you modify a patch, the previous version is the new version's parent, and when you rebase a series, you include as a parent any parent of the input that isn't also in the input (but never include the input itself as a parent of the output; the point of rebasing is to pretend that it was the newer mainline that you modified). This should mean that the internal history of a patch consists of the present version, based on each version that was replaced due to changing the patch rather than rebasing it.

Of course, there's an interesting situation with the commits earlier in a series from a patch that was changed not being ancestors of the newer versions of those patches (because they weren't interesting in the development of those patches) but accessible as the commits that an interesting patch was based on.

A possible solution is just to consider the revision of any patch a significant event in the history of the whole stack, causing all of the patches to get a new retained version.

	-Daniel
*This .sig left intentionally blank*
Catalin Marinas· Aug 23, 2005, 16:55 UTC · re: Daniel Barkalow · lore

Re: [RFC] Stgit - patch history / add extra parents

Daniel Barkalow <barkalow@iabervon.org> wrote:
Show 5 quoted lines
> One factor not mentioned there is that, as things move upstream, we often
> want to discard a lot of history; if someone commits constantly to deal
> with editor malfunction or something, we don't really want to take all of
> this junk into the project history when it is cleaned up and
> accepted.

That's true but Jan's proposal is to choose which commits to preserve in the history via a 'freeze' command. That's a bit difficult to implement in a clean way since the patches are floating on top of the stack base and they change every time the base changes. It is possible that a previous frozen state might no longer apply on top of a new base.

> So the point is that there are things which are, in fact, parents, but we
> don't want to list them, because it's not desired information.

What's the definition of a parent in GIT terms? What are the restriction for a commit object to be a parent? Can a parent be an arbitrarily chosen commit?

Show 5 quoted lines
> Probably the right thing is to have two views of the stack: the internal
> view, showing what actually happened, and the external view, showing what
> would have happened if the developers had done everything right the first
> time. When you make changes to the series, this adds to the internal view
> and entirely replaces the external view.

That's what I've been thinking. StGIT currently only implements the external view.

An StGIT patch is a represented by a top and bottom commit objects. The bottom one is the same as the parent of the top commit. The patch is the diff between the top's tree id and the bottom's tree id.

Jan's proposal is to allow a freeze command to save the current top hash and later be used as a second parent for the newly generated top. The problem I see with this approach is that (even for the internal view you described) the newly generated top will have two parents, new-bottom and old-top, but only the diff between new-top and new-bottom is meaningful. The diff between new-top and old-top (as a parent-child relation) wouldn't contain anything relevant to the patch but all the new changes to the base of the stack.

Is the above an acceptable usage of the GIT DAG structure?
Show 5 quoted lines
> I think that users will also want to discard the commits from the stack
> before rebasing in favor of the commits after, because (a) rebasing isn't
> all that interesting, especially if there's minimal merging, and (b)
> otherwise you'd get a ton of boring commits that obscure the interesting
> ones.

StGIT does this currently. The old commit is no longer available and can be pruned (I still need to save the commits corresponding to the unpushed patches since prune would remove those as well).

Show 8 quoted lines
> I think that the best rule would be that, when you modify a patch, the
> previous version is the new version's parent, and when you rebase a
> series, you include as a parent any parent of the input that isn't also in
> the input (but never include the input itself as a parent of the output;
> the point of rebasing is to pretend that it was the newer mainline that
> you modified). This should mean that the internal history of a patch
> consists of the present version, based on each version that was replaced
> due to changing the patch rather than rebasing it.

Since you proposed this, my above usage of the DAG structure would be fine then.

Show 9 quoted lines
> Of course, there's an interesting situation with the commits earlier in a
> series from a patch that was changed not being ancestors of the newer
> versions of those patches (because they weren't interesting in the
> development of those patches) but accessible as the commits that an
> interesting patch was based on.
>
> A possible solution is just to consider the revision of any patch a
> significant event in the history of the whole stack, causing all of the
> patches to get a new retained version.
That's another idea to think about.
-- 
Catalin


P.S. I'll be away until next week and not able to follow-up the
discussions.
Daniel Barkalow· Aug 23, 2005, 18:05 UTC · re: Catalin Marinas · lore

Re: [RFC] Stgit - patch history / add extra parents

On Tue, 23 Aug 2005, Catalin Marinas wrote:
Show 6 quoted lines
> > So the point is that there are things which are, in fact, parents, but we
> > don't want to list them, because it's not desired information.
>
> What's the definition of a parent in GIT terms? What are the
> restriction for a commit object to be a parent? Can a parent be an
> arbitrarily chosen commit?

Something is legitimate as a parent if someone took that commit and did something to it to get the new commit. The operation which caused the change is not specified. But you only want to include it if anyone cares about the parent.

(For example, I often start with a chunk of work that does multiple things and is committed; I take mainline and generate a series of commits from there. It would be legitimate to list my development commit as a parent of each of these, since I did actually take it and strip out the unrelated changes. This would be a bit confusing in the log, but would make merges between something based on the "messy" version and something based on the "refined" version work well. On the other hand, I don't want to report the existance of the messy version, so I don't include it.)

Show 13 quoted lines
> An StGIT patch is a represented by a top and bottom commit
> objects. The bottom one is the same as the parent of the top
> commit. The patch is the diff between the top's tree id and the
> bottom's tree id.
>
> Jan's proposal is to allow a freeze command to save the current top
> hash and later be used as a second parent for the newly generated
> top. The problem I see with this approach is that (even for the
> internal view you described) the newly generated top will have two
> parents, new-bottom and old-top, but only the diff between new-top and
> new-bottom is meaningful. The diff between new-top and old-top (as a
> parent-child relation) wouldn't contain anything relevant to the patch
> but all the new changes to the base of the stack.

Having a useful diff isn't really a requirement for a parent; the diff in the case of a merge is going to be the total of everything that happened elsewhere. The point is to be able to reach some commits between which there are interesting diffs.

This also depends on how exactly freeze is used; if you use it before commiting a modification to the patch without rebasing, you get:

old-top -> new-top
      ^    ^
       \  /
      bottom

bottom to old-top is the old patch bottom to new-top is the new patch old-top to new-top is the change to the patch

Then you want to keep new-top as a parent for rebasings until one of these is frozen. These links are not interesting to look at, but preserve the path to the old-top:new-top change, which is interesting.

Ignoring the links to the corresponding bottoms, the development therefore looks like:

local1 -> local2 -> merge -> local3 -> merge ^ ^ ^ mainline---->-->--------->------>-->----->

And this is how development is normally supposed to look. The trick is to only include a minimal number of merges.

	-Daniel
*This .sig left intentionally blank*
Jan Veldeman· Aug 23, 2005, 21:23 UTC · re: Daniel Barkalow · lore

Re: [RFC] Stgit - patch history / add extra parents

Daniel Barkalow wrote:
Show 6 quoted lines
> On Tue, 23 Aug 2005, Catalin Marinas wrote:
> 
> Something is legitimate as a parent if someone took that commit and did
> something to it to get the new commit. The operation which caused the
> change is not specified. But you only want to include it if anyone cares
> about the parent.

This is indeed what I thought a parent should be used for. As an adition, I'll try to explain why I would sometimes want to care about some parents:

I want to track a mailine tree, but have quite a few changes, which shoudn't be commited to the mainline immediately (let's call it my development tree). This is why I would use stgit. But I would also want to colaborate with other developers on this development tree, so I sometimes want to make updates available of this development tree to the others. This is where current stgit falls short. To easily share this development tree, I want some history (not all, only the ones I choose) of this development tree included, so that the other developers can easily follow my development.

The parents which should be visible to the outside, will always be versions
of my development tree, which I have previously pushed out. My way of
working would become:
* make changes, all over the place, using stgit
* still make changes (none of these gets tracked, intermittent versions are
  lost)
* having a good day: changes looks good, I want to push this out:
  * push my tree out
  * stgit-free (which makes the pushed out commits, the new parents of my
    stgit patches)
* restart from top
[...]
Show 15 quoted lines
> This also depends on how exactly freeze is used; if you use it before
> commiting a modification to the patch without rebasing, you get:
> 
> old-top -> new-top
>       ^    ^
>        \  /
>       bottom
> 
> bottom to old-top is the old patch
> bottom to new-top is the new patch
> old-top to new-top is the change to the patch
> 
> Then you want to keep new-top as a parent for rebasings until one of these
> is frozen. These links are not interesting to look at, but preserve the
> path to the old-top:new-top change, which is interesting.

my proposal does something like this, but a little more: not only does it keep track of the link between old-top and new-top, it also keeps track of the links between old-patch-in-between and new-patch-in-between. (This makes sense when the top is being removed or reordered)

I hope this kind of clarifies my intension.
Thank you for clarifying the meaning of parents.

Best regards, Jan

Daniel Barkalow· Aug 23, 2005, 22:23 UTC · re: Jan Veldeman · lore

Re: [RFC] Stgit - patch history / add extra parents

On Tue, 23 Aug 2005, Jan Veldeman wrote:
Show 32 quoted lines
> Daniel Barkalow wrote:
>
> > On Tue, 23 Aug 2005, Catalin Marinas wrote:
> >
> > Something is legitimate as a parent if someone took that commit and did
> > something to it to get the new commit. The operation which caused the
> > change is not specified. But you only want to include it if anyone cares
> > about the parent.
>
> This is indeed what I thought a parent should be used for. As an adition,
> I'll try to explain why I would sometimes want to care about some parents:
>
> I want to track a mailine tree, but have quite a few changes, which shoudn't
> be commited to the mainline immediately (let's call it my development tree).
> This is why I would use stgit. But I would also want to colaborate with
> other developers on this development tree, so I sometimes want to make
> updates available of this development tree to the others. This is where
> current stgit falls short. To easily share this development tree, I want
> some history (not all, only the ones I choose) of this development tree
> included, so that the other developers can easily follow my development.
>
> The parents which should be visible to the outside, will always be versions
> of my development tree, which I have previously pushed out. My way of
> working would become:
> * make changes, all over the place, using stgit
> * still make changes (none of these gets tracked, intermittent versions are
>   lost)
> * having a good day: changes looks good, I want to push this out:
>   * push my tree out
>   * stgit-free (which makes the pushed out commits, the new parents of my
>     stgit patches)
> * restart from top

I'm not sure how applicable to this situation stgit really is; I see stgit as optimized for the case of a patch set which is basically done, where you want to keep it applicable to the mainline as the mainline advances.

For your application, I'd just have a git branch full of various stuff, and then generate clean commits by branching mainline, diffing development against it, cutting the diff down to just what I want to push, and applying that. Then the clean patch goes into stgit.

Show 21 quoted lines
> [...]
> > This also depends on how exactly freeze is used; if you use it before
> > commiting a modification to the patch without rebasing, you get:
> >
> > old-top -> new-top
> >       ^    ^
> >        \  /
> >       bottom
> >
> > bottom to old-top is the old patch
> > bottom to new-top is the new patch
> > old-top to new-top is the change to the patch
> >
> > Then you want to keep new-top as a parent for rebasings until one of these
> > is frozen. These links are not interesting to look at, but preserve the
> > path to the old-top:new-top change, which is interesting.
>
> my proposal does something like this, but a little more: not only does it
> keep track of the link between old-top and new-top, it also keeps track of
> the links between old-patch-in-between and new-patch-in-between.
> (This makes sense when the top is being removed or reordered)

I was thinking of this as being the top and bottom commits for a single tracked patch, not as a whole series. I think patches lower wouldn't be affected, and patches higher would see this as a rebase.

	-Daniel
*This .sig left intentionally blank*
Jan Veldeman· Aug 25, 2005, 07:09 UTC · re: Daniel Barkalow · lore

Re: [RFC] Stgit - patch history / add extra parents

Daniel Barkalow wrote:
[...]
Show 15 quoted lines
> > The parents which should be visible to the outside, will always be versions
> > of my development tree, which I have previously pushed out. My way of
> > working would become:
> > * make changes, all over the place, using stgit
> > * still make changes (none of these gets tracked, intermittent versions are
> >   lost)
> > * having a good day: changes looks good, I want to push this out:
> >   * push my tree out
> >   * stgit-free (which makes the pushed out commits, the new parents of my
> >     stgit patches)
> > * restart from top
> 
> I'm not sure how applicable to this situation stgit really is; I see stgit
> as optimized for the case of a patch set which is basically done, where
> you want to keep it applicable to the mainline as the mainline advances.

Maybe I forgot to mention this: I would also like to have my development tree split up in a patch stack. The separate patches makes tracking the mainline a lot easier (conflicts are a lot easier to solve)

Show 5 quoted lines
> 
> For your application, I'd just have a git branch full of various stuff,
> and then generate clean commits by branching mainline, diffing development
> against it, cutting the diff down to just what I want to push, and
> applying that. Then the clean patch goes into stgit.

But this would assume that once the patch goes into stgit, it won't change except when the parent gets updated. I think we will still change the patches quite a bit and simultanious by a couple of people.

Show 11 quoted lines
> 
> > [...]
> > my proposal does something like this, but a little more: not only does it
> > keep track of the link between old-top and new-top, it also keeps track of
> > the links between old-patch-in-between and new-patch-in-between.
> > (This makes sense when the top is being removed or reordered)
> 
> I was thinking of this as being the top and bottom commits for a single
> tracked patch, not as a whole series. I think patches lower wouldn't be
> affected, and patches higher would see this as a rebase.
> 
ah, ok, I misunderstood that part

Best regards, Jan

PS. sorry if my responses are sometimes a bit late, I'm trying to find more time to spend on this list ;-)

Daniel Barkalow· Aug 25, 2005, 19:41 UTC · re: Jan Veldeman · lore

Re: [RFC] Stgit - patch history / add extra parents

On Thu, 25 Aug 2005, Jan Veldeman wrote:
Show 9 quoted lines
> Daniel Barkalow wrote:
>
> > I'm not sure how applicable to this situation stgit really is; I see stgit
> > as optimized for the case of a patch set which is basically done, where
> > you want to keep it applicable to the mainline as the mainline advances.
>
> Maybe I forgot to mention this: I would also like to have my development
> tree split up in a patch stack. The separate patches makes tracking the
> mainline a lot easier (conflicts are a lot easier to solve)

I just try to keep things in this state sufficiently briefly that it doesn't become a problem. I also split things up into a bunch of branches, rather than into a stack of patches, and only work on parallel development before I've actually got a candidate for a series.

> But this would assume that once the patch goes into stgit, it won't
> change except when the parent gets updated. I think we will still change
> the patches quite a bit and simultanious by a couple of people.

The extension I had proposed to stgit should work for this; it would let you version control each patch just like other git projects. I just think it wouldn't work so well before the group has agreed on what patches there are.

	-Daniel
*This .sig left intentionally blank*
Catalin Marinas· Aug 31, 2005, 09:27 UTC · re: Daniel Barkalow · lore

Re: [RFC] Stgit - patch history / add extra parents

Daniel Barkalow <barkalow@iabervon.org> wrote:
Show 21 quoted lines
> On Tue, 23 Aug 2005, Jan Veldeman wrote:
>> The parents which should be visible to the outside, will always be versions
>> of my development tree, which I have previously pushed out. My way of
>> working would become:
>> * make changes, all over the place, using stgit
>> * still make changes (none of these gets tracked, intermittent versions are
>>   lost)
>> * having a good day: changes looks good, I want to push this out:
>>   * push my tree out
>>   * stgit-free (which makes the pushed out commits, the new parents of my
>>     stgit patches)
>> * restart from top
>
> I'm not sure how applicable to this situation stgit really is; I see stgit
> as optimized for the case of a patch set which is basically done, where
> you want to keep it applicable to the mainline as the mainline advances.
>
> For your application, I'd just have a git branch full of various stuff,
> and then generate clean commits by branching mainline, diffing development
> against it, cutting the diff down to just what I want to push, and
> applying that. Then the clean patch goes into stgit.

StGIT has the 'refresh' command which allows a patch to be indefinitely modified (that's pretty much how I use StGIT). I use it for the case where I would like to keep the changes in separate logical entities (patches) but they are not independent enough to create separate branches.

Take for example a new platform port, you can have some of the existing code re-worked in a patch, some CPU support in the next patch, basic platform support in another patch, some device drivers specific to this platform in yet another patch. Subsequent patches are dependent on the previous ones. Branching from the mainline for each of these features might not make sense and keeping all of them in the same branch can make maintenance a bit more difficult.

-- 
Catalin
Catalin Marinas· Aug 31, 2005, 09:12 UTC · re: Jan Veldeman · lore

Re: [RFC] Stgit - patch history / add extra parents

Jan Veldeman <jan.veldeman@gmail.com> wrote:
Show 10 quoted lines
> The parents which should be visible to the outside, will always be versions
> of my development tree, which I have previously pushed out. My way of
> working would become:
> * make changes, all over the place, using stgit
> * still make changes (none of these gets tracked, intermittent versions are
>   lost)
> * having a good day: changes looks good, I want to push this out:
>   * push my tree out
>   * stgit-free (which makes the pushed out commits, the new parents of my
>     stgit patches)

When the tree status looks good, you could tag it and the whole stack (the commits corresponding to the patches) would be available via this tag.

As I said, I agree with the idea of freezing a patch and even having multiple parents to a commit but this commit should not be visible via HEAD. We could have another id for a patch, 'frozen', which stores the last frozen state with the parents being the previous frozen states. Its SHA1 value could be accessed via 'stg id <patch>/frozen' (in the same way as /top and /bottom ids).

-- 
Catalin
Catalin Marinas· Aug 30, 2005, 21:41 UTC · re: Daniel Barkalow · lore

Re: [RFC] Stgit - patch history / add extra parents

Back from holiday. Thanks to all who replied to this thread.
On Tue, 2005-08-23 at 14:05 -0400, Daniel Barkalow wrote:
Show 20 quoted lines
> Having a useful diff isn't really a requirement for a parent; the diff in
> the case of a merge is going to be the total of everything that happened
> elsewhere. The point is to be able to reach some commits between which
> there are interesting diffs.
> 
> This also depends on how exactly freeze is used; if you use it before
> commiting a modification to the patch without rebasing, you get:
> 
> old-top -> new-top
>       ^    ^
>        \  /
>       bottom
> 
> bottom to old-top is the old patch
> bottom to new-top is the new patch
> old-top to new-top is the change to the patch
> 
> Then you want to keep new-top as a parent for rebasings until one of these
> is frozen. These links are not interesting to look at, but preserve the
> path to the old-top:new-top change, which is interesting.

This was my initial StGIT implementation (up to version 0.3), only that there was no freeze command. Since I want an StGIT tree to be clean to the outside world, I wouldn't keep multiple parents for the visible top of a patch.

As I understand from Junio's and Linus' e-mails (on the 23rd of August), there might be problems with merging the HEAD of an StGIT-managed tree if the above method is accessible via HEAD.

Show 9 quoted lines
> Ignoring the links to the corresponding bottoms, the development therefore
> looks like:
> 
> local1 -> local2 -> merge -> local3 -> merge
> ^                   ^                  ^
> mainline---->-->--------->------>-->----->
> 
> And this is how development is normally supposed to look. The trick is to
> only include a minimal number of merges.

A merge occurs every time a patch is rebased. Anyway, having the bottoms in the graph (which is the main idea of StGIT) together with the old-top (or frozen state) parents make the graph pretty complicated.

-- 
Catalin
Daniel Barkalow· Aug 31, 2005, 17:17 UTC · re: Catalin Marinas · lore

Re: [RFC] Stgit - patch history / add extra parents

On Tue, 30 Aug 2005, Catalin Marinas wrote:
Show 32 quoted lines
> Back from holiday. Thanks to all who replied to this thread.
> 
> On Tue, 2005-08-23 at 14:05 -0400, Daniel Barkalow wrote:
> > Having a useful diff isn't really a requirement for a parent; the diff in
> > the case of a merge is going to be the total of everything that happened
> > elsewhere. The point is to be able to reach some commits between which
> > there are interesting diffs.
> > 
> > This also depends on how exactly freeze is used; if you use it before
> > commiting a modification to the patch without rebasing, you get:
> > 
> > old-top -> new-top
> >       ^    ^
> >        \  /
> >       bottom
> > 
> > bottom to old-top is the old patch
> > bottom to new-top is the new patch
> > old-top to new-top is the change to the patch
> > 
> > Then you want to keep new-top as a parent for rebasings until one of these
> > is frozen. These links are not interesting to look at, but preserve the
> > path to the old-top:new-top change, which is interesting.
> 
> This was my initial StGIT implementation (up to version 0.3), only that
> there was no freeze command. Since I want an StGIT tree to be clean to
> the outside world, I wouldn't keep multiple parents for the visible top
> of a patch.
> 
> As I understand from Junio's and Linus' e-mails (on the 23rd of August),
> there might be problems with merging the HEAD of an StGIT-managed tree
> if the above method is accessible via HEAD.

Right, you'd want a separate head which is what you ask people to merge; the rest is only visible to people who are working on preparing the patch. But you could keep both sets of stuff (sharing tree objects but not commits).

Show 13 quoted lines
> > Ignoring the links to the corresponding bottoms, the development therefore
> > looks like:
> > 
> > local1 -> local2 -> merge -> local3 -> merge
> > ^                   ^                  ^
> > mainline---->-->--------->------>-->----->
> > 
> > And this is how development is normally supposed to look. The trick is to
> > only include a minimal number of merges.
> 
> A merge occurs every time a patch is rebased. Anyway, having the bottoms
> in the graph (which is the main idea of StGIT) together with the old-top
> (or frozen state) parents make the graph pretty complicated.

It should be possible to drop merges such that there's only one between any pair of local changes. That is, if you rebase at the end of the line above, it would get as parents local3 and the new bottom, not the last merge and the new bottom.

The mainline changes only come in through the bottoms, so higher levels should look the same, but with the lower levels in the place of mainline.

	-Daniel
*This .sig left intentionally blank*
Junio C Hamano· Aug 24, 2005, 00:23 UTC · re: Catalin Marinas · lore

Re: [RFC] Stgit - patch history / add extra parents

Catalin Marinas <catalin.marinas@gmail.com> writes:
> What's the definition of a parent in GIT terms? What are the
> restriction for a commit object to be a parent? Can a parent be an
> arbitrarily chosen commit?

Yes it can. GIT does not care if the commit ancestry does not make sense in contents terms (i.e. you can record one tree object in a commit object, and record another, completely unrelated tree object in a commit object that has the first commit object as its parent). The "git-diff-tree" output from comparing those two commits may not make _any_ sense at all to the human, though, but that is not a problem for GIT to do its work.

Show 18 quoted lines
> That's what I've been thinking. StGIT currently only implements the
> external view.
>
> An StGIT patch is a represented by a top and bottom commit
> objects. The bottom one is the same as the parent of the top
> commit. The patch is the diff between the top's tree id and the
> bottom's tree id.
>
> Jan's proposal is to allow a freeze command to save the current top
> hash and later be used as a second parent for the newly generated
> top. The problem I see with this approach is that (even for the
> internal view you described) the newly generated top will have two
> parents, new-bottom and old-top, but only the diff between new-top and
> new-bottom is meaningful. The diff between new-top and old-top (as a
> parent-child relation) wouldn't contain anything relevant to the patch
> but all the new changes to the base of the stack.
>
> Is the above an acceptable usage of the GIT DAG structure?

Surely, as long as it stays internal as StGIT patch stacks. Even when it is exported (i.e. you manage to have other non StGIT managed tree to pull and merge with such a commit), the development history _might_ look funny and complicated, but it should not be a problem for GIT to handle. After all, you are doing complicated thing in the StGIT patch stacks when you pop, push and freeze number of times, so your history is complicated. If you want to record that history at least for StGIT internal bookkeeping purposes, GIT should not prevent you from doing it.

Having said that, I would 100% agree with you that having a cleansed external view would be a good idea. It makes merging with StGIT managed repositories much more acceptable to ordinary GIT managed history.

Linus Torvalds· Aug 24, 2005, 02:26 UTC · re: Junio C Hamano · lore

Re: [RFC] Stgit - patch history / add extra parents

On Tue, 23 Aug 2005, Junio C Hamano wrote:
Show 9 quoted lines
> 
> Yes it can.  GIT does not care if the commit ancestry does not
> make sense in contents terms (i.e. you can record one tree
> object in a commit object, and record another, completely
> unrelated tree object in a commit object that has the first
> commit object as its parent).  The "git-diff-tree" output from
> comparing those two commits may not make _any_ sense at all to
> the human, though, but that is not a problem for GIT to do its
> work.
One issue is later merges.

If you have incorrect parent information, trying to do a merge may prove impossible and/or ignore changes from the "already merged" stream. By marking another head as a parent, you basically say "I have merged _everything_ leading up to that other head", and if you have only actually done a partial merge (and gotten just a part of the changes, ignoring the others), you'll have to notice that yourself, and forward-port the rest by hand.

For stgit, this probably doesn't matter.
		Linus
Junio C Hamano· Aug 23, 2005, 01:06 UTC · re: Jan Veldeman · lore

Re: [RFC] Stgit - patch history / add extra parents

Jan Veldeman <jan.veldeman@gmail.com> writes:
> ... So as you said: we indeed need a git designer to clear this up
> for us.

I understand I am counted as one of the "git designers"; I have been aware of this thread but I have to admit that I am way behind and have not caught up with the issues. I'll try to comment on them when I feel I know enough about the issues involved, but not right now. Sorry.

Please do not start explain them to me yet, I am behind due to lack of time and you would end up wasting your time.

← back to recent threads