threads / discuss / 26980

[GSoC 2011] Git Sequencer

Subject: [GSoC 2011] Git Sequencer

## tl;dr

20 messages between Apr 3, 2011 and Apr 6, 2011.

replies: 19people: 7as markdown or json

Ramkumar Ramachandra· Apr 3, 2011, 17:20 UTC · lore
Hi,

I'd like to re-apply this year as a student because I really want to see (among other things), a sequencer in git.git. Also, since I worked on areas related to fast-import and remote helpers last year, I thought I should work on something completely orthogonal this year.

I now have a draft of my proposal ready, and I'd really appreciate feedback. Also, could someone mentor me?

======================================================================
Project Proposal: Git Sequencer
Student: Ramkumar Ramachandra
Mentor: ?
== The Objective ==

To write git-sequencer, a new builtin command, and implement existing commands on top of that. This should give the commands more functionality, improve their error handling, and make them faster. The project can only be considered successful if all (or most) of the code written gets merged into upstream.

The Git Sequencer was a 2008 GSoC project as well; unfortunately most of the code did not get merged into git.git. The learning from all that work should serve as a huge headstart this year.

=== The Plan ===
1. Extend 'cherry-pick' with '--continue', '--abort', and '--skip'
features, so that it works like (a subset of) the current
'git-rebase--interactive.sh'.  This will require patching
'builtin/revert.c' in place, and merging it immediately.  I plan to
roughly follow the road laid out by Christian's 2010 series [1].

1.1. Factor out all calls to 'die' with 'return error' so so that we can pause the entire process when a commit doesn't apply automatically.

1.2. Create and populate TODO and DONE files, similar to the one that 'git-rebase--interactive.sh' creates. For now, it should simply give us information about why a 'cherry-pick' failed. Use these files with 'git-rebase--interactive.sh' to resume.

1.3. Port selective tests from the current 't3404' to make sure that TODO and DONE are populated correctly; "stop on conflicting pick" is a good candidate.

1.4. Decouple the 'revert' functionality from the 'cherry-pick' functionality in 'revert.c'. Implement '--abort' for 'cherry-pick' and port "abort" test from 't3404'.

1.5. Implement parsing the TODO and DONE files into suitable data structures. Derive inspiration from the code written in 2008 to do this.

1.6. Implement '--continue' and '--skip', and write suitable tests. Merge into upstream.

2. Factor out the 'cherry-pick' code from 'revert.c' into a new
'builtin/sequencer.c', and expose a simple cherry-picking API in
'sequencer.h'.
3. Implement a fresh 'cherry-pick.c' as a simple API call to the
sequencer, and make sure that all the existing tests pass.  After
this, cherry-pick will not be a builtin command anymore*.  Merge into
upstream.
4. Extend the sequncer to parse commands like 'execute', 'reword',
'squash', and 'fixup' that are specific to interactive rebasing.
Carefully implement the functionality for each of these keywords in a
step-wise manner, making sure that it inter-operates with 'rebase -i'
seamlessly.
5. Port the entire testsuite from 'rebase -i', and rewrite
'git-rebase.sh', 'git-rebase--interactive.sh' to call the sequncer.
The script should essentially build to a list of TODO instructions and
pass it to the 'git-sequencer' binary.  Merge into upstream.
6. [Optional] Lib'ify the sequncer: modify the sequencer API to
include rebase-related functionality.  Write 'rebase.c' as a bunch of
API calls to the sequencer.  Make sure that the existing tests pass.
Merge into upstream.
7. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the
sequncer: 'am.c'.  Bulk of this should be mbox parsing code.  Make sure
that all existing tests pass.  Merge into upstream.
* Is this a good change? Are there any forseeable issues?
** [Optional] should be read as "If time permits"
== Timeline ===

Deriving from last year's experience, I've decided not to present a tight timeline. Instead, I simply have an outline: Get the extended cherry-pick functionality merged before mid-term evaluations, and the rest before the final evaluations.

== Who am I? ==

I'm Ramkumar Ramachandra, and I first started contributing to git.git in January 2010. Apart from doing fast-import and remote helper related work last year, I also authored and merged svnrdump into Subversion trunk in the same period.

== Notes ==

[1]: http://thread.gmane.org/gmane.comp.version-control.git/162183 ======================================================================

Thanks for reading.
-- Ram
Sverre Rabbelier· Apr 3, 2011, 17:24 UTC · re: Ramkumar Ramachandra · lore

Re: [GSoC 2011] Git Sequencer

Heya,
On Sun, Apr 3, 2011 at 19:20, Ramkumar Ramachandra <artagnon@gmail.com> wrote:
Show 7 quoted lines
> I'd like to re-apply this year as a student because I really want to
> see (among other things), a sequencer in git.git.  Also, since I
> worked on areas related to fast-import and remote helpers last year, I
> thought I should work on something completely orthogonal this year.
>
> I now have a draft of my proposal ready, and I'd really appreciate
> feedback.  Also, could someone mentor me?

While I'm very interested in this project, I have no relevant experience. I'll definitely +1 your proposal though :).

-- 
Cheers,

Sverre Rabbelier
Stephan Beyer· Apr 3, 2011, 19:07 UTC · re: Ramkumar Ramachandra · lore

Re: [GSoC 2011] Git Sequencer

Hi Ram,

first, some notes on my git-sequencer 2008 branches that can be found at http://repo.or.cz/w/git/sbeyer.git ... (Not sure if I remember everything correctly)

I've settled to develop within the "seq-builtin-dev" branch and I sometimes merged Junio's "master" into that branch to catch up. The "seq-builtin-dev" branch is the important one.

Using git rebase -i (using git-sequencer) I sometimes remanaged the branch to "seq-builtin-rfc" that should represent a snapshot of a potential patch queue. My last rebase processes of the seq-builtin-rfc branch were pretty unmotivated and hence messy.

I have not touched the repo very often after GSOC'08 and I stopped touching it (as I stopped following recent Git development) "20 months ago" apparently. Quite many things may have changed since then.

The file A_SEQUENCER_TODO_FILE (added 2009-08-03) in the repo describes the missing and buggy pieces to fix so that _I_ (only me) would have been 100 per cent satisfied with that git-sequencer. http://repo.or.cz/w/git/sbeyer.git/blob/9e4b4d92f681a47e3d7ad2152d2391b2ab125a0c:/A_SEQUENCER_TODO_FILE [Some notes are also "strategy notes" to get things accepted, like the changes on "rebase -i -p" which are "not loved by everyone". ;-)]

On 2011-04-03, 22:50 +0530, Ramkumar Ramachandra wrote: 
> * Is this a good change? Are there any forseeable issues?

I want to mention an issue that I have not foreseen before: merges. (You need merges, for example, when doing rebase -i -p ... -p as in --preserve-merges.)

When I began, there was code in the "next" branch that added the TODO instructions "mark", "reset" and "merge" to do merges properly and I based my work on it. The original pieces by Jörg Sommer can still be found here: http://repo.or.cz/w/git/sbeyer.git/shortlog/6fabd85e8a777c26f3ae8ce11cb7f4265502ea7f

However, there have been strong opinions that the "mark"/"reset"/"merge" instructions are ugly and unpleasant to users and not even necessary (at least for rebase--interactive... and for sequencer, maybe, maybe not). Hence, the code in "next" has been rejected later.

During GSOC 2008 I regrettably underestimated the importance to communicate with the Git folks about these things. That's one of the main reasons the sequencer pieces did not get into master. And after GSOC'08 I had too little time for this... :-/

Well, the merging thing is the only *real* issue I remember.
Good luck and regards,
  Stephan
Ramkumar Ramachandra· Apr 3, 2011, 20:00 UTC · re: Stephan Beyer · lore

Re: [GSoC 2011] Git Sequencer

Hi Stephen,
Stephan Beyer writes:
Show 7 quoted lines
> first, some notes on my git-sequencer 2008 branches that can be found at
> http://repo.or.cz/w/git/sbeyer.git ... (Not sure if I remember
> everything correctly)
> 
> I've settled to develop within the "seq-builtin-dev" branch and I
> sometimes merged Junio's "master" into that branch to catch up.
> The "seq-builtin-dev" branch is the important one.

Thanks! Jonathan told me about it earlier, and I've already started ripping out code from the seq-builtin-dev branch :) I found your 't3350-sequencer.sh' especially interesting.

Show 8 quoted lines
> Using git rebase -i (using git-sequencer) I sometimes remanaged the
> branch to "seq-builtin-rfc" that should represent a snapshot of a
> potential patch queue. My last rebase processes of the seq-builtin-rfc
> branch were pretty unmotivated and hence messy.
> 
> I have not touched the repo very often after GSOC'08 and I stopped
> touching it (as I stopped following recent Git development) "20 months
> ago" apparently. Quite many things may have changed since then.

Okay, got it. I saw a few patches in 'master' that were based on your work though. Some of the patches in Christian's series also refer to your work.

Show 6 quoted lines
> The file A_SEQUENCER_TODO_FILE (added 2009-08-03) in the repo describes
> the missing and buggy pieces to fix so that _I_ (only me) would have
> been 100 per cent satisfied with that git-sequencer.
> http://repo.or.cz/w/git/sbeyer.git/blob/9e4b4d92f681a47e3d7ad2152d2391b2ab125a0c:/A_SEQUENCER_TODO_FILE
> [Some notes are also "strategy notes" to get things accepted, like the
> changes on "rebase -i -p" which are "not loved by everyone". ;-)]
Okay.
Show 6 quoted lines
> On 2011-04-03, 22:50 +0530, Ramkumar Ramachandra wrote: 
> > * Is this a good change? Are there any forseeable issues?
> 
> I want to mention an issue that I have not foreseen before: merges.
> (You need merges, for example, when doing rebase -i -p ... -p as in
> --preserve-merges.)
Ah, that's not something I thought about immediately.
Show 10 quoted lines
> When I began, there was code in the "next" branch that added the TODO
> instructions "mark", "reset" and "merge" to do merges properly and I
> based my work on it. The original pieces by Jörg Sommer can still be
> found here:
> http://repo.or.cz/w/git/sbeyer.git/shortlog/6fabd85e8a777c26f3ae8ce11cb7f4265502ea7f
> 
> However, there have been strong opinions that the "mark"/"reset"/"merge"
> instructions are ugly and unpleasant to users and not even necessary (at
> least for rebase--interactive... and for sequencer, maybe, maybe not). 
> Hence, the code in "next" has been rejected later.
Interesting historical note.
Show 6 quoted lines
> During GSOC 2008 I regrettably underestimated the importance to
> communicate with the Git folks about these things. That's one of the
> main reasons the sequencer pieces did not get into master. And after
> GSOC'08 I had too little time for this... :-/
> 
> Well, the merging thing is the only *real* issue I remember.

Point noted. Yes, I noticed that your sequencer was mostly functionally complete. I'll make sure that I spend a lot of interacting with the community.

Thank you for your elaborate note! I really appreciate it :) Hopefully, we will have that sequencer by next year.

-- Ram
Jonathan Nieder· Apr 3, 2011, 20:08 UTC · re: Stephan Beyer · lore

Re: [GSoC 2011] Git Sequencer

Stephan Beyer wrote:
Show 9 quoted lines
> I want to mention an issue that I have not foreseen before: merges.
> (You need merges, for example, when doing rebase -i -p ... -p as in
> --preserve-merges.)
>
> When I began, there was code in the "next" branch that added the TODO
> instructions "mark", "reset" and "merge" to do merges properly and I
> based my work on it. The original pieces by Jörg Sommer can still be
> found here:
> http://repo.or.cz/w/git/sbeyer.git/shortlog/6fabd85e8a777c26f3ae8ce11cb7f4265502ea7f
[etc]

Some more pointers: http://thread.gmane.org/gmane.comp.version-control.git/148059 IIRC there's some rough consensus about the design, even if I'm not sure what it is :).

Of course a dream would be a way to rebase merge conflict resolutions instead of being limited to replaying conflict-free merges or resolving conflicts by hand. But that's a more complex story.

Daniel Barkalow· Apr 3, 2011, 19:49 UTC · re: Ramkumar Ramachandra · lore

Re: [GSoC 2011] Git Sequencer

On Sun, 3 Apr 2011, Ramkumar Ramachandra wrote:
Show 26 quoted lines
> Hi,
> 
> I'd like to re-apply this year as a student because I really want to
> see (among other things), a sequencer in git.git.  Also, since I
> worked on areas related to fast-import and remote helpers last year, I
> thought I should work on something completely orthogonal this year.
> 
> I now have a draft of my proposal ready, and I'd really appreciate
> feedback.  Also, could someone mentor me?
> 
> ======================================================================
> Project Proposal: Git Sequencer
> Student: Ramkumar Ramachandra
> Mentor: ?
> 
> == The Objective ==
> 
> To write git-sequencer, a new builtin command, and implement existing
> commands on top of that.  This should give the commands more
> functionality, improve their error handling, and make them faster.
> The project can only be considered successful if all (or most) of the
> code written gets merged into upstream.
> 
> The Git Sequencer was a 2008 GSoC project as well; unfortunately most
> of the code did not get merged into git.git.  The learning from all
> that work should serve as a huge headstart this year.

One of the things that is hard about sequencer is that it is ultimately a complete replacement for several differently-implemented programs in different languages, with different temporary file formats and differrent supported operations. As such, you could probably spend an entire summer just getting it reviewed, revised, and accepted, starting with a working implementation.

So I think your proposal is good in how [1/5] includes getting something useful merged. My suspicion is that the outcome will be something like that you implemented all 7 tasks and got 4 of them merged, assuming that you really push getting things merged as soon as they're ready, without spending too much time porting other things to use the core and getting the ports reviewed before the core is accepted.

I actually think that it would be a worthwhile feature for git's library code to have a uniform mechanism for communicating that it is requesting human intervention in the middle of a particular operation, where library operations which conflict with being able to continue this operation are either blocked or abort the operation, and the library is able to be told in general that the human intervention is done and the library operation should be finished now (or produce complaints about the user's work). That is, a library-level, single-interrupted-step "sequencer". For that matter, it should also apply to the common '"git merge" gets a conflict' case, and it would be useful to get some representational uniformity between that and cherry-pick getting a conflict.

I think replacing existing multi-step processes is going to be a lot more contentious and involve user-visible changes which involve matters of taste and such. But I think you can make a valuable contribution in how a single current step is handled before getting tangled in that, and be much more likely to get a useful outcome than if you try to tackle the whole problem.

	-Daniel
*This .sig left intentionally blank*
Ramkumar Ramachandra· Apr 4, 2011, 04:06 UTC · re: Daniel Barkalow · lore

Re: [GSoC 2011] Git Sequencer

Hi Daniel,
Daniel Barkalow writes:
Show 17 quoted lines
> On Sun, 3 Apr 2011, Ramkumar Ramachandra wrote:
> > To write git-sequencer, a new builtin command, and implement existing
> > commands on top of that.  This should give the commands more
> > functionality, improve their error handling, and make them faster.
> > The project can only be considered successful if all (or most) of the
> > code written gets merged into upstream.
> > 
> > The Git Sequencer was a 2008 GSoC project as well; unfortunately most
> > of the code did not get merged into git.git.  The learning from all
> > that work should serve as a huge headstart this year.
> 
> One of the things that is hard about sequencer is that it is ultimately a 
> complete replacement for several differently-implemented programs in 
> different languages, with different temporary file formats and differrent 
> supported operations. As such, you could probably spend an entire summer 
> just getting it reviewed, revised, and accepted, starting with a working 
> implementation.

Agreed. I've chosen to use the same commands and temporary files as 'git-rebase--interactive.sh', because I think those commands are sufficient to implement everything else.

Show 6 quoted lines
> So I think your proposal is good in how [1/5] includes getting something 
> useful merged. My suspicion is that the outcome will be something like 
> that you implemented all 7 tasks and got 4 of them merged, assuming that 
> you really push getting things merged as soon as they're ready, without 
> spending too much time porting other things to use the core and getting 
> the ports reviewed before the core is accepted.

Hm. In that case, we'll just have a sequencer that can cherry-pick -- I personally wouldn't be too happy with this outcome either.

Show 11 quoted lines
> I actually think that it would be a worthwhile feature for git's library 
> code to have a uniform mechanism for communicating that it is requesting 
> human intervention in the middle of a particular operation, where library 
> operations which conflict with being able to continue this operation are 
> either blocked or abort the operation, and the library is able to be told 
> in general that the human intervention is done and the library operation 
> should be finished now (or produce complaints about the user's work). That 
> is, a library-level, single-interrupted-step "sequencer". For that matter, 
> it should also apply to the common '"git merge" gets a conflict' case, and 
> it would be useful to get some representational uniformity between that 
> and cherry-pick getting a conflict.

Until 4/7, I'd only planned to make the 'git-sequencer' binary like the 'git-rebase--interactive.sh' script, except that it would accept a TODO file on stdin, instead of interactively opening up an editor.

Your idea is a slightly more ambitious version of what I'd planned for 6/7, especially since 'merge' contains a lot of cruft like MERGE_HEAD and CHERRY_PICK_HEAD. I can shift my focus after 4/7 though -- here's what I have in mind. Do you have something similar in mind?

enum commit_todo_action {
     ACTION_PICK;
     ACTION_REWORD;
     ACTION_EDIT;
     ACTION_SQUASH;
     ACTION_FIXUP;
     ACTION_EXEC;
};
struct commit_todo_list {
       struct commit *item;
       enum commit_todo_action action;
       struct commit_todo_list *next;
};

int sequencer_cherry_pick(struct commit *base, struct commit_list *list); int sequncer_rebase(struct commit *base, struct commit_todo_list *list); int sequencer_handle_conflict(); /* Returns ABORT (1) or RESOLVED (0) */

/**
 * The sequencer_handle_conflict function essentially starts with a
 * working tree with unmerged files and results in either a working
 * tree without unmerged files (in which case it returns 0), or simply
 * returns 1.  Advantage: Consistency. Each individual script will not
 * have to maintain its own temporary files.
 */
Show 6 quoted lines
> I think replacing existing multi-step processes is going to be a lot more 
> contentious and involve user-visible changes which involve matters of 
> taste and such. But I think you can make a valuable contribution in how a 
> single current step is handled before getting tangled in that, and be much 
> more likely to get a useful outcome than if you try to tackle the whole 
> problem.

Okay. I'll replace 5/7 - 7/7 in my proposal with an alternative as soon as I sketch out the details more clearly.

Thanks for your suggestions!
-- Ram
Ramkumar Ramachandra· Apr 4, 2011, 04:54 UTC · re: Ramkumar Ramachandra · lore

Re: [GSoC 2011] Git Sequencer

Hi Daniel,
Ramkumar Ramachandra writes:
Show 12 quoted lines
> Daniel Barkalow writes:
> > I actually think that it would be a worthwhile feature for git's library 
> > code to have a uniform mechanism for communicating that it is requesting 
> > human intervention in the middle of a particular operation, where library 
> > operations which conflict with being able to continue this operation are 
> > either blocked or abort the operation, and the library is able to be told 
> > in general that the human intervention is done and the library operation 
> > should be finished now (or produce complaints about the user's work). That 
> > is, a library-level, single-interrupted-step "sequencer". For that matter, 
> > it should also apply to the common '"git merge" gets a conflict' case, and 
> > it would be useful to get some representational uniformity between that 
> > and cherry-pick getting a conflict.
[...]
Show 9 quoted lines
> int sequencer_handle_conflict(); /* Returns ABORT (1) or RESOLVED (0) */
> 
> /**
>  * The sequencer_handle_conflict function essentially starts with a
>  * working tree with unmerged files and results in either a working
>  * tree without unmerged files (in which case it returns 0), or simply
>  * returns 1.  Advantage: Consistency. Each individual script will not
>  * have to maintain its own temporary files.
>  */

Uh, no. I wrote this part in too quickly. Clearly needs more thought.

-- Ram
Daniel Barkalow· Apr 4, 2011, 18:59 UTC · re: Ramkumar Ramachandra · lore

Re: [GSoC 2011] Git Sequencer

On Mon, 4 Apr 2011, Ramkumar Ramachandra wrote:
Show 30 quoted lines
> Hi Daniel,
> 
> Ramkumar Ramachandra writes:
> > Daniel Barkalow writes:
> > > I actually think that it would be a worthwhile feature for git's library 
> > > code to have a uniform mechanism for communicating that it is requesting 
> > > human intervention in the middle of a particular operation, where library 
> > > operations which conflict with being able to continue this operation are 
> > > either blocked or abort the operation, and the library is able to be told 
> > > in general that the human intervention is done and the library operation 
> > > should be finished now (or produce complaints about the user's work). That 
> > > is, a library-level, single-interrupted-step "sequencer". For that matter, 
> > > it should also apply to the common '"git merge" gets a conflict' case, and 
> > > it would be useful to get some representational uniformity between that 
> > > and cherry-pick getting a conflict.
> 
> [...]
> 
> > int sequencer_handle_conflict(); /* Returns ABORT (1) or RESOLVED (0) */
> > 
> > /**
> >  * The sequencer_handle_conflict function essentially starts with a
> >  * working tree with unmerged files and results in either a working
> >  * tree without unmerged files (in which case it returns 0), or simply
> >  * returns 1.  Advantage: Consistency. Each individual script will not
> >  * have to maintain its own temporary files.
> >  */
> 
> Uh, no.  I wrote this part in too quickly.  Clearly needs more
> thought.
Here's how I'm thinking about a single step:
The non-conflict case is:
  $ git cherry-pick ...
  figure out what we're asked to do
  make the change to the working directory and index
  make the commit with info from the commit we're cherry-picking
The conflict case should be:
  $ git cherry-pick ...
  figure out what we're asked to do
  make the change to the working directory and index
  discover problem, set up for human assistance, tweak info to say that we 
    needed help
  $ fix stuff
  $ git continue
  check that everything is how it should be
  make the commit with info from the commit we're cherry-picking

That is, the code that cherry-picks one commit can quit in the middle and resume after the user finishes helping, and the main entry point to git can resume that operation.

So, my thought was that you'd have something like:
cherry_pick_conflict = { 
  "cherry-pick", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,
  cherry_pick_verify_resolution,
  cherry_pick_abort,
  cherry_pick_post_resolution
};
int cherry_pick(struct commit *item)
{
  save info on the commit, flags, etc needed to understand what we're doing
  try to apply diff...
  if (!rejected)
    return cherry_pick_post_resolution();
  try to use merge recursive...
  if (!conflict)
    return cherry_pick_post_resolution();
  return report_conflict(cherry_pick_conflict);
}
int cherry_pick_verify_resolution(void)
{
  if (still unmerged files) {
    describe work still needed
    return 1;
  }
  return 0;
}
int cherry_pick_abort(void)
{
  restore info on what we'd started and delete state
  clean up the working directory, index, etc
  return 0;
}
int cherry_pick_post_resolution(void)
{
  restore info on what we'd started and delete state
  make the commit
  return 0;
}
The important things are:
 - arbitrary code can determine that you're in the middle of resolving 
   some conflict, that the resolution of that conflict is about doing
   something to your current branch, and how to abort what you're doing,
   and how to finish it
 - the same code gets run after the conflict has been resolved that would 
   have been run immediately if the merge went smoothly
 - cherry-pick can save whatever it needs to in its state file; that's 
   its business, and the semantics here don't have to interact with other 
   commands, because report_conflict() has taken care of interaction with 
   other commands
The next step is to be able to have:
int sequencer()
{
  save the list of steps, with no in-progress step
  return sequencer_post_resolution();
}
int sequencer_post_resolution()
{
  finish any in-progress step
  get the next step
  if (no steps left)
    return 0;
  attempt step
  if (!got_conflict)
    return sequencer_post_resolution();
  return report_conflict(sequencer_conflict);
}

Where the sequencer-level conflict nests around the cherry-pick-level conflict, and the generic "continue" completes things from the inside out.

I think, ultimately, that with this code structure in place, the am/rebase/rebase--interactive/sequencer details of how the multi-step process is recorded becomes less important. That way, your project can be successful even if you can't find a syntax for the sequencer file that meets the needs of all of these use cases. (Which is where I suspect you'll get bogged down.) If you can get all of the cases where git exits in order to get human intervention to share "everything that matters" and the core to "know what's in progress as far as anything else cares", I think that would be success, even if the various multi-step programs continue using their own state files.

	-Daniel
*This .sig left intentionally blank*
Ramkumar Ramachandra· Apr 5, 2011, 17:50 UTC · re: Daniel Barkalow · lore

Re: [GSoC 2011] Git Sequencer

Hi Daniel,
Daniel Barkalow writes:
Show 16 quoted lines
> On Mon, 4 Apr 2011, Ramkumar Ramachandra wrote:
> > Ramkumar Ramachandra writes:
> > > Daniel Barkalow writes:
> > > > I actually think that it would be a worthwhile feature for git's library 
> > > > code to have a uniform mechanism for communicating that it is requesting 
> > > > human intervention in the middle of a particular operation, where library 
> > > > operations which conflict with being able to continue this operation are 
> > > > either blocked or abort the operation, and the library is able to be told 
> > > > in general that the human intervention is done and the library operation 
> > > > should be finished now (or produce complaints about the user's work). That 
> > > > is, a library-level, single-interrupted-step "sequencer". For that matter, 
> > > > it should also apply to the common '"git merge" gets a conflict' case, and 
> > > > it would be useful to get some representational uniformity between that 
> > > > and cherry-pick getting a conflict.
> > 
> > [...]

Thanks for the detailed response- I've rearragned your response and added some comments. I initially wanted to design it so that all state is persisted by the sequencer, but I can clearly see what's wrong with that approach now.

Show 10 quoted lines
> I think, ultimately, that with this code structure in place, the 
> am/rebase/rebase--interactive/sequencer details of how the multi-step 
> process is recorded becomes less important. That way, your project can be 
> successful even if you can't find a syntax for the sequencer file that 
> meets the needs of all of these use cases. (Which is where I suspect 
> you'll get bogged down.) If you can get all of the cases where git exits 
> in order to get human intervention to share "everything that matters" and 
> the core to "know what's in progress as far as anything else cares", I 
> think that would be success, even if the various multi-step programs 
> continue using their own state files.

Excellent. The crux of the idea: The sequencer should serve as the entry/ exit point for Git when any operation requires user intervention to proceed. For this, it should have information about how we got to this point, and how to proceed after the user intervention is complete; this information is contained in:

Show 6 quoted lines
> cherry_pick_conflict = { 
>   "cherry-pick", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,
>   cherry_pick_verify_resolution,
>   cherry_pick_abort,
>   cherry_pick_post_resolution
> };
Wait -- isn't it missing a skip callback?
cherry_pick_conflict = { 
  "cherry-pick", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,
  cherry_pick_verify_resolution,
  cherr_pick_skip,
  cherry_pick_abort,
  cherry_pick_post_resolution
};

This information is passed to report_conflict(), which takes care of user intervention. The user can do whatever she wants and then ask the sequencer to "continue", "skip" or "abort":

> Where the sequencer-level conflict nests around the cherry-pick-level 
> conflict, and the generic "continue" completes things from the inside out.

Right. And then the sequencer fires the appropriate callback and returns control to the parent command. More notes:

>  - cherry-pick can save whatever it needs to in its state file; that's 
>    its business, and the semantics here don't have to interact with other 
>    commands, because report_conflict() has taken care of interaction with 
>    other commands

At the end of a merge for example, the MERGE_MSG needs to be retrieved to create a new merge commit. The sequencer des not need to know anything about this, since this is specific to 'merge'.

>  - arbitrary code can determine that you're in the middle of resolving 
>    some conflict, that the resolution of that conflict is about doing
>    something to your current branch, and how to abort what you're doing,
>    and how to finish it

Any arbitrary code simply has to ask the sequencer about the state of the intermediate files that report_conflict() uses. They don't have to worry about command-specific intermediate files.

>  - the same code gets run after the conflict has been resolved that would 
>    have been run immediately if the merge went smoothly

Using these callbacks, there is no need for if-else ugliness inside the specific command to decide what to do next.

I suppose we can call this idea a "generic conflict handler". I like it very much, and I'll definitely include this as part of my GSoC work. Thanks for taking the time to explain it in such detail :)

-- Ram
Daniel Barkalow· Apr 5, 2011, 18:24 UTC · re: Ramkumar Ramachandra · lore

Re: [GSoC 2011] Git Sequencer

On Tue, 5 Apr 2011, Ramkumar Ramachandra wrote:
Show 39 quoted lines
> Hi Daniel,
> 
> Daniel Barkalow writes:
> > On Mon, 4 Apr 2011, Ramkumar Ramachandra wrote:
> > > Ramkumar Ramachandra writes:
> > > > Daniel Barkalow writes:
> > > > > I actually think that it would be a worthwhile feature for git's library 
> > > > > code to have a uniform mechanism for communicating that it is requesting 
> > > > > human intervention in the middle of a particular operation, where library 
> > > > > operations which conflict with being able to continue this operation are 
> > > > > either blocked or abort the operation, and the library is able to be told 
> > > > > in general that the human intervention is done and the library operation 
> > > > > should be finished now (or produce complaints about the user's work). That 
> > > > > is, a library-level, single-interrupted-step "sequencer". For that matter, 
> > > > > it should also apply to the common '"git merge" gets a conflict' case, and 
> > > > > it would be useful to get some representational uniformity between that 
> > > > > and cherry-pick getting a conflict.
> > > 
> > > [...]
> 
> Thanks for the detailed response- I've rearragned your response and
> added some comments.  I initially wanted to design it so that all
> state is persisted by the sequencer, but I can clearly see what's
> wrong with that approach now.
> 
> > I think, ultimately, that with this code structure in place, the 
> > am/rebase/rebase--interactive/sequencer details of how the multi-step 
> > process is recorded becomes less important. That way, your project can be 
> > successful even if you can't find a syntax for the sequencer file that 
> > meets the needs of all of these use cases. (Which is where I suspect 
> > you'll get bogged down.) If you can get all of the cases where git exits 
> > in order to get human intervention to share "everything that matters" and 
> > the core to "know what's in progress as far as anything else cares", I 
> > think that would be success, even if the various multi-step programs 
> > continue using their own state files.
> 
> Excellent.  The crux of the idea: The sequencer should serve as the
> entry/ exit point for Git when any operation requires user
> intervention to proceed.

I'm a bit surprised by the idea of calling that "the sequencer" (rather than having "the sequencer" be a command), but I actually think you're entirely right to do so. Be sure to be very explicit about that, though, because people will probably start with the wrong idea of what you're proposing otherwise.

Show 12 quoted lines
> For this, it should have information about
> how we got to this point, and how to proceed after the user
> intervention is complete; this information is contained in:
> 
> > cherry_pick_conflict = { 
> >   "cherry-pick", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,
> >   cherry_pick_verify_resolution,
> >   cherry_pick_abort,
> >   cherry_pick_post_resolution
> > };
> 
> Wait -- isn't it missing a skip callback?

I think "skip" is actually: abort the lowest-level conflict and continue the next-level conflict. If you're doing a rebase, and the rebase is doing a "pick", and the pick got a conflict, --skip means that you abort the pick (to get back to the state where the earlier commits have been picked but this one hasn't been started, followed by having the rebase continue with what it was going to do after the pick completed.

So I don't think you need a "skip" callback, as long as you've untangled the levels cleanly and get the nesting support right.

Show 11 quoted lines
> cherry_pick_conflict = { 
>   "cherry-pick", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,
>   cherry_pick_verify_resolution,
>   cherr_pick_skip,
>   cherry_pick_abort,
>   cherry_pick_post_resolution
> };
> 
> This information is passed to report_conflict(), which takes care of
> user intervention.  The user can do whatever she wants and then ask
> the sequencer to "continue", "skip" or "abort":
Right, although I think:
  $ git cheery-pick some-sha1
  Conflict needs to be fixed now!
  $ git skip

should give an error message about the current conflict not being a step of a larger process. That is, you can always "continue" or "abort", but you can only "skip" if there's something to skip to, even if it's only the higher-order sequence reporting that it's completed successfully.

Show 14 quoted lines
> > Where the sequencer-level conflict nests around the cherry-pick-level 
> > conflict, and the generic "continue" completes things from the inside out.
> 
> Right.  And then the sequencer fires the appropriate callback and
> returns control to the parent command.  More notes:
> 
> >  - cherry-pick can save whatever it needs to in its state file; that's 
> >    its business, and the semantics here don't have to interact with other 
> >    commands, because report_conflict() has taken care of interaction with 
> >    other commands
> 
> At the end of a merge for example, the MERGE_MSG needs to be retrieved
> to create a new merge commit.  The sequencer des not need to know
> anything about this, since this is specific to 'merge'.

Right. And remove_branch_state() wouldn't even need to know about MERGE_MSG like it does now, because that would be handled by aborting any in-progress merge.

Show 8 quoted lines
> >  - arbitrary code can determine that you're in the middle of resolving 
> >    some conflict, that the resolution of that conflict is about doing
> >    something to your current branch, and how to abort what you're doing,
> >    and how to finish it
> 
> Any arbitrary code simply has to ask the sequencer about the state of
> the intermediate files that report_conflict() uses.  They don't have
> to worry about command-specific intermediate files.
Right.
Show 9 quoted lines
> >  - the same code gets run after the conflict has been resolved that would 
> >    have been run immediately if the merge went smoothly
> 
> Using these callbacks, there is no need for if-else ugliness inside
> the specific command to decide what to do next.
> 
> I suppose we can call this  idea a "generic conflict handler".  I like
> it very  much, and  I'll definitely  include this as  part of  my GSoC
> work.  Thanks for taking the time to explain it in such detail :)
You're welcome. Thanks for proposing to actually implement it. :)
	-Daniel
*This .sig left intentionally blank*
Ramkumar Ramachandra· Apr 5, 2011, 18:59 UTC · re: Daniel Barkalow · lore

Re: [GSoC 2011] Git Sequencer

Hi Daniel,
Daniel Barkalow writes:
Show 10 quoted lines
> On Tue, 5 Apr 2011, Ramkumar Ramachandra wrote:
> > Excellent.  The crux of the idea: The sequencer should serve as the
> > entry/ exit point for Git when any operation requires user
> > intervention to proceed.
> 
> I'm a bit surprised by the idea of calling that "the sequencer" (rather 
> than having "the sequencer" be a command), but I actually think you're 
> entirely right to do so. Be sure to be very explicit about that, though, 
> because people will probably start with the wrong idea of what you're 
> proposing otherwise.

Ah. I'll make sure to word it unambiguously in the proposal, and link to this thread :)

Show 22 quoted lines
> > For this, it should have information about
> > how we got to this point, and how to proceed after the user
> > intervention is complete; this information is contained in:
> > 
> > > cherry_pick_conflict = { 
> > >   "cherry-pick", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,
> > >   cherry_pick_verify_resolution,
> > >   cherry_pick_abort,
> > >   cherry_pick_post_resolution
> > > };
> > 
> > Wait -- isn't it missing a skip callback?
> 
> I think "skip" is actually: abort the lowest-level conflict and continue 
> the next-level conflict. If you're doing a rebase, and the rebase is doing 
> a "pick", and the pick got a conflict, --skip means that you abort the 
> pick (to get back to the state where the earlier commits have been picked 
> but this one hasn't been started, followed by having the rebase continue 
> with what it was going to do after the pick completed.
> 
> So I don't think you need a "skip" callback, as long as you've untangled 
> the levels cleanly and get the nesting support right.

Okay. I'm not yet entirely clear about this yet, but I think it should be sorted out during implementation.

Show 23 quoted lines
> > cherry_pick_conflict = { 
> >   "cherry-pick", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,
> >   cherry_pick_verify_resolution,
> >   cherr_pick_skip,
> >   cherry_pick_abort,
> >   cherry_pick_post_resolution
> > };
> > 
> > This information is passed to report_conflict(), which takes care of
> > user intervention.  The user can do whatever she wants and then ask
> > the sequencer to "continue", "skip" or "abort":
> 
> Right, although I think:
> 
>   $ git cheery-pick some-sha1
>   Conflict needs to be fixed now!
> 
>   $ git skip
> 
> should give an error message about the current conflict not being a step 
> of a larger process. That is, you can always "continue" or "abort", but 
> you can only "skip" if there's something to skip to, even if it's only the 
> higher-order sequence reporting that it's completed successfully.
Right, got it.
-- Ram
Christian Couder· Apr 4, 2011, 04:43 UTC · re: Ramkumar Ramachandra · lore

Re: [GSoC 2011] Git Sequencer

On Sunday 03 April 2011 19:20:56 Ramkumar Ramachandra wrote:
Show 9 quoted lines
> Hi,
> 
> I'd like to re-apply this year as a student because I really want to
> see (among other things), a sequencer in git.git.  Also, since I
> worked on areas related to fast-import and remote helpers last year, I
> thought I should work on something completely orthogonal this year.
> 
> I now have a draft of my proposal ready, and I'd really appreciate
> feedback.  Also, could someone mentor me?

Yeah! I would be happy to mentor you (or co-mentor you if someone else want to be involved)!

Show 10 quoted lines
> ======================================================================
> Project Proposal: Git Sequencer
> Student: Ramkumar Ramachandra
> Mentor: ?
> 
> == The Objective ==
> 
> To write git-sequencer, a new builtin command, and implement existing
> commands on top of that.  This should give the commands more
> functionality, improve their error handling, and make them faster.

You should first talk about extending git cherry-pick with --continue, --abort and --skip, because it can be very valuable already if done properly with many tests and if it's merged of course.

> The project can only be considered successful if all (or most) of the
> code written gets merged into upstream.
Yeah, just say "most of the code". It is definitely good enough.
Show 11 quoted lines
> The Git Sequencer was a 2008 GSoC project as well; unfortunately most
> of the code did not get merged into git.git.  The learning from all
> that work should serve as a huge headstart this year.
> 
> === The Plan ===
> 
> 1. Extend 'cherry-pick' with '--continue', '--abort', and '--skip'
> features, so that it works like (a subset of) the current
> 'git-rebase--interactive.sh'.  This will require patching
> 'builtin/revert.c' in place, and merging it immediately.  I plan to
> roughly follow the road laid out by Christian's 2010 series [1].

Yeah, the first step should be 'cherry-pick' with '--continue', '--abort', and '--skip' merged.

Show 8 quoted lines
> 1.1. Factor out all calls to 'die' with 'return error' so so that we
> can pause the entire process when a commit doesn't apply
> automatically.
> 
> 1.2. Create and populate TODO and DONE files, similar to the one that
> 'git-rebase--interactive.sh' creates.  For now, it should simply give
> us information about why a 'cherry-pick' failed.  Use these files with
> 'git-rebase--interactive.sh' to resume.

I am not sure it's a good thing to use 'git-rebase--interactive.sh' to resume the cherry-pick. The parsing code already exists, is not very big, is in C and has been reviewed and tested, so I think it's better to use.

Show 7 quoted lines
> 1.3. Port selective tests from the current 't3404' to make sure that
> TODO and DONE are populated correctly; "stop on conflicting pick" is a
> good candidate.
> 
> 1.4. Decouple the 'revert' functionality from the 'cherry-pick'
> functionality in 'revert.c'.  Implement '--abort' for 'cherry-pick'
> and port "abort" test from 't3404'.

I am not sure decoupling revert and cherry-pick functionnalities is really needed, or I don't know what you mean exactly.

> 1.5. Implement parsing the TODO and DONE files into suitable data
> structures.  Derive inspiration from the code written in 2008 to do
> this.
Yeah, you can use this patch:
http://article.gmane.org/gmane.comp.version-control.git/162198
> 1.6. Implement '--continue' and '--skip', and write suitable tests.
> Merge into upstream.
Great!
> 2. Factor out the 'cherry-pick' code from 'revert.c' into a new
> 'builtin/sequencer.c', and expose a simple cherry-picking API in
> 'sequencer.h'.
Yeah!
> 3. Implement a fresh 'cherry-pick.c' as a simple API call to the
> sequencer, and make sure that all the existing tests pass.  After
> this, cherry-pick will not be a builtin command anymore*.  Merge into
> upstream.

Yeah, it is closely linked with the previous point. So maybe this can be 2.2 and the previous one can be 2.1. And this way we know that at the end of 1) and at the end of 2) everything should be merged upstream.

Show 5 quoted lines
> 4. Extend the sequncer to parse commands like 'execute', 'reword',
> 'squash', and 'fixup' that are specific to interactive rebasing.
> Carefully implement the functionality for each of these keywords in a
> step-wise manner, making sure that it inter-operates with 'rebase -i'
> seamlessly.
Great!
> 5. Port the entire testsuite from 'rebase -i', and rewrite
> 'git-rebase.sh', 'git-rebase--interactive.sh' to call the sequncer.
> The script should essentially build to a list of TODO instructions and
> pass it to the 'git-sequencer' binary.  Merge into upstream.

Great! But as it is closely llinked with the previous point, maybe these 2 points should be 3.1 and 3.2.

Show 10 quoted lines
> 6. [Optional] Lib'ify the sequncer: modify the sequencer API to
> include rebase-related functionality.  Write 'rebase.c' as a bunch of
> API calls to the sequencer.  Make sure that the existing tests pass.
> Merge into upstream.
> 
> 7. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the
> sequncer: 'am.c'.  Bulk of this should be mbox parsing code.  Make sure
> that all existing tests pass.  Merge into upstream.
> 
> * Is this a good change? 
I think it is a good proposal.
> Are there any forseeable issues?
I don't see anything that others didn't told you about.
Show 8 quoted lines
> ** [Optional] should be read as "If time permits"
> 
> == Timeline ===
> 
> Deriving from last year's experience, I've decided not to present a
> tight timeline.  Instead, I simply have an outline: Get the extended
> cherry-pick functionality merged before mid-term evaluations, and the
> rest before the final evaluations.
I agree, but still you could perhaps state something like this:
- before mid june:
	some patch series for everything in 1) should have been sent to the list
- before midterm evaluation:
	everything in 1) should be merged upstream
	some patch series for everything in 2) and 3) (or 2.1 and 2.2 if you use 
the numbering I suggest) should have been sent to the list
- before the end of July:
	everything in 2) and 3) should be merged upstream
	some patch series for everything in 4) and 5) (or 3.1 and 3.2 if you use 
the numbering I suggest) should have been sent to the list
- before final evaluation:
	everything should be merged

I think it is better to have more details like the above because this way we can realize early that there is not a lot of time after the midterm evaluation.

Show 13 quoted lines
> == Who am I? ==
> 
> I'm Ramkumar Ramachandra, and I first started contributing to git.git
> in January 2010.  Apart from doing fast-import and remote helper
> related work last year, I also authored and merged svnrdump into
> Subversion trunk in the same period.
> 
> == Notes ==
> 
> [1]: http://thread.gmane.org/gmane.comp.version-control.git/162183
> ======================================================================
> 
> Thanks for reading.

Thanks for applying, Christian.

Junio C Hamano· Apr 4, 2011, 05:20 UTC · re: Christian Couder · lore

Re: [GSoC 2011] Git Sequencer

Christian Couder <chriscool@tuxfamily.org> writes:
> Yeah, the first step should be 'cherry-pick' with '--continue', '--abort', and 
> '--skip' merged.

I haven't looked at rebase-i machinery recently, but I wonder if it would just be a matter of making a multi-commit cherry-pick just prepare a bunch of "pick XXX" lines into .git/rebase-merge/rebase-todo file, make other trivial setups (like detaching HEAD, writing head-name and head files) and then execing "git rebase --continue"?

Christian Couder· Apr 5, 2011, 06:23 UTC · re: Junio C Hamano · lore

Re: [GSoC 2011] Git Sequencer

On Monday 04 April 2011 07:20:18 Junio C Hamano wrote:
Show 9 quoted lines
> Christian Couder <chriscool@tuxfamily.org> writes:
> > Yeah, the first step should be 'cherry-pick' with '--continue',
> > '--abort', and '--skip' merged.
> 
> I haven't looked at rebase-i machinery recently, but I wonder if it would
> just be a matter of making a multi-commit cherry-pick just prepare a bunch
> of "pick XXX" lines into .git/rebase-merge/rebase-todo file, make other
> trivial setups (like detaching HEAD, writing head-name and head files) and
> then execing "git rebase --continue"?

It is probably quite easy to do that, but it would result in cherry-pick in C calling rebase-i in shell that itself calls cherry-pick in C (to pick individual commit). Instead with this GSoC I think we have the opportunity to have everything we needed in C.

Best regards, Christian.

Ramkumar Ramachandra· Apr 5, 2011, 06:46 UTC · re: Christian Couder · lore

Re: [GSoC 2011] Git Sequencer

Hi Christian and Junio,

On Tue, Apr 5, 2011 at 11:53 AM, Christian Couder <chriscool@tuxfamily.org> wrote:

Show 15 quoted lines
> On Monday 04 April 2011 07:20:18 Junio C Hamano wrote:
>> Christian Couder <chriscool@tuxfamily.org> writes:
>> > Yeah, the first step should be 'cherry-pick' with '--continue',
>> > '--abort', and '--skip' merged.
>>
>> I haven't looked at rebase-i machinery recently, but I wonder if it would
>> just be a matter of making a multi-commit cherry-pick just prepare a bunch
>> of "pick XXX" lines into .git/rebase-merge/rebase-todo file, make other
>> trivial setups (like detaching HEAD, writing head-name and head files) and
>> then execing "git rebase --continue"?
>
> It is probably quite easy to do that, but it would result in cherry-pick in C
> calling rebase-i in shell that itself calls cherry-pick in C (to pick
> individual commit). Instead with this GSoC I think we have the opportunity to
> have everything we needed in C.

I thought I should clarify- in the original proposal, I didn't mean for cherry-pick to call "rebase -i" at all. When I said "use rebase to resume", I meant "try resuming by hand using rebase --continue to verify that cherry-pick has written the state information correctly" as an intermediate step in the cherry-pick development.

-- Ram
Ramkumar Ramachandra· Apr 4, 2011, 16:57 UTC · re: Christian Couder · lore

Re: [GSoC 2011] Git Sequencer

Hi Christian,
Christian Couder writes:
Show 10 quoted lines
> On Sunday 03 April 2011 19:20:56 Ramkumar Ramachandra wrote:
> > I'd like to re-apply this year as a student because I really want to
> > see (among other things), a sequencer in git.git.  Also, since I
> > worked on areas related to fast-import and remote helpers last year, I
> > thought I should work on something completely orthogonal this year.
> > 
> > I now have a draft of my proposal ready, and I'd really appreciate
> > feedback.  Also, could someone mentor me?
> Yeah! I would be happy to mentor you (or co-mentor you if someone else want to 
> be involved)!
Awesome! Thanks :)
Show 14 quoted lines
> > == The Objective ==
> > 
> > To write git-sequencer, a new builtin command, and implement existing
> > commands on top of that.  This should give the commands more
> > functionality, improve their error handling, and make them faster.
> 
> You should first talk about extending git cherry-pick with --continue, --abort 
> and --skip, because it can be very valuable already if done properly with many 
> tests and if it's merged of course.
> 
> > The project can only be considered successful if all (or most) of the
> > code written gets merged into upstream.
> 
> Yeah, just say "most of the code". It is definitely good enough.
Okay, here's the new objective:

Extend 'git cherry-pick' with '--continue', '--abort', and '--skip' features. This will ultimately be used to write git-sequencer, a new builtin command. The sequencer will provide a uniform interface over which existing commands like 'rebase', 'rebase -i' and 'am' can be re-implented. This should give the commands more functionality, improve their error handling, and make them faster. The project can only be considered successful if most of the code written gets merged into upstream.

Show 12 quoted lines
> > 1.1. Factor out all calls to 'die' with 'return error' so so that we
> > can pause the entire process when a commit doesn't apply
> > automatically.
> > 
> > 1.2. Create and populate TODO and DONE files, similar to the one that
> > 'git-rebase--interactive.sh' creates.  For now, it should simply give
> > us information about why a 'cherry-pick' failed.  Use these files with
> > 'git-rebase--interactive.sh' to resume.
> 
> I am not sure it's a good thing to use 'git-rebase--interactive.sh' to resume 
> the cherry-pick. The parsing code already exists, is not very big, is in C and 
> has been reviewed and tested, so I think it's better to use.
Okay, noted.
Show 10 quoted lines
> > 1.3. Port selective tests from the current 't3404' to make sure that
> > TODO and DONE are populated correctly; "stop on conflicting pick" is a
> > good candidate.
> > 
> > 1.4. Decouple the 'revert' functionality from the 'cherry-pick'
> > functionality in 'revert.c'.  Implement '--abort' for 'cherry-pick'
> > and port "abort" test from 't3404'.
> 
> I am not sure decoupling revert and cherry-pick functionnalities is really 
> needed, or I don't know what you mean exactly.

What I was thinking when I wrote that: we should have a 'revert.c' independent of 'cherry-pick.c' after the sequencer is implemented (see 2). By "decouple" I meant: move code around so we don't have to use the enum { REVERT, CHERRY_PICK }.

Show 7 quoted lines
> > 1.5. Implement parsing the TODO and DONE files into suitable data
> > structures.  Derive inspiration from the code written in 2008 to do
> > this.
> 
> Yeah, you can use this patch:
> 
> http://article.gmane.org/gmane.comp.version-control.git/162198

Yeah, this is exactly the patch I was referring to. I just forgot to include the link :p

Show 19 quoted lines
> > 1.6. Implement '--continue' and '--skip', and write suitable tests.
> > Merge into upstream.
> 
> Great!
> 
> > 2. Factor out the 'cherry-pick' code from 'revert.c' into a new
> > 'builtin/sequencer.c', and expose a simple cherry-picking API in
> > 'sequencer.h'.
> 
> Yeah!
> 
> > 3. Implement a fresh 'cherry-pick.c' as a simple API call to the
> > sequencer, and make sure that all the existing tests pass.  After
> > this, cherry-pick will not be a builtin command anymore*.  Merge into
> > upstream.
> 
> Yeah, it is closely linked with the previous point. So maybe this can be 2.2 
> and the previous one can be 2.1. And this way we know that at the end of 1) 
> and at the end of 2) everything should be merged upstream.
Okay.
Show 15 quoted lines
> > 4. Extend the sequncer to parse commands like 'execute', 'reword',
> > 'squash', and 'fixup' that are specific to interactive rebasing.
> > Carefully implement the functionality for each of these keywords in a
> > step-wise manner, making sure that it inter-operates with 'rebase -i'
> > seamlessly.
> 
> Great!
> 
> > 5. Port the entire testsuite from 'rebase -i', and rewrite
> > 'git-rebase.sh', 'git-rebase--interactive.sh' to call the sequncer.
> > The script should essentially build to a list of TODO instructions and
> > pass it to the 'git-sequencer' binary.  Merge into upstream.
> 
> Great! But as it is closely llinked with the previous point, maybe these 2 
> points should be 3.1 and 3.2.
Okay.
Show 8 quoted lines
> > 6. [Optional] Lib'ify the sequncer: modify the sequencer API to
> > include rebase-related functionality.  Write 'rebase.c' as a bunch of
> > API calls to the sequencer.  Make sure that the existing tests pass.
> > Merge into upstream.
> > 
> > 7. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the
> > sequncer: 'am.c'.  Bulk of this should be mbox parsing code.  Make sure
> > that all existing tests pass.  Merge into upstream.

I'll append Daniel's single-step-interrupt idea here, once I understand how to implement it fully.

Show 25 quoted lines
> > == Timeline ===
> > 
> > Deriving from last year's experience, I've decided not to present a
> > tight timeline.  Instead, I simply have an outline: Get the extended
> > cherry-pick functionality merged before mid-term evaluations, and the
> > rest before the final evaluations.
> 
> I agree, but still you could perhaps state something like this:
> 
> - before mid june:
> 	some patch series for everything in 1) should have been sent to the list
> - before midterm evaluation:
> 	everything in 1) should be merged upstream
> 	some patch series for everything in 2) and 3) (or 2.1 and 2.2 if you use 
> the numbering I suggest) should have been sent to the list
> - before the end of July:
> 	everything in 2) and 3) should be merged upstream
> 	some patch series for everything in 4) and 5) (or 3.1 and 3.2 if you use 
> the numbering I suggest) should have been sent to the list
> - before final evaluation:
> 	everything should be merged
> 
> I think it is better to have more details like the above because this way we 
> can realize early that there is not a lot of time after the midterm 
> evaluation.

Great suggestion! I'll include this in the next iteration of my proposal.

Thanks for the detailed review and the suggestions.
-- Ram
Ramkumar Ramachandra· Apr 5, 2011, 20:00 UTC · re: Ramkumar Ramachandra · lore

[GSoC 2011 v2] Git Sequencer

Hi,
Thanks for all the feedback on the first iteration!

This iteration of the proposal has already been submitted via the Melange interface. More comments/ feedback are always welcome.

======================================================================
Project Proposal: Git Sequencer
Student: Ramkumar Ramachandra
Mentor: Christian Couder
== The Objective ==

Extend 'git cherry-pick' with '--continue', '--abort', and '--skip' features. This will ultimately be used to write git-sequencer, a new builtin command. The sequencer will provide a uniform interface over which existing commands like 'rebase', 'rebase -i' and 'am' can be re-implented. This should give the commands more functionality, improve their error handling, and make them faster. The project can only be considered successful if most of the code written gets merged into upstream.

The Git Sequencer was a 2008 GSoC project as well; unfortunately most of the code did not get merged into git.git. The learning from all that work should serve as a huge headstart this year [1].

=== The Plan ===
1. Extend 'cherry-pick' with '--continue', '--abort', and '--skip'
features, so that it works like (a subset of) the current
'git-rebase--interactive.sh'.  This will require patching
'builtin/revert.c' in place, and merging it immediately.  I plan to
roughly follow the road laid out by Christian's 2010 series [2].

1.1. Factor out all calls to 'die' with 'return error' so so that we can pause the entire process when a commit doesn't apply automatically.

1.2. Create and populate TODO and DONE files, similar to the one that 'git-rebase--interactive.sh' creates. For now, it should simply give us information about why a 'cherry-pick' failed.

1.3. Port selective tests from the current 't3404' to make sure that TODO and DONE are populated correctly; "stop on conflicting pick" is a good candidate.

1.4. Decouple the 'revert' functionality from the 'cherry-pick' functionality in 'revert.c'. Implement '--abort' for 'cherry-pick' and port "abort" test from 't3404'.

1.5. Implement parsing the TODO and DONE files into suitable data structures. Derive inspiration from the code written in 2008 to do this [3].

1.6. Implement '--continue' and '--skip', and write suitable tests.
2. Build a sequencer so that just has cherry-picking functionality.
This mostly involves moving code written in (1) around, and crafting a
general API for handling conflicts.

2.1. Factor out the 'cherry-pick' code from 'revert.c' into a new 'builtin/sequencer.c'.

2.2. Write an API for handling conflicts, so that the sequencer is ultimate entry/ exit point for all user intervention in a multi-step process [4].

2.3. Implement a fresh 'cherry-pick.c' on top of the sequencer. Make sure that all the existing tests pass.

2.4 [Optional] Patch 'builtin/merge.c' to use the conflict handler in the sequencer.

3. Extend the sequencer to accomodate the functionality provided by
'rebase -i'.

3.1. Parse commands like 'execute', 'reword', 'squash', and 'fixup' that are specific to interactive rebasing. Carefully implement the functionality for each of these keywords in a step-wise manner.

3.2. [Optional] Port the '--preserve-merges' option of 'rebase' to the sequencer. Port relevant tests from 't3409'.

4. [Optional] Lib'ify the sequncer. Modify the API to
include rebase-related functionality.  Write 'rebase.c' as a bunch of
API calls to the sequencer.  Make sure that the existing tests pass.
5. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the
sequncer: 'am.c'.  Bulk of this should be mbox parsing code.  Make sure
that all existing tests pass.
[Optional] should be read as "If time permits"
== Timeline ===
- Before mid june: (1) should be implemented, and a series should be
  sent out to the list for review.
- Before midterm evaluation: (1) should be merged, and an
  implementation of (2) should be sent to the list.
- Before the end of July: (2) should be merged, and an implementation
  of (3) should be sent to the list.
- Before final evaluation: (3) should be merged.
== Who am I? ==

I'm Ramkumar Ramachandra, and I first started contributing to git.git in January 2010. Apart from doing fast-import and remote helper related work last year, I also authored and merged svnrdump into Subversion trunk in the same period.

== Notes ==

[1]: http://repo.or.cz/w/git/sbeyer.git [2]: http://thread.gmane.org/gmane.comp.version-control.git/162183 [3]: http://article.gmane.org/gmane.comp.version-control.git/162198 [4]: http://thread.gmane.org/gmane.comp.version-control.git/170834 ======================================================================

Thanks for reading.
-- Ram
Christian Couder· Apr 6, 2011, 08:11 UTC · re: Ramkumar Ramachandra · lore

Re: [GSoC 2011 v2] Git Sequencer

Hi,

On Tue, Apr 5, 2011 at 10:00 PM, Ramkumar Ramachandra <artagnon@gmail.com> wrote:

Show 11 quoted lines
>
> 2. Build a sequencer so that just has cherry-picking functionality.
> This mostly involves moving code written in (1) around, and crafting a
> general API for handling conflicts.
>
> 2.1. Factor out the 'cherry-pick' code from 'revert.c' into a new
> 'builtin/sequencer.c'.
>
> 2.2. Write an API for handling conflicts, so that the sequencer is
> ultimate entry/ exit point for all user intervention in a multi-step
> process [4].

This one may be more difficult than we can guess right now, and it is linked with 2.4 that is optional too. And I wouldn't like you to spend too much time on it if it appears to be quite difficult. So I'd suggest that you switch 2.2 and 2.3 and make the new 2.3 optional.

Show 8 quoted lines
> 2.3. Implement a fresh 'cherry-pick.c' on top of the sequencer.  Make
> sure that all the existing tests pass.
>
> 2.4 [Optional] Patch 'builtin/merge.c' to use the conflict handler in
> the sequencer.
>
> 3. Extend the sequencer to accomodate the functionality provided by
> 'rebase -i'.
s/accomodate/accommodate/
> 3.1. Parse commands like 'execute', 'reword', 'squash', and 'fixup'
> that are specific to interactive rebasing.  Carefully implement the
> functionality for each of these keywords in a step-wise manner.
I think that here you could add something like:

3.2. Make 'rebase -i' use the sequencer when '--preserve-merges' option is not used.

> 3.2. [Optional] Port the '--preserve-merges' option of 'rebase' to the
> sequencer.  Port relevant tests from 't3409'.
> 4. [Optional] Lib'ify the sequncer.
s/sequncer/sequencer/
Show 6 quoted lines
> Modify the API to
> include rebase-related functionality.  Write 'rebase.c' as a bunch of
> API calls to the sequencer.  Make sure that the existing tests pass.
>
> 5. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the
> sequncer: 'am.c'.
s/sequncer/sequencer/
> Bulk of this should be mbox parsing code.  Make sure
> that all existing tests pass.
>
> [Optional] should be read as "If time permits"
Otherwise I like it very much.

Thanks, Christian.

Ramkumar Ramachandra· Apr 6, 2011, 09:01 UTC · re: Christian Couder · lore

Re: [GSoC 2011 v2] Git Sequencer

Hi Christian,
Christian Couder writes:
Show 20 quoted lines
> On Tue, Apr 5, 2011 at 10:00 PM, Ramkumar Ramachandra
> <artagnon@gmail.com> wrote:
> >
> > 2. Build a sequencer so that just has cherry-picking functionality.
> > This mostly involves moving code written in (1) around, and crafting a
> > general API for handling conflicts.
> >
> > 2.1. Factor out the 'cherry-pick' code from 'revert.c' into a new
> > 'builtin/sequencer.c'.
> >
> > 2.2. Write an API for handling conflicts, so that the sequencer is
> > ultimate entry/ exit point for all user intervention in a multi-step
> > process [4].
> 
> This one may be more difficult than we can guess right now, and it is
> linked with 2.4 that is optional too. And I wouldn't like you to spend
> too much time on it if it appears to be quite difficult. So I'd
> suggest that you switch 2.2 and 2.3 and make the new 2.3 optional.
> 
> [...]
Okay.  Fixed other nits as well.
Thanks.
-- Ram

← back to recent threads