{"thread":{"id":"26980","subject":"[GSoC 2011] Git Sequencer","startedAt":"2011-04-03T17:20:56Z","lastAt":"2011-04-06T09:01:19Z","messageCount":20,"participants":["Ramkumar Ramachandra","Sverre Rabbelier","Stephan Beyer","Daniel Barkalow","Jonathan Nieder","Christian Couder","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"165059","messageId":"20110403172054.GA10220@kytes","threadId":"26980","inReplyTo":null,"subject":"[GSoC 2011] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-03T17:20:56Z","receivedAt":"2011-04-03T17:20:56Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nI'd like to re-apply this year as a student because I really want to\nsee (among other things), a sequencer in git.git.  Also, since I\nworked on areas related to fast-import and remote helpers last year, I\nthought I should work on something completely orthogonal this year.\n\nI now have a draft of my proposal ready, and I'd really appreciate\nfeedback.  Also, could someone mentor me?\n\n======================================================================\nProject Proposal: Git Sequencer\nStudent: Ramkumar Ramachandra\nMentor: ?\n\n== The Objective ==\n\nTo write git-sequencer, a new builtin command, and implement existing\ncommands on top of that.  This should give the commands more\nfunctionality, improve their error handling, and make them faster.\nThe project can only be considered successful if all (or most) of the\ncode written gets merged into upstream.\n\nThe Git Sequencer was a 2008 GSoC project as well; unfortunately most\nof the code did not get merged into git.git.  The learning from all\nthat work should serve as a huge headstart this year.\n\n=== The Plan ===\n\n1. Extend 'cherry-pick' with '--continue', '--abort', and '--skip'\nfeatures, so that it works like (a subset of) the current\n'git-rebase--interactive.sh'.  This will require patching\n'builtin/revert.c' in place, and merging it immediately.  I plan to\nroughly follow the road laid out by Christian's 2010 series [1].\n\n1.1. Factor out all calls to 'die' with 'return error' so so that we\ncan pause the entire process when a commit doesn't apply\nautomatically.\n\n1.2. Create and populate TODO and DONE files, similar to the one that\n'git-rebase--interactive.sh' creates.  For now, it should simply give\nus information about why a 'cherry-pick' failed.  Use these files with\n'git-rebase--interactive.sh' to resume.\n\n1.3. Port selective tests from the current 't3404' to make sure that\nTODO and DONE are populated correctly; \"stop on conflicting pick\" is a\ngood candidate.\n\n1.4. Decouple the 'revert' functionality from the 'cherry-pick'\nfunctionality in 'revert.c'.  Implement '--abort' for 'cherry-pick'\nand port \"abort\" test from 't3404'.\n\n1.5. Implement parsing the TODO and DONE files into suitable data\nstructures.  Derive inspiration from the code written in 2008 to do\nthis.\n\n1.6. Implement '--continue' and '--skip', and write suitable tests.\nMerge into upstream.\n\n2. Factor out the 'cherry-pick' code from 'revert.c' into a new\n'builtin/sequencer.c', and expose a simple cherry-picking API in\n'sequencer.h'.\n\n3. Implement a fresh 'cherry-pick.c' as a simple API call to the\nsequencer, and make sure that all the existing tests pass.  After\nthis, cherry-pick will not be a builtin command anymore*.  Merge into\nupstream.\n\n4. Extend the sequncer to parse commands like 'execute', 'reword',\n'squash', and 'fixup' that are specific to interactive rebasing.\nCarefully implement the functionality for each of these keywords in a\nstep-wise manner, making sure that it inter-operates with 'rebase -i'\nseamlessly.\n\n5. Port the entire testsuite from 'rebase -i', and rewrite\n'git-rebase.sh', 'git-rebase--interactive.sh' to call the sequncer.\nThe script should essentially build to a list of TODO instructions and\npass it to the 'git-sequencer' binary.  Merge into upstream.\n\n6. [Optional] Lib'ify the sequncer: modify the sequencer API to\ninclude rebase-related functionality.  Write 'rebase.c' as a bunch of\nAPI calls to the sequencer.  Make sure that the existing tests pass.\nMerge into upstream.\n\n7. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the\nsequncer: 'am.c'.  Bulk of this should be mbox parsing code.  Make sure\nthat all existing tests pass.  Merge into upstream.\n\n* Is this a good change? Are there any forseeable issues?\n** [Optional] should be read as \"If time permits\"\n\n== Timeline ===\n\nDeriving from last year's experience, I've decided not to present a\ntight timeline.  Instead, I simply have an outline: Get the extended\ncherry-pick functionality merged before mid-term evaluations, and the\nrest before the final evaluations.\n\n== Who am I? ==\n\nI'm Ramkumar Ramachandra, and I first started contributing to git.git\nin January 2010.  Apart from doing fast-import and remote helper\nrelated work last year, I also authored and merged svnrdump into\nSubversion trunk in the same period.\n\n== Notes ==\n\n[1]: http://thread.gmane.org/gmane.comp.version-control.git/162183\n======================================================================\n\nThanks for reading.\n\n-- Ram\n"},{"id":"165060","messageId":"BANLkTi=HZ1ev8G4Of+=h7k2TrLqEPeonrg@mail.gmail.com","threadId":"26980","inReplyTo":"20110403172054.GA10220@kytes","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-04-03T17:24:55Z","receivedAt":"2011-04-03T17:24:55Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Sun, Apr 3, 2011 at 19:20, Ramkumar Ramachandra <artagnon@gmail.com> wrote:\n> I'd like to re-apply this year as a student because I really want to\n> see (among other things), a sequencer in git.git.  Also, since I\n> worked on areas related to fast-import and remote helpers last year, I\n> thought I should work on something completely orthogonal this year.\n>\n> I now have a draft of my proposal ready, and I'd really appreciate\n> feedback.  Also, could someone mentor me?\n\nWhile I'm very interested in this project, I have no relevant\nexperience. I'll definitely +1 your proposal though :).\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"165073","messageId":"1301857622.3448.134.camel@lambda","threadId":"26980","inReplyTo":"20110403172054.GA10220@kytes","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Stephan Beyer","fromEmail":"s-beyer@gmx.net","sentAt":"2011-04-03T19:07:01Z","receivedAt":"2011-04-03T19:07:01Z","isPatch":false,"sender":{"key":"s-beyer@gmx.net","avatar":"https://avatars.githubusercontent.com/u/143889?v=4"},"body":"Hi Ram,\n\nfirst, some notes on my git-sequencer 2008 branches that can be found at\nhttp://repo.or.cz/w/git/sbeyer.git ... (Not sure if I remember\neverything correctly)\n\nI've settled to develop within the \"seq-builtin-dev\" branch and I\nsometimes merged Junio's \"master\" into that branch to catch up.\nThe \"seq-builtin-dev\" branch is the important one.\n\nUsing git rebase -i (using git-sequencer) I sometimes remanaged the\nbranch to \"seq-builtin-rfc\" that should represent a snapshot of a\npotential patch queue. My last rebase processes of the seq-builtin-rfc\nbranch were pretty unmotivated and hence messy.\n\nI have not touched the repo very often after GSOC'08 and I stopped\ntouching it (as I stopped following recent Git development) \"20 months\nago\" apparently. Quite many things may have changed since then.\n\nThe file A_SEQUENCER_TODO_FILE (added 2009-08-03) in the repo describes\nthe missing and buggy pieces to fix so that _I_ (only me) would have\nbeen 100 per cent satisfied with that git-sequencer.\nhttp://repo.or.cz/w/git/sbeyer.git/blob/9e4b4d92f681a47e3d7ad2152d2391b2ab125a0c:/A_SEQUENCER_TODO_FILE\n[Some notes are also \"strategy notes\" to get things accepted, like the\nchanges on \"rebase -i -p\" which are \"not loved by everyone\". ;-)]\n\nOn 2011-04-03, 22:50 +0530, Ramkumar Ramachandra wrote: \n> * Is this a good change? Are there any forseeable issues?\n\nI want to mention an issue that I have not foreseen before: merges.\n(You need merges, for example, when doing rebase -i -p ... -p as in\n--preserve-merges.)\n\nWhen I began, there was code in the \"next\" branch that added the TODO\ninstructions \"mark\", \"reset\" and \"merge\" to do merges properly and I\nbased my work on it. The original pieces by Jörg Sommer can still be\nfound here:\nhttp://repo.or.cz/w/git/sbeyer.git/shortlog/6fabd85e8a777c26f3ae8ce11cb7f4265502ea7f\n\nHowever, there have been strong opinions that the \"mark\"/\"reset\"/\"merge\"\ninstructions are ugly and unpleasant to users and not even necessary (at\nleast for rebase--interactive... and for sequencer, maybe, maybe not). \nHence, the code in \"next\" has been rejected later.\n\nDuring GSOC 2008 I regrettably underestimated the importance to\ncommunicate with the Git folks about these things. That's one of the\nmain reasons the sequencer pieces did not get into master. And after\nGSOC'08 I had too little time for this... :-/\n\nWell, the merging thing is the only *real* issue I remember.\n\nGood luck and regards,\n  Stephan\n"},{"id":"165079","messageId":"alpine.LNX.2.00.1104031407480.14365@iabervon.org","threadId":"26980","inReplyTo":"20110403172054.GA10220@kytes","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2011-04-03T19:49:32Z","receivedAt":"2011-04-03T19:49:32Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sun, 3 Apr 2011, Ramkumar Ramachandra wrote:\n\n> Hi,\n> \n> I'd like to re-apply this year as a student because I really want to\n> see (among other things), a sequencer in git.git.  Also, since I\n> worked on areas related to fast-import and remote helpers last year, I\n> thought I should work on something completely orthogonal this year.\n> \n> I now have a draft of my proposal ready, and I'd really appreciate\n> feedback.  Also, could someone mentor me?\n> \n> ======================================================================\n> Project Proposal: Git Sequencer\n> Student: Ramkumar Ramachandra\n> Mentor: ?\n> \n> == The Objective ==\n> \n> To write git-sequencer, a new builtin command, and implement existing\n> commands on top of that.  This should give the commands more\n> functionality, improve their error handling, and make them faster.\n> The project can only be considered successful if all (or most) of the\n> code written gets merged into upstream.\n> \n> The Git Sequencer was a 2008 GSoC project as well; unfortunately most\n> of the code did not get merged into git.git.  The learning from all\n> that work should serve as a huge headstart this year.\n\nOne of the things that is hard about sequencer is that it is ultimately a \ncomplete replacement for several differently-implemented programs in \ndifferent languages, with different temporary file formats and differrent \nsupported operations. As such, you could probably spend an entire summer \njust getting it reviewed, revised, and accepted, starting with a working \nimplementation.\n\nSo I think your proposal is good in how [1/5] includes getting something \nuseful merged. My suspicion is that the outcome will be something like \nthat you implemented all 7 tasks and got 4 of them merged, assuming that \nyou really push getting things merged as soon as they're ready, without \nspending too much time porting other things to use the core and getting \nthe ports reviewed before the core is accepted.\n\nI actually think that it would be a worthwhile feature for git's library \ncode to have a uniform mechanism for communicating that it is requesting \nhuman intervention in the middle of a particular operation, where library \noperations which conflict with being able to continue this operation are \neither blocked or abort the operation, and the library is able to be told \nin general that the human intervention is done and the library operation \nshould be finished now (or produce complaints about the user's work). That \nis, a library-level, single-interrupted-step \"sequencer\". For that matter, \nit should also apply to the common '\"git merge\" gets a conflict' case, and \nit would be useful to get some representational uniformity between that \nand cherry-pick getting a conflict.\n\nI think replacing existing multi-step processes is going to be a lot more \ncontentious and involve user-visible changes which involve matters of \ntaste and such. But I think you can make a valuable contribution in how a \nsingle current step is handled before getting tangled in that, and be much \nmore likely to get a useful outcome than if you try to tackle the whole \nproblem.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"165080","messageId":"20110403200043.GA18704@kytes","threadId":"26980","inReplyTo":"1301857622.3448.134.camel@lambda","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-03T20:00:45Z","receivedAt":"2011-04-03T20:00:45Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Stephen,\n\nStephan Beyer writes:\n> first, some notes on my git-sequencer 2008 branches that can be found at\n> http://repo.or.cz/w/git/sbeyer.git ... (Not sure if I remember\n> everything correctly)\n> \n> I've settled to develop within the \"seq-builtin-dev\" branch and I\n> sometimes merged Junio's \"master\" into that branch to catch up.\n> The \"seq-builtin-dev\" branch is the important one.\n\nThanks! Jonathan told me about it earlier, and I've already started\nripping out code from the seq-builtin-dev branch :) I found your\n't3350-sequencer.sh' especially interesting.\n\n> Using git rebase -i (using git-sequencer) I sometimes remanaged the\n> branch to \"seq-builtin-rfc\" that should represent a snapshot of a\n> potential patch queue. My last rebase processes of the seq-builtin-rfc\n> branch were pretty unmotivated and hence messy.\n> \n> I have not touched the repo very often after GSOC'08 and I stopped\n> touching it (as I stopped following recent Git development) \"20 months\n> ago\" apparently. Quite many things may have changed since then.\n\nOkay, got it.  I saw a few patches in 'master' that were based on your\nwork though.  Some of the patches in Christian's series also refer to\nyour work.\n\n> The file A_SEQUENCER_TODO_FILE (added 2009-08-03) in the repo describes\n> the missing and buggy pieces to fix so that _I_ (only me) would have\n> been 100 per cent satisfied with that git-sequencer.\n> http://repo.or.cz/w/git/sbeyer.git/blob/9e4b4d92f681a47e3d7ad2152d2391b2ab125a0c:/A_SEQUENCER_TODO_FILE\n> [Some notes are also \"strategy notes\" to get things accepted, like the\n> changes on \"rebase -i -p\" which are \"not loved by everyone\". ;-)]\n\nOkay.\n\n> On 2011-04-03, 22:50 +0530, Ramkumar Ramachandra wrote: \n> > * Is this a good change? Are there any forseeable issues?\n> \n> I want to mention an issue that I have not foreseen before: merges.\n> (You need merges, for example, when doing rebase -i -p ... -p as in\n> --preserve-merges.)\n\nAh, that's not something I thought about immediately.\n\n> When I began, there was code in the \"next\" branch that added the TODO\n> instructions \"mark\", \"reset\" and \"merge\" to do merges properly and I\n> based my work on it. The original pieces by Jörg Sommer can still be\n> found here:\n> http://repo.or.cz/w/git/sbeyer.git/shortlog/6fabd85e8a777c26f3ae8ce11cb7f4265502ea7f\n> \n> However, there have been strong opinions that the \"mark\"/\"reset\"/\"merge\"\n> instructions are ugly and unpleasant to users and not even necessary (at\n> least for rebase--interactive... and for sequencer, maybe, maybe not). \n> Hence, the code in \"next\" has been rejected later.\n\nInteresting historical note.\n\n> During GSOC 2008 I regrettably underestimated the importance to\n> communicate with the Git folks about these things. That's one of the\n> main reasons the sequencer pieces did not get into master. And after\n> GSOC'08 I had too little time for this... :-/\n> \n> Well, the merging thing is the only *real* issue I remember.\n\nPoint noted.  Yes, I noticed that your sequencer was mostly\nfunctionally complete.  I'll make sure that I spend a lot of\ninteracting with the community.\n\nThank you for your elaborate note! I really appreciate it :)\nHopefully, we will have that sequencer by next year.\n\n-- Ram\n"},{"id":"165081","messageId":"20110403200839.GG3830@elie","threadId":"26980","inReplyTo":"1301857622.3448.134.camel@lambda","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-04-03T20:08:39Z","receivedAt":"2011-04-03T20:08:39Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Stephan Beyer wrote:\n\n> I want to mention an issue that I have not foreseen before: merges.\n> (You need merges, for example, when doing rebase -i -p ... -p as in\n> --preserve-merges.)\n>\n> When I began, there was code in the \"next\" branch that added the TODO\n> instructions \"mark\", \"reset\" and \"merge\" to do merges properly and I\n> based my work on it. The original pieces by Jörg Sommer can still be\n> found here:\n> http://repo.or.cz/w/git/sbeyer.git/shortlog/6fabd85e8a777c26f3ae8ce11cb7f4265502ea7f\n[etc]\n\nSome more pointers:\nhttp://thread.gmane.org/gmane.comp.version-control.git/148059\nIIRC there's some rough consensus about the design, even if I'm not\nsure what it is :).\n\nOf course a dream would be a way to rebase merge conflict resolutions\ninstead of being limited to replaying conflict-free merges or\nresolving conflicts by hand.  But that's a more complex story.\n"},{"id":"165085","messageId":"20110404040610.GA30737@kytes","threadId":"26980","inReplyTo":"alpine.LNX.2.00.1104031407480.14365@iabervon.org","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-04T04:06:15Z","receivedAt":"2011-04-04T04:06:15Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Daniel,\n\nDaniel Barkalow writes:\n> On Sun, 3 Apr 2011, Ramkumar Ramachandra wrote:\n> > To write git-sequencer, a new builtin command, and implement existing\n> > commands on top of that.  This should give the commands more\n> > functionality, improve their error handling, and make them faster.\n> > The project can only be considered successful if all (or most) of the\n> > code written gets merged into upstream.\n> > \n> > The Git Sequencer was a 2008 GSoC project as well; unfortunately most\n> > of the code did not get merged into git.git.  The learning from all\n> > that work should serve as a huge headstart this year.\n> \n> One of the things that is hard about sequencer is that it is ultimately a \n> complete replacement for several differently-implemented programs in \n> different languages, with different temporary file formats and differrent \n> supported operations. As such, you could probably spend an entire summer \n> just getting it reviewed, revised, and accepted, starting with a working \n> implementation.\n\nAgreed.  I've chosen to use the same commands and temporary files as\n'git-rebase--interactive.sh', because I think those commands are\nsufficient to implement everything else.\n\n> So I think your proposal is good in how [1/5] includes getting something \n> useful merged. My suspicion is that the outcome will be something like \n> that you implemented all 7 tasks and got 4 of them merged, assuming that \n> you really push getting things merged as soon as they're ready, without \n> spending too much time porting other things to use the core and getting \n> the ports reviewed before the core is accepted.\n\nHm.  In that case, we'll just have a sequencer that can cherry-pick --\nI personally wouldn't be too happy with this outcome either.\n\n> I actually think that it would be a worthwhile feature for git's library \n> code to have a uniform mechanism for communicating that it is requesting \n> human intervention in the middle of a particular operation, where library \n> operations which conflict with being able to continue this operation are \n> either blocked or abort the operation, and the library is able to be told \n> in general that the human intervention is done and the library operation \n> should be finished now (or produce complaints about the user's work). That \n> is, a library-level, single-interrupted-step \"sequencer\". For that matter, \n> it should also apply to the common '\"git merge\" gets a conflict' case, and \n> it would be useful to get some representational uniformity between that \n> and cherry-pick getting a conflict.\n\nUntil 4/7, I'd only planned to make the 'git-sequencer' binary like\nthe 'git-rebase--interactive.sh' script, except that it would accept a\nTODO file on stdin, instead of interactively opening up an editor.\n\nYour idea is a slightly more ambitious version of what I'd planned for\n6/7, especially since 'merge' contains a lot of cruft like MERGE_HEAD\nand CHERRY_PICK_HEAD.  I can shift my focus after 4/7 though -- here's\nwhat I have in mind.  Do you have something similar in mind?\n\nenum commit_todo_action {\n     ACTION_PICK;\n     ACTION_REWORD;\n     ACTION_EDIT;\n     ACTION_SQUASH;\n     ACTION_FIXUP;\n     ACTION_EXEC;\n};\n\nstruct commit_todo_list {\n       struct commit *item;\n       enum commit_todo_action action;\n       struct commit_todo_list *next;\n};\n\nint sequencer_cherry_pick(struct commit *base, struct commit_list *list);\nint sequncer_rebase(struct commit *base, struct commit_todo_list *list);\nint sequencer_handle_conflict(); /* Returns ABORT (1) or RESOLVED (0) */\n\n/**\n * The sequencer_handle_conflict function essentially starts with a\n * working tree with unmerged files and results in either a working\n * tree without unmerged files (in which case it returns 0), or simply\n * returns 1.  Advantage: Consistency. Each individual script will not\n * have to maintain its own temporary files.\n */\n\n> I think replacing existing multi-step processes is going to be a lot more \n> contentious and involve user-visible changes which involve matters of \n> taste and such. But I think you can make a valuable contribution in how a \n> single current step is handled before getting tangled in that, and be much \n> more likely to get a useful outcome than if you try to tackle the whole \n> problem.\n\nOkay.  I'll replace 5/7 - 7/7 in my proposal with an alternative as\nsoon as I sketch out the details more clearly.\n\nThanks for your suggestions!\n\n-- Ram\n"},{"id":"165086","messageId":"201104040643.35583.chriscool@tuxfamily.org","threadId":"26980","inReplyTo":"20110403172054.GA10220@kytes","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2011-04-04T04:43:35Z","receivedAt":"2011-04-04T04:43:35Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sunday 03 April 2011 19:20:56 Ramkumar Ramachandra wrote:\n> Hi,\n> \n> I'd like to re-apply this year as a student because I really want to\n> see (among other things), a sequencer in git.git.  Also, since I\n> worked on areas related to fast-import and remote helpers last year, I\n> thought I should work on something completely orthogonal this year.\n> \n> I now have a draft of my proposal ready, and I'd really appreciate\n> feedback.  Also, could someone mentor me?\n\nYeah! I would be happy to mentor you (or co-mentor you if someone else want to \nbe involved)!\n\n> ======================================================================\n> Project Proposal: Git Sequencer\n> Student: Ramkumar Ramachandra\n> Mentor: ?\n> \n> == The Objective ==\n> \n> To write git-sequencer, a new builtin command, and implement existing\n> commands on top of that.  This should give the commands more\n> functionality, improve their error handling, and make them faster.\n\nYou should first talk about extending git cherry-pick with --continue, --abort \nand --skip, because it can be very valuable already if done properly with many \ntests and if it's merged of course.\n\n> The project can only be considered successful if all (or most) of the\n> code written gets merged into upstream.\n\nYeah, just say \"most of the code\". It is definitely good enough.\n\n> The Git Sequencer was a 2008 GSoC project as well; unfortunately most\n> of the code did not get merged into git.git.  The learning from all\n> that work should serve as a huge headstart this year.\n> \n> === The Plan ===\n> \n> 1. Extend 'cherry-pick' with '--continue', '--abort', and '--skip'\n> features, so that it works like (a subset of) the current\n> 'git-rebase--interactive.sh'.  This will require patching\n> 'builtin/revert.c' in place, and merging it immediately.  I plan to\n> roughly follow the road laid out by Christian's 2010 series [1].\n\nYeah, the first step should be 'cherry-pick' with '--continue', '--abort', and \n'--skip' merged.\n\n> 1.1. Factor out all calls to 'die' with 'return error' so so that we\n> can pause the entire process when a commit doesn't apply\n> automatically.\n> \n> 1.2. Create and populate TODO and DONE files, similar to the one that\n> 'git-rebase--interactive.sh' creates.  For now, it should simply give\n> us information about why a 'cherry-pick' failed.  Use these files with\n> 'git-rebase--interactive.sh' to resume.\n\nI am not sure it's a good thing to use 'git-rebase--interactive.sh' to resume \nthe cherry-pick. The parsing code already exists, is not very big, is in C and \nhas been reviewed and tested, so I think it's better to use.\n\n> 1.3. Port selective tests from the current 't3404' to make sure that\n> TODO and DONE are populated correctly; \"stop on conflicting pick\" is a\n> good candidate.\n> \n> 1.4. Decouple the 'revert' functionality from the 'cherry-pick'\n> functionality in 'revert.c'.  Implement '--abort' for 'cherry-pick'\n> and port \"abort\" test from 't3404'.\n\nI am not sure decoupling revert and cherry-pick functionnalities is really \nneeded, or I don't know what you mean exactly.\n\n> 1.5. Implement parsing the TODO and DONE files into suitable data\n> structures.  Derive inspiration from the code written in 2008 to do\n> this.\n\nYeah, you can use this patch:\n\nhttp://article.gmane.org/gmane.comp.version-control.git/162198\n\n> 1.6. Implement '--continue' and '--skip', and write suitable tests.\n> Merge into upstream.\n\nGreat!\n\n> 2. Factor out the 'cherry-pick' code from 'revert.c' into a new\n> 'builtin/sequencer.c', and expose a simple cherry-picking API in\n> 'sequencer.h'.\n\nYeah!\n\n> 3. Implement a fresh 'cherry-pick.c' as a simple API call to the\n> sequencer, and make sure that all the existing tests pass.  After\n> this, cherry-pick will not be a builtin command anymore*.  Merge into\n> upstream.\n\nYeah, it is closely linked with the previous point. So maybe this can be 2.2 \nand the previous one can be 2.1. And this way we know that at the end of 1) \nand at the end of 2) everything should be merged upstream.\n\n> 4. Extend the sequncer to parse commands like 'execute', 'reword',\n> 'squash', and 'fixup' that are specific to interactive rebasing.\n> Carefully implement the functionality for each of these keywords in a\n> step-wise manner, making sure that it inter-operates with 'rebase -i'\n> seamlessly.\n\nGreat!\n\n> 5. Port the entire testsuite from 'rebase -i', and rewrite\n> 'git-rebase.sh', 'git-rebase--interactive.sh' to call the sequncer.\n> The script should essentially build to a list of TODO instructions and\n> pass it to the 'git-sequencer' binary.  Merge into upstream.\n\nGreat! But as it is closely llinked with the previous point, maybe these 2 \npoints should be 3.1 and 3.2.\n\n> 6. [Optional] Lib'ify the sequncer: modify the sequencer API to\n> include rebase-related functionality.  Write 'rebase.c' as a bunch of\n> API calls to the sequencer.  Make sure that the existing tests pass.\n> Merge into upstream.\n> \n> 7. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the\n> sequncer: 'am.c'.  Bulk of this should be mbox parsing code.  Make sure\n> that all existing tests pass.  Merge into upstream.\n> \n> * Is this a good change? \n\nI think it is a good proposal.\n\n> Are there any forseeable issues?\n\nI don't see anything that others didn't told you about.\n\n> ** [Optional] should be read as \"If time permits\"\n> \n> == Timeline ===\n> \n> Deriving from last year's experience, I've decided not to present a\n> tight timeline.  Instead, I simply have an outline: Get the extended\n> cherry-pick functionality merged before mid-term evaluations, and the\n> rest before the final evaluations.\n\nI agree, but still you could perhaps state something like this:\n\n- before mid june:\n\tsome patch series for everything in 1) should have been sent to the list\n- before midterm evaluation:\n\teverything in 1) should be merged upstream\n\tsome patch series for everything in 2) and 3) (or 2.1 and 2.2 if you use \nthe numbering I suggest) should have been sent to the list\n- before the end of July:\n\teverything in 2) and 3) should be merged upstream\n\tsome patch series for everything in 4) and 5) (or 3.1 and 3.2 if you use \nthe numbering I suggest) should have been sent to the list\n- before final evaluation:\n\teverything should be merged\n\nI think it is better to have more details like the above because this way we \ncan realize early that there is not a lot of time after the midterm \nevaluation.\n\n> == Who am I? ==\n> \n> I'm Ramkumar Ramachandra, and I first started contributing to git.git\n> in January 2010.  Apart from doing fast-import and remote helper\n> related work last year, I also authored and merged svnrdump into\n> Subversion trunk in the same period.\n> \n> == Notes ==\n> \n> [1]: http://thread.gmane.org/gmane.comp.version-control.git/162183\n> ======================================================================\n> \n> Thanks for reading.\n\nThanks for applying,\nChristian.\n"},{"id":"165087","messageId":"20110404045437.GA2208@kytes","threadId":"26980","inReplyTo":"20110404040610.GA30737@kytes","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-04T04:54:41Z","receivedAt":"2011-04-04T04:54:41Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Daniel,\n\nRamkumar Ramachandra writes:\n> Daniel Barkalow writes:\n> > I actually think that it would be a worthwhile feature for git's library \n> > code to have a uniform mechanism for communicating that it is requesting \n> > human intervention in the middle of a particular operation, where library \n> > operations which conflict with being able to continue this operation are \n> > either blocked or abort the operation, and the library is able to be told \n> > in general that the human intervention is done and the library operation \n> > should be finished now (or produce complaints about the user's work). That \n> > is, a library-level, single-interrupted-step \"sequencer\". For that matter, \n> > it should also apply to the common '\"git merge\" gets a conflict' case, and \n> > it would be useful to get some representational uniformity between that \n> > and cherry-pick getting a conflict.\n\n[...]\n\n> int sequencer_handle_conflict(); /* Returns ABORT (1) or RESOLVED (0) */\n> \n> /**\n>  * The sequencer_handle_conflict function essentially starts with a\n>  * working tree with unmerged files and results in either a working\n>  * tree without unmerged files (in which case it returns 0), or simply\n>  * returns 1.  Advantage: Consistency. Each individual script will not\n>  * have to maintain its own temporary files.\n>  */\n\nUh, no.  I wrote this part in too quickly.  Clearly needs more\nthought.\n\n-- Ram\n"},{"id":"165088","messageId":"7vy63qa8z1.fsf@alter.siamese.dyndns.org","threadId":"26980","inReplyTo":"201104040643.35583.chriscool@tuxfamily.org","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-04-04T05:20:18Z","receivedAt":"2011-04-04T05:20:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder <chriscool@tuxfamily.org> writes:\n\n> Yeah, the first step should be 'cherry-pick' with '--continue', '--abort', and \n> '--skip' merged.\n\nI haven't looked at rebase-i machinery recently, but I wonder if it would\njust be a matter of making a multi-commit cherry-pick just prepare a bunch\nof \"pick XXX\" lines into .git/rebase-merge/rebase-todo file, make other\ntrivial setups (like detaching HEAD, writing head-name and head files) and\nthen execing \"git rebase --continue\"?\n"},{"id":"165126","messageId":"20110404165659.GA28587@kytes","threadId":"26980","inReplyTo":"201104040643.35583.chriscool@tuxfamily.org","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-04T16:57:02Z","receivedAt":"2011-04-04T16:57:02Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Christian,\n\nChristian Couder writes:\n> On Sunday 03 April 2011 19:20:56 Ramkumar Ramachandra wrote:\n> > I'd like to re-apply this year as a student because I really want to\n> > see (among other things), a sequencer in git.git.  Also, since I\n> > worked on areas related to fast-import and remote helpers last year, I\n> > thought I should work on something completely orthogonal this year.\n> > \n> > I now have a draft of my proposal ready, and I'd really appreciate\n> > feedback.  Also, could someone mentor me?\n> Yeah! I would be happy to mentor you (or co-mentor you if someone else want to \n> be involved)!\n\nAwesome! Thanks :)\n\n> > == The Objective ==\n> > \n> > To write git-sequencer, a new builtin command, and implement existing\n> > commands on top of that.  This should give the commands more\n> > functionality, improve their error handling, and make them faster.\n> \n> You should first talk about extending git cherry-pick with --continue, --abort \n> and --skip, because it can be very valuable already if done properly with many \n> tests and if it's merged of course.\n> \n> > The project can only be considered successful if all (or most) of the\n> > code written gets merged into upstream.\n> \n> Yeah, just say \"most of the code\". It is definitely good enough.\n\nOkay, here's the new objective:\n\nExtend 'git cherry-pick' with '--continue', '--abort', and '--skip'\nfeatures.  This will ultimately be used to write git-sequencer, a new\nbuiltin command.  The sequencer will provide a uniform interface over\nwhich existing commands like 'rebase', 'rebase -i' and 'am' can be\nre-implented.  This should give the commands more functionality,\nimprove their error handling, and make them faster.  The project can\nonly be considered successful if most of the code written gets merged\ninto upstream.\n\n> > 1.1. Factor out all calls to 'die' with 'return error' so so that we\n> > can pause the entire process when a commit doesn't apply\n> > automatically.\n> > \n> > 1.2. Create and populate TODO and DONE files, similar to the one that\n> > 'git-rebase--interactive.sh' creates.  For now, it should simply give\n> > us information about why a 'cherry-pick' failed.  Use these files with\n> > 'git-rebase--interactive.sh' to resume.\n> \n> I am not sure it's a good thing to use 'git-rebase--interactive.sh' to resume \n> the cherry-pick. The parsing code already exists, is not very big, is in C and \n> has been reviewed and tested, so I think it's better to use.\n\nOkay, noted.\n\n> > 1.3. Port selective tests from the current 't3404' to make sure that\n> > TODO and DONE are populated correctly; \"stop on conflicting pick\" is a\n> > good candidate.\n> > \n> > 1.4. Decouple the 'revert' functionality from the 'cherry-pick'\n> > functionality in 'revert.c'.  Implement '--abort' for 'cherry-pick'\n> > and port \"abort\" test from 't3404'.\n> \n> I am not sure decoupling revert and cherry-pick functionnalities is really \n> needed, or I don't know what you mean exactly.\n\nWhat I was thinking when I wrote that: we should have a 'revert.c'\nindependent of 'cherry-pick.c' after the sequencer is implemented (see\n2).  By \"decouple\" I meant: move code around so we don't have to use\nthe enum { REVERT, CHERRY_PICK }.\n\n> > 1.5. Implement parsing the TODO and DONE files into suitable data\n> > structures.  Derive inspiration from the code written in 2008 to do\n> > this.\n> \n> Yeah, you can use this patch:\n> \n> http://article.gmane.org/gmane.comp.version-control.git/162198\n\nYeah, this is exactly the patch I was referring to.  I just forgot to\ninclude the link :p\n\n> > 1.6. Implement '--continue' and '--skip', and write suitable tests.\n> > Merge into upstream.\n> \n> Great!\n> \n> > 2. Factor out the 'cherry-pick' code from 'revert.c' into a new\n> > 'builtin/sequencer.c', and expose a simple cherry-picking API in\n> > 'sequencer.h'.\n> \n> Yeah!\n> \n> > 3. Implement a fresh 'cherry-pick.c' as a simple API call to the\n> > sequencer, and make sure that all the existing tests pass.  After\n> > this, cherry-pick will not be a builtin command anymore*.  Merge into\n> > upstream.\n> \n> Yeah, it is closely linked with the previous point. So maybe this can be 2.2 \n> and the previous one can be 2.1. And this way we know that at the end of 1) \n> and at the end of 2) everything should be merged upstream.\n\nOkay.\n\n> > 4. Extend the sequncer to parse commands like 'execute', 'reword',\n> > 'squash', and 'fixup' that are specific to interactive rebasing.\n> > Carefully implement the functionality for each of these keywords in a\n> > step-wise manner, making sure that it inter-operates with 'rebase -i'\n> > seamlessly.\n> \n> Great!\n> \n> > 5. Port the entire testsuite from 'rebase -i', and rewrite\n> > 'git-rebase.sh', 'git-rebase--interactive.sh' to call the sequncer.\n> > The script should essentially build to a list of TODO instructions and\n> > pass it to the 'git-sequencer' binary.  Merge into upstream.\n> \n> Great! But as it is closely llinked with the previous point, maybe these 2 \n> points should be 3.1 and 3.2.\n\nOkay.\n\n> > 6. [Optional] Lib'ify the sequncer: modify the sequencer API to\n> > include rebase-related functionality.  Write 'rebase.c' as a bunch of\n> > API calls to the sequencer.  Make sure that the existing tests pass.\n> > Merge into upstream.\n> > \n> > 7. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the\n> > sequncer: 'am.c'.  Bulk of this should be mbox parsing code.  Make sure\n> > that all existing tests pass.  Merge into upstream.\n\nI'll append Daniel's single-step-interrupt idea here, once I\nunderstand how to implement it fully.\n\n> > == Timeline ===\n> > \n> > Deriving from last year's experience, I've decided not to present a\n> > tight timeline.  Instead, I simply have an outline: Get the extended\n> > cherry-pick functionality merged before mid-term evaluations, and the\n> > rest before the final evaluations.\n> \n> I agree, but still you could perhaps state something like this:\n> \n> - before mid june:\n> \tsome patch series for everything in 1) should have been sent to the list\n> - before midterm evaluation:\n> \teverything in 1) should be merged upstream\n> \tsome patch series for everything in 2) and 3) (or 2.1 and 2.2 if you use \n> the numbering I suggest) should have been sent to the list\n> - before the end of July:\n> \teverything in 2) and 3) should be merged upstream\n> \tsome patch series for everything in 4) and 5) (or 3.1 and 3.2 if you use \n> the numbering I suggest) should have been sent to the list\n> - before final evaluation:\n> \teverything should be merged\n> \n> I think it is better to have more details like the above because this way we \n> can realize early that there is not a lot of time after the midterm \n> evaluation.\n\nGreat suggestion! I'll include this in the next iteration of my\nproposal.\n\nThanks for the detailed review and the suggestions.\n\n-- Ram\n"},{"id":"165133","messageId":"alpine.LNX.2.00.1104041319570.14365@iabervon.org","threadId":"26980","inReplyTo":"20110404045437.GA2208@kytes","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2011-04-04T18:59:59Z","receivedAt":"2011-04-04T18:59:59Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 4 Apr 2011, Ramkumar Ramachandra wrote:\n\n> Hi Daniel,\n> \n> Ramkumar Ramachandra writes:\n> > Daniel Barkalow writes:\n> > > I actually think that it would be a worthwhile feature for git's library \n> > > code to have a uniform mechanism for communicating that it is requesting \n> > > human intervention in the middle of a particular operation, where library \n> > > operations which conflict with being able to continue this operation are \n> > > either blocked or abort the operation, and the library is able to be told \n> > > in general that the human intervention is done and the library operation \n> > > should be finished now (or produce complaints about the user's work). That \n> > > is, a library-level, single-interrupted-step \"sequencer\". For that matter, \n> > > it should also apply to the common '\"git merge\" gets a conflict' case, and \n> > > it would be useful to get some representational uniformity between that \n> > > and cherry-pick getting a conflict.\n> \n> [...]\n> \n> > int sequencer_handle_conflict(); /* Returns ABORT (1) or RESOLVED (0) */\n> > \n> > /**\n> >  * The sequencer_handle_conflict function essentially starts with a\n> >  * working tree with unmerged files and results in either a working\n> >  * tree without unmerged files (in which case it returns 0), or simply\n> >  * returns 1.  Advantage: Consistency. Each individual script will not\n> >  * have to maintain its own temporary files.\n> >  */\n> \n> Uh, no.  I wrote this part in too quickly.  Clearly needs more\n> thought.\n\nHere's how I'm thinking about a single step:\n\nThe non-conflict case is:\n\n  $ git cherry-pick ...\n  figure out what we're asked to do\n  make the change to the working directory and index\n  make the commit with info from the commit we're cherry-picking\n\nThe conflict case should be:\n\n  $ git cherry-pick ...\n  figure out what we're asked to do\n  make the change to the working directory and index\n  discover problem, set up for human assistance, tweak info to say that we \n    needed help\n  $ fix stuff\n  $ git continue\n  check that everything is how it should be\n  make the commit with info from the commit we're cherry-picking\n\nThat is, the code that cherry-picks one commit can quit in the middle and \nresume after the user finishes helping, and the main entry point to git \ncan resume that operation.\n\nSo, my thought was that you'd have something like:\n\ncherry_pick_conflict = { \n  \"cherry-pick\", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,\n  cherry_pick_verify_resolution,\n  cherry_pick_abort,\n  cherry_pick_post_resolution\n};\n\nint cherry_pick(struct commit *item)\n{\n  save info on the commit, flags, etc needed to understand what we're doing\n\n  try to apply diff...\n  if (!rejected)\n    return cherry_pick_post_resolution();\n  try to use merge recursive...\n  if (!conflict)\n    return cherry_pick_post_resolution();\n  return report_conflict(cherry_pick_conflict);\n}\n\nint cherry_pick_verify_resolution(void)\n{\n  if (still unmerged files) {\n    describe work still needed\n    return 1;\n  }\n  return 0;\n}\n\nint cherry_pick_abort(void)\n{\n  restore info on what we'd started and delete state\n\n  clean up the working directory, index, etc\n\n  return 0;\n}\n\nint cherry_pick_post_resolution(void)\n{\n  restore info on what we'd started and delete state\n\n  make the commit\n\n  return 0;\n}\n\nThe important things are:\n\n - arbitrary code can determine that you're in the middle of resolving \n   some conflict, that the resolution of that conflict is about doing\n   something to your current branch, and how to abort what you're doing,\n   and how to finish it\n\n - the same code gets run after the conflict has been resolved that would \n   have been run immediately if the merge went smoothly\n\n - cherry-pick can save whatever it needs to in its state file; that's \n   its business, and the semantics here don't have to interact with other \n   commands, because report_conflict() has taken care of interaction with \n   other commands\n\nThe next step is to be able to have:\n\nint sequencer()\n{\n  save the list of steps, with no in-progress step\n  return sequencer_post_resolution();\n}\n\nint sequencer_post_resolution()\n{\n  finish any in-progress step\n  get the next step\n  if (no steps left)\n    return 0;\n  attempt step\n  if (!got_conflict)\n    return sequencer_post_resolution();\n  return report_conflict(sequencer_conflict);\n}\n\nWhere the sequencer-level conflict nests around the cherry-pick-level \nconflict, and the generic \"continue\" completes things from the inside out.\n\nI think, ultimately, that with this code structure in place, the \nam/rebase/rebase--interactive/sequencer details of how the multi-step \nprocess is recorded becomes less important. That way, your project can be \nsuccessful even if you can't find a syntax for the sequencer file that \nmeets the needs of all of these use cases. (Which is where I suspect \nyou'll get bogged down.) If you can get all of the cases where git exits \nin order to get human intervention to share \"everything that matters\" and \nthe core to \"know what's in progress as far as anything else cares\", I \nthink that would be success, even if the various multi-step programs \ncontinue using their own state files.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"165168","messageId":"201104050823.29762.chriscool@tuxfamily.org","threadId":"26980","inReplyTo":"7vy63qa8z1.fsf@alter.siamese.dyndns.org","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2011-04-05T06:23:28Z","receivedAt":"2011-04-05T06:23:28Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Monday 04 April 2011 07:20:18 Junio C Hamano wrote:\n> Christian Couder <chriscool@tuxfamily.org> writes:\n> > Yeah, the first step should be 'cherry-pick' with '--continue',\n> > '--abort', and '--skip' merged.\n> \n> I haven't looked at rebase-i machinery recently, but I wonder if it would\n> just be a matter of making a multi-commit cherry-pick just prepare a bunch\n> of \"pick XXX\" lines into .git/rebase-merge/rebase-todo file, make other\n> trivial setups (like detaching HEAD, writing head-name and head files) and\n> then execing \"git rebase --continue\"?\n\nIt is probably quite easy to do that, but it would result in cherry-pick in C \ncalling rebase-i in shell that itself calls cherry-pick in C (to pick \nindividual commit). Instead with this GSoC I think we have the opportunity to \nhave everything we needed in C.\n\nBest regards,\nChristian.\n"},{"id":"165169","messageId":"BANLkTimfSvGeZcwExB_RW5X_coN908s8Rw@mail.gmail.com","threadId":"26980","inReplyTo":"201104050823.29762.chriscool@tuxfamily.org","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-05T06:46:22Z","receivedAt":"2011-04-05T06:46:22Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Christian and Junio,\n\nOn Tue, Apr 5, 2011 at 11:53 AM, Christian Couder\n<chriscool@tuxfamily.org> wrote:\n> On Monday 04 April 2011 07:20:18 Junio C Hamano wrote:\n>> Christian Couder <chriscool@tuxfamily.org> writes:\n>> > Yeah, the first step should be 'cherry-pick' with '--continue',\n>> > '--abort', and '--skip' merged.\n>>\n>> I haven't looked at rebase-i machinery recently, but I wonder if it would\n>> just be a matter of making a multi-commit cherry-pick just prepare a bunch\n>> of \"pick XXX\" lines into .git/rebase-merge/rebase-todo file, make other\n>> trivial setups (like detaching HEAD, writing head-name and head files) and\n>> then execing \"git rebase --continue\"?\n>\n> It is probably quite easy to do that, but it would result in cherry-pick in C\n> calling rebase-i in shell that itself calls cherry-pick in C (to pick\n> individual commit). Instead with this GSoC I think we have the opportunity to\n> have everything we needed in C.\n\nI thought I should clarify- in the original proposal, I didn't mean\nfor cherry-pick to call \"rebase -i\" at all. When I said \"use rebase to\nresume\", I meant \"try resuming by hand using rebase --continue to\nverify that cherry-pick has written the state information correctly\"\nas an intermediate step in the cherry-pick development.\n\n-- Ram\n"},{"id":"165193","messageId":"20110405175003.GA12159@kytes","threadId":"26980","inReplyTo":"alpine.LNX.2.00.1104041319570.14365@iabervon.org","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-05T17:50:08Z","receivedAt":"2011-04-05T17:50:08Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Daniel,\n\nDaniel Barkalow writes:\n> On Mon, 4 Apr 2011, Ramkumar Ramachandra wrote:\n> > Ramkumar Ramachandra writes:\n> > > Daniel Barkalow writes:\n> > > > I actually think that it would be a worthwhile feature for git's library \n> > > > code to have a uniform mechanism for communicating that it is requesting \n> > > > human intervention in the middle of a particular operation, where library \n> > > > operations which conflict with being able to continue this operation are \n> > > > either blocked or abort the operation, and the library is able to be told \n> > > > in general that the human intervention is done and the library operation \n> > > > should be finished now (or produce complaints about the user's work). That \n> > > > is, a library-level, single-interrupted-step \"sequencer\". For that matter, \n> > > > it should also apply to the common '\"git merge\" gets a conflict' case, and \n> > > > it would be useful to get some representational uniformity between that \n> > > > and cherry-pick getting a conflict.\n> > \n> > [...]\n\nThanks for the detailed response- I've rearragned your response and\nadded some comments.  I initially wanted to design it so that all\nstate is persisted by the sequencer, but I can clearly see what's\nwrong with that approach now.\n\n> I think, ultimately, that with this code structure in place, the \n> am/rebase/rebase--interactive/sequencer details of how the multi-step \n> process is recorded becomes less important. That way, your project can be \n> successful even if you can't find a syntax for the sequencer file that \n> meets the needs of all of these use cases. (Which is where I suspect \n> you'll get bogged down.) If you can get all of the cases where git exits \n> in order to get human intervention to share \"everything that matters\" and \n> the core to \"know what's in progress as far as anything else cares\", I \n> think that would be success, even if the various multi-step programs \n> continue using their own state files.\n\nExcellent.  The crux of the idea: The sequencer should serve as the\nentry/ exit point for Git when any operation requires user\nintervention to proceed.  For this, it should have information about\nhow we got to this point, and how to proceed after the user\nintervention is complete; this information is contained in:\n\n> cherry_pick_conflict = { \n>   \"cherry-pick\", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,\n>   cherry_pick_verify_resolution,\n>   cherry_pick_abort,\n>   cherry_pick_post_resolution\n> };\n\nWait -- isn't it missing a skip callback?\n\ncherry_pick_conflict = { \n  \"cherry-pick\", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,\n  cherry_pick_verify_resolution,\n  cherr_pick_skip,\n  cherry_pick_abort,\n  cherry_pick_post_resolution\n};\n\nThis information is passed to report_conflict(), which takes care of\nuser intervention.  The user can do whatever she wants and then ask\nthe sequencer to \"continue\", \"skip\" or \"abort\":\n\n> Where the sequencer-level conflict nests around the cherry-pick-level \n> conflict, and the generic \"continue\" completes things from the inside out.\n\nRight.  And then the sequencer fires the appropriate callback and\nreturns control to the parent command.  More notes:\n\n>  - cherry-pick can save whatever it needs to in its state file; that's \n>    its business, and the semantics here don't have to interact with other \n>    commands, because report_conflict() has taken care of interaction with \n>    other commands\n\nAt the end of a merge for example, the MERGE_MSG needs to be retrieved\nto create a new merge commit.  The sequencer des not need to know\nanything about this, since this is specific to 'merge'.\n\n>  - arbitrary code can determine that you're in the middle of resolving \n>    some conflict, that the resolution of that conflict is about doing\n>    something to your current branch, and how to abort what you're doing,\n>    and how to finish it\n\nAny arbitrary code simply has to ask the sequencer about the state of\nthe intermediate files that report_conflict() uses.  They don't have\nto worry about command-specific intermediate files.\n\n>  - the same code gets run after the conflict has been resolved that would \n>    have been run immediately if the merge went smoothly\n\nUsing these callbacks, there is no need for if-else ugliness inside\nthe specific command to decide what to do next.\n\nI suppose we can call this  idea a \"generic conflict handler\".  I like\nit very  much, and  I'll definitely  include this as  part of  my GSoC\nwork.  Thanks for taking the time to explain it in such detail :)\n\n-- Ram\n"},{"id":"165196","messageId":"alpine.LNX.2.00.1104051354500.14365@iabervon.org","threadId":"26980","inReplyTo":"20110405175003.GA12159@kytes","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2011-04-05T18:24:46Z","receivedAt":"2011-04-05T18:24:46Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 5 Apr 2011, Ramkumar Ramachandra wrote:\n\n> Hi Daniel,\n> \n> Daniel Barkalow writes:\n> > On Mon, 4 Apr 2011, Ramkumar Ramachandra wrote:\n> > > Ramkumar Ramachandra writes:\n> > > > Daniel Barkalow writes:\n> > > > > I actually think that it would be a worthwhile feature for git's library \n> > > > > code to have a uniform mechanism for communicating that it is requesting \n> > > > > human intervention in the middle of a particular operation, where library \n> > > > > operations which conflict with being able to continue this operation are \n> > > > > either blocked or abort the operation, and the library is able to be told \n> > > > > in general that the human intervention is done and the library operation \n> > > > > should be finished now (or produce complaints about the user's work). That \n> > > > > is, a library-level, single-interrupted-step \"sequencer\". For that matter, \n> > > > > it should also apply to the common '\"git merge\" gets a conflict' case, and \n> > > > > it would be useful to get some representational uniformity between that \n> > > > > and cherry-pick getting a conflict.\n> > > \n> > > [...]\n> \n> Thanks for the detailed response- I've rearragned your response and\n> added some comments.  I initially wanted to design it so that all\n> state is persisted by the sequencer, but I can clearly see what's\n> wrong with that approach now.\n> \n> > I think, ultimately, that with this code structure in place, the \n> > am/rebase/rebase--interactive/sequencer details of how the multi-step \n> > process is recorded becomes less important. That way, your project can be \n> > successful even if you can't find a syntax for the sequencer file that \n> > meets the needs of all of these use cases. (Which is where I suspect \n> > you'll get bogged down.) If you can get all of the cases where git exits \n> > in order to get human intervention to share \"everything that matters\" and \n> > the core to \"know what's in progress as far as anything else cares\", I \n> > think that would be success, even if the various multi-step programs \n> > continue using their own state files.\n> \n> Excellent.  The crux of the idea: The sequencer should serve as the\n> entry/ exit point for Git when any operation requires user\n> intervention to proceed.\n\nI'm a bit surprised by the idea of calling that \"the sequencer\" (rather \nthan having \"the sequencer\" be a command), but I actually think you're \nentirely right to do so. Be sure to be very explicit about that, though, \nbecause people will probably start with the wrong idea of what you're \nproposing otherwise.\n\n> For this, it should have information about\n> how we got to this point, and how to proceed after the user\n> intervention is complete; this information is contained in:\n> \n> > cherry_pick_conflict = { \n> >   \"cherry-pick\", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,\n> >   cherry_pick_verify_resolution,\n> >   cherry_pick_abort,\n> >   cherry_pick_post_resolution\n> > };\n> \n> Wait -- isn't it missing a skip callback?\n\nI think \"skip\" is actually: abort the lowest-level conflict and continue \nthe next-level conflict. If you're doing a rebase, and the rebase is doing \na \"pick\", and the pick got a conflict, --skip means that you abort the \npick (to get back to the state where the earlier commits have been picked \nbut this one hasn't been started, followed by having the rebase continue \nwith what it was going to do after the pick completed.\n\nSo I don't think you need a \"skip\" callback, as long as you've untangled \nthe levels cleanly and get the nesting support right.\n\n> cherry_pick_conflict = { \n>   \"cherry-pick\", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,\n>   cherry_pick_verify_resolution,\n>   cherr_pick_skip,\n>   cherry_pick_abort,\n>   cherry_pick_post_resolution\n> };\n> \n> This information is passed to report_conflict(), which takes care of\n> user intervention.  The user can do whatever she wants and then ask\n> the sequencer to \"continue\", \"skip\" or \"abort\":\n\nRight, although I think:\n\n  $ git cheery-pick some-sha1\n  Conflict needs to be fixed now!\n\n  $ git skip\n\nshould give an error message about the current conflict not being a step \nof a larger process. That is, you can always \"continue\" or \"abort\", but \nyou can only \"skip\" if there's something to skip to, even if it's only the \nhigher-order sequence reporting that it's completed successfully.\n\n> > Where the sequencer-level conflict nests around the cherry-pick-level \n> > conflict, and the generic \"continue\" completes things from the inside out.\n> \n> Right.  And then the sequencer fires the appropriate callback and\n> returns control to the parent command.  More notes:\n> \n> >  - cherry-pick can save whatever it needs to in its state file; that's \n> >    its business, and the semantics here don't have to interact with other \n> >    commands, because report_conflict() has taken care of interaction with \n> >    other commands\n> \n> At the end of a merge for example, the MERGE_MSG needs to be retrieved\n> to create a new merge commit.  The sequencer des not need to know\n> anything about this, since this is specific to 'merge'.\n\nRight. And remove_branch_state() wouldn't even need to know about \nMERGE_MSG like it does now, because that would be handled by aborting any \nin-progress merge.\n\n> >  - arbitrary code can determine that you're in the middle of resolving \n> >    some conflict, that the resolution of that conflict is about doing\n> >    something to your current branch, and how to abort what you're doing,\n> >    and how to finish it\n> \n> Any arbitrary code simply has to ask the sequencer about the state of\n> the intermediate files that report_conflict() uses.  They don't have\n> to worry about command-specific intermediate files.\n\nRight.\n\n> >  - the same code gets run after the conflict has been resolved that would \n> >    have been run immediately if the merge went smoothly\n> \n> Using these callbacks, there is no need for if-else ugliness inside\n> the specific command to decide what to do next.\n> \n> I suppose we can call this  idea a \"generic conflict handler\".  I like\n> it very  much, and  I'll definitely  include this as  part of  my GSoC\n> work.  Thanks for taking the time to explain it in such detail :)\n\nYou're welcome. Thanks for proposing to actually implement it. :)\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"165200","messageId":"20110405185929.GA25644@kytes","threadId":"26980","inReplyTo":"alpine.LNX.2.00.1104051354500.14365@iabervon.org","subject":"Re: [GSoC 2011] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-05T18:59:32Z","receivedAt":"2011-04-05T18:59:32Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Daniel,\n\nDaniel Barkalow writes:\n> On Tue, 5 Apr 2011, Ramkumar Ramachandra wrote:\n> > Excellent.  The crux of the idea: The sequencer should serve as the\n> > entry/ exit point for Git when any operation requires user\n> > intervention to proceed.\n> \n> I'm a bit surprised by the idea of calling that \"the sequencer\" (rather \n> than having \"the sequencer\" be a command), but I actually think you're \n> entirely right to do so. Be sure to be very explicit about that, though, \n> because people will probably start with the wrong idea of what you're \n> proposing otherwise.\n\nAh.  I'll make sure to word it unambiguously in the proposal, and link\nto this thread :)\n\n> > For this, it should have information about\n> > how we got to this point, and how to proceed after the user\n> > intervention is complete; this information is contained in:\n> > \n> > > cherry_pick_conflict = { \n> > >   \"cherry-pick\", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,\n> > >   cherry_pick_verify_resolution,\n> > >   cherry_pick_abort,\n> > >   cherry_pick_post_resolution\n> > > };\n> > \n> > Wait -- isn't it missing a skip callback?\n> \n> I think \"skip\" is actually: abort the lowest-level conflict and continue \n> the next-level conflict. If you're doing a rebase, and the rebase is doing \n> a \"pick\", and the pick got a conflict, --skip means that you abort the \n> pick (to get back to the state where the earlier commits have been picked \n> but this one hasn't been started, followed by having the rebase continue \n> with what it was going to do after the pick completed.\n> \n> So I don't think you need a \"skip\" callback, as long as you've untangled \n> the levels cleanly and get the nesting support right.\n\nOkay.  I'm not yet entirely clear about this yet, but I think it\nshould be sorted out during implementation.\n\n> > cherry_pick_conflict = { \n> >   \"cherry-pick\", APPLIES_TO_CURRENT_BRANCH | IN_MIDDLE_OF_COMMIT,\n> >   cherry_pick_verify_resolution,\n> >   cherr_pick_skip,\n> >   cherry_pick_abort,\n> >   cherry_pick_post_resolution\n> > };\n> > \n> > This information is passed to report_conflict(), which takes care of\n> > user intervention.  The user can do whatever she wants and then ask\n> > the sequencer to \"continue\", \"skip\" or \"abort\":\n> \n> Right, although I think:\n> \n>   $ git cheery-pick some-sha1\n>   Conflict needs to be fixed now!\n> \n>   $ git skip\n> \n> should give an error message about the current conflict not being a step \n> of a larger process. That is, you can always \"continue\" or \"abort\", but \n> you can only \"skip\" if there's something to skip to, even if it's only the \n> higher-order sequence reporting that it's completed successfully.\n\nRight, got it.\n\n-- Ram\n"},{"id":"165205","messageId":"20110405200008.GC25644@kytes","threadId":"26980","inReplyTo":"20110403172054.GA10220@kytes","subject":"[GSoC 2011 v2] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-05T20:00:10Z","receivedAt":"2011-04-05T20:00:10Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nThanks for all the feedback on the first iteration!\n\nThis iteration of the proposal has already been submitted via the\nMelange interface. More comments/ feedback are always welcome.\n\n======================================================================\nProject Proposal: Git Sequencer\nStudent: Ramkumar Ramachandra\nMentor: Christian Couder\n\n== The Objective ==\n\nExtend 'git cherry-pick' with '--continue', '--abort', and '--skip'\nfeatures.  This will ultimately be used to write git-sequencer, a new\nbuiltin command.  The sequencer will provide a uniform interface over\nwhich existing commands like 'rebase', 'rebase -i' and 'am' can be\nre-implented.  This should give the commands more functionality,\nimprove their error handling, and make them faster.  The project can\nonly be considered successful if most of the code written gets merged\ninto upstream.\n\nThe Git Sequencer was a 2008 GSoC project as well; unfortunately most\nof the code did not get merged into git.git.  The learning from all\nthat work should serve as a huge headstart this year [1].\n\n=== The Plan ===\n\n1. Extend 'cherry-pick' with '--continue', '--abort', and '--skip'\nfeatures, so that it works like (a subset of) the current\n'git-rebase--interactive.sh'.  This will require patching\n'builtin/revert.c' in place, and merging it immediately.  I plan to\nroughly follow the road laid out by Christian's 2010 series [2].\n\n1.1. Factor out all calls to 'die' with 'return error' so so that we\ncan pause the entire process when a commit doesn't apply\nautomatically.\n\n1.2. Create and populate TODO and DONE files, similar to the one that\n'git-rebase--interactive.sh' creates.  For now, it should simply give\nus information about why a 'cherry-pick' failed.\n\n1.3. Port selective tests from the current 't3404' to make sure that\nTODO and DONE are populated correctly; \"stop on conflicting pick\" is a\ngood candidate.\n\n1.4. Decouple the 'revert' functionality from the 'cherry-pick'\nfunctionality in 'revert.c'.  Implement '--abort' for 'cherry-pick'\nand port \"abort\" test from 't3404'.\n\n1.5. Implement parsing the TODO and DONE files into suitable data\nstructures.  Derive inspiration from the code written in 2008 to do\nthis [3].\n\n1.6. Implement '--continue' and '--skip', and write suitable tests.\n\n2. Build a sequencer so that just has cherry-picking functionality.\nThis mostly involves moving code written in (1) around, and crafting a\ngeneral API for handling conflicts.\n\n2.1. Factor out the 'cherry-pick' code from 'revert.c' into a new\n'builtin/sequencer.c'.\n\n2.2. Write an API for handling conflicts, so that the sequencer is\nultimate entry/ exit point for all user intervention in a multi-step\nprocess [4].\n\n2.3. Implement a fresh 'cherry-pick.c' on top of the sequencer.  Make\nsure that all the existing tests pass.\n\n2.4 [Optional] Patch 'builtin/merge.c' to use the conflict handler in\nthe sequencer.\n\n3. Extend the sequencer to accomodate the functionality provided by\n'rebase -i'.\n\n3.1. Parse commands like 'execute', 'reword', 'squash', and 'fixup'\nthat are specific to interactive rebasing.  Carefully implement the\nfunctionality for each of these keywords in a step-wise manner.\n\n3.2. [Optional] Port the '--preserve-merges' option of 'rebase' to the\nsequencer.  Port relevant tests from 't3409'.\n\n4. [Optional] Lib'ify the sequncer. Modify the API to\ninclude rebase-related functionality.  Write 'rebase.c' as a bunch of\nAPI calls to the sequencer.  Make sure that the existing tests pass.\n\n5. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the\nsequncer: 'am.c'.  Bulk of this should be mbox parsing code.  Make sure\nthat all existing tests pass.\n\n[Optional] should be read as \"If time permits\"\n\n== Timeline ===\n\n- Before mid june: (1) should be implemented, and a series should be\n  sent out to the list for review.\n\n- Before midterm evaluation: (1) should be merged, and an\n  implementation of (2) should be sent to the list.\n\n- Before the end of July: (2) should be merged, and an implementation\n  of (3) should be sent to the list.\n\n- Before final evaluation: (3) should be merged.\n\n== Who am I? ==\n\nI'm Ramkumar Ramachandra, and I first started contributing to git.git\nin January 2010.  Apart from doing fast-import and remote helper\nrelated work last year, I also authored and merged svnrdump into\nSubversion trunk in the same period.\n\n== Notes ==\n\n[1]: http://repo.or.cz/w/git/sbeyer.git\n[2]: http://thread.gmane.org/gmane.comp.version-control.git/162183\n[3]: http://article.gmane.org/gmane.comp.version-control.git/162198\n[4]: http://thread.gmane.org/gmane.comp.version-control.git/170834\n======================================================================\n\nThanks for reading.\n\n-- Ram\n"},{"id":"165251","messageId":"BANLkTim5B1PGHr+TKGFaekywUh9r6K_Htg@mail.gmail.com","threadId":"26980","inReplyTo":"20110405200008.GC25644@kytes","subject":"Re: [GSoC 2011 v2] Git Sequencer","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2011-04-06T08:11:47Z","receivedAt":"2011-04-06T08:11:47Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nOn Tue, Apr 5, 2011 at 10:00 PM, Ramkumar Ramachandra\n<artagnon@gmail.com> wrote:\n>\n> 2. Build a sequencer so that just has cherry-picking functionality.\n> This mostly involves moving code written in (1) around, and crafting a\n> general API for handling conflicts.\n>\n> 2.1. Factor out the 'cherry-pick' code from 'revert.c' into a new\n> 'builtin/sequencer.c'.\n>\n> 2.2. Write an API for handling conflicts, so that the sequencer is\n> ultimate entry/ exit point for all user intervention in a multi-step\n> process [4].\n\nThis one may be more difficult than we can guess right now, and it is\nlinked with 2.4 that is optional too. And I wouldn't like you to spend\ntoo much time on it if it appears to be quite difficult. So I'd\nsuggest that you switch 2.2 and 2.3 and make the new 2.3 optional.\n\n> 2.3. Implement a fresh 'cherry-pick.c' on top of the sequencer.  Make\n> sure that all the existing tests pass.\n>\n> 2.4 [Optional] Patch 'builtin/merge.c' to use the conflict handler in\n> the sequencer.\n>\n> 3. Extend the sequencer to accomodate the functionality provided by\n> 'rebase -i'.\n\ns/accomodate/accommodate/\n\n> 3.1. Parse commands like 'execute', 'reword', 'squash', and 'fixup'\n> that are specific to interactive rebasing.  Carefully implement the\n> functionality for each of these keywords in a step-wise manner.\n\nI think that here you could add something like:\n\n3.2. Make 'rebase -i' use the sequencer when '--preserve-merges'\noption is not used.\n\n> 3.2. [Optional] Port the '--preserve-merges' option of 'rebase' to the\n> sequencer.  Port relevant tests from 't3409'.\n\n> 4. [Optional] Lib'ify the sequncer.\n\ns/sequncer/sequencer/\n\n> Modify the API to\n> include rebase-related functionality.  Write 'rebase.c' as a bunch of\n> API calls to the sequencer.  Make sure that the existing tests pass.\n>\n> 5. [Optional] Re-implement 'git-am.sh' as a thin wrapper over the\n> sequncer: 'am.c'.\n\ns/sequncer/sequencer/\n\n> Bulk of this should be mbox parsing code.  Make sure\n> that all existing tests pass.\n>\n> [Optional] should be read as \"If time permits\"\n\nOtherwise I like it very much.\n\nThanks,\nChristian.\n"},{"id":"165252","messageId":"20110406090115.GA9210@kytes","threadId":"26980","inReplyTo":"BANLkTim5B1PGHr+TKGFaekywUh9r6K_Htg@mail.gmail.com","subject":"Re: [GSoC 2011 v2] Git Sequencer","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-06T09:01:19Z","receivedAt":"2011-04-06T09:01:19Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Christian,\n\nChristian Couder writes:\n> On Tue, Apr 5, 2011 at 10:00 PM, Ramkumar Ramachandra\n> <artagnon@gmail.com> wrote:\n> >\n> > 2. Build a sequencer so that just has cherry-picking functionality.\n> > This mostly involves moving code written in (1) around, and crafting a\n> > general API for handling conflicts.\n> >\n> > 2.1. Factor out the 'cherry-pick' code from 'revert.c' into a new\n> > 'builtin/sequencer.c'.\n> >\n> > 2.2. Write an API for handling conflicts, so that the sequencer is\n> > ultimate entry/ exit point for all user intervention in a multi-step\n> > process [4].\n> \n> This one may be more difficult than we can guess right now, and it is\n> linked with 2.4 that is optional too. And I wouldn't like you to spend\n> too much time on it if it appears to be quite difficult. So I'd\n> suggest that you switch 2.2 and 2.3 and make the new 2.3 optional.\n> \n> [...]\n\nOkay.  Fixed other nits as well.\n\nThanks.\n\n-- Ram\n"}]}