{"thread":{"id":"48070","subject":"[RFC][GSoC] Project proposal: convert interactive rebase to C","startedAt":"2018-03-17T19:14:49Z","lastAt":"2018-03-25T00:43:35Z","messageCount":8,"participants":["Alban Gruin","Christian Couder","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"342043","messageId":"2564070.6kGkNsz7Qa@andromeda","threadId":"48070","inReplyTo":null,"subject":"[RFC][GSoC] Project proposal: convert interactive rebase to C","fromName":"Alban Gruin","fromEmail":"alban.gruin@gmail.com","sentAt":"2018-03-17T19:14:29Z","receivedAt":"2018-03-17T19:14:49Z","isPatch":false,"sender":{"key":"alban.gruin@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6310153?v=4"},"body":"Hi,\n\nhere is my first draft of my proposal for the GSoC, about the \"convert \ninteractive rebase to C\" project. Any feedback is welcome :)\n\n---\nABSTRACT\ngit is a modular source control management software, and all of its \nsubcommands are programs on their own. A lot of them are written in C, but a \ncouple of them are shell or Perl scripts. This is the case of git-rebase--\ninteractive (or interactive rebase), which is a shell script. Rewriting it in \nC would improve its performance, its portability, and maybe its robustness.\n\n\nABOUT `git-rebase{,--interactive}`\n\ngit-rebase allows to re-apply changes on top of another branch. For instance, \nwhen a local branch and a remote branch have diverged, git-rebase can re-unify \nthem, applying each change made on the local branch on top of the remote \nbranch.\n\ngit-rebase--interactive is used to reorganize commits by reordering, \nrewording, or squashing them. To achieve this purpose, git opens the list of \ncommits to be modified in a text editor (hence the interactivity), as well as \nthe actions to be performed for each of them.\n\n\nPROJECT GOALS\n\nThe goal of this project is to rewrite git-rebase--interactive in C as it has \nbeen discussed on the git mailing list[1], for multiple reasons :\n\nPerformance improvements\nShell scripts are inherently slow. That’s because each command is a program by \nitself. So, for each command, the shell interpreter has to spawn a new process \nand to load a new program.\n\nThose commands can be other git commands. Sometimes, they are wrappers to call \ninternal C functions (eg. git-rebase--helper), something shell scripts can’t \ndo natively. These wrappers basically parse the parameters, then start the \nappropriate function, which is\nobviously slower than just calling a function from C.\n\nOther commands can be POSIX utilities (eg. sed, cut, etc.). They have their \nown problems (speed aside), namely portability.\n\nPortability improvements\nShell scripts often relies on many of those POSIX utilities, which are not \nnecessarily natively available on all platforms (most notably, Windows), or \nmay have more or less features depending on the implementation.\n\n\nAPPROXIMATIVE TIMELINE\n\nCommunity bonding — April 23, 2018 – May 14, 2018\nDuring the community bonding, I would like to dive into git’s codebase, and to \nunderstand what git-rebase--interactive does under the hood. At the same time, \nI’d communicate with the community and my mentor, seeking for clarifications, \nand asking questions about how things should or should not be done.\n\nWeeks 1 & 2 — May 14, 2018 – May 28, 2018\nFirst, I would refactor --preserve-merges in its own shell script, as \ndescribed in Dscho’s email.\n\nWeeks 3 & 4 — May 18, 2018 – June 11, 2018\nThen, I would start to rewrite git-rebase--interactive, and get rid of git-\nrebase--helper.\n\nWeeks 5 to 9 — June 11, 2018 – July 15, 2018\nDuring this period, I would continue to rewrite git-rebase--interactive.\n\nWeeks 10 & 11 — July 16, 2018 – July 29, 2018\nIn the second half of July, I would look for bugs in the new code, test it, \nand improve its coverage.\n\nWeeks 12 — July 30, 2018 – August 5, 2018\nIn the last week, I would polish the code where needed, in order to improve \nfor performance or to make the code more readable.\n\n\nABOUT ME\n\nMy name is Alban Gruin, I am an undergraduate at the Paul Sabatier University \nin Toulouse, France, where I have been studying Computer Sciences for the past \nyear and a half. My timezone currently is UTC+01:00, but will be UTC+02:00 \nstarting from March 25th, because of the daylight saving time in Europe.\n\nI have been programming in C for the last 5 years. I learned using freely \navailable resources online, and by attending class ever since last year.\n\nI am also quite familiar with shell scripts, and I have been using git for the \nlast 3 years. \n\nMy e-mail address is alban <dot> gruin <at> gmail <dot> com. My IRC nick is \nabngrn.\n\nMy micro-project was \"userdiff: add built-in pattern for golang\"[2][3].\n\n---\n\nYou can find the Google Doc version here[4].\n\nRegards,\nAlban Gruin\n\n[1] https://public-inbox.org/git/alpine.DEB.2.20.1609021432070.129229@virtualbox/\n[2] https://public-inbox.org/git/20180228172906.30582-1-alban.gruin@gmail.com/\n[3] https://git.kernel.org/pub/scm/git/git.git/commit/?id=1dbf0c0a\n[4] https://docs.google.com/document/d/1Jx0w867tVAht7QI1_prieiXg_iQ_nTloOyaIIOnm85g/edit?usp=sharing\n\n\n"},{"id":"342048","messageId":"CAP8UFD27rWmp5+y8dHgqag=uuU1G_kRFvv1x_XqXk+ur8-HauQ@mail.gmail.com","threadId":"48070","inReplyTo":"2564070.6kGkNsz7Qa@andromeda","subject":"Re: [RFC][GSoC] Project proposal: convert interactive rebase to C","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-17T20:29:25Z","receivedAt":"2018-03-17T20:29:32Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nOn Sat, Mar 17, 2018 at 8:14 PM, Alban Gruin <alban.gruin@gmail.com> wrote:\n>\n> Weeks 3 & 4 — May 18, 2018 – June 11, 2018\n> Then, I would start to rewrite git-rebase--interactive, and get rid of git-\n> rebase--helper.\n\nUsually to rewrite a shell script in C, we first rewrite shell\nfunctions into option arguments in a C builtin helper and make the\nschell script call the builtin helper (instead of the original shell\nfunctions). Eventually when the shell script is mostly only calling\nthe builtin helper, we add what is needed into the builtin helper and\nwe rename it to make it fully replace the shell script.\n\nSee for example 0cce4a2756 (rebase -i -x: add exec commands via the\nrebase--helper, 2017-12-05) or b903674b35 (bisect--helper:\n`is_expected_rev` & `check_expected_revs` shell function in C,\n2017-09-29). These examples show that we can do step by step rewrites.\n\nI would suggest planning to use the same approach, and describing in\nyour proposal which shell functions you would like to rewrite into the\nC builtin helper in which order, before planning to fully replace the\ncurrent git-rebase--interactive.\n\nThanks,\nChristian.\n"},{"id":"342338","messageId":"nycvar.QRO.7.76.6.1803201720090.55@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"48070","inReplyTo":"2564070.6kGkNsz7Qa@andromeda","subject":"Re: [RFC][GSoC] Project proposal: convert interactive rebase to C","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-20T16:29:28Z","receivedAt":"2018-03-20T16:30:00Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Alban,\n\nthank you for your proposal!\n\nI will only comment on the parts that I feel could use improvement, the\nrest is fine by me.\n\nOn Sat, 17 Mar 2018, Alban Gruin wrote:\n\n> APPROXIMATIVE TIMELINE\n> \n> Community bonding — April 23, 2018 – May 14, 2018\n> During the community bonding, I would like to dive into git’s codebase,\n> and to understand what git-rebase--interactive does under the hood. At\n> the same time, I’d communicate with the community and my mentor, seeking\n> for clarifications, and asking questions about how things should or\n> should not be done.\n> \n> Weeks 1 & 2 — May 14, 2018 – May 28, 2018\n> First, I would refactor --preserve-merges in its own shell script, as\n> described in Dscho’s email.\n\nCould you go into detail a bit here? Like, describe what parts of the\ngit-rebase--interactive.sh script would need to be duplicated, which ones\nwould be truly moved, etc\n\n> Weeks 3 & 4 — May 18, 2018 – June 11, 2018\n> Then, I would start to rewrite git-rebase--interactive, and get rid of git-\n> rebase--helper.\n\nI think this is a bit premature, as the rebase--helper would not only\nserve the --interactive part, but in later conversions also --am and\n--merge, and only in the very end, when git-rebase.sh gets converted,\nwould we be able to simply rename the rebase--helper to rebase.\n\nAlso, I have a hunch that there is actually almost nothing left to rewrite\nafter my sequencer improvements that made it into Git v2.13.0, together\nwith the upcoming changes (which are on top of the --recreate-merges patch\nseries, hence I did not send them to the mailing list yet) in\nhttps://github.com/dscho/git/commit/c261f17a4a3e\n\nSo I would like to see more details here... ;-)\n\n> Weeks 5 to 9 — June 11, 2018 – July 15, 2018\n> During this period, I would continue to rewrite git-rebase--interactive.\n\nIt would be good if the proposal said what parts of the conversion are\ntricky, to merit spending a month on them.\n\n> Weeks 10 & 11 — July 16, 2018 – July 29, 2018\n> In the second half of July, I would look for bugs in the new code, test it, \n> and improve its coverage.\n\nAs I mentioned in a related mail, the test suite coverage would be most\nsensibly extended *before* starting to rewrite code in C, as it helps\ncatching bugs early and avoids having to merge buggy code that needs to be\nfixed immediately.\n\n> Weeks 12 — July 30, 2018 – August 5, 2018\n> In the last week, I would polish the code where needed, in order to\n> improve for performance or to make the code more readable.\n\nThank you for sharing this draft with us!\nJohannes"},{"id":"342399","messageId":"46112353.MVnaADNoVi@andromeda","threadId":"48070","inReplyTo":"nycvar.QRO.7.76.6.1803201720090.55@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: [RFC][GSoC] Project proposal: convert interactive rebase to C","fromName":"Alban Gruin","fromEmail":"alban.gruin@gmail.com","sentAt":"2018-03-21T11:04:28Z","receivedAt":"2018-03-21T11:04:42Z","isPatch":false,"sender":{"key":"alban.gruin@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6310153?v=4"},"body":"Hi Johannes,\n\nLe mardi 20 mars 2018 17:29:28 CET, vous avez écrit :\n> > Weeks 1 & 2 — May 14, 2018 – May 28, 2018\n> > First, I would refactor --preserve-merges in its own shell script, as\n> > described in Dscho’s email.\n> \n> Could you go into detail a bit here? Like, describe what parts of the\n> git-rebase--interactive.sh script would need to be duplicated, which ones\n> would be truly moved, etc\n> \n\nIt would lead to duplicate a good chunk of git_rebase__interactive(), \napparently. The moved parts would be everything in `if test t = \n\"$preserve_merges\"; then …; fi` statements. That is, about 50 lines of shell \ncode.\n\nJudging by that, beginning by that is probably not the right thing to do. \nAlso, somebody is already working on that[1]. \n\n> > Weeks 3 & 4 — May 18, 2018 – June 11, 2018\n> > Then, I would start to rewrite git-rebase--interactive, and get rid of\n> > git-\n> > rebase--helper.\n> \n> I think this is a bit premature, as the rebase--helper would not only\n> serve the --interactive part, but in later conversions also --am and\n> --merge, and only in the very end, when git-rebase.sh gets converted,\n> would we be able to simply rename the rebase--helper to rebase.\n> \n\nYes, Christian Couder told me that it would not be a good methodology too.\n\n> Also, I have a hunch that there is actually almost nothing left to rewrite\n> after my sequencer improvements that made it into Git v2.13.0, together\n> with the upcoming changes (which are on top of the --recreate-merges patch\n> series, hence I did not send them to the mailing list yet) in\n> https://github.com/dscho/git/commit/c261f17a4a3e\n\nOne year ago, you said[2] that converting this script \"will fill up 3 month, \nvery easily\". Is this not accurate anymore?\n\n> \n> So I would like to see more details here... ;-)\n\nYep, I’m working on that. \n\n> > Weeks 5 to 9 — June 11, 2018 – July 15, 2018\n> > During this period, I would continue to rewrite git-rebase--interactive.\n> \n> It would be good if the proposal said what parts of the conversion are\n> tricky, to merit spending a month on them.\n> \n> > Weeks 10 & 11 — July 16, 2018 – July 29, 2018\n> > In the second half of July, I would look for bugs in the new code, test\n> > it,\n> > and improve its coverage.\n> \n> As I mentioned in a related mail, the test suite coverage would be most\n> sensibly extended *before* starting to rewrite code in C, as it helps\n> catching bugs early and avoids having to merge buggy code that needs to be\n> fixed immediately.\n\nMakes sense.\n\n> \n> > Weeks 12 — July 30, 2018 – August 5, 2018\n> > In the last week, I would polish the code where needed, in order to\n> > improve for performance or to make the code more readable.\n> \n> Thank you for sharing this draft with us!\n> Johannes\n\nI’ll send a new draft as soon as possible (hopefully this afternoon).\n\nThank you for your enthousiasm :)\nAlban\n\n[1] https://public-inbox.org/git/20180320204507.12623-1-wink@saville.com/\n[2] https://public-inbox.org/git/alpine.DEB.\n2.20.1703231827060.3767@virtualbox/\n\n\n"},{"id":"342487","messageId":"nycvar.QRO.7.76.6.1803220043180.77@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"48070","inReplyTo":"46112353.MVnaADNoVi@andromeda","subject":"Re: [RFC][GSoC] Project proposal: convert interactive rebase to C","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-21T23:51:17Z","receivedAt":"2018-03-21T23:51:25Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Alban,\n\nOn Wed, 21 Mar 2018, Alban Gruin wrote:\n\n> Le mardi 20 mars 2018 17:29:28 CET, vous avez écrit :\n> \n> > Also, I have a hunch that there is actually almost nothing left to\n> > rewrite after my sequencer improvements that made it into Git v2.13.0,\n> > together with the upcoming changes (which are on top of the\n> > --recreate-merges patch series, hence I did not send them to the\n> > mailing list yet) in\n> > https://github.com/dscho/git/commit/c261f17a4a3e\n> \n> One year ago, you said[2] that converting this script \"will fill up 3\n> month, very easily\". Is this not accurate anymore?\n\nLet me read that mail ;-)\n\n*goes and reads*\n\nWell, I was talking about two different aspects to Ivan and to you. I\nshould have been clearer. So let me try again:\n\nTo convert `git-rebase--interactive.sh`, I think the most important part\nis to factor out the preserve-merges code into its own script. After that,\nthere is little I can think of (apart from support for --root, which a\nnot-yet-contributed patch in my sequencer-shears branch on\nhttps://github.com/dscho/git addresses) that still needs to be converted.\nFor somebody familiar with Git's source code, I would estimate one week\n(and therefore 3 weeks would be a realistic estimate :-)).\n\nCome to think of it, a better approach might be to leave the\npreserve-merges stuff in, and teach `git-rebase.sh` to call the sequencer\ndirectly for --interactive without --preserve-merges, then rename the\nscript to git-rebase--preserve.sh\n\nThe other aspect, the one I thought would take up to 3 months, easily, was\nto convert the entirety of rebase -i into C. That would entail also the\noption parsing, for which you would have to convert also git-rebase.sh\n(and if you do not convert git-rebase--am.sh and git-rebase--merge.sh\nfirst, you would then have to teach builtin/rebase.c to populate the\nenvironment variables expected by those shell scripts while spawning\nthem).\n\nI still think that the latter is too big a task for a single GSoC.\n\n> I’ll send a new draft as soon as possible (hopefully this afternoon).\n\nI look forward to reading it!\n\nCiao,\nJohannes"},{"id":"342546","messageId":"ef8552ba-edb0-ea10-29fd-32152adbc992@gmail.com","threadId":"48070","inReplyTo":"2564070.6kGkNsz7Qa@andromeda","subject":"Re: [RFC][GSoC] Project proposal: convert interactive rebase to C","fromName":"Alban Gruin","fromEmail":"alban.gruin@gmail.com","sentAt":"2018-03-22T22:03:30Z","receivedAt":"2018-03-22T22:03:46Z","isPatch":false,"sender":{"key":"alban.gruin@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6310153?v=4"},"body":"Hi,\n\nhere is my second draft of my proposal. As last time, any feedback is\nwelcome :)\n\nI did not write my phone number and address here for obvious reasons,\nbut they will be in the “about me” section of the final proposal.\n\nApart from that, do you think there is something to add?\n\n---\nABSTRACT\n\ngit is a modular source control management software, and all of its\nsubcommands are programs on their own. A lot of them are written in C,\nbut a couple of them are shell or Perl scripts. This is the case of\ngit-rebase--interactive (or interactive rebase), which is a shell\nscript. Rewriting it in C would improve its performance, its\nportability, and maybe its robustness.\n\n\nABOUT `git-rebase` and `git-rebase--interactive`\n\ngit-rebase allows to re-apply changes on top of another branch. For\ninstance, when a local branch and a remote branch have diverged,\ngit-rebase can re-unify them, applying each change made on the\nlocal branch on top of the remote branch.\n\ngit-rebase--interactive is used to reorganize commits by reordering,\nrewording, or squashing them. To achieve this purpose, git opens the\nlist of commits to be modified in a text editor (hence the\ninteractivity), as well as the actions to be performed for each of\nthem.\n\n\nPROJECT GOALS\n\nThe goal of this project is to rewrite git-rebase--interactive in C\nas it has been discussed on the git mailing list[1], for multiple\nreasons :\n\n\nPerformance improvements\nShell scripts are inherently slow. That’s because each command is a\nprogram by itself. So, for each command, the shell interpreter has to\nspawn a new process and to load a new program (with fork() and\nexec() syscalls), which is an expensive process.\n\nThose commands can be other git commands. Sometimes, they are\nwrappers to call internal C functions (eg. git-rebase--helper),\nsomething shell scripts can’t do natively. These wrappers basically\nparse the parameters, then start the appropriate function, which is\nobviously slower than just calling a function from C.\n\nOther commands can be POSIX utilities (eg. sed, cut, etc.). They\nhave their own problems (speed aside), namely portability.\n\nPortability improvements\nShell scripts often relies on many of those POSIX utilities, which are\nnot necessarily natively available on all platforms (most notably,\nWindows), or may have more or less features depending on the\nimplementation.\n\nAlthough C is not perfect portability-wise, it’s still better than\nshell scripts. For instance, the resulting binaries will not\nnecessarily depend on third-party programs or libraries.\n\n\nRISKS\n\nOf course, rewriting a piece of software takes time, and can lead to\nregressions (ie. new bugs). To mitigate that risk, I should understand\nwell the functions I want to rewrite, run tests on a regular basis and\nwrite new if needed, and of course discuss about my code with the\ncommunity during reviews.\n\n\nAPPROXIMATIVE TIMELINE\n\nCommunity bonding -- April 23, 2018 - May 14, 2018\nDuring the community bonding, I would like to dive into git’s\ncodebase, and to understand what git-rebase--interactive does under\nthe hood. At the same time, I’d communicate with the community and my\nmentor, seeking for clarifications, and asking questions about how\nthings should or should not be done.\n\nWeeks 1 & 2 -- May 14, 2018 - May 27, 2018\n/From May 14 to 18, I have exams at the university, so I won’t be able\nto work full time./\n\nI would search for edge cases not covered by current tests and write\nsome if needed.\n\nWeek 3 -- May 28, 2018 - June 3, 2018\nAt the same time, I would refactor --preserve-merges in its own\nshell script (as described in Dscho’s email[1]), if it has\nnot been deprecated or moved in the meantime. This operation is not\nreally tricky by itself, as --preserve-merges is about only 50 lines\nof code into git_rebase__interactive().\n\nWeeks 4 to 7 -- May 4, 2018 - July 1, 2018\nThen, I would start to incrementally rewrite\ngit-rebase--interactive.sh functions in C, and move them\ngit-rebase--helper.c (as in commits 0cce4a2756[2] (rebase -i\n-x: add exec commands via the rebase--helper) and b903674b35[3]\n(bisect--helper: `is_expected_rev` & `check_expected_revs` shell\nfunction in C)).\n\nThere is a lot of functions into git-rebase--interactive.sh to\nrewrite. Most of them are small, and some of them are even wrappers\nfor a single command (eg. is_merge_commit()), so they shouldn’t be\nreally problematic.\n\nA couple of them are quite long (eg. pick_one()), and will probably\nbe even longer once rewritten in C due to the low-level nature of the\nlanguage. They also tend to depend a lot on other smaller functions.\n\nThe plan here would be to start rewriting the smaller functions when\napplicable (ie. they’re not a simple command wrapper) before\nworking on the biggest of them.\n\nWeek 8 -- July 2, 2018 - July 8, 2018\nWhen all majors functions from git-rebase--interactive.sh have been\nrewritten in C, I would retire the script in favor of a builtin.\n\nWeeks 9 & 10 -- July 9, 2018 - July 22, 2018\nI plan to spend theses two weeks to improve the code coverage where\nneeded.\n\nWeeks 11 & 12 -- July 23, 2018 - August 5, 2018\nIn the last two weeks, I would polish the code where needed, in order\nto improve its performance or to make it more readable.\n\nI also plan to use this time as a backup if I am late on my planning.\n\n\nIF TIME PERMITS\n\n * I see there is a known breakage in test t3404. Maybe I could\n   investigate and try to fix it.\n * Add an option to include the patch in the message of commits to be\n   reworded, as proposed by Ævar Arnfjörð Bjarmason[4].\n\n\nABOUT ME\n\nMy name is Alban Gruin. I am an undergraduate at the Paul Sabatier\nUniversity in Toulouse, France, where I have been studying Computer\nSciences for a year and a half. My timezone currently is UTC+01:00,\nbut will be UTC+02:00 starting from March 25th, because of the\ndaylight saving time in Europe.\n\nI have been programming in C for the last 5 years. I learned using\nfreely available resources online, and by attending class ever since\nlast year.\n\nI am also quite familiar with shell scripts, and I have been using\ngit for the last 3 years.\n\nMy e-mail address is alban <dot> gruin <at> gmail <dot> com, my IRC\nnick is abngrn.\n\nMy micro-project was \"userdiff: add built-in pattern for\ngolang\"[5][6].\n\n---\n\nYou can find the Google Doc version here[7].\n\n\nRegards,\nAlban Gruin\n\n[1]\nhttps://public-inbox.org/git/alpine.DEB.2.20.1609021432070.129229@virtualbox/\n[2] https://git.kernel.org/pub/scm/git/git.git/commit/?id=0cce4a2756\n[3] https://git.kernel.org/pub/scm/git/git.git/commit/?id=b903674b35\n[4] https://public-inbox.org/git/87in9ucsbb.fsf@evledraar.gmail.com/\n[5]\nhttps://public-inbox.org/git/20180228172906.30582-1-alban.gruin@gmail.com/\n[6] https://git.kernel.org/pub/scm/git/git.git/commit/?id=1dbf0c0a\n[7]\nhttps://docs.google.com/document/d/1Jx0w867tVAht7QI1_prieiXg_iQ_nTloOyaIIOnm85g/edit?usp=sharing\n"},{"id":"342752","messageId":"CAP8UFD0eU150p2puqdgkYQuZpZ8Yzxo9wMw2PAVUrwy+8zUZ2g@mail.gmail.com","threadId":"48070","inReplyTo":"ef8552ba-edb0-ea10-29fd-32152adbc992@gmail.com","subject":"Re: [RFC][GSoC] Project proposal: convert interactive rebase to C","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-24T07:43:18Z","receivedAt":"2018-03-24T07:43:24Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nOn Thu, Mar 22, 2018 at 11:03 PM, Alban Gruin <alban.gruin@gmail.com> wrote:\n> Hi,\n>\n> here is my second draft of my proposal. As last time, any feedback is\n> welcome :)\n>\n> I did not write my phone number and address here for obvious reasons,\n> but they will be in the “about me” section of the final proposal.\n>\n> Apart from that, do you think there is something to add?\n\nPlease take a look at the comments that have been made on other's\nproposal. Many proposals could be improved in the same way and it is a\nbit annoying for us to repeat the same things many times.\n\n[...]\n\n> The goal of this project is to rewrite git-rebase--interactive in C\n> as it has been discussed on the git mailing list[1], for multiple\n> reasons :\n\nIn general when the project or some issues related to the project have\nalready been worked on or discussed on the mailing list, it is a good\nthing to summarize those discussions and link to them in your\nproposal. It shows that you want to take the time to gather existing\ninformation, to understand that information and to take it into\naccount in your proposal, and it can also make your proposal easier to\nread and understand.\n\nMore specifically your proposal has some links which is nice, but I\nthink it would be better if it summarized a bit more what the links\ncontain.\n\n[...]\n\n> Weeks 1 & 2 -- May 14, 2018 - May 27, 2018\n> /From May 14 to 18, I have exams at the university, so I won’t be able\n> to work full time./\n>\n> I would search for edge cases not covered by current tests and write\n> some if needed.\n>\n> Week 3 -- May 28, 2018 - June 3, 2018\n> At the same time, I would refactor --preserve-merges in its own\n> shell script (as described in Dscho’s email[1]), if it has\n> not been deprecated or moved in the meantime.\n\nHere for example it is better if we could get a better idea about how\nyou plan to do it without having to read Dscho's email.\n\n> This operation is not\n> really tricky by itself, as --preserve-merges is about only 50 lines\n> of code into git_rebase__interactive().\n>\n> Weeks 4 to 7 -- May 4, 2018 - July 1, 2018\n> Then, I would start to incrementally rewrite\n> git-rebase--interactive.sh functions in C, and move them\n> git-rebase--helper.c (as in commits 0cce4a2756[2] (rebase -i\n> -x: add exec commands via the rebase--helper) and b903674b35[3]\n> (bisect--helper: `is_expected_rev` & `check_expected_revs` shell\n> function in C)).\n\nI know what you mean but I would still appreciate if you could summarize it.\n\n> There is a lot of functions into git-rebase--interactive.sh to\n> rewrite. Most of them are small, and some of them are even wrappers\n> for a single command (eg. is_merge_commit()), so they shouldn’t be\n> really problematic.\n>\n> A couple of them are quite long (eg. pick_one()), and will probably\n> be even longer once rewritten in C due to the low-level nature of the\n> language. They also tend to depend a lot on other smaller functions.\n>\n> The plan here would be to start rewriting the smaller functions when\n> applicable (ie. they’re not a simple command wrapper) before\n> working on the biggest of them.\n>\n> Week 8 -- July 2, 2018 - July 8, 2018\n> When all majors functions from git-rebase--interactive.sh have been\n> rewritten in C, I would retire the script in favor of a builtin.\n>\n> Weeks 9 & 10 -- July 9, 2018 - July 22, 2018\n> I plan to spend theses two weeks to improve the code coverage where\n> needed.\n>\n> Weeks 11 & 12 -- July 23, 2018 - August 5, 2018\n> In the last two weeks, I would polish the code where needed, in order\n> to improve its performance or to make it more readable.\n\nWe like to have big improvements be split into batches of patches,\nalso called patch series, and polishing of the first batches happening\nas soon as possible so that they can be ready to be merged soon. The\npatch series that are sent to the mailing list often need a number of\nround of reviews and improvements which can take a long time, so it is\nbetter if this process starts as soon as possible.\n\nThanks.\n"},{"id":"342856","messageId":"cfadb1da-4d8b-ff70-5e5c-7c2fa0b8950a@gmail.com","threadId":"48070","inReplyTo":"ef8552ba-edb0-ea10-29fd-32152adbc992@gmail.com","subject":"Re: [RFC v3][GSoC] Project proposal: convert interactive rebase to C","fromName":"Alban Gruin","fromEmail":"alban.gruin@gmail.com","sentAt":"2018-03-25T00:43:13Z","receivedAt":"2018-03-25T00:43:35Z","isPatch":false,"sender":{"key":"alban.gruin@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6310153?v=4"},"body":"Hi,\nthis is the first draft of my proposal.\n\n---\nABSTRACT\n\ngit is a modular source control management software, and all of its\nsubcommands are programs on their own. A lot of them are written in C,\nbut a couple of them are shell or Perl scripts. This is the case of\n=git-rebase--interactive= (or interactive rebase), which is a shell\nscript. Rewriting it in C would improve its performance, its\nportability, and maybe its robustness.\n\n\nABOUT `git-rebase` AND `git-rebase--interactive`\n\ngit-rebase allows to re-apply changes on top of another branch. For\ninstance, when a local branch and a remote branch have diverged,\ngit-rebase can re-unify them, applying each change made on the\nlocal branch on top of the remote branch.\n\ngit-rebase--interactive is used to reorganize commits by reordering,\nrewording, or squashing them. To achieve this purpose, =git= opens the\nlist of commits to be modified in a text editor (hence the\ninteractivity), as well as the actions to be performed for each of\nthem.\n\n\nPROJECT GOALS\n\nBack in 2016, Johannes Schindelin discussed[1] about retiring\ngit-rebase.sh (described here as a “hacky shell script”) in favor of\na builtin. He explained that, as it’s only a command-line parser for\ngit-rebase--am.sh, git-rebase--interactive.sh, and\ngit-rebase--merge.sh, these 3 scripts should be rewritten first.\n\nThe goal of this project is to rewrite git-rebase--interactive.sh in\nC, for multiple reasons :\n\nPerformance improvements\nShell scripts are inherently slow. When Johannes Schindelin began to\nrewrite some parts of git-rebase--interactive in C, its performance\nincreased from 3 to 5 times, depending on the platform[2].\n\nThat’s because each command is a program by itself. So, for each\ncommand, the shell interpreter has to spawn a new process and to load\na new program (with fork() and exec() syscalls), which is an\nexpensive process.\n\nThose commands can be other git commands. Sometimes, they are\nwrappers to call internal C functions (eg. git-rebase--helper),\nsomething shell scripts can’t do natively. These wrappers basically\nparse the parameters, then start the appropriate function, which is\nobviously slower than just calling a function from C.\n\nOther commands can be POSIX utilities (eg. sed, cut, etc.). They\nhave their own problems (speed aside), namely portability.\n\nPortability improvements\nShell scripts often relies on many of those POSIX utilities, which are\nnot necessarily natively available on all platforms (most notably,\nWindows), or may have more or less features depending on the\nimplementation.\n\nAlthough C is not perfect portability-wise, it’s still better than\nshell scripts. For instance, the resulting binaries will not\nnecessarily depend on third-party programs or libraries.\n\n\nRISKS\n\nOf course, rewriting a piece of software takes time, and can lead to\nregressions (ie. new bugs). To mitigate that risk, I should understand\nwell the functions I want to rewrite, run tests on a regular basis and\nwrite new if needed, and of course discuss about my code with the\ncommunity during reviews.\n\n\nAPPROXIMATIVE TIMELINE\n\nNormally, I would be able to work 35 to 40 hours a week. When I have\ncourses or exams at university, I could work between 20 and 25 hours a\nweek.\n\nCommunity bonding --- April 23, 2018 -- May 14, 2018\n/I’ll still have courses at the university during this period./\n\nDuring the community bonding, I would like to dive into git’s\ncodebase, and to understand what git-rebase--interactive does under\nthe hood. At the same time, I’d communicate with the community and my\nmentor, seeking for clarifications, and asking questions about how\nthings should or should not be done.\n\nWeeks 1 & 2 --- May 14, 2018 -- May 27, 2018\n/From May 14 to 18, I have exams at the university, so I won’t be able\nto work full time./\n\nI would search for edge cases not covered by current tests and write\nsome if needed.\n\nWeek 3 --- May 28, 2018 -- June 3, 2018\nAt the same time, I would refactor --preserve-merges in its own\nshell script, if it has not been deprecated or moved in the\nmeantime. Dscho explained that this would be the first step of the\nconversion[1]. This operation is not really tricky by itself,\nas --preserve-merges is about only 50 lines of code into\ngit_rebase__interactive(), plus some specific functions\n(eg. pick_one()).\n\nWeeks 4 to 7 --- June 4, 2018 -- July 1, 2018\nThen, I would start to incrementally rewrite\ngit-rebase--interactive.sh functions in C, and move them\ngit-rebase--helper.c. Newly-rewritten C functions are then\nassociated to command-line parameters to be able to use them from\nshell scripts.\n\nExamples of such conversion can be found in commits\n0cce4a2756[3] (rebase -i -x: add exec commands via the\nrebase--helper) and b903674b35[4] (bisect--helper:\n`is_expected_rev` & `check_expected_revs` shell function in C).\n\nThere is a lot of functions into git-rebase--interactive.sh to\nrewrite. Most of them are small, and some of them are even wrappers\nfor a single command (eg. do_next()), so they shouldn’t be really\nproblematic.\n\nA couple of them are quite long (eg. pick_one()), and will probably\nbe even longer once rewritten in C due to the low-level nature of the\nlanguage. They also tend to depend a lot on other smaller functions.\n\nThe plan here would be to start rewriting the smaller functions when\napplicable (ie. they’re not a simple command wrapper) before\nworking on the biggest of them.\n\nOf course, rewriting a function should not cause any breakage in the\ntest suite.\n\nWeek 8 --- July 2, 2018 -- July 8, 2018\nThen, I plan to polish the new code, in order to improve its\nperformance or to make it more readable.\n\nBenchmarking will be done using the existing script in t/perf.\n\nWeeks 9 & 10 --- July 9, 2018 -- July 22, 2018\nWhen all majors functions from git-rebase--interactive.sh have been\nrewritten in C, I would retire the script in favor of a builtin.\n\nWeeks 11 & 12 --- July 23, 2018 -- August 5, 2018\nIn the last two weeks, I would improve the code coverage where needed.\n\nI also plan to use this time as a backup if I am late on my planning.\n\n\nIF TIMES PERMITS\n\n * Add an option to include the patch in the message of commits to be\n   reworded, as proposed by Ævar Arnfjörð Bjarmason[5].\n\n\nCOMMUNICATING ABOUT MY WORK\n\nI will communicate every week about the state of my work on the\nmailing list.\n\n\nABOUT ME\n\nMy name is Alban Gruin. I am an undergraduate at the Paul Sabatier\nUniversity in Toulouse, France, where I have been studying Computer\nSciences for a year and a half. My timezone is UTC+02:00.\n\nI have been programming in C for the last 5 years. I learned using\nfreely available resources online, and by attending class ever since\nlast year.\n\nI am also quite familiar with shell scripts, and I have been using\ngit for the last 3 years.\n\nMy e-mail address is alban <dot> gruin <at> gmail <dot> com, my IRC\nnick is abngrn.\n\nMy micro-project was “userdiff: add built-in pattern for\ngolang”[6][7].\n---\n\nYou can find the Google Doc version here[8].\n\nRegards,\nAlban Gruin\n\n[1]\nhttps://public-inbox.org/git/alpine.DEB.2.20.1609021432070.129229@virtualbox/\n[2]\nhttps://public-inbox.org/git/cover.1472457609.git.johannes.schindelin@gmx.de/\n[3] https://git.kernel.org/pub/scm/git/git.git/commit/?id=0cce4a2756\n[4] https://git.kernel.org/pub/scm/git/git.git/commit/?id=b903674b35\n[5] https://public-inbox.org/git/87in9ucsbb.fsf@evledraar.gmail.com/\n[6]\nhttps://public-inbox.org/git/20180228172906.30582-1-alban.gruin@gmail.com/\n[7] https://git.kernel.org/pub/scm/git/git.git/commit/?id=1dbf0c0a\n[8]\nhttps://docs.google.com/document/d/1Jx0w867tVAht7QI1_prieiXg_iQ_nTloOyaIIOnm85g/edit?usp=sharing\n"}]}