{"thread":{"id":"45514","subject":"[GSoC] Proposal: turn git-add--interactive.perl into a builtin","startedAt":"2017-03-26T03:16:46Z","lastAt":"2017-04-02T22:51:06Z","messageCount":7,"participants":["Daniel Ferreira (theiostream)","Stefan Beller","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"315448","messageId":"CAEA2_RJEf4vjgcaux8a1KWh1-vxLLmv1--Vjf9wiEQoF+gVDtA@mail.gmail.com","threadId":"45514","inReplyTo":null,"subject":"[GSoC] Proposal: turn git-add--interactive.perl into a builtin","fromName":"Daniel Ferreira (theiostream)","fromEmail":"bnmvco@gmail.com","sentAt":"2017-03-26T03:15:11Z","receivedAt":"2017-03-26T03:16:46Z","isPatch":false,"sender":{"key":"bnmvco@gmail.com","avatar":"https://avatars.githubusercontent.com/u/735397?v=4"},"body":"Hi there. First of all, I'd like to thank all of the support up to now\nwith my microproject :). Here's a first draft of my proposal for\nGoogle Summer of Code '17, based on the \"Convert scripts to builtins\"\nidea. Please let me know what you think.\n\n---\n\nSYNOPSIS\nThere are many advantages to converting parts of git that are still\nscripts to C builtins, among which execution speed, improved\ncompatibility and code deduplication. This proposal aims to apply this\nto git-add--interactive, one of the most useful features of Git.\n\nFEASIBILITY\nMany git scripts have attracted attention for being turned into\nbuiltins. There is ongoing work on git-stash\n(https://public-inbox.org/git/20170321053135.thk77soxc4irxbqj@sigill.intra.peff.net/),\nand porting interactive rebase is one of the ideas for this edition of\nGSoC. Not as much attention, however, has been directed to\ngit-add--interactive.\n\nThere was only one discussion regarding the feasibility of its porting\n(https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1HLx+Q@mail.gmail.com/).\nIt resulted in a consensus that doing it would be a task too large –\nalthough interesting – for GSoC 2015 based on the amount of its lines\nof code. It is, however, only a few lines larger than\ngit-rebase--interactive, which has been considered an appropriate\nidea. As such, it looks like a possible project for three months of\nfull-time work.\n\nAside from the benefits cited above, turning git-add--interactive into\na builtin can reduce Git's dependency on Perl to the point where no\n\"common\" command would continue to rely on it.\n\nPROJECTED TIMELINE\n- Prior to May 4\n-- Refine my basic knowledge of Perl\n-- Craft one or two small patches to some of Git's Perl components\n(preferentially to git-add--interactive itself) to improve my\nunderstanding of the language and of how Git's Perl scripts actually\nwork\n\n- May 4 - May 30\n-- Clarify implementation details with my mentor, and work on a more\ndetailed roadmap for the project\n-- Investigate roughly how to replace command invocations from the\nscript with actual builtin functions; which Git APIs in Perl already\nhave functional equivalents in C; which parts will require a full\nrewrite.\n\n- May 30 - June 30 (start of coding period)\n-- Define the architecture of the builtin within git (which\nfunctions/interfaces will it have? where will its code reside?).\n-- Implement a small subset of the builtin (to be defined with my\nmentor) and glue it into the existing Perl script. Present this as a\nfirst patch to get feedback early regarding the implementation and\navoid piling up mistakes early.\n-- Do necessary changes based on this initial review.\n-- Have roughly 1/3 of the script's functionality ported to C.\n\n- June 30 - July 28\n-- Port the remainder of the script to a builtin.\n-- Have a weekly roadmap, sending a part of the patch every 15 days to\nthe mailing list for review and to avoid massive commits by the end of\nGSoC.\n-- Apply suggestions from community reviews when possible; if not,\nsave them for doing toward the end of GSoC (see below).\n(Note: due to a previous commitment, during a five-day period of July\nI will only be able to work part-time on GSoC. The actual week will be\nknown over the next weeks.)\n\n- July 28 - August 29\n-- By the start of this period, send a patch with the builtin fully\nimplemented to the mailing list.\n-- Fix bugs, test extensively, possibly extend test coverage for\ngit-add--interactive.\n-- Respond to the (predictably big) community feedback regarding the change.\n\nI currently work full-time in a payments company (see below), but in\ncase of being accepted I am willing to quit my job some months early\nto dedicate myself fully to GSoC starting June.\n\nBIOGRAPHICAL INFORMATION\nMy name is Daniel Ferreira and I'm a student from São Paulo, Brazil. I\nwas accepted by Stanford University last year and I will start college\nthis fall. I started coding C about six years ago writing up system\nmodifications (\"tweaks\") for jailbroken iPhones. Since then, I have\nwritten/contributed to a couple of open-source projects like an IRC\nbot and other assorted things – all of them tracked on Git\n(https://github.com/theiostream). I have also developed a\n(closed-source) library in C for interacting with payment terminals in\nthe company I have worked for over the last two years (Pagar.me).\nThere, we use Git extensively for managing projects with around 20\npeople working concurrently.\n\nMICROPROJECT\nI have sent a series of patches to complete the microproject of\nconverting recursive calls to readdir() into calls to dir_iterator.\nThe most recent version can be found in\nhttps://public-inbox.org/git/1490465551-71056-2-git-send-email-bnmvco@gmail.com/T/#u.\n\nThanks,\n-- Daniel.\n"},{"id":"315580","messageId":"CAGZ79kbyW79wToWqoL_F5n+jOFwFH=z2jY3Du2YTyv9tS9L=JA@mail.gmail.com","threadId":"45514","inReplyTo":"CAEA2_RJEf4vjgcaux8a1KWh1-vxLLmv1--Vjf9wiEQoF+gVDtA@mail.gmail.com","subject":"Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-03-28T18:05:07Z","receivedAt":"2017-03-28T18:05:18Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sat, Mar 25, 2017 at 8:15 PM, Daniel Ferreira (theiostream)\n<bnmvco@gmail.com> wrote:\n\n> SYNOPSIS\n> There are many advantages to converting parts of git that are still\n> scripts to C builtins, among which execution speed, improved\n> compatibility and code deduplication.\n\nagreed.\n\n> git-add--interactive, one of the most useful features of Git.\n\nknee jerk reaction: I never used it, so it cannot be that important ;)\n(I use git-gui, which is essentially the same workflow. There are tons\nof ways to accomplish a given goal using Git, so I guess we don't\nwant to get in an argument here).\n\n>\n> FEASIBILITY\n>\n> There was only one discussion regarding the feasibility of its porting\n> (https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1HLx+Q@mail.gmail.com/).\n> It resulted in a consensus that doing it would be a task too large –\n> although interesting – for GSoC 2015 based on the amount of its lines\n> of code. It is, however, only a few lines larger than\n> git-rebase--interactive, which has been considered an appropriate\n> idea. As such, it looks like a possible project for three months of\n> full-time work.\n\nok, it sounds a challenging project. (currently counting 1750 lines of\ncode). Scrolling over the source code, there are quite a couple of\nfunctions, where the direct equivalent in C springs to mind.\n\nrun_cmd_pipe -> see run-command.h\nunquote_path -> unquote_c_style ?\nrefresh -> update_index_if_able()\nlist_modified -> iterate over \"const struct cache_entry *ce = active_cache[i];\"\n\n\n> PROJECTED TIMELINE\n> - Prior to May 4\n> -- Refine my basic knowledge of Perl\n> -- Craft one or two small patches to some of Git's Perl components\n> (preferentially to git-add--interactive itself) to improve my\n> understanding of the language and of how Git's Perl scripts actually\n> work\n\n\n\n>\n> - May 4 - May 30\n> -- Clarify implementation details with my mentor, and work on a more\n> detailed roadmap for the project\n> -- Investigate roughly how to replace command invocations from the\n> script with actual builtin functions; which Git APIs in Perl already\n> have functional equivalents in C; which parts will require a full\n> rewrite.\n\nThere are different approaches for replacing functionality in another\nlanguage. Examples:\n* Implement the functionality in C and then have a \"flag-day\" commit\n  783d7e865e (builtin-am: remove redirection to git-am.sh, 2015-08-04)\n  This only works when the whole functionality was replaced in prior commits\n* Implement partial functionality in C and call it via a helper function.\n  3604242f08 (submodule: port init from shell to C, 2016-04-15)\n  This works well for only partial conversions (the larger the thing to\n  convert the more appealing this is, as it gets code shipped early.)\n  When choosing this strategy, this part of the Project would be to\n  identify parts that could be ported on its own without much\n  additional glue-code.\n\n> - May 30 - June 30 (start of coding period)\n> -- Define the architecture of the builtin within git (which\n> functions/interfaces will it have? where will its code reside?).\n> -- Implement a small subset of the builtin (to be defined with my\n> mentor) and glue it into the existing Perl script. Present this as a\n> first patch to get feedback early regarding the implementation and\n> avoid piling up mistakes early.\n> -- Do necessary changes based on this initial review.\n> -- Have roughly 1/3 of the script's functionality ported to C.\n>\n> - June 30 - July 28\n> -- Port the remainder of the script to a builtin.\n> -- Have a weekly roadmap, sending a part of the patch every 15 days to\n> the mailing list for review and to avoid massive commits by the end of\n> GSoC.\n\nyeah; send early, send often. ;)\n\n> -- Apply suggestions from community reviews when possible; if not,\n> save them for doing toward the end of GSoC (see below).\n\nPlease do not underestimate the discussion by community, finding\nconsensus on list consumes a bit of time in some cases.\n\n> (Note: due to a previous commitment, during a five-day period of July\n> I will only be able to work part-time on GSoC. The actual week will be\n> known over the next weeks.)\n\nMaybe you want to shift the schedule up to here by one week then?\n(e.g. the first period would be April 27  - May 23)\n\n>\n> - July 28 - August 29\n> -- By the start of this period, send a patch with the builtin fully\n> implemented to the mailing list.\n\n/a patch/a patch series consisting of many patches/\nExperience shows that smaller patches are easier to review as\nit is more focused. Consider e.g. e86ab2c1cd (wt-status: convert\nto struct object_id, 2017-02-21) and the parents leading up to this\ncommit. They work on the same big topic, but focus on very\nregional areas to ease review.\n\n> -- Fix bugs, test extensively, possibly extend test coverage for\n> git-add--interactive.\n\nAFAICT ('$ git grep \"git add -i\"') there is only t3701 testing the\ninteractive add. Maybe we need to add tests first to document\ncurrent behavior, before attempting a conversion?\n\nThis could go well into the period \"May 4 - May 30\", as writing\ntests would count as \"Clarify implementation details\".\n\nHope this helps,\nStefan\n"},{"id":"315725","messageId":"alpine.DEB.2.20.1703300142230.4068@virtualbox","threadId":"45514","inReplyTo":"CAGZ79kbyW79wToWqoL_F5n+jOFwFH=z2jY3Du2YTyv9tS9L=JA@mail.gmail.com","subject":"Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-03-30T00:01:17Z","receivedAt":"2017-03-30T00:01:36Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Stefan & Daniel,\n\nOn Tue, 28 Mar 2017, Stefan Beller wrote:\n\n> On Sat, Mar 25, 2017 at 8:15 PM, Daniel Ferreira (theiostream)\n> <bnmvco@gmail.com> wrote:\n> \n> > SYNOPSIS\n> > There are many advantages to converting parts of git that are still\n> > scripts to C builtins, among which execution speed, improved\n> > compatibility and code deduplication.\n> \n> agreed.\n\nI would even add portability. But yeah, speed is a big thing. I am an\nextensive user of `git add -p` (which is backed by\ngit-add--interactive.perl) and it is slow as molasses on Windows, just\nbecause it is a Perl script (and the Perl interpreter needs to emulate\nPOSIX functionality that is frequently not even needed, such as: copying\nall memory and reopening all file descriptors in a fork() call only to\nexec() git.exe right away, tossing all of the diligently work into the\ndustbin).\n\n> > git-add--interactive, one of the most useful features of Git.\n> \n> knee jerk reaction: I never used it, so it cannot be that important ;)\n> (I use git-gui, which is essentially the same workflow. There are tons\n> of ways to accomplish a given goal using Git, so I guess we don't\n> want to get in an argument here).\n\nWell, I make up for your lack of `git add -i` usage.\n\nOf course, since you use git-gui, you are simply using another dependency\nthat bloats Git for Windows: Tcl/Tk.\n\n> > FEASIBILITY\n> >\n> > There was only one discussion regarding the feasibility of its porting\n> > (https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1HLx+Q@mail.gmail.com/).\n> > It resulted in a consensus that doing it would be a task too large –\n> > although interesting – for GSoC 2015 based on the amount of its lines\n> > of code. It is, however, only a few lines larger than\n> > git-rebase--interactive, which has been considered an appropriate\n> > idea. As such, it looks like a possible project for three months of\n> > full-time work.\n> \n> ok, it sounds a challenging project. (currently counting 1750 lines of\n> code). Scrolling over the source code, there are quite a couple of\n> functions, where the direct equivalent in C springs to mind.\n> \n> run_cmd_pipe -> see run-command.h\n> unquote_path -> unquote_c_style ?\n> refresh -> update_index_if_able()\n> list_modified -> iterate over \"const struct cache_entry *ce = active_cache[i];\"\n\nYes, I think it would be more important to acquaint oneself with the\nidiosynchracies of Git's internal \"API\" than to get familiar with Perl:\ninterpreting what obscure Perl code does is something I would gladly do as\na mentor.\n\n> > PROJECTED TIMELINE\n> > - Prior to May 4\n> > -- Refine my basic knowledge of Perl\n> > -- Craft one or two small patches to some of Git's Perl components\n> > (preferentially to git-add--interactive itself) to improve my\n> > understanding of the language and of how Git's Perl scripts actually\n> > work\n\nAs I mentioned above, the Perl code should be fairly intuitive for the\nmost part, with maybe a couple of pieces of code using more advanced Perl\ntechniques. Given the scope of the project, I would recommend working\nclosely together with the mentor(s) to clarify what those code parts do.\n\n> > - May 4 - May 30\n> > -- Clarify implementation details with my mentor, and work on a more\n> > detailed roadmap for the project\n> > -- Investigate roughly how to replace command invocations from the\n> > script with actual builtin functions; which Git APIs in Perl already\n> > have functional equivalents in C; which parts will require a full\n> > rewrite.\n> \n> There are different approaches for replacing functionality in another\n> language. Examples:\n> * Implement the functionality in C and then have a \"flag-day\" commit\n>   783d7e865e (builtin-am: remove redirection to git-am.sh, 2015-08-04)\n>   This only works when the whole functionality was replaced in prior commits\n> * Implement partial functionality in C and call it via a helper function.\n>   3604242f08 (submodule: port init from shell to C, 2016-04-15)\n>   This works well for only partial conversions (the larger the thing to\n>   convert the more appealing this is, as it gets code shipped early.)\n>   When choosing this strategy, this part of the Project would be to\n>   identify parts that could be ported on its own without much\n>   additional glue-code.\n\nTo offer my perspective: I strongly prefer the latter approach. Not only\ndoes it yield earlier results, it also makes it substantially easier to\nhandle the project even if it should turn out to be a little larger than\njust 3 months.\n\n> > - May 30 - June 30 (start of coding period)\n> > -- Define the architecture of the builtin within git (which\n> > functions/interfaces will it have? where will its code reside?).\n> > -- Implement a small subset of the builtin (to be defined with my\n> > mentor) and glue it into the existing Perl script. Present this as a\n> > first patch to get feedback early regarding the implementation and\n> > avoid piling up mistakes early.\n> > -- Do necessary changes based on this initial review.\n> > -- Have roughly 1/3 of the script's functionality ported to C.\n> >\n> > - June 30 - July 28\n> > -- Port the remainder of the script to a builtin.\n> > -- Have a weekly roadmap, sending a part of the patch every 15 days to\n> > the mailing list for review and to avoid massive commits by the end of\n> > GSoC.\n> \n> yeah; send early, send often. ;)\n\nEven better: push multiple times a day to a public repository, say, on\nGitHub. That allows for \"Work In Progress\" commits that not only serve as\na backup but also as transparent report what was done.\n\n> > -- Apply suggestions from community reviews when possible; if not,\n> > save them for doing toward the end of GSoC (see below).\n> \n> Please do not underestimate the discussion by community, finding\n> consensus on list consumes a bit of time in some cases.\n\nI agree with this statement. Ideally, the first patch series would be\nready to submit very soon into the project, something like 2 weeks (which\nis another point in favor of the helper approach outlined by Stefan).\n\nThe smaller the patch series are, the more likely they will get in\nquickly. And the easier is it to address reviewers' comments, because you\nwon't have to send out gazillion unchanged patches of a multi-dozen patch\nseries just to address one typo in one patch.\n\n> > (Note: due to a previous commitment, during a five-day period of July\n> > I will only be able to work part-time on GSoC. The actual week will be\n> > known over the next weeks.)\n> \n> Maybe you want to shift the schedule up to here by one week then?\n> (e.g. the first period would be April 27  - May 23)\n> \n> >\n> > - July 28 - August 29\n> > -- By the start of this period, send a patch with the builtin fully\n> > implemented to the mailing list.\n> \n> /a patch/a patch series consisting of many patches/\n> Experience shows that smaller patches are easier to review as\n> it is more focused.\n\nNot only that. Reviewers' time is scarce. I frequently have to triage\nwhich mails I can read in the time I have for the mailing list. If I see a\npatch series that is too large, I simply cannot review it. I gather that\nis the same issue for pretty much all available reviewers.\n\n> Consider e.g. e86ab2c1cd (wt-status: convert to struct object_id,\n> 2017-02-21) and the parents leading up to this commit. They work on the\n> same big topic, but focus on very regional areas to ease review.\n> \n> > -- Fix bugs, test extensively, possibly extend test coverage for\n> > git-add--interactive.\n> \n> AFAICT ('$ git grep \"git add -i\"') there is only t3701 testing the\n> interactive add. Maybe we need to add tests first to document\n> current behavior, before attempting a conversion?\n\nThat should indeed be the first step, before doing much else.\n\nHowever, it does look as if t3701's 40 test cases provide fairly thorough\ncoverage.\n\n> This could go well into the period \"May 4 - May 30\", as writing\n> tests would count as \"Clarify implementation details\".\n\nI agree with that, too.\n\nThanks for working on this!\nJohannes"},{"id":"315871","messageId":"CAEA2_RLX+0Yz-wcdAaEj3Pp0qOKWdHu32T9Vvkk2KSFkzUx7Cw@mail.gmail.com","threadId":"45514","inReplyTo":"alpine.DEB.2.20.1703300142230.4068@virtualbox","subject":"Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin","fromName":"Daniel Ferreira (theiostream)","fromEmail":"bnmvco@gmail.com","sentAt":"2017-03-31T05:07:46Z","receivedAt":"2017-03-31T05:08:25Z","isPatch":false,"sender":{"key":"bnmvco@gmail.com","avatar":"https://avatars.githubusercontent.com/u/735397?v=4"},"body":"Hi Stefan & Johannes,\n\nThank you for the precious feedback on the proposal. I don't see much\nsense in sending a full \"v2\" of it and have you read it all over\nagain, so I'll just answer to your comments directly.\n\nAlso, although the GSoC website allows me to send a \"proposal draft\"\nto you through the website, since I've already sent it here that\nshouldn't be necessary, correct? I intend to use it just to send the\nfinal thing.\n\nOn Wed, Mar 29, 2017 at 9:01 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> On Tue, 28 Mar 2017, Stefan Beller wrote:\n>\n>> On Sat, Mar 25, 2017 at 8:15 PM, Daniel Ferreira (theiostream)\n>> <bnmvco@gmail.com> wrote:\n>>\n>> > SYNOPSIS\n>> > There are many advantages to converting parts of git that are still\n>> > scripts to C builtins, among which execution speed, improved\n>> > compatibility and code deduplication.\n>>\n>> agreed.\n>\n> I would even add portability. But yeah, speed is a big thing. I am an\n> extensive user of `git add -p` (which is backed by\n> git-add--interactive.perl) and it is slow as molasses on Windows, just\n> because it is a Perl script (and the Perl interpreter needs to emulate\n> POSIX functionality that is frequently not even needed, such as: copying\n> all memory and reopening all file descriptors in a fork() call only to\n> exec() git.exe right away, tossing all of the diligently work into the\n> dustbin).\n\nThanks for this example – it hadn't come to my mind since I don't use\nGit on Windows. I'll be sure to complement the synopsis with it. :)\n\n>\n>> > FEASIBILITY\n>> >\n>> > There was only one discussion regarding the feasibility of its porting\n>> > (https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1HLx+Q@mail.gmail.com/).\n>> > It resulted in a consensus that doing it would be a task too large –\n>> > although interesting – for GSoC 2015 based on the amount of its lines\n>> > of code. It is, however, only a few lines larger than\n>> > git-rebase--interactive, which has been considered an appropriate\n>> > idea. As such, it looks like a possible project for three months of\n>> > full-time work.\n>>\n>> ok, it sounds a challenging project. (currently counting 1750 lines of\n>> code). Scrolling over the source code, there are quite a couple of\n>> functions, where the direct equivalent in C springs to mind.\n>>\n>> run_cmd_pipe -> see run-command.h\n>> unquote_path -> unquote_c_style ?\n>> refresh -> update_index_if_able()\n>> list_modified -> iterate over \"const struct cache_entry *ce = active_cache[i];\"\n\nThank you for these functions. I don't think I will be able to specify\nthem in detail as part of the projected timeline (e.g. \"June 1:\nconvert calls to refresh() to use update_index_if_able()\") already\nbecause there is not enough time prior to the proposal deadline to\nstudy their behavior in detail, and I like to avoid talking about\nthings I don't fully understand. Although I think I can cite them as\nexamples for a thesis I had put elsewhere in the proposal that \"Git\nAPIs in Perl already have functional equivalents in C\".\n\nAlso, they will be great for my early investigation stage into\ngit-add--interactive. :) Once more, thanks for having listed them.\n\n> Yes, I think it would be more important to acquaint oneself with the\n> idiosynchracies of Git's internal \"API\" than to get familiar with Perl:\n> interpreting what obscure Perl code does is something I would gladly do as\n> a mentor.\n\nThat's really nice! I usually don't get stuck when trying to\nunderstand code in languages I'm not too well acquainted with, but I\nfigured getting more familiar with Perl would speed development up.\nBut it does make sense that this \"prior to May 4\" might be better\ninvested learning about git's internals than Perl.\n\nQuestion: do you suggest any pending bugfix to git-add--interactive or\nto something related that might give some useful knowledge in advance?\n(for the pre-code period). My microproject involves playing with the\ndir_iterator interface, which is a great exercise in code refactoring\nbut really does not teach me too much about Git's architecture.\n\nEven if you do not have an answer to this, I'm pretty sure I'll keep\nthis commitment to submitting some patch series somehow related to\ngit-add before GSoC begins, especially after this comment from\nJohannes.\n\n>\n>> > PROJECTED TIMELINE\n>> > - Prior to May 4\n>> > -- Refine my basic knowledge of Perl\n>> > -- Craft one or two small patches to some of Git's Perl components\n>> > (preferentially to git-add--interactive itself) to improve my\n>> > understanding of the language and of how Git's Perl scripts actually\n>> > work\n\nSo yeah, I think this could be rewritten as:\n\n- Prior to May 4\n-- Craft two or three small patch series to git-add--interactive or\nrelated components to improve my understanding of Git's internal\narchitecture, especially that related to git-add.\n\n>\n>> > - May 4 - May 30\n>> > -- Clarify implementation details with my mentor, and work on a more\n>> > detailed roadmap for the project\n>> > -- Investigate roughly how to replace command invocations from the\n>> > script with actual builtin functions; which Git APIs in Perl already\n>> > have functional equivalents in C; which parts will require a full\n>> > rewrite.\n>> There are different approaches for replacing functionality in another\n>> language. Examples:\n>> * Implement the functionality in C and then have a \"flag-day\" commit\n>>   783d7e865e (builtin-am: remove redirection to git-am.sh, 2015-08-04)\n>>   This only works when the whole functionality was replaced in prior commits\n>> * Implement partial functionality in C and call it via a helper function.\n>>   3604242f08 (submodule: port init from shell to C, 2016-04-15)\n>>   This works well for only partial conversions (the larger the thing to\n>>   convert the more appealing this is, as it gets code shipped early.)\n>>   When choosing this strategy, this part of the Project would be to\n>>   identify parts that could be ported on its own without much\n>>   additional glue-code.\n>\n> To offer my perspective: I strongly prefer the latter approach. Not only\n> does it yield earlier results, it also makes it substantially easier to\n> handle the project even if it should turn out to be a little larger than\n> just 3 months.\n\nI agree. I even think that the latter parts of this projected timeline\nimply a choice for the second approach.\n\nFor now, Stefan's implementation in 3604242f08 (create a helper\nbuiltin command, call it from a script, do it to other functions until\nthe whole script is replaced) seems the most natural way to do it. But\nsadly, I still do not have enough knowledge about the project to be\nable to specify in this proposal how (e.g. in what order) exactly I\nintend to \"divide\" the code to do this. But I'm sure the answer to\nthis question will be clearer through contact with my mentor and with\ngrowing experience in the project.\n\n>\n>> > - May 30 - June 30 (start of coding period)\n>> > -- Define the architecture of the builtin within git (which\n>> > functions/interfaces will it have? where will its code reside?).\n>> > -- Implement a small subset of the builtin (to be defined with my\n>> > mentor) and glue it into the existing Perl script. Present this as a\n>> > first patch to get feedback early regarding the implementation and\n>> > avoid piling up mistakes early.\n>> > -- Do necessary changes based on this initial review.\n>> > -- Have roughly 1/3 of the script's functionality ported to C.\n>> >\n>> > - June 30 - July 28\n>> > -- Port the remainder of the script to a builtin.\n>> > -- Have a weekly roadmap, sending a part of the patch every 15 days to\n>> > the mailing list for review and to avoid massive commits by the end of\n>> > GSoC.\n>>\n>> yeah; send early, send often. ;)\n>\n> Even better: push multiple times a day to a public repository, say, on\n> GitHub. That allows for \"Work In Progress\" commits that not only serve as\n> a backup but also as transparent report what was done.\n\nGreat idea. It can also be a good way to report my progress to my\nmentor. (I'll do my best to avoid \"[WIP]\" commits with barely any\nexplanation).\n\nAs for the \"timeline\", I think it can be improved by some better\nspecification regarding the gradual process of the implementation\nopposed to switching from the script to a builtin all at once.\n\n>> > -- Apply suggestions from community reviews when possible; if not,\n>> > save them for doing toward the end of GSoC (see below).\n>>\n>> Please do not underestimate the discussion by community, finding\n>> consensus on list consumes a bit of time in some cases.\n> I agree with this statement. Ideally, the first patch series would be\n> ready to submit very soon into the project, something like 2 weeks (which\n> is another point in favor of the helper approach outlined by Stefan).\n> The smaller the patch series are, the more likely they will get in\n> quickly. And the easier is it to address reviewers' comments, because you\n> won't have to send out gazillion unchanged patches of a multi-dozen patch\n> series just to address one typo in one patch.\n\nSeems reasonable. I'll amend the proposal to commit to a first patch\nseries in 2 weeks and to sending a new one every week or less (to\navoid series getting massive). I'll also refrain from postponing\nresponding to feedback and, when necessary, work in parallel between\namending patches and working on new code. After all, I know how my\nschedule will be during the whole GSoC period, but I don't know how\navailable those in the list will be to review it, so it's probably\nbest to respond early.\n\n>> > (Note: due to a previous commitment, during a five-day period of July\n>> > I will only be able to work part-time on GSoC. The actual week will be\n>> > known over the next weeks.)\n>>\n>> Maybe you want to shift the schedule up to here by one week then?\n>> (e.g. the first period would be April 27  - May 23)\n>>\n\nI cannot shift the schedule one week earlier during the pre-code\nperiod because Google only releases mentor information on May 4, and\nthe things I'm planning for the second period depend on the mentor.\nHowever, my classes only start in September 22, so I would have\nabsolutely no problem continuing to work full-time on Git after the\nGSoC period is officially over to compensate for these five days.\n\n>> >\n>> > - July 28 - August 29\n>> > -- By the start of this period, send a patch with the builtin fully\n>> > implemented to the mailing list.\n>>\n>> /a patch/a patch series consisting of many patches/\n>> Experience shows that smaller patches are easier to review as\n>> it is more focused.\n\nI will reword it.\n\n>> Consider e.g. e86ab2c1cd (wt-status: convert to struct object_id,\n>> 2017-02-21) and the parents leading up to this commit. They work on the\n>> same big topic, but focus on very regional areas to ease review.\n>>\n>> > -- Fix bugs, test extensively, possibly extend test coverage for\n>> > git-add--interactive.\n>>\n>> AFAICT ('$ git grep \"git add -i\"') there is only t3701 testing the\n>> interactive add. Maybe we need to add tests first to document\n>> current behavior, before attempting a conversion?\n>\n> That should indeed be the first step, before doing much else.\n>\n> However, it does look as if t3701's 40 test cases provide fairly thorough\n> coverage.\n>\n>> This could go well into the period \"May 4 - May 30\", as writing\n>> tests would count as \"Clarify implementation details\".\n>\n> I agree with that, too.\n\nSeems reasonable. As I said above, working on patches (if possible)\nrelated to git-add--interactive will probably lead me to discovering\ncoverage issues, which could then be dealt with in this pre-code\nperiod already with some guidance from my mentor.\n\n> Thanks for working on this!\n\nThank *you*, and Stefan, for the careful review of it. It means a lot.\n\n-- Daniel.\n"},{"id":"315946","messageId":"CAEA2_R+RnAFx5PZGL3abMXOpau4P2FPhdL1e26xftgDijZTS-w@mail.gmail.com","threadId":"45514","inReplyTo":"CAEA2_RLX+0Yz-wcdAaEj3Pp0qOKWdHu32T9Vvkk2KSFkzUx7Cw@mail.gmail.com","subject":"Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin","fromName":"Daniel Ferreira (theiostream)","fromEmail":"bnmvco@gmail.com","sentAt":"2017-03-31T19:06:29Z","receivedAt":"2017-03-31T19:07:08Z","isPatch":false,"sender":{"key":"bnmvco@gmail.com","avatar":"https://avatars.githubusercontent.com/u/735397?v=4"},"body":"Well, Google requires me to have a draft on a Google Doc anyway for\nthe proposal, and I am unsure who exactly it will reach. Since it *is*\npart of the discussion regarding my proposal, I suppose it is worth\nposting here for anyone to comment:\nhttps://docs.google.com/document/d/1dvF2PNRQvvZ351jCdKzOLs7tzaDqhR7ci7TDgzYQg9I/edit?usp=sharing.\n\n-- Daniel.\n\nOn Fri, Mar 31, 2017 at 2:07 AM, Daniel Ferreira (theiostream)\n<bnmvco@gmail.com> wrote:\n> Hi Stefan & Johannes,\n>\n> Thank you for the precious feedback on the proposal. I don't see much\n> sense in sending a full \"v2\" of it and have you read it all over\n> again, so I'll just answer to your comments directly.\n>\n> Also, although the GSoC website allows me to send a \"proposal draft\"\n> to you through the website, since I've already sent it here that\n> shouldn't be necessary, correct? I intend to use it just to send the\n> final thing.\n>\n> On Wed, Mar 29, 2017 at 9:01 PM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n>> On Tue, 28 Mar 2017, Stefan Beller wrote:\n>>\n>>> On Sat, Mar 25, 2017 at 8:15 PM, Daniel Ferreira (theiostream)\n>>> <bnmvco@gmail.com> wrote:\n>>>\n>>> > SYNOPSIS\n>>> > There are many advantages to converting parts of git that are still\n>>> > scripts to C builtins, among which execution speed, improved\n>>> > compatibility and code deduplication.\n>>>\n>>> agreed.\n>>\n>> I would even add portability. But yeah, speed is a big thing. I am an\n>> extensive user of `git add -p` (which is backed by\n>> git-add--interactive.perl) and it is slow as molasses on Windows, just\n>> because it is a Perl script (and the Perl interpreter needs to emulate\n>> POSIX functionality that is frequently not even needed, such as: copying\n>> all memory and reopening all file descriptors in a fork() call only to\n>> exec() git.exe right away, tossing all of the diligently work into the\n>> dustbin).\n>\n> Thanks for this example – it hadn't come to my mind since I don't use\n> Git on Windows. I'll be sure to complement the synopsis with it. :)\n>\n>>\n>>> > FEASIBILITY\n>>> >\n>>> > There was only one discussion regarding the feasibility of its porting\n>>> > (https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1HLx+Q@mail.gmail.com/).\n>>> > It resulted in a consensus that doing it would be a task too large –\n>>> > although interesting – for GSoC 2015 based on the amount of its lines\n>>> > of code. It is, however, only a few lines larger than\n>>> > git-rebase--interactive, which has been considered an appropriate\n>>> > idea. As such, it looks like a possible project for three months of\n>>> > full-time work.\n>>>\n>>> ok, it sounds a challenging project. (currently counting 1750 lines of\n>>> code). Scrolling over the source code, there are quite a couple of\n>>> functions, where the direct equivalent in C springs to mind.\n>>>\n>>> run_cmd_pipe -> see run-command.h\n>>> unquote_path -> unquote_c_style ?\n>>> refresh -> update_index_if_able()\n>>> list_modified -> iterate over \"const struct cache_entry *ce = active_cache[i];\"\n>\n> Thank you for these functions. I don't think I will be able to specify\n> them in detail as part of the projected timeline (e.g. \"June 1:\n> convert calls to refresh() to use update_index_if_able()\") already\n> because there is not enough time prior to the proposal deadline to\n> study their behavior in detail, and I like to avoid talking about\n> things I don't fully understand. Although I think I can cite them as\n> examples for a thesis I had put elsewhere in the proposal that \"Git\n> APIs in Perl already have functional equivalents in C\".\n>\n> Also, they will be great for my early investigation stage into\n> git-add--interactive. :) Once more, thanks for having listed them.\n>\n>> Yes, I think it would be more important to acquaint oneself with the\n>> idiosynchracies of Git's internal \"API\" than to get familiar with Perl:\n>> interpreting what obscure Perl code does is something I would gladly do as\n>> a mentor.\n>\n> That's really nice! I usually don't get stuck when trying to\n> understand code in languages I'm not too well acquainted with, but I\n> figured getting more familiar with Perl would speed development up.\n> But it does make sense that this \"prior to May 4\" might be better\n> invested learning about git's internals than Perl.\n>\n> Question: do you suggest any pending bugfix to git-add--interactive or\n> to something related that might give some useful knowledge in advance?\n> (for the pre-code period). My microproject involves playing with the\n> dir_iterator interface, which is a great exercise in code refactoring\n> but really does not teach me too much about Git's architecture.\n>\n> Even if you do not have an answer to this, I'm pretty sure I'll keep\n> this commitment to submitting some patch series somehow related to\n> git-add before GSoC begins, especially after this comment from\n> Johannes.\n>\n>>\n>>> > PROJECTED TIMELINE\n>>> > - Prior to May 4\n>>> > -- Refine my basic knowledge of Perl\n>>> > -- Craft one or two small patches to some of Git's Perl components\n>>> > (preferentially to git-add--interactive itself) to improve my\n>>> > understanding of the language and of how Git's Perl scripts actually\n>>> > work\n>\n> So yeah, I think this could be rewritten as:\n>\n> - Prior to May 4\n> -- Craft two or three small patch series to git-add--interactive or\n> related components to improve my understanding of Git's internal\n> architecture, especially that related to git-add.\n>\n>>\n>>> > - May 4 - May 30\n>>> > -- Clarify implementation details with my mentor, and work on a more\n>>> > detailed roadmap for the project\n>>> > -- Investigate roughly how to replace command invocations from the\n>>> > script with actual builtin functions; which Git APIs in Perl already\n>>> > have functional equivalents in C; which parts will require a full\n>>> > rewrite.\n>>> There are different approaches for replacing functionality in another\n>>> language. Examples:\n>>> * Implement the functionality in C and then have a \"flag-day\" commit\n>>>   783d7e865e (builtin-am: remove redirection to git-am.sh, 2015-08-04)\n>>>   This only works when the whole functionality was replaced in prior commits\n>>> * Implement partial functionality in C and call it via a helper function.\n>>>   3604242f08 (submodule: port init from shell to C, 2016-04-15)\n>>>   This works well for only partial conversions (the larger the thing to\n>>>   convert the more appealing this is, as it gets code shipped early.)\n>>>   When choosing this strategy, this part of the Project would be to\n>>>   identify parts that could be ported on its own without much\n>>>   additional glue-code.\n>>\n>> To offer my perspective: I strongly prefer the latter approach. Not only\n>> does it yield earlier results, it also makes it substantially easier to\n>> handle the project even if it should turn out to be a little larger than\n>> just 3 months.\n>\n> I agree. I even think that the latter parts of this projected timeline\n> imply a choice for the second approach.\n>\n> For now, Stefan's implementation in 3604242f08 (create a helper\n> builtin command, call it from a script, do it to other functions until\n> the whole script is replaced) seems the most natural way to do it. But\n> sadly, I still do not have enough knowledge about the project to be\n> able to specify in this proposal how (e.g. in what order) exactly I\n> intend to \"divide\" the code to do this. But I'm sure the answer to\n> this question will be clearer through contact with my mentor and with\n> growing experience in the project.\n>\n>>\n>>> > - May 30 - June 30 (start of coding period)\n>>> > -- Define the architecture of the builtin within git (which\n>>> > functions/interfaces will it have? where will its code reside?).\n>>> > -- Implement a small subset of the builtin (to be defined with my\n>>> > mentor) and glue it into the existing Perl script. Present this as a\n>>> > first patch to get feedback early regarding the implementation and\n>>> > avoid piling up mistakes early.\n>>> > -- Do necessary changes based on this initial review.\n>>> > -- Have roughly 1/3 of the script's functionality ported to C.\n>>> >\n>>> > - June 30 - July 28\n>>> > -- Port the remainder of the script to a builtin.\n>>> > -- Have a weekly roadmap, sending a part of the patch every 15 days to\n>>> > the mailing list for review and to avoid massive commits by the end of\n>>> > GSoC.\n>>>\n>>> yeah; send early, send often. ;)\n>>\n>> Even better: push multiple times a day to a public repository, say, on\n>> GitHub. That allows for \"Work In Progress\" commits that not only serve as\n>> a backup but also as transparent report what was done.\n>\n> Great idea. It can also be a good way to report my progress to my\n> mentor. (I'll do my best to avoid \"[WIP]\" commits with barely any\n> explanation).\n>\n> As for the \"timeline\", I think it can be improved by some better\n> specification regarding the gradual process of the implementation\n> opposed to switching from the script to a builtin all at once.\n>\n>>> > -- Apply suggestions from community reviews when possible; if not,\n>>> > save them for doing toward the end of GSoC (see below).\n>>>\n>>> Please do not underestimate the discussion by community, finding\n>>> consensus on list consumes a bit of time in some cases.\n>> I agree with this statement. Ideally, the first patch series would be\n>> ready to submit very soon into the project, something like 2 weeks (which\n>> is another point in favor of the helper approach outlined by Stefan).\n>> The smaller the patch series are, the more likely they will get in\n>> quickly. And the easier is it to address reviewers' comments, because you\n>> won't have to send out gazillion unchanged patches of a multi-dozen patch\n>> series just to address one typo in one patch.\n>\n> Seems reasonable. I'll amend the proposal to commit to a first patch\n> series in 2 weeks and to sending a new one every week or less (to\n> avoid series getting massive). I'll also refrain from postponing\n> responding to feedback and, when necessary, work in parallel between\n> amending patches and working on new code. After all, I know how my\n> schedule will be during the whole GSoC period, but I don't know how\n> available those in the list will be to review it, so it's probably\n> best to respond early.\n>\n>>> > (Note: due to a previous commitment, during a five-day period of July\n>>> > I will only be able to work part-time on GSoC. The actual week will be\n>>> > known over the next weeks.)\n>>>\n>>> Maybe you want to shift the schedule up to here by one week then?\n>>> (e.g. the first period would be April 27  - May 23)\n>>>\n>\n> I cannot shift the schedule one week earlier during the pre-code\n> period because Google only releases mentor information on May 4, and\n> the things I'm planning for the second period depend on the mentor.\n> However, my classes only start in September 22, so I would have\n> absolutely no problem continuing to work full-time on Git after the\n> GSoC period is officially over to compensate for these five days.\n>\n>>> >\n>>> > - July 28 - August 29\n>>> > -- By the start of this period, send a patch with the builtin fully\n>>> > implemented to the mailing list.\n>>>\n>>> /a patch/a patch series consisting of many patches/\n>>> Experience shows that smaller patches are easier to review as\n>>> it is more focused.\n>\n> I will reword it.\n>\n>>> Consider e.g. e86ab2c1cd (wt-status: convert to struct object_id,\n>>> 2017-02-21) and the parents leading up to this commit. They work on the\n>>> same big topic, but focus on very regional areas to ease review.\n>>>\n>>> > -- Fix bugs, test extensively, possibly extend test coverage for\n>>> > git-add--interactive.\n>>>\n>>> AFAICT ('$ git grep \"git add -i\"') there is only t3701 testing the\n>>> interactive add. Maybe we need to add tests first to document\n>>> current behavior, before attempting a conversion?\n>>\n>> That should indeed be the first step, before doing much else.\n>>\n>> However, it does look as if t3701's 40 test cases provide fairly thorough\n>> coverage.\n>>\n>>> This could go well into the period \"May 4 - May 30\", as writing\n>>> tests would count as \"Clarify implementation details\".\n>>\n>> I agree with that, too.\n>\n> Seems reasonable. As I said above, working on patches (if possible)\n> related to git-add--interactive will probably lead me to discovering\n> coverage issues, which could then be dealt with in this pre-code\n> period already with some guidance from my mentor.\n>\n>> Thanks for working on this!\n>\n> Thank *you*, and Stefan, for the careful review of it. It means a lot.\n>\n> -- Daniel.\n"},{"id":"316070","messageId":"alpine.DEB.2.20.1704022140420.3742@virtualbox","threadId":"45514","inReplyTo":"CAEA2_RLX+0Yz-wcdAaEj3Pp0qOKWdHu32T9Vvkk2KSFkzUx7Cw@mail.gmail.com","subject":"Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-04-02T19:43:56Z","receivedAt":"2017-04-02T19:44:07Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Daniel,\n\nOn Fri, 31 Mar 2017, Daniel Ferreira (theiostream) wrote:\n\n> Question: do you suggest any pending bugfix to git-add--interactive or\n> to something related that might give some useful knowledge in advance?\n> (for the pre-code period). My microproject involves playing with the\n> dir_iterator interface, which is a great exercise in code refactoring\n> but really does not teach me too much about Git's architecture.\n\nWe ask to accomplish a microproject before evaluating the proposals for\none reason: to have a good understanding how well the students would\ninteract with the project if they were accepted. As such, the\nmicroprojects really are about the flow of the contribution, not to tackle\nthe project already.\n\nMeaning: I would recommend staying with your microproject, in particular\nif it is already in full swing.\n\nCiao,\nJohannes\n"},{"id":"316077","messageId":"CAEA2_RJ31r-9ROWKKQY8JwWF0iQk9d4hdLJZdkZgRHaFGphzfg@mail.gmail.com","threadId":"45514","inReplyTo":"alpine.DEB.2.20.1704022140420.3742@virtualbox","subject":"Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin","fromName":"Daniel Ferreira (theiostream)","fromEmail":"bnmvco@gmail.com","sentAt":"2017-04-02T22:50:28Z","receivedAt":"2017-04-02T22:51:06Z","isPatch":false,"sender":{"key":"bnmvco@gmail.com","avatar":"https://avatars.githubusercontent.com/u/735397?v=4"},"body":"On Sun, Apr 2, 2017 at 4:43 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> We ask to accomplish a microproject before evaluating the proposals for\n> one reason: to have a good understanding how well the students would\n> interact with the project if they were accepted. As such, the\n> microprojects really are about the flow of the contribution, not to tackle\n> the project already.\n> Meaning: I would recommend staying with your microproject, in particular\n> if it is already in full swing.\n\nOh, when I mentioned these bugfixes I meant I'd be willing to do those\n*plus* my microproject before the coding period begins as a \"warm-up\"\nto the project. I'm certainly staying with the microproject until the\nend!\n\n-- Daniel.\n"}]}