{"thread":{"id":"45586","subject":"[RFC] [GSoC] Proposal Draft for GSoC 2017 : Incremental Rewrite of git-submodules","startedAt":"2017-04-01T15:40:45Z","lastAt":"2017-04-01T15:40:45Z","messageCount":1,"participants":["Prathamesh Chavan"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"316010","messageId":"CAME+mvXBuLbbRJu1DAA8o-u6DeZATKypH=W=hPEks3KL5WMX+Q@mail.gmail.com","threadId":"45586","inReplyTo":null,"subject":"[RFC] [GSoC] Proposal Draft for GSoC 2017 : Incremental Rewrite of git-submodules","fromName":"Prathamesh Chavan","fromEmail":"pc44800@gmail.com","sentAt":"2017-04-01T15:40:37Z","receivedAt":"2017-04-01T15:40:45Z","isPatch":false,"sender":{"key":"pc44800@gmail.com","avatar":"https://avatars.githubusercontent.com/u/17272661?v=4"},"body":"Hello everyone,\n\nThis is the first draft of my project proposal. I decided to change my project\nsince the project I initially intended to do and also proposed was slightly\na smaller project for 13 weeks. Also when I came across this project,\nand also found out that there are left over bits (git submodule $cmd\n$pathspec may want to error out when the $pathspec does not match any\nsubmodules) which also can be included in this conversion process, I was\nmotivated to change my project and hence made my proposal regarding it.\n\nAlso I wish to complete the previously proposed task, probably after finishing\nGSoC first.\n\nHere is a Google doc for my proposal,\nhttps://docs.google.com/document/d/1krxVLooWl--75Pot3dazhfygR3wCUUWZWzTXtK1L-xU\n\nBelow I have also included the text only version of my proposal, so that\nall the discussion related to it would also be there on the public-inbox of git.\nIt would be great to have your suggestion, so that I can improve it futher.\n\nThanks,\nPrathamesh Chavan\n---\n\nIncremental Rewrite of git-submodules\n01.04.2017\n\nAbout Me\n\nName              Prathamesh Chavan\nUniversity        Indian Institute of Technology, Kharagpur\nMajor             Computer Science and Engineering(Dual Degree)\nEmail             pc44800@gmail.com\nIRC               pratham_pc\nBlog              pratham-pc.github.io\nContact No.       +91-993-235-8333\nTime Zone         IST (UTC +5:30)\n\n\nBackground\n\nI am a second-year student from the Department of Computer Science and\nEngineering IIT Kharagpur. I’m an active member at Kharagpur open\nsource society and also of Kharagpur Linux Users Group since my first\nyear. I got introduced to open source, Linux and git since the\nbeginning of my first year.\nI use to try to complete any task on my laptop without the use of the\ncursor and hence eventually got familiar with shell script and in the\nsame manner, I still find something new every day. I’m also quite\nfamiliar with C language. I always wanted to get involved in the\ndevelopment process. As Google Summer of Code is a great way to\nachieve it, hence I would like to participate.\nThis is my first attempt to GSoC. Since I use Git on regular basis\nand would also continue to use it, I would love to be a part of its\ncontributors and hence would also be able to contribute to the project\neven after GSoC.\n\nThe Project\n\nAbstract\n\nTo manage the external dependencies, git submodule commands have been\nfrequently used by the developers in their projects. But as most of the\nsubcommands of git submodule are still written in shell scripts (All, but\ngit submodule init), my project intends to convert the subcommands into\nC code, thus making them builtins. This will increase Git's portability\nand hence the efficiency of working with the git-submodule commands.\nThe function cmd_init has been ported to its built-in successfully and\nsimilar operation needs to be done to the remaining functions.\n\nProblems encountered while using Shell Scripts\n\nGit submodule subcommands are currently implemented by using shell script\n'git-submodule.sh'. There are several reasons why we'll prefer not to\nuse the shell script:\n\n1. Less Efficient\n\nSince being a part of shell script, the commands of git submodule do not\nhave access to the low-level internal API of git, which are designed to\ncarry out the task more efficiently.\nAlso, since the subcommands in  ‘git-submodule.sh’ do not have direct\naccess to the native low-level internal API's of git, carrying out even\ntrivial processes make git be spawned into a separate process with the\nhelp of plumbing functions. Also since everything comes from external\ncommands even in the shell scripts, everything needs to undergo through\na fork and exec. Whenever fork takes place, the git command requires\nanother fully functioning POSIX shell. This results in making the git\nsubmodule commands slow and relatively inefficient, and also reduces git's\nportability, since it is difficult to port such commands to a non-POSIX\nan environment like Windows.\n\n2. Spawning makes the process slower in many cases\n\nMost of the operations in the shell scripts require the scripts to call\nthe external executables, even for performing trivial tasks. When we\ntake a look at the file 'git-submodule.sh', then its is seen that\nmany git-executables like : 'git config', 'git checkout' 'git add',\n'git rev-list', 'git fetch', 'git rev-parse', 'git describe', 'git\nhash-object', 'git log' and since these and other shell command comes\nfrom other executables, every command undergoes a through a fork and\nexec, thus making the processes slow.\n\n3. Additional dependencies are introduced\n\nWhen we take a look at the 'git-submodule.sh' file, we can notice that\nit is dependent upon various other executables like: 'test', 'printf',\n'echo', 'sed', 'grep', 'mkdir', 'rm'. This leads to the requirement of\nadding these various dependencies to a non-POSIX environment.\n\n4. Coverage checking is not implemented for shell scripts:\n\nSince currently kcov is not implemented, checking the coverage of the shell\nscripts is still an issue. It was also proposed in previous GSoC proposals, but\nwasn’t implemented.  I also checked out its implementation for linux kernel\nproject , but it seems quite difficult to cover this in my GSoC proposal.\nInstead, I would run ‘make coverage’ after I port all functions to C. This will\nalso help me find the code for which the test suites aren’t created and\nimprove the test coverage over submodule code.\n\nBrief introduction of git-submodules\n\nGit submodules are used for including a git repository inside a\nsubdirectory of another git repository. Submodules have their own\nhistory, and the it doesn't interfere with the superproject's history.\nHence it helps the developers include external dependencies like third\nparty libraries in their project source tree. Submodules have been there\nsince quite a while, yet there are still areas to work on.\nCurrently the git submodule is implemented using various file which are:\n'git-submodule.sh', 'submodule.c', 'submodule.h' and\n'builtin/sumodule--helper.c'. Hence to convert the script file to\nbuitin, I would first port and implement all the algorithmic and\nnon-algorithmic part in 'submodule.c' and in 'submodule--helper.c'\nrespectively and then finally create 'builtin/submodule.c', which\ncan call any function from submodule--helper.c directly from C itself.\nEventually, I will be ensuring that the user interface part is\nimplemented in the builtin/submodule.c , having submodule--helper.c\nfor using it as plumbing command and will use submodule.c\nfor writing the reusable part for the code.\n\nGoals\n\n1. Convert submodule subcommands to C.\n2. Finally creating a builtin/submodule.c\n\nPlan\n\n1. Understand the code written in 'git-submodule.sh'\n\n   In the community bonding period, my main aim will be to understand\n   the code is written in the 'git-submodule.sh' and get a clear\n   understanding of how each submodule subcommand is being implemented.\n   Understanding the subcommand's logical flow is very important since\n   I will be retaining it in the rewritten code as well.\n\n2. Understand what API's could be used and learn about these API's and\n   also learn about other plumbing functions which will be required.\n\n   In the community bonding period, once I understand the code written\n   in 'git-submodule.sh', I would start the discussion of the procedure\n   which will be appropriate for the rewriting the code to builtin.\n   Since the procedure of rewriting is not simply translating the code\n   into another language, but also to utilize the API's and functions\n   available in the later language and to create new API to ease the\n   future implementations. Here I will learn about the various API's\n   I'll be using during the conversion, will clarify my doubts about\n   the implementation of the functions and also check for the need of\n   creation of any new APIs. Also, as submodule--helper is a plumbing function,\n   other such functions will also be used in the conversion and hence is\n   learning about them will be useful.\n\n3. Start Converting Submodule subcommands\n\n   As done in the porting of the submodule subcommand 'init', I'll be\n   first porting to the submodule--helper, and call the function from\n   git-submodule.sh till I convert every subcommand. Then I'll make the\n   required changes in the command structure to call submodule on\n   place of submodule--helper and also make the similar changes for\n   the complete conversion.\n   Also while conversion of a single functions to builtin is over,\nI’ll immediately\n   perform step 4 also, so that while sending out patches, they are well\n   tested, as well as the documentation is also updated at the same time.\n4. Testing the rewritten code, Optimizing, Documenting and other\nchanges required:\n\n   The working of rewritten code can be checked with the help of\n   available test suites. Also taking into account the bug to be fixed :\n   git submodule $cmd $pathspec may want to error out when the\n   $pathspec does not match any submodules. [1],\n   there will also need to add new test cases in the suite. Also\n   the makefile would require changes for updating the build\n   process. Once testing is completed, I would start working on\n   optimizing various functions written in C so that the code will be\n   more efficient. Then finally I would document all the changes. Once\n   all of this is done, after sending the patches, the reviews will also\n   help in the processes of optimizing the code, as well as improving\n   the documentation. Also, I will try to send patches as early as possible\n   since, the reviewing of patches takes time, and till then I may start to\n   work on next function as well. Also I will try my best to get the code\n   merged in time, so as to ensure that I’m on the correct path at any\n   point of time.\n\nPotential difficulties during the conversion\n\n1. The rewritten code may introduce bugs, which formerly weren't\n   present. Most of these bugs would be detected by the test suites,\n   which exists. Apart from them, I would stick to following the same\n   the structure of the code, as is written in the script form. And even\n   after that, if still there are introduced bugs left, I have allotted\n   time for fixing these bugs.\n2. As it's witnessed in the porting of 'git submodule init' command,\n   that the code written in C language takes more lines than the\n   scripted code. Hence, this problem may be addressed by creating\n   functions and once we identify the recurring code structures, we\n   may also want to want to create a GIT API of them for future\n   Convenience.\n\nTimeline(Tentative)\n\nApril 1 to April 21\n\nSince I started up with this project late. Hence in this time period,\nI would like to convert the submodule subcommand 'foreach' from the\nshell script to builtin. Also, in this time span, I would go through\nthe codebase as much as possible.\n\nApril 22 to April 28\n\nEnd Semester examinations at college. Also I have some small class tests\nIn between 5th April to 9th April, but other than this, I do not\nhave any other test in between the described timeline( 1 April to\nAugust 31, 2017).\n\nApril 29 to May 15:\n\nWe like to spend this time, reading the codebase to understand the\nsoftware very well and also get my doubts about implementation clarified\nfrom the mentor.Also, this period will be utilized by me to understand\nthe various git APIs so that they can be used for the code's implementation.\nAlso, in this time span, I'll be discussing the way I\nhave decided to proceed further, and address any issue related to that.\nSince my next semester at college starts on July 14th, I will prefer to\nstart actual coding by 16th May itself, giving sufficient time for\nunderstanding the and for community bonding as well.\n\nMay 16 - July 3: (Week 1 - Week 7):\n\nThere are around 7 subcommands(excluding for each) of git submodule in\n'git-submodule.sh'. I would require on an average 4 days to complete\nthe conversion of each subcommand. And finally to merge 'submodule.c'\nand 'builtin/submodule--helper.c'. Also during this process of code the\nconversion itself, I would like to address the BUG:  git submodule $cmd\n$pathspec may want to error out when the $pathspec does not match any\nsubmodules. [1]\nI will also be keeping some additional buffer time here.\n\n\nJuly 4 to July 17: (Week 8 - Week 9):\n\nIn this span of 2 weeks, I would like to fix the bugs introduced by the\nnew code. Also, will be working on fixing the issue: git submodule $cmd\n$pathspec may want to error out when the $pathspec does not match any\nsubmodules. [1]\n\nJuly 18 to July 31: (Week 10 - Week 11):\n\nCheck the test coverage of the code, and improve it, wherever required.\nAlso will work on optimizing the code written, as required.\n\nAugust 1 to August 7: (Week 12):\n\nI would keep this final week to polish the code in written till then,\nand give it the final touches as required.\n\nAugust 8 to August 29 :\n\nI would like to keep this time as buffer, for fixing any bugs created by the\nrewrite. Also in this time I would like to work on my wish list if I have time\nremaining.\n\nWish-list\n\nI have already discussed these topics before [2], but since I was more\ninterested in doing the above project, I’m adding these to my wishlist.\n1. \"git -C sub add .\" might behave just like \"git add sub\"\n2. Teach \"git -C <submodule-path> status\" in an unpopulated\n   submodule to report the submodule being unpopulated, do not\n   fall back to the superproject.\n3. Teach \"git log -- <path/into/submodule/and/further>\" to behave\n   like \"git -C <path/into/submodule> log -- <and/further>\"\n\nMicroproject Attempted\n\nThe microproject which I attempted is: “Avoid pipes in git related\ncommands for test suite”. I hope it would soon get merged.\nYou can find my patch here.[3]\n\nPlans for Summer\n\nI recently went through an ACL-reconstruction surgery, and hence in\nsummers I'll be just continuing with the physiotherapy I'm\nundergoing from last 3 months.\n\n\n[1] : https://git-blame.blogspot.in/p/leftover-bits.html\n[2] : https://public-inbox.org/git/CAME+mvW1x6fnGKt1_auGOp+wFYFR=Y_Qhxfd50E7KFe6t+X4kw@mail.gmail.com/\n[3] : https://public-inbox.org/git/20170324120433.2890-1-pc44800@gmail.com/\n"}]}