{"thread":{"id":"45343","subject":"[GSoC] Discussion of \"Submodule related work\" project","startedAt":"2017-03-10T11:28:37Z","lastAt":"2017-03-15T20:48:50Z","messageCount":7,"participants":["Valery Tolstov","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"313743","messageId":"1489145258.10535.0@smtp.yandex.ru","threadId":"45343","inReplyTo":null,"subject":"[GSoC] Discussion of \"Submodule related work\" project","fromName":"Valery Tolstov","fromEmail":"me@vtolstov.org","sentAt":"2017-03-10T11:27:38Z","receivedAt":"2017-03-10T11:28:37Z","isPatch":false,"sender":{"key":"me@vtolstov.org","avatar":"https://avatars.githubusercontent.com/u/1188278?v=4"},"body":"Have some questions about \"Submodule related work\" project\n\nFirst of all, I would like to add this task to the project, if I'll \ntake it:\nhttps://public-inbox.org/git/1488913150.8812.0@smtp.yandex.ru/T/\nWhat do you think about this task?\n\n > Cleanup our test suite. Do not use a repo itself as a submodule for \nitself\n\nNot quite familiar with submodules yet, why this is considered to be \nineligible\n(i.e. using repo as a submodule for itself)?\n\n > (Advanced datastructure knowledge required?) Protect submodule from \ngc-ing\n > interesting HEADS.\n\nCan you provide a small example that shows the problem, please?\nAnd why advanced datastructure knowledge is expected?\n\nMaybe you have something else about this project to say.\n\nThanks,\n  Valery Tolstov\n\n\n"},{"id":"313767","messageId":"CAGZ79kZHjpptQAF4NM5y47PF+YXHRADAhzFUYG_+CSwRozo+qQ@mail.gmail.com","threadId":"45343","inReplyTo":"1489145258.10535.0@smtp.yandex.ru","subject":"Re: [GSoC] Discussion of \"Submodule related work\" project","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-03-10T18:47:43Z","receivedAt":"2017-03-10T18:47:50Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Mar 10, 2017 at 3:27 AM, Valery Tolstov <me@vtolstov.org> wrote:\n> Have some questions about \"Submodule related work\" project\n>\n> First of all, I would like to add this task to the project, if I'll take it:\n> https://public-inbox.org/git/1488913150.8812.0@smtp.yandex.ru/T/\n> What do you think about this task?\n\nThat is a nice project, though my gut feeling is that it is too small\nfor a GSoC project on itself.\n\n>> Cleanup our test suite. Do not use a repo itself as a submodule for itself\n>\n> Not quite familiar with submodules yet, why this is considered to be\n> ineligible (i.e. using repo as a submodule for itself)?\n\n(a bit of background on submodules)\n\nman gitglossary (then searching for submodule):\n       submodule\n           A repository that holds the history of a separate project inside\n           another repository (the latter of which is called superproject).\n\n       superproject\n           A repository that references repositories of other projects in its\n           working tree as submodules. The superproject knows about the names\n           of (but does not hold copies of) commit objects of the contained\n           submodules.\n\nAn example that I just found on Github[1]. It is a game\n(so it includes graphics, game code etc). But it makes use of a library[2],\nwhich could be used by different projects.\n\n[1] https://github.com/stephank/orona\n[2] https://github.com/stephank/villain\n\nNow why would a repo be ineligible to use itself as a submodule?\nThere is nothing wrong with it *technically* (which is why we do such things\nin the test suite.)\n\nBut what are the use cases for it? Why would a project bind itself\nas a submodule (you can get the same effect of having the source code\nby just checking out that other version.) ? Well now that I think about it,\nit may be useful if you want to test old versions of yourself for e.g.\nnetworking compatibility. But for that you'd probably still not use submodules.\n\nSo the use case of using submodules for another copy of itself is\n*very rare* if it exists at all out there. And the Git test suite\nshould rather test\nuse cases that are not these weird corner cases, but rather pay attention to\nthe common case first.\n\nI thought this project would have been solved parially already, but I was wrong.\n($ git grep \"submodule add \\./\\.\"). This also doesn't seem large enough for\na summer project, after thinking about it further.\n\n>> (Advanced datastructure knowledge required?) Protect submodule from gc-ing\n>> interesting HEADS.\n>\n> Can you provide a small example that shows the problem, please?\n\nLet's use this example from above:\n\n$ git clone --recursive https://github.com/stephank/orona\n    # now we have 2 repositories, the orona repo as well as its submodule\n    # at node_modules/villain\n    #\n    # \"Let's inspect the Readmes/license files, if they are ok to use\n    # Oh! the submodule is MIT licensed but doesn't have the full\n    # license text, I can contribute and make a patch for it.\"\n$ cd node_modules/villain\n$ $EDIT LICENSE\n$ git add LICENSE\n$ git commit -a -m \"add license full text\"\n$\n$ cd ../.. # go back to the superproject\n$ git add  node_modules/villain\n$ git commit -a -m \"update game to include latest lib\"\n$ git checkout -b \"fix_license\"\n    # note how I forget to actually push it / pull request it!\n    # All we need for the demonstration is a local commit\n    # in the submodule that is referenced by the superproject...\n    #\n    # ... \"Let's test the pristine copy of the game!\" ...\n$ git checkout origin/master\n$ git submodule update\n    # ... which gets lost here. The submodule commit\n    # is only referenced by a superproject commit.\n\n.. time passes ..\n\n    # \"My disk is so full, maybe I can clean up all these random\n    # accumulated projects, to have more disk space again.\"\n    # my cleanup script may do this:\n\n$ cd node_modules/villain\n$ git reflog expire --all --expire=all\n$ git gc --prune=all\n$ cd ../..\n\n$ git branch\n    # \"Oh what about this 'fix_license branch' ?\n    #  Did I actually send that upstream?\"\n$ git checkout fix_license\n$ git submodule update\nerror: no such remote ref 96016818b2ed9eb0ca72552b18f8339fc20850b4\nFetched in submodule path 'villain', but it did not contain\n96016818b2ed9eb0ca72552b18f8339fc20850b4. Direct fetching of that\ncommit failed.\n\n> And why advanced datastructure knowledge is expected?\n\nI am not quite sure how to approach this problem, so I put\na \"warning; it may be complicated\" sticker on it. ;)\n\nThe problem is that a submodule until now was considered\nits own repository, in full control what to keep and delete,\nhow to name its branches and so on.\n\ngit-gc only pays attention to commits (and its history) of all\nbranches and commits mentioned in the reflog.\n(which is why we had to delete the reflog, and as we\nwere making the license commit on a \"detached HEAD\",\nthere was no need to delete its branch).\n\nHowever it should also consider all commits referenced\nby the superproject valuable.\n\nIn this case the superproject has a branch \"fix_license\",\nso that commit is considered too valuable for gc in the\nsuperproject, but it breaks with the submodule pointer\nas the pointer changes in the superproject, but the\ngc operation in the submodule doesn't care.\n\nOne way to fix it is to figure out if there is a superproject\nat gc time and then collect all valuable hashes (submodule\npointers) before actually performing the gc.\n\nBut that may be expensive, so we would rather record\nit on the fly, e.g. when making the commit in the superproject\nwe'd record in the submodule that the given hash by the\nsubmodule pointer is valuable.\n\nThis could be done by having a ref (=branch) in the submodule\nthat points at all the interesting submodule commits.\n\nSo despite being prominent on the ideas page (because of a lot\nof text), this may be controversial how to actually solve it.\n\n>\n> Maybe you have something else about this project to say.\n\nIf I remember correctly, shell -> C conversion projects are\neasy (both for writing the code as well as for mentoring)\n\n> git archive(/bundle) to have a --recurse-submodules flag\n> to include the submodule contents.\n\nis an actual interesting project as well despite its short description.\n\nThanks,\nStefan\n"},{"id":"313776","messageId":"20170310193040.16816-1-me@vtolstov.org","threadId":"45343","inReplyTo":"CAGZ79kZHjpptQAF4NM5y47PF+YXHRADAhzFUYG_+CSwRozo+qQ@mail.gmail.com","subject":"Re: [GSoC] Discussion of \"Submodule related work\" project","fromName":"Valery Tolstov","fromEmail":"me@vtolstov.org","sentAt":"2017-03-10T19:30:40Z","receivedAt":"2017-03-10T19:30:54Z","isPatch":false,"sender":{"key":"me@vtolstov.org","avatar":"https://avatars.githubusercontent.com/u/1188278?v=4"},"body":"So... I thought those items listed in \"Submodule related work\" are\nconsidered too small to be complete projects separately, and they\nare just \"subprojects\" of bigger project (maybe I have this thought\nbecause I can't estimate complexity before truly digging in).\nIn your response you talk about them as independent projects...\nThis means I can take only any one of them as starting point for\nmy proposal? Or maybe I misunderstood you?\n\n\nThanks,\n  Valery Tolstov\n"},{"id":"313798","messageId":"20170310211348.18887-1-me@vtolstov.org","threadId":"45343","inReplyTo":"20170310193040.16816-1-me@vtolstov.org","subject":"Re: [GSoC] Discussion of \"Submodule related work\" project","fromName":"Valery Tolstov","fromEmail":"me@vtolstov.org","sentAt":"2017-03-10T21:13:48Z","receivedAt":"2017-03-10T21:14:02Z","isPatch":false,"sender":{"key":"me@vtolstov.org","avatar":"https://avatars.githubusercontent.com/u/1188278?v=4"},"body":"> This means I can take only any one of them as starting point for\n> my proposal?\n\nIf it is true, than i'll try to take sh->C transition for submodule\ncommand, and as addirional part of my whole project also this:\nhttps://public-inbox.org/git/1488913150.8812.0@smtp.yandex.ru/T/\n\n>> Have some questions about \"Submodule related work\" project\n>>\n>> First of all, I would like to add this task to the project, if I'll take it:\n>> https://public-inbox.org/git/1488913150.8812.0@smtp.yandex.ru/T/\n>> What do you think about this task?\n>\n> That is a nice project, though my gut feeling is that it is too small\n> for a GSoC project on itself.\n\nDoes it sound good? If does, then I'll begin to work on my proposal.\n\nThanks,\n  Valery Tolstov\n"},{"id":"313918","messageId":"CAGZ79kbLr=aSu-g83obmj3XCMMYbuody6Q_FreCmQcqpyX+_rQ@mail.gmail.com","threadId":"45343","inReplyTo":"20170310211348.18887-1-me@vtolstov.org","subject":"Re: [GSoC] Discussion of \"Submodule related work\" project","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-03-13T18:24:58Z","receivedAt":"2017-03-13T18:25:07Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> So... I thought those items listed in \"Submodule related work\" are\n> considered too small to be complete projects separately, and they\n> are just \"subprojects\" of bigger project (maybe I have this thought\n> because I can't estimate complexity before truly digging in).\n\nWhen writing these points, I was not sure about the complexity\nmyself, but rather I wanted to produce a lot of different potential\nprojects, which can be discussed if they sound exciting and are\nof good size.\n\n> In your response you talk about them as independent projects...\n> This means I can take only any one of them as starting point for\n> my proposal? Or maybe I misunderstood you?\n\nWell I think some of them are too small to stand alone for a full GSoC\nproject. Others have a good size and complexity for GSoC already.\n\n> If it is true, than i'll try to take sh->C transition for submodule\n> command,\n\nFor shell -> C transitions, see\n65e1449614d\nb7d2a15b9f5\n307de75c4\ndec034a34e\nas all of them are rewrites from sh -> C for different commands.\nYou might find common patterns (e.g. what is all needed for a conversion,\nsuch as slight updates to tests or documentation; certainly updating\nthe build process in the Makefile, and of course the code translated).\n\nMost of these conversions, start out with a patch that is quite a literal\ntranslation and then afterwards add in optimizations.\n\nAnother approach for the conversion is outlined in73c2779f42\n(builtin-am: implement skeletal builtin am).  I am not sure how\neasy this approach is for a submodule specific command.\n\n> and as addirional part of my whole project also this:\n> https://public-inbox.org/git/1488913150.8812.0@smtp.yandex.ru/T/\n\nYeah, that is also a good task :) Thanks for bringing it up.\n\n> Does it sound good? If does, then I'll begin to work on my proposal.\n\nSure.\n\nThanks,\nStefan\n"},{"id":"314169","messageId":"20170315204324.9875-1-me@vtolstov.org","threadId":"45343","inReplyTo":"CAGZ79kbLr=aSu-g83obmj3XCMMYbuody6Q_FreCmQcqpyX+_rQ@mail.gmail.com","subject":"Re: [GSoC] Discussion of \"Submodule related work\" project","fromName":"Valery Tolstov","fromEmail":"me@vtolstov.org","sentAt":"2017-03-15T20:43:24Z","receivedAt":"2017-03-15T20:43:50Z","isPatch":false,"sender":{"key":"me@vtolstov.org","avatar":"https://avatars.githubusercontent.com/u/1188278?v=4"},"body":"I have a thought. At the moment when submodule command is already translated\nto C, possibly we want to rename submodule--helper.c, and maybe hide\nsome of it's functions from subcommands list. Is there any examples of\nsimilar situation that already happened before?\nDon't quite sure about this.\n\nThanks,\n  Valery Tolstov\n"},{"id":"314170","messageId":"CAGZ79kZU555GyMNkuED2efep9oYAag_FJ7ETRPH2tvipLO9K4Q@mail.gmail.com","threadId":"45343","inReplyTo":"20170315204324.9875-1-me@vtolstov.org","subject":"Re: [GSoC] Discussion of \"Submodule related work\" project","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-03-15T20:48:41Z","receivedAt":"2017-03-15T20:48:50Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Mar 15, 2017 at 1:43 PM, Valery Tolstov <me@vtolstov.org> wrote:\n> I have a thought. At the moment when submodule command is already translated\n> to C, possibly we want to rename submodule--helper.c, and maybe hide\n> some of it's functions from subcommands list. Is there any examples of\n> similar situation that already happened before?\n\nI would keep the submodule--helper around as it is\nand maybe even promote it to be a \"plumbing\" command.\nCurrently it is just an undocumented, internal command.\n\nOnce all commands in git-submodule.sh are translated to C\n(when they all look similar to \"git submodule init\", which\nis just a relay in the shell script), then we can make a full grown\nbuiltin/submodule.c that calls the functions of the submodule--helper\ndirectly in C.\n\nStefan\n"}]}