{"thread":{"id":"25779","subject":"Submodules or similar for exercise/exam management","startedAt":"2010-11-18T10:09:08Z","lastAt":"2010-11-22T13:56:19Z","messageCount":7,"participants":["Thomas Rast","Seth Robertson","Jens Lehmann","Junio C Hamano","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"156086","messageId":"201011181109.08345.trast@student.ethz.ch","threadId":"25779","inReplyTo":null,"subject":"Submodules or similar for exercise/exam management","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-11-18T10:09:08Z","receivedAt":"2010-11-18T10:09:08Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Hi all\n\n[Jens, I was hoping you'd have some insight, but of course everyone is\nwelcome to join in.  I honestly never thought I'd get into the\nsubmodule business so shortly after gittogether...]\n\n\nOut of $DAYJOB issues with tracking exercises and exams, I had a\nlittle brainstorming session with a fellow assistant yesterday.  It\nseems none of this has been done before?  Is anyone else interested\n(mainly in getting the requirements right so that the next poor soul\nwon't have to roll their own)?\n\nThe scenario is that we have a bunch of exercise questions stored in\none or several files, each living in a directory.  Then, we assemble\nthose into exercise sheets (super-repos) spanning a group of questions\n(submodules).  We would like to track this in such a way that changes\nto the questions propagate to other sheets (possibly in other\nlectures) that use them.\n\nTraining everyone in full git usage is probably not an option, so any\nsolution would have to have some frontend that can be explained\neasily.\n\nThe requirements we came up with are roughly:\n\n1) commit across all sub-repos at the same time (atomicity nice but\n   optional)\n\n  1b) tag across all sub-repos\n\n2) do recursive clone/update of one super-repo easily\n\n3) never need to be aware of repo boundaries or manipulate sub-repo\n\n4) sanely cope with branches etc. in case the user wants them\n\nA sample workflow might be:\n\n  foo clone git@example.com/some/super/repo\n  # time passes\n  cd repo\n  foo update\n  # work\n  foo status\n  foo diff\n  foo commit -m 'one message fits all'\n  # possibly, but this could also be commit --no-push instead\n  foo push\n\nMerges are expected to be rare but would happen every so often.\n\nIt seems repo can do (2) and (4) but violates (3) [requires use of\ngit-commit and others in the sub-repo].  git-subtree can do (1) and\n(2) but probably violates (3) [need to rebase changes to sub-repo\nbranch] and (4).  Submodules can do (2) and (4) but violate (3) and\ncurrently have no support for (1).  I think svn externals could do\n(1)-(3) but violate (4) and probably (1b).\n\nHas this been done before?  Or do we need to hack up a new wrapper\naround submodules?\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"156115","messageId":"201011181636.oAIGaRxm021543@no.baka.org","threadId":"25779","inReplyTo":"201011181109.08345.trast@student.ethz.ch","subject":"Re: Submodules or similar for exercise/exam management","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2010-11-18T16:36:27Z","receivedAt":"2010-11-18T16:36:27Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message <201011181109.08345.trast@student.ethz.ch>, Thomas Rast writes:\n\n    The requirements we came up with are roughly:\n\n    1) commit across all sub-repos at the same time (atomicity nice but\n       optional)\n      1b) tag across all sub-repos\n    2) do recursive clone/update of one super-repo easily\n    3) never need to be aware of repo boundaries or manipulate sub-repo\n    4) sanely cope with branches etc. in case the user wants them\n\nThis is exactly the sort of situation I wrote gitslave for.\nhttp://gitslave.sf.net  I have a superproject of many subprojects where\nI want to perform the same git operation on all of them at the same\ntime as if it was one giant repository (without the drawbacks of a\ngiant repository while losing as few as the benefits as possible).\n\nIt does not perform atomic commits across sub-repos, but otherwise\nhandles 1 and 1b.  2 and 4 are handled as well.  3 is mostly handled,\nand by mostly, I mean if you get, for instance, a merge conflict in\nthe superproject or the subproject, you would need to resolve that\nusing the normal git commands and then typically retry the gitslave\ncommand.  gitslave does not prevent you from using any git command in\nany way you might desire, it simply allows you to use \"gits\" instead\nof \"git\" when you want your command to run over all repos.\n\n    A sample workflow might be:\n\n      foo clone git@example.com/some/super/repo\n      # time passes\n      cd repo\n      foo update\n      # work\n      foo status\n      foo diff\n      foo commit -m 'one message fits all'\n      # possibly, but this could also be commit --no-push instead\n      foo push\n\nIn the gitslave world, this would be done:\n\ngit clone git@example.com/some/super/repo\ncd repo\ngits checkout master\n# time passes\ngits pull\n# work\ngits status\ngits diff\ngits commit -a -m 'one message fits none'\n# I am not familiar with the \"git commit --no-push\" option\ngits push\n\n    Merges are expected to be rare but would happen every so often.\n\ngits checkout featurebranch\n# work\ngits commit -a -m 'features are misunderstood'\ngits checkout master\ngits merge featurebranch\ngits tag v1.0 -a -m \"feature available\"\ngits push\ngits push --tags\n\n    Has this been done before?  Or do we need to hack up a new wrapper\n    around submodules?\n\nHopefully gitslave will work for you (or could be modified to work for\nyou).  It is not perfect but has been working well enough for a team\nof developers for over two years.\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"156139","messageId":"4CE5988F.7050309@web.de","threadId":"25779","inReplyTo":"201011181109.08345.trast@student.ethz.ch","subject":"Re: Submodules or similar for exercise/exam management","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-11-18T21:20:15Z","receivedAt":"2010-11-18T21:20:15Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Hi Thomas\n\nAm 18.11.2010 11:09, schrieb Thomas Rast:\n> The scenario is that we have a bunch of exercise questions stored in\n> one or several files, each living in a directory.  Then, we assemble1\n> those into exercise sheets (super-repos) spanning a group of questions\n> (submodules).\n\nThat sounds like each directory is maintained by a different group of\npeople and then there is another bunch of people choosing the content\nof the next exercise sheets, right?\n\n> We would like to track this in such a way that changes\n> to the questions propagate to other sheets (possibly in other\n> lectures) that use them.\n\nThis could mean you would want an 'always-tip' mode for the\nsubmodules, or am I misunderstanding you here?\n\n> Training everyone in full git usage is probably not an option, so any\n> solution would have to have some frontend that can be explained\n> easily.\n\nYup, makes sense (at least until something goes wrong, see 3). ;-)\n\n> The requirements we came up with are roughly:\n> \n> 1) commit across all sub-repos at the same time (atomicity nice but\n>    optional)\n> \n>   1b) tag across all sub-repos\n\n\"git commit\" and \"git tag\" could be taught the \"--recurse-submodule\"\noption (but the commit part only makes sense when \"git branch\" can\ndo that too, so you have something to commit on. And I think you\nwant to enable a - yet to be implemented - file-based recursive diff\nand status too, to see what changes your next commit will include).\nAnd until all that materializes for submodules, a script would have\nto do that.\n\n> 2) do recursive clone/update of one super-repo easily\n\nThat will be handled by recursive checkout and is already achieved\nby \"git clone --recursive\" (but at least in the first version both\ndon't support an \"always-tip\" mode).\n\n> 3) never need to be aware of repo boundaries or manipulate sub-repo\n\nI think that this requirement is the hardest for any solution I know\nof or can imagine, as you hit these boundaries sooner or later either\nwhen you want to commit, push and/or when you have to resolve merge\nconflicts.\n\n> 4) sanely cope with branches etc. in case the user wants them\n\nA \"--recurse-submodules\" option to \"git-branch\" might be what you\nwant here, but as this isn't there yet a script will have to do that\nfor now.\n\n> A sample workflow might be:\n> \n>   foo clone git@example.com/some/super/repo\n>   # time passes\n>   cd repo\n>   foo update\n>   # work\n>   foo status\n>   foo diff\n>   foo commit -m 'one message fits all'\n>   # possibly, but this could also be commit --no-push instead\n>   foo push\n> \n> Merges are expected to be rare but would happen every so often.\n> \n> It seems repo can do (2) and (4) but violates (3) [requires use of\n> git-commit and others in the sub-repo].  git-subtree can do (1) and\n> (2) but probably violates (3) [need to rebase changes to sub-repo\n> branch] and (4).  Submodules can do (2) and (4) but violate (3) and\n> currently have no support for (1).  I think svn externals could do\n> (1)-(3) but violate (4) and probably (1b).\n> \n> Has this been done before?  Or do we need to hack up a new wrapper\n> around submodules?\n\nIf you would base that on submodule functionality, you would have\nto hack up a wrapper script for the foreseeable future because the\n\"fully integrated\" world view you seem to need is not worked on yet\n(and I didn't see anyone coming forward to do that).\n\nI took a cursory glance at \"gitslave\" Seth mentioned, it might do\nwhat you want, but I can't tell for sure as I never used it myself.\n\n\nJens\n"},{"id":"156142","messageId":"7vsjyyb6ui.fsf@alter.siamese.dyndns.org","threadId":"25779","inReplyTo":"4CE5988F.7050309@web.de","subject":"Re: Submodules or similar for exercise/exam management","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-11-18T22:32:53Z","receivedAt":"2010-11-18T22:32:53Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n>> 3) never need to be aware of repo boundaries or manipulate sub-repo\n>\n> I think that this requirement is the hardest for any solution I know\n> of or can imagine, as you hit these boundaries sooner or later either\n> when you want to commit, push and/or when you have to resolve merge\n> conflicts.\n\nJust a quick sanity check.  When this \"requirement\" makes sense, does the\nwhole thing need to be a superproject with bunch of submodules, and why?\n"},{"id":"156145","messageId":"20101118234916.GB21621@burratino","threadId":"25779","inReplyTo":"7vsjyyb6ui.fsf@alter.siamese.dyndns.org","subject":"Re: Submodules or similar for exercise/exam management","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-18T23:49:16Z","receivedAt":"2010-11-18T23:49:16Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n>>> 3) never need to be aware of repo boundaries or manipulate sub-repo\n>>\n>> I think that this requirement is the hardest for any solution I know\n>> of or can imagine, as you hit these boundaries sooner or later either\n>> when you want to commit, push and/or when you have to resolve merge\n>> conflicts.\n>\n> Just a quick sanity check.  When this \"requirement\" makes sense, does the\n> whole thing need to be a superproject with bunch of submodules, and why?\n\nIn this example: because the submodule represents individual\nquestions that are used by multiple exams.\n\nAnother instance of the same (slightly uncomfortable :)) practice:\nsuppose a certain chapter is part of multiple works I have\npublished --- maybe in an article and a larger book.  When working on\nthe book:\n\n - I do not want to make changes to that chapter and forget to commit\n   them.\n\n - After making changes to many chapters, I do not want the fuss of\n   going from chapter to chapter and commiting them one by one.\n\n - I certainly do not want to publish a version of the book that\n   \"includes\" versions of the chapters as dead links, so to speak.\n   That is, when I publish the current version of the book, I want\n   to publish the current version of all chapters, too.\n\n - When starting work on the book again after long absence, I would\n   like to be able to see and have the chance to adopt changes to\n   chapters made as part of this book and as part of others.\n\nIf I understand correctly.\n"},{"id":"156312","messageId":"201011221420.52872.trast@student.ethz.ch","threadId":"25779","inReplyTo":"201011181636.oAIGaRxm021543@no.baka.org","subject":"Re: Submodules or similar for exercise/exam management","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-11-22T13:20:52Z","receivedAt":"2010-11-22T13:20:52Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Seth Robertson wrote:\n> git clone git@example.com/some/super/repo\n> cd repo\n> gits checkout master\n> # time passes\n> gits pull\n> # work\n> gits status\n> gits diff\n> gits commit -a -m 'one message fits none'\n> # I am not familiar with the \"git commit --no-push\" option\n> gits push\n\nThanks a lot for pointing that out!  I just did a few simple tests and\nit looks like it's the tool for the job.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"156318","messageId":"201011221456.19784.trast@student.ethz.ch","threadId":"25779","inReplyTo":"20101118234916.GB21621@burratino","subject":"Re: Submodules or similar for exercise/exam management","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-11-22T13:56:19Z","receivedAt":"2010-11-22T13:56:19Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Jonathan Nieder wrote:\n> Junio C Hamano wrote:\n> > Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n> >>> 3) never need to be aware of repo boundaries or manipulate sub-repo\n> >>\n> >> I think that this requirement is the hardest for any solution I know\n> >> of or can imagine, as you hit these boundaries sooner or later either\n> >> when you want to commit, push and/or when you have to resolve merge\n> >> conflicts.\n> >\n> > Just a quick sanity check.  When this \"requirement\" makes sense, does the\n> > whole thing need to be a superproject with bunch of submodules, and why?\n> \n> In this example: because the submodule represents individual\n> questions that are used by multiple exams.\n\nIndeed.\n\nI tried to not go too deeply into the status quo to keep the original\nmail short, but if you really care, here goes (there's nothing else\nbelow this point so you can also stop reading):\n\nIn my own group (theoretical CS), we have a Big Ugly SVN repository\nwith a directory per class taught, with subdirectories for the\nexercises of every year.  Either at the beginning or during the year,\nwe copy the previous exercises and solutions to a new directory and\nmodify/fix them to taste.  Similar for exams.\n\nThis works reasonably well because all our lectures are on mostly\ndisjoint topics, so we rarely share anything between them.  The\ndirectories act more as a \"tag\" of the old version than anything else,\nand also help lazy users find the old version without talking to SVN.\nConverting it to git would be fairly trivial since the usage maps\nstraight to lecture-projects with tags at the end of every semester,\nbut largely due to inertia nobody wants to do it.\n\nThe statistics institute where my colleague works is different in two\nmajor respects.\n\nFirst, they do not currently have any kind of enforced versioning.\nThere apparently are a bunch of directories along similar lines as the\nabove, but they live on a big shared NFS and thus it's easy to lose\nexisting state simply by overwriting it.  This is good for the\npurposes of this discussion because we can weigh the pain of living\nwith the existing solution against the pain of using the new solution\n;-)\n\nMore importantly, and this spawned my question, they have a lot more\noverlaps between lectures.  (I assume this is because they give\nstatistics courses at various levels of difficulty to various other\ndepartments/subjects of study, but I don't really know.)\n\nThus it often happens that an exercise is copied from lecture A to B,\nthen later some mistake is fixed in A and the fix does not propagate.\nWorse, for whatever reason they might later copy back the exercise\nfrom B to A and thus have a regression.\n\nHandling at least this case sanely would already be good.  The\nrequirement that Junio criticized above is geared not towards sanity\nof the use-case, but towards not making adoption so complicated as to\nmake it impossible.\n\nIt is true that in cases where submodules really pay off, such\nexercises (or perhaps more realistically, guides and lecture notes)\nbeing customized slightly for each course.  This would need branches.\nFor now I am putting this case a bit into the background since it can\nbe solved by working on such documents in a separate repo, splitting\nthe customizations into branches and merging the fixes from the main\ncopy; and then just binding the appropriate version as a sub-repo.\n\nAnd yes, this would in theory also go for books as you described:\n\n> suppose a certain chapter is part of multiple works I have\n> published --- maybe in an article and a larger book.  When working on\n> the book:\n> \n>  - I do not want to make changes to that chapter and forget to commit\n>    them.\n> \n>  - After making changes to many chapters, I do not want the fuss of\n>    going from chapter to chapter and commiting them one by one.\n> \n>  - I certainly do not want to publish a version of the book that\n>    \"includes\" versions of the chapters as dead links, so to speak.\n>    That is, when I publish the current version of the book, I want\n>    to publish the current version of all chapters, too.\n> \n>  - When starting work on the book again after long absence, I would\n>    like to be able to see and have the chance to adopt changes to\n>    chapters made as part of this book and as part of others.\n\nThough I wonder how that pans out in practice since it is rarely\npossible to just \"drop in\" a chapter without rewriting significant\nparts of it to match the rest of the book.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"}]}