{"thread":{"id":"52236","subject":"tying files to git repository","startedAt":"2019-11-11T15:28:29Z","lastAt":"2019-11-26T15:28:28Z","messageCount":2,"participants":["Jim Edwards","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"385911","messageId":"CAPuR+ZhwnHCp8j76PscuBqG2rCLkgG0+6Y3WwLgNRhaoj4OR9A@mail.gmail.com","threadId":"52236","inReplyTo":null,"subject":"tying files to git repository","fromName":"Jim Edwards","fromEmail":"jedwards@ucar.edu","sentAt":"2019-11-11T15:28:15Z","receivedAt":"2019-11-11T15:28:29Z","isPatch":false,"sender":{"key":"jedwards@ucar.edu","avatar":null},"body":"Hi,\n\nI am a developer of scientific software, as science software we expect\nand encourage users to modify source code in order to customize their\nexperiments.   The mechanism which has been developed to do this\npredates having source code in git and we are trying to figure out a\nway to minimize changes to the scientists workflow, while leveraging\nthe power of git to improve the process.  In the workflow the\nscientist creates a 'case' using script in the repository to create a\ndirectory structure from which they will conduct their experiment.\nPart of that directory structure is a SourceMods directory where the\nuser can drop modified source files that will be compiled in place of\na file of the same name in the source tree.  These files are sometimes\nlong lived and passed from case to case and even user to user and it\nis not hard to have the files get out of sync with the source tree.\nWe have discussed at length removing the SourceMods capability and\nrequiring scientists to create branches in git, but there is a lot of\nresistance to this in the community.   What I would like to explore is\nallowing scientists to keep the method that they are used to but at\nthe same time tying these modified files to their history in git.   Is\nthere a way to get the git metadata associated with an individual file\nso that we can treat that file as if it were in the repo?\n\n-- \nJim Edwards\n\nCESM Software Engineer\nNational Center for Atmospheric Research\nBoulder, CO\n"},{"id":"387086","messageId":"07afaf21-9731-9068-962b-2e089ddd576c@iee.email","threadId":"52236","inReplyTo":"CAPuR+ZhwnHCp8j76PscuBqG2rCLkgG0+6Y3WwLgNRhaoj4OR9A@mail.gmail.com","subject":"Re: tying files to git repository","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2019-11-26T15:28:25Z","receivedAt":"2019-11-26T15:28:28Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Jim\n\nThis \n(https://public-inbox.org/git/CAPuR+ZhwnHCp8j76PscuBqG2rCLkgG0+6Y3WwLgNRhaoj4OR9A@mail.gmail.com/) \nhas been lying around for a long while..\n\nOn 11/11/2019 15:28, Jim Edwards wrote:\n> Hi,\n>\n> I am a developer of scientific software, as science software we expect\n> and encourage users to modify source code in order to customize their\n> experiments.\nSounds an ideal usage for git - giving back control to the users so they \ncan easily maintain their versions.\n> The mechanism which has been developed to do this\n> predates having source code in git and we are trying to figure out a\n> way to minimize changes to the scientists workflow, while leveraging\n> the power of git to improve the process.\nGood, though sounds like you will need some terminology mapping.. Is \nthere a public reference to the method?\n> In the workflow the\n> scientist creates a 'case' using script in the repository to create a\n> directory structure from which they will conduct their experiment.\nIn what sense do you use \"case\"?\n  e.g. a [single] suitcase that holds a choice of clothes to wear / \nexperimental methods;\nOr, a long list of experimental setups, each with a name, selecting a \n'case' statement (like a software 'case' statement);\nOr, a use-case that gives a half complete suggestion about how it may \nwork, but with some details still to be filled in due to lack of space \non the post-it note..?\n\nGiven that the scientists create a 'directory structure' (containing \nfiles?) for each experiment, this sounds very much like creating a \n'branch' (line of development) from the initial template of that \nstructure, and as they develop their experiment's directory structure, \nthey record their development in 'commits' on that branch (and sub \nbranches if they are looking at alternatives). Finally, when they have a \ngood structure ready, they can 'tag' that commit so it's easy to find.\n\n\n> Part of that directory structure is a SourceMods directory where the\n> user can drop modified source files that will be compiled in place of\n> a file of the same name in the source tree.\nThis 'dropping' has a strong _conceptual_ similarity to the staging area \nor 'index', where git users 'add' files that they feel are ready to an \narea that is used (like an outbox awaiting collection) as a temporary \nholding area waiting till all the bits are ready and waiting before they \ncommit the ensemble.\n\nSo this \"SourceMods\" is very similar to the staging area, except that in \nyour case it sounds like it is a specific place, while in git it is more \nconceptual as the user will 'git add <file>', and that change is \nregistered in \"the index\" (a local file in a hidden .git directory), \nready for the big commit.\n\nBehind the scenes, a copy of the file is saved (in the object store) and \nhashed ready for inclusion in the commit hierarchy . Later the files \n(objects) stored in the object store are 'packed' resulting in a very \ncompact storage, particularly for source files. The git repository can \nbe 'pushed' to other servers, and other repositories 'fetched' from \nservers (and mixed together if they have common ancestry).\n\n\n> These files are sometimes\n> long lived and passed from case to case and even user to user and it\n> is not hard to have the files get out of sync with the source tree.\nIn Git, because the current files stay 'in place', you can start a \nbranch (new experiment definition) from anywhere in any line of \ndevelopment. You/they simply checkout that particular commit and, voila, \nall the files are back as they were exactly.\n> We have discussed at length removing the SourceMods capability and\n> requiring scientists to create branches in git, but there is a lot of\n> resistance to this in the community.\nMost of that will be fear of the unknown and the unfamiliarity of the \ngit terminology. There can also be confusion about how Git has changed \nthe old ways of working. Because you get 100% verification and \nvalidation you no longer need to worry about requiring a central golden \nreference store (though usually the \"organisation\" will want to have a \n_copy_ ;-)\n> What I would like to explore is\n> allowing scientists to keep the method that they are used to but at\n> the same time tying these modified files to their history in git.\nThe key part will be in how you map what they already do and know to the \ngit commands and structure, and how you show them that it will remove a \nlot of the pain points and bottle necks.\n> Is\n> there a way to get the git metadata associated with an individual file\n> so that we can treat that file as if it were in the repo?\n>\nThat [mental model view] way is a Sisyphean task, a never ending up-hill \nstruggle. With a few careful words (mapping out solutions to their pain \npoints) you should be able to get the scientists to pester you to \nimplement git sooner rather than later. Choose the lead experimenters of \ngit carefully.\n\n-- \nPhilip\n"}]}