{"thread":{"id":"56925","subject":"Config spec for git","startedAt":"2021-11-17T09:36:52Z","lastAt":"2021-11-22T18:19:30Z","messageCount":5,"participants":["Wallace, Brooke T (US 349D-Affiliate)","Ævar Arnfjörð Bjarmason","Philip Oakley","Johannes Schindelin","Martin von Zweigbergk"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"441439","messageId":"D5EE9939-F639-4E69-BD81-10B05EC43A8E@jpl.nasa.gov","threadId":"56925","inReplyTo":null,"subject":"Config spec for git","fromName":"Wallace, Brooke T (US 349D-Affiliate)","fromEmail":"brooke.t.wallace@jpl.nasa.gov","sentAt":"2021-11-17T09:30:55Z","receivedAt":"2021-11-17T09:36:52Z","isPatch":false,"sender":{"key":"brooke.t.wallace@jpl.nasa.gov","avatar":null},"body":"Has any one considered adding a config spec feature to Git or does Git alreadt have some way to support the same features?\n\nI've been using Git for a while now for small projects but taking on a new larger project I've come to realize that Git does not have config specs and so seems to be missing an important feature for managing large projects.\n\nWe use configuration specs to select directories from a common code base (repo) and map them into different baselines to creat multiple product builds with different feature sets. We used this feature in VCSs such as Clearcase and Perforce. Ultimately this allows us to manage the repo in one directory structure and create product builds with a different one. For example the repo has multiple directories for different products/targets, but a baseline, the workspace, has only one target directory always with the same name mapped to the same location. Obviously the corresponding directories in the repo have different names.\n\nGit supports the notion of submodules, but I see no way to map a submodule directory to a different name, remove unwanted subdirs of a submodule, or map a submodule over a subdirectory of the primary repo. Config specs also allow you to specify a specific branch or version that you want to map to your workspace independent of other directories, branches and versions.\n\nI suppose it may be possible to achieve the same result by treating the primary repo as the configspec. But I feel like there are some features config specs support that i do not have using submodules, but might need down the road.\n\nI can see that omitting, obscuring, or overwriting parts of a repo would not play well with the commit id. So I imagine there could be some real complications trying to add support for the notion of a flexible config spec.\n\nAppreciate any comments/feedback\n-Brooke"},{"id":"441478","messageId":"211117.86fsrv6jq7.gmgdl@evledraar.gmail.com","threadId":"56925","inReplyTo":"D5EE9939-F639-4E69-BD81-10B05EC43A8E@jpl.nasa.gov","subject":"Re: Config spec for git","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-17T12:32:18Z","receivedAt":"2021-11-17T12:37:57Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 17 2021, Wallace, Brooke T (US 349D-Affiliate) wrote:\n\n> Has any one considered adding a config spec feature to Git or does Git alreadt have some way to support the same features?\n>\n> I've been using Git for a while now for small projects but taking on a\n> new larger project I've come to realize that Git does not have config\n> specs and so seems to be missing an important feature for managing\n> large projects.\n>\n> We use configuration specs to select directories from a common code\n> base (repo) and map them into different baselines to creat multiple\n> product builds with different feature sets. We used this feature in\n> VCSs such as Clearcase and Perforce. Ultimately this allows us to\n> manage the repo in one directory structure and create product builds\n> with a different one. For example the repo has multiple directories\n> for different products/targets, but a baseline, the workspace, has\n> only one target directory always with the same name mapped to the same\n> location. Obviously the corresponding directories in the repo have\n> different names.\n>\n> Git supports the notion of submodules, but I see no way to map a\n> submodule directory to a different name, remove unwanted subdirs of a\n> submodule, or map a submodule over a subdirectory of the primary\n> repo. Config specs also allow you to specify a specific branch or\n> version that you want to map to your workspace independent of other\n> directories, branches and versions.\n>\n> I suppose it may be possible to achieve the same result by treating\n> the primary repo as the configspec. But I feel like there are some\n> features config specs support that i do not have using submodules, but\n> might need down the road.\n>\n> I can see that omitting, obscuring, or overwriting parts of a repo\n> would not play well with the commit id. So I imagine there could be\n> some real complications trying to add support for the notion of a\n> flexible config spec.\n>\n> Appreciate any comments/feedback\n\nI understand all the terms involved in your E-Mail except \"config spec\",\nso on the first couple of readings I was thoroughly confused.\n\nI gather from some Google searching that you may be referring to\nClearCase SCM jargon:\nhttps://en.wikipedia.org/wiki/Rational_ClearCase#The_configuration_specification\n&\nhttps://www.ibm.com/docs/en/rational-clearcase/8.0.0?topic=views-how-config-spec-works\n\nFrom your description it seems like you're talking about some\ncombination of the work-in-progress \"sparse checkout\" feature, and a\nfeature to compose arbitrary subdirectories and overlays of existing\nrepositories.\n\nAs far as I know nobody's working on the latter, although I suppose some\nclever combination of submodules and sparse checkouts might make it\npossible.\n\nAll of that's really a shot in the dark, I think I'm probably not the\nonly one who'd benefit from a description of what you'd expect a \"config\nspec\" to do for you that doesn't assume pre-existing knowledge of the\nterm.\n\nMore generally it's a very common initial migration stategy between\nSCM's and X SCM -> Git in particular to first consider how you could 1=1\nmap existing behavior to Git.\n\nThose sorts of migrations are generally much more painful in the longer\nterm than considering how you'd map the software or assets you have to\nGit if you were starting out today, which may be something to think\nabout.\n"},{"id":"441501","messageId":"37ef900a-d9bf-1941-75eb-ea8556e2ba8c@iee.email","threadId":"56925","inReplyTo":"211117.86fsrv6jq7.gmgdl@evledraar.gmail.com","subject":"Re: Config spec for git","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2021-11-17T15:38:31Z","receivedAt":"2021-11-17T15:38:37Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 17/11/2021 12:32, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, Nov 17 2021, Wallace, Brooke T (US 349D-Affiliate) wrote:\n>\n>> Has any one considered adding a config spec feature to Git or does Git alreadt have some way to support the same features?\n>>\n>> I've been using Git for a while now for small projects but taking on a\n>> new larger project I've come to realize that Git does not have config\n>> specs and so seems to be missing an important feature for managing\n>> large projects.\n>>\n>> We use configuration specs to select directories from a common code\n>> base (repo) and map them into different baselines to creat multiple\n>> product builds with different feature sets. We used this feature in\n>> VCSs such as Clearcase and Perforce. Ultimately this allows us to\n>> manage the repo in one directory structure and create product builds\n>> with a different one. For example the repo has multiple directories\n>> for different products/targets, but a baseline, the workspace, has\n>> only one target directory always with the same name mapped to the same\n>> location. Obviously the corresponding directories in the repo have\n>> different names.\n>>\n>> Git supports the notion of submodules, but I see no way to map a\n>> submodule directory to a different name, remove unwanted subdirs of a\n>> submodule, or map a submodule over a subdirectory of the primary\n>> repo. Config specs also allow you to specify a specific branch or\n>> version that you want to map to your workspace independent of other\n>> directories, branches and versions.\n>>\n>> I suppose it may be possible to achieve the same result by treating\n>> the primary repo as the configspec. But I feel like there are some\n>> features config specs support that i do not have using submodules, but\n>> might need down the road.\n>>\n>> I can see that omitting, obscuring, or overwriting parts of a repo\n>> would not play well with the commit id. So I imagine there could be\n>> some real complications trying to add support for the notion of a\n>> flexible config spec.\n>>\n>> Appreciate any comments/feedback\n> I understand all the terms involved in your E-Mail except \"config spec\",\n> so on the first couple of readings I was thoroughly confused.\n>\n> I gather from some Google searching that you may be referring to\n> ClearCase SCM jargon:\n> https://en.wikipedia.org/wiki/Rational_ClearCase#The_configuration_specification\n> &\n> https://www.ibm.com/docs/en/rational-clearcase/8.0.0?topic=views-how-config-spec-works\n>\n> From your description it seems like you're talking about some\n> combination of the work-in-progress \"sparse checkout\" feature, and a\n> feature to compose arbitrary subdirectories and overlays of existing\n> repositories.\n>\n> As far as I know nobody's working on the latter, although I suppose some\n> clever combination of submodules and sparse checkouts might make it\n> possible.\n>\n> All of that's really a shot in the dark, I think I'm probably not the\n> only one who'd benefit from a description of what you'd expect a \"config\n> spec\" to do for you that doesn't assume pre-existing knowledge of the\n> term.\n>\n> More generally it's a very common initial migration stategy between\n> SCM's and X SCM -> Git in particular to first consider how you could 1=1\n> map existing behavior to Git.\n>\n> Those sorts of migrations are generally much more painful in the longer\n> term than considering how you'd map the software or assets you have to\n> Git if you were starting out today, which may be something to think\n> about.\nAlso intrigued, I tried \"mapping clearcase config spec to Git\" in my\nsearch engine to see what it came up with.\n\nThere's a YouTube webinar  \"ClearCase to Git - November 2016\"\nhttps://www.youtube.com/watch?v=z2odE0CKxCQ  which looked like it may\nhelp clarify issues.\nThen there are a few StackOverflow Q&As that may help with terminology.\nhttps://stackoverflow.com/questions/763099/flexible-vs-static-branching-git-vs-clearcase-accurev\nhttps://stackoverflow.com/questions/28280685/toward-an-ideal-workflow-with-clearcase-and-git\n\nIt feels like your config specs are like feature branches, but that what\nis missing (from Git, relative to the config spec) is a merge strategy\nthat can define which particular files/folders are merged at the one\ntime, rather than the current 'all files' being merged. This desire has\ncome up a few times when large (corporate?)  projects need to merge\nlarge independent feature branches that will need different specialists\nto handle different groups of files (i.e. partial merges, e.g. [1]), but\nthat hasn't been implemented (yet) as it would need someone to think it\nthrough and work on it.\n--\nPhilip\n[1]\nhttps://lore.kernel.org/git/BY5PR19MB3400EB9AD87DFE612AFD5CC390810@BY5PR19MB3400.namprd19.prod.outlook.com/\n\nCollaborative conflict resolution feature request\n\n"},{"id":"441541","messageId":"nycvar.QRO.7.76.6.2111180033130.11028@tvgsbejvaqbjf.bet","threadId":"56925","inReplyTo":"D5EE9939-F639-4E69-BD81-10B05EC43A8E@jpl.nasa.gov","subject":"Re: Config spec for git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-11-18T00:07:06Z","receivedAt":"2021-11-18T00:07:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brooke,\n\nOn Wed, 17 Nov 2021, Wallace, Brooke T (US 349D-Affiliate) wrote:\n\n> Has any one considered adding a config spec feature to Git or does Git\n> alreadt have some way to support the same features?\n\nSince config specs are clearly not (yet) a Git feature, it would make\nsense to begin the discussion with a description of the concept of config\nspecs, as the majority of the readers on the Git mailing list will be\nunfamiliar with them.\n\n> I've been using Git for a while now for small projects but taking on a\n> new larger project I've come to realize that Git does not have config\n> specs and so seems to be missing an important feature for managing large\n> projects.\n>\n> We use configuration specs to select directories from a common code base\n> (repo) and map them into different baselines to creat multiple product\n> builds with different feature sets. We used this feature in VCSs such as\n> Clearcase and Perforce. Ultimately this allows us to manage the repo in\n> one directory structure and create product builds with a different one.\n> For example the repo has multiple directories for different\n> products/targets, but a baseline, the workspace, has only one target\n> directory always with the same name mapped to the same location.\n> Obviously the corresponding directories in the repo have different\n> names.\n\nSo from what I gather after reading this, I suspect that you have a main\nbranch with a full tree, and you want to have a way to check out only\nparts of the tree.\n\nThis concept has been brought up before, in\nhttps://lore.kernel.org/git/pull.627.git.1588857462.gitgitgadget@gmail.com/:\nproposing a way to define what parts of the tree should be checked out in\na sparse checkout.\n\nHowever, it looks as if your config specs also allow to map the\ndirectories in the Git revisions to different locations and maybe even\nnames?\n\nSuch a concept has not come up on the Git mailing list.\n\nThere _have_ been ideas floating around in Git for Windows, mainly to\nallow for checking out revisions that rely on file names that are illegal\non Windows (such as file names containing backslashes, or reserved names\nsuch as `aux.c`).\n\nNothing came of those ideas, though, mainly because nobody snatched up the\nbaton to work on a concrete patch to implement this.\n\nI should point out, though, that the concept of a sparse checkout is\nindependent from the concept of mapping file/directory names in the Git\nrevision to different ones in the Git worktree.\n\n> Git supports the notion of submodules, but I see no way to map a\n> submodule directory to a different name, remove unwanted subdirs of a\n> submodule, or map a submodule over a subdirectory of the primary repo.\n> Config specs also allow you to specify a specific branch or version that\n> you want to map to your workspace independent of other directories,\n> branches and versions.\n\nThe idea of letting directories in the same Git worktree originate from\n_different_ revisions is very, very foreign to the fundamental Git concept\nof what constitutes a commit. A commit is very much a snapshot of the\nentire tree. And when you make a new commit, it is again very much a\nsnapshot of the entire tree, based on a single parent commit.\n\nSo I doubt that you will be able to come up with a workable design to let\nGit replicate this functionality.\n\n> I suppose it may be possible to achieve the same result by treating the\n> primary repo as the configspec. But I feel like there are some features\n> config specs support that i do not have using submodules, but might need\n> down the road.\n\nI agree that submodules are unlikely to give you what you want.\n\n> I can see that omitting, obscuring, or overwriting parts of a repo would\n> not play well with the commit id. So I imagine there could be some real\n> complications trying to add support for the notion of a flexible config\n> spec.\n\nIndeed.\n\nThe only way I can see that you can _somehow_ combine parts of multiple\nrevisions into one worktree is by transforming those parts into a single\ncommit, quite possibly by scripting the transformation.\n\nFor example, if you wanted to map, say, `Documentation/technical/` of the\ntag `v2.34.0` to `tech-specs/` and `compat/poll/` of the tag `v2.30.0` to\n`poll-emulation/` in a clone of https://github.com/git/git, you could use\nsomething like this to create a new branch:\n\n(\n\tGIT_INDEX_FILE=.tmp-index &&\n\texport GIT_INDEX_FILE &&\n\tgit read-tree --prefix=tech-specs/ v2.34.0:Documentation/technical &&\n\tgit read-tree --prefix=poll-emulation/ v2.30.0:compat/poll &&\n\ttree=$(git write-tree) &&\n\tcommit=$(git commit-tree $tree -p v2.34.0^0 -p v2.30.0) &&\n\tgit branch my-generated-branch $commit\n)\n\nThis would give you a full Git branch that could be checked out and has\nthe mapping.\n\nYou would have to play similar tricks if you wanted to transport committed\nchanges from that branch back to the originating commit histories.\n\nSo yes, it is _somewhat_ possible to replicate what you can do with config\nspecs, it is just unlikely to ever offer a good user experience.\n\nCiao,\nJohannes\n"},{"id":"441997","messageId":"CANiSa6ieZrsjQD0+5dCMQCyCZzEVeLpQij8AOFS1R4RYGqbf7g@mail.gmail.com","threadId":"56925","inReplyTo":"37ef900a-d9bf-1941-75eb-ea8556e2ba8c@iee.email","subject":"Re: Config spec for git","fromName":"Martin von Zweigbergk","fromEmail":"martinvonz@gmail.com","sentAt":"2021-11-22T18:19:11Z","receivedAt":"2021-11-22T18:19:30Z","isPatch":false,"sender":{"key":"martinvonz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/891642?v=4"},"body":"On Wed, Nov 17, 2021 at 8:38 AM Philip Oakley <philipoakley@iee.email> wrote:\n>\n> It feels like your config specs are like feature branches, but that what\n> is missing (from Git, relative to the config spec) is a merge strategy\n> that can define which particular files/folders are merged at the one\n> time, rather than the current 'all files' being merged. This desire has\n> come up a few times when large (corporate?)  projects need to merge\n> large independent feature branches that will need different specialists\n> to handle different groups of files (i.e. partial merges, e.g. [1]), but\n> that hasn't been implemented (yet) as it would need someone to think it\n> through and work on it.\n> --\n> Philip\n> [1]\n> https://lore.kernel.org/git/BY5PR19MB3400EB9AD87DFE612AFD5CC390810@BY5PR19MB3400.namprd19.prod.outlook.com/\n>\n> Collaborative conflict resolution feature request\n>\n\nFWIW, I've spent a lot of time thinking about this and implementing it\nin https://github.com/martinvonz/jj. Being able to represent a\nconflicted file state has a lot of benefits, of which collaborative\nconflict resolution is among the smaller ones (allowing automatic\nrebase every time a commit is rewritten is much more useful, IMO). I\nneed to document the design but you can find an old version of it\n(from before I rewrote the README to target users) is at [1]. The\ndesign can of course be copied to Git.\n\n[1] https://github.com/martinvonz/jj/blob/2879d817dd1021f8dc2ea5e42000c1d5d50e4fc7/README.md#commits-can-contain-conflicts\n"}]}