{"thread":{"id":"44913","subject":"The design of refs backends, linked worktrees and submodules","startedAt":"2017-01-19T11:56:13Z","lastAt":"2017-02-07T15:08:29Z","messageCount":6,"participants":["Duy Nguyen","Michael Haggerty","Junio C Hamano","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"309720","messageId":"CACsJy8CHoroX2k9GqOFmXkvvPCPN4SBeCg+6aC2WSWNSKVmWQw@mail.gmail.com","threadId":"44913","inReplyTo":null,"subject":"The design of refs backends, linked worktrees and submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2017-01-19T11:55:30Z","receivedAt":"2017-01-19T11:56:13Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"I've started working on fixing the \"git gc\" issue with multiple\nworktrees, which brings me back to this. Just some thoughts. Comments\nare really appreciated.\n\nIn the current code, files backend has special cases for both\nsubmodules (explicitly) and linked worktrees (hidden behind git_path).\nBut if a backend has to handle this stuff, all future backends have to\ntoo. Which does not sound great. Imagine we have \"something\" in\naddition to worktrees and submodules in future, then all backends have\nto learn about it.\n\nSo how about we move worktree and submodule support back to front-end?\n\nThe backend deals with refs, period. The file-based backend will be\ngiven a directory where refs live in and it work on that. Backends do\nnot use git_path(). Backends do not care about $GIT_DIR. Access to odb\n(e.g. sha-1 validation) if needed is abstracted out via a set of\ncallbacks. This allows submodules to give access to submodule's\nseparate odb. And it's getting close to the \"struct repository\"\nmentioned somewhere in refs \"TODO\" comments, even though we are\nnowhere close to introducing that struct.\n\nHow does that sound? In particular, is there anything wrong, or\nunrealistic, with that?\n\nFor that to work, I'll probably need to add a \"composite\" ref_store\nthat combines two file-based backends (for per-repo and per-worktree\nrefs) to represent a unified ref store. I think your work on ref\niterator makes way for that. A bit worried about transactions though,\nbecause I think per-repo and per-worktree updates will be separated in\ntwo transactions. But that's probably ok because future backends, like\nlmdb, will have to go through the same route anyway.\n-- \nDuy\n"},{"id":"309726","messageId":"341999fc-4496-b974-c117-c18a2fca1358@alum.mit.edu","threadId":"44913","inReplyTo":"CACsJy8CHoroX2k9GqOFmXkvvPCPN4SBeCg+6aC2WSWNSKVmWQw@mail.gmail.com","subject":"Re: The design of refs backends, linked worktrees and submodules","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2017-01-19T13:30:20Z","receivedAt":"2017-01-19T13:31:14Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/19/2017 12:55 PM, Duy Nguyen wrote:\n> I've started working on fixing the \"git gc\" issue with multiple\n> worktrees, which brings me back to this. Just some thoughts. Comments\n> are really appreciated.\n> \n> In the current code, files backend has special cases for both\n> submodules (explicitly) and linked worktrees (hidden behind git_path).\n\nThere is another terrible hack also needed to implement linked\nworktrees, namely that the `refs/bisect/` hierarchy is manually inserted\ninto the `ref_cache`, because otherwise it wouldn't be noticed when\niterating over loose references via `readdir()`.\n\nOther similar hacks would be required if other reference subtrees are\ndeclared to be per-worktree.\n\n> But if a backend has to handle this stuff, all future backends have to\n> too. Which does not sound great. Imagine we have \"something\" in\n> addition to worktrees and submodules in future, then all backends have\n> to learn about it.\n\nAgreed, the status quo is not pretty.\n\nI kindof think that it would have been a better design to store the\nreferences for all linked worktrees in the main repository's ref-store.\nFor example, the \"bisect\" refs for a worktree named \"\" could have\nbeen stored under \"refs/worktrees//bisect/*\". Then either:\n\n* teach the associated tools to read/write references there directly\n(probably with DWIM rules to make command-line use easier), or\n* treat these references as if they were actually at a standard place\nlike `refs/worktree/bisect/*`; i.e., users would need to know that they\nwere per-worktree references, but wouldn't need to worry about the true\nlocations, or\n* treat these references as if they were actually in their traditional\nlocations (though it is not obvious how this scheme could be expanded to\ncover new per-worktree references).\n\n> So how about we move worktree and submodule support back to front-end?\n> \n> The backend deals with refs, period. The file-based backend will be\n> given a directory where refs live in and it work on that. Backends do\n> not use git_path(). Backends do not care about $GIT_DIR. Access to odb\n> (e.g. sha-1 validation) if needed is abstracted out via a set of\n> callbacks. This allows submodules to give access to submodule's\n> separate odb. And it's getting close to the \"struct repository\"\n> mentioned somewhere in refs \"TODO\" comments, even though we are\n> nowhere close to introducing that struct.\n\nThis is a topic that I have thought a lot about. I definitely like this\ndirection. In fact I've dabbled around with some first steps; see branch\n`submodule-hash` in my fork on GitHub [1]. That branch associates a\n`ref_store` more closely with the directory where the references are\nstored, as opposed to having a 1:1 relationship between `ref_store`s and\nsubmodules.\n\nI would like to see the separation of a concept \"iterate over all\nreachability roots\" that is independent of other ref iteration. Then it\nwouldn't have to include reference names, except basically for use in\nerror messages. So for linked worktrees, in place of the reference name\nit might emit a string like \":\". (Of course it would\nget its information by iterating over all of the linked reference stores\nusing their reference iteration APIs.)\n\n> For that to work, I'll probably need to add a \"composite\" ref_store\n> that combines two file-based backends (for per-repo and per-worktree\n> refs) to represent a unified ref store. I think your work on ref\n> iterator makes way for that. A bit worried about transactions though,\n> because I think per-repo and per-worktree updates will be separated in\n> two transactions. But that's probably ok because future backends, like\n> lmdb, will have to go through the same route anyway.\n\nYes, that was the main motivation for the ref-iterator work.\n\nRegarding transactions, the commit pointed at by branch\n`split-transaction` in my fork shows how I think the\n`transaction_commit()` method could be split into two parts,\n`transaction_prepare()` and `transaction_finish()`. The idea would be\nthat the driver function, `ref_transaction_commit()`, calls\n`transaction_prepare()` for each `ref_store` involved in the\ntransaction, passing each one the reference updates for references that\nlive in that reference store. Those methods would verify that the part\nof the transaction that lives in that ref-store \"should\" go through,\nwithout actually committing anything. Then `transaction_finish()` would\nbe called on each ref store, and that method would commit the updates.\nYou probably couldn't get a bulletproof kind of compound transaction out\nof this (e.g., if the computer's power goes out, one `ref_store`'s\nupdates might be committed but another one's not). But it would probably\nbe good enough to cover everyday reasons for transaction failures, like\npre-checksums not matching up.\n\nLet me braindump some more information about this topic. A files backend\nfor a repository (ignoring submodules for the moment) currently consists\nof five interacting parts, each of which looks a lot like a ref-store\nitself:\n\n* A loose reference ref-store for the main repo\n* A loose reference ref-store for the per-subtree references\n* A ref_cache in front of the two loose reference stores\n* A packed ref-store\n* A second ref_cache in front of the packed ref-store\n\nBut these ref-stores are currently coupled very tightly and have\npeculiarities:\n\n* The caching code is tightly coupled to the ref-store behind it.\n* It is hard to imagine a packed refs-store that doesn't have some kind\nof caching in front of it.\n* There are tricky ordering constraints between writes to packed and\nloose references to avoid races.\n* The packed ref-store can't store symbolic refs, nor can it store\nreflogs. It currently relies on the loose ref-store for those things.\n* There is no packed-refs ref-store associated with per-worktree refs.\n* Packed references are currently locked via `*.lock` files located near\nthe corresponding loose references.\n* There are constraints that span refstores. For example, you aren't\nallowed to create a packed ref that D/F conflicts with a loose ref or\nvice versa.\n* Symrefs, which are loose, can point at packed references.\n\nI've taken some stabs at picking these apart into separate ref stores,\nbut haven't had time to make very satisfying progress. By the time of\nGitMerge I might have a better feeling for whether I can devote some\ntime to this project.\n\nMichael\n\n[1] https://github.com/mhagger/git\n\n"},{"id":"309763","messageId":"xmqq8tq625ti.fsf@gitster.mtv.corp.google.com","threadId":"44913","inReplyTo":"CACsJy8CHoroX2k9GqOFmXkvvPCPN4SBeCg+6aC2WSWNSKVmWQw@mail.gmail.com","subject":"Re: The design of refs backends, linked worktrees and submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-01-19T19:44:09Z","receivedAt":"2017-01-19T19:47:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> ... A bit worried about transactions though,\n> because I think per-repo and per-worktree updates will be separated in\n> two transactions. But that's probably ok because future backends, like\n> lmdb, will have to go through the same route anyway.\n\nIf we have per-worktree ref-store, what can be done as a useful\nthing by future backends like lmdb may be to use the same single\ndatabase to store shared and per-worktree refs, similar to the way\nMichael mentioned in his response to your message I am responding\nto, i.e.\n\n I kindof think that it would have been a better design to store the\n references for all linked worktrees in the main repository's ref-store.\n For example, the \"bisect\" refs for a worktree named \"\" could have\n been stored under \"refs/worktrees//bisect/*\".\n\nThe current design is heavily influenced by how \"contrib/workdir\"\nlays things out, for the latter of which I take the blame X-<, but\nperhaps the files backend can be retrofitted to use that layout in\nthe longer term?\n\n"},{"id":"309766","messageId":"alpine.DEB.2.20.1701192057530.3469@virtualbox","threadId":"44913","inReplyTo":"341999fc-4496-b974-c117-c18a2fca1358@alum.mit.edu","subject":"Re: The design of refs backends, linked worktrees and submodules","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-01-19T20:04:24Z","receivedAt":"2017-01-19T20:04:35Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 19 Jan 2017, Michael Haggerty wrote:\n\n> On 01/19/2017 12:55 PM, Duy Nguyen wrote:\n> > I've started working on fixing the \"git gc\" issue with multiple\n> > worktrees, which brings me back to this. Just some thoughts. Comments\n> > are really appreciated.\n> > \n> > In the current code, files backend has special cases for both\n> > submodules (explicitly) and linked worktrees (hidden behind git_path).\n> \n> There is another terrible hack also needed to implement linked\n> worktrees, namely that the `refs/bisect/` hierarchy is manually inserted\n> into the `ref_cache`, because otherwise it wouldn't be noticed when\n> iterating over loose references via `readdir()`.\n> \n> Other similar hacks would be required if other reference subtrees are\n> declared to be per-worktree.\n> \n> > But if a backend has to handle this stuff, all future backends have to\n> > too. Which does not sound great. Imagine we have \"something\" in\n> > addition to worktrees and submodules in future, then all backends have\n> > to learn about it.\n> \n> Agreed, the status quo is not pretty.\n> \n> I kindof think that it would have been a better design to store the\n> references for all linked worktrees in the main repository's ref-store.\n> For example, the \"bisect\" refs for a worktree named \"\" could have\n> been stored under \"refs/worktrees//bisect/*\".\n\nThat strikes me as a good design, indeed. It addresses very explicitly the\nroot cause of the worktree problems: Git's source code was developed for\nyears with the assumption that there is a 1:1 mapping between ref names\nand SHA-1s in each repository, and the way worktrees were implemented\nbroke that assumption.\n\nSo introducing a new refs/ namespace -- that is visible to all other\nworktrees -- would have addressed that problem.\n\nThis, BTW, is related to my concerns about introducing a \"shadow\" config\nlayer for worktrees: I still think it would be a bad idea, and very likely\nto cause regressions in surprising ways, to allow such config \"overlays\"\nper-worktree, as Git's current code's assumption is that there is only one\nconfig per repository, and that it can, say, set one config setting to\nmatch another (which in the per-worktree case would possibly hold true in\nonly one worktree only). Instead, introducing a new \"namespace\" in the\n(single) config similar to refs/worktrees/ could address that\nproblem preemptively.\n\nCiao,\nJohannes\n"},{"id":"309821","messageId":"CACsJy8AkLMG6SSgu0KP6C4GSt1hUu0bjqavpVSsS-qQ90wbKcA@mail.gmail.com","threadId":"44913","inReplyTo":"341999fc-4496-b974-c117-c18a2fca1358@alum.mit.edu","subject":"Re: The design of refs backends, linked worktrees and submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2017-01-20T11:22:57Z","receivedAt":"2017-01-20T11:23:38Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 19, 2017 at 8:30 PM, Michael Haggerty wrote:\n> I kindof think that it would have been a better design to store the\n> references for all linked worktrees in the main repository's ref-store.\n> For example, the \"bisect\" refs for a worktree named \"\" could have\n> been stored under \"refs/worktrees//bisect/*\". Then either:\n>\n> * teach the associated tools to read/write references there directly\n> (probably with DWIM rules to make command-line use easier), or\n> * treat these references as if they were actually at a standard place\n> like `refs/worktree/bisect/*`; i.e., users would need to know that they\n> were per-worktree references, but wouldn't need to worry about the true\n> locations, or\n> * treat these references as if they were actually in their traditional\n> locations (though it is not obvious how this scheme could be expanded to\n> cover new per-worktree references).\n\nWell. In one direction, we store everything at one place and construct\ndifferent slices of view of the unified store. On the other far end,\nwe have plenty of one-purpose stores, then combine them as we need.\nIt's probably personal taste, but I prefer the latter.\n\nMaking a single big store could bring us closer to the \"big number\"\nproblem. Yeah we will have to handle million of refs anyway, someday.\nThat does not mean we're free to increase the number of refs a few\nmore times. Then there are separate stores by nature like submodules\n(caveat: I haven't checked out your submodule-hash branch), or the\nproblem with multiple repos sharing objects/info/alternates.\n\n> This is a topic that I have thought a lot about. I definitely like this\n> direction. In fact I've dabbled around with some first steps; see branch\n> `submodule-hash` in my fork on GitHub [1]. That branch associates a\n> `ref_store` more closely with the directory where the references are\n> stored, as opposed to having a 1:1 relationship between `ref_store`s and\n> submodules.\n\nThanks. Will check it out.\n\n> Let me braindump some more information about this topic.\n> ...\n\nJuicy stuff :D It's hard to know these without staring really long and\nhard at refs code. Thank you.\n\n> I've taken some stabs at picking these apart into separate ref stores,\n> but haven't had time to make very satisfying progress. By the time of\n> GitMerge I might have a better feeling for whether I can devote some\n> time to this project.\n\nI think sending WIP patches to the list from time to time is also\nhelpful, even if it's not perfect. For one thing I would know you were\ndoing (or thinking at least, which also counts) and not stepping on\neach other. On my part I'm not attempting to make any more changes (*)\nuntil after I've read your branches.\n\n(*) I took git_path() out of refs code and was surprised that multi\nworktree broke. Silly me. Wrong first step.\n-- \nDuy\n"},{"id":"310989","messageId":"CACsJy8CYrbSJqbG7h+-wQPNBCNAVs94Hx7m8qXXb88L0o-rw0Q@mail.gmail.com","threadId":"44913","inReplyTo":"CACsJy8CHoroX2k9GqOFmXkvvPCPN4SBeCg+6aC2WSWNSKVmWQw@mail.gmail.com","subject":"Re: The design of refs backends, linked worktrees and submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2017-02-07T15:07:53Z","receivedAt":"2017-02-07T15:08:29Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 19, 2017 at 6:55 PM, Duy Nguyen wrote:\n> I've started working on fixing the \"git gc\" issue with multiple\n> worktrees, which brings me back to this. Just some thoughts. Comments\n> are really appreciated.\n>\n> In the current code, files backend has special cases for both\n> submodules (explicitly) and linked worktrees (hidden behind git_path).\n\nIt just occurs to me that, since the refs directory structure of a\nlinked worktree is exactly like one in a normal single-worktree setup,\nminus the shared (or packed) refs. The \"files\" refs backend can just\nsee this \"per-worktree only\" refs directory as a remote refs storage,\nwhich is just another name for \"submodule\".\n\nSo, we could just use the exact same submodule code path in refs to\ncreate a per-worktree refs storage. Doing it this way, files backedn\ndo not need to learn about linked worktrees at all. To retrieve a\nper-worktree refs storage, we do\n\"get_ref_store(\".git/worktrees/foobar\")\". To get all per-worktree refs\ndo for_each_ref_submodule(\".git/worktrees/foobar\", ...).\n\nDoes it make sense? Should we go this way?\n-- \nDuy\n"}]}