{"thread":{"id":"43896","subject":"Adding more namespace support to git","startedAt":"2016-08-20T19:07:06Z","lastAt":"2016-08-24T17:51:13Z","messageCount":7,"participants":["Richard","Josh Triplett","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"299797","messageId":"CAB2VqoZh6zZV3Lk2B7dBPVE5h6H1LCwyxWrUo4yJFhqB=hAjig@mail.gmail.com","threadId":"43896","inReplyTo":null,"subject":"Adding more namespace support to git","fromName":"Richard","fromEmail":"richard.maw@gmail.com","sentAt":"2016-08-20T19:07:00Z","receivedAt":"2016-08-20T19:07:06Z","isPatch":false,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"I work on a git server called gitano.\nWe've been using and recommending cgit for the web UI.\n\nI've been working on adding git namespace support to both,\nso that we can separate administrative branches from code branches.\n\nBecause git is not namespace aware for anything but git-upload-pack\nand git-receive-pack,\nI've had to implement namespace parsing in cgit\nfor listing branches, showing logs, displaying notes and commit decorations.\nIt might be more useful if this support was added to git itself,\nso other git servers could make use of it so there's less duplicated code.\n\nI think the way to do this would be to make the low-level ref reading functions,\nread_raw_ref, for_each_reflog_ent*, reflog_exists etc.,\ninterpret the ref they are passed as being relative to the current git\nnamespace.\n\nSince when upload-pack and receive-pack run hooks they leave GIT_NAMESPACE set\nthere are hook scripts that expect that the current namespace is ignored,\nso commands that now want to be namespace aware would have to opt-in.\n\nI tried adding a flag to the functions\nwith the intention of changing everything to conditionally set the flag\nand adding wrapper functions to provide the old ABIs,\nbut quickly abandoned that after seeing how much change would be required.\n\nIs anyone more familiar with the codebase able to recommend how I\nmight make it work,\nand would there any interest in accepting such patches if I got it to work?\n\nPlease reply-all in response, I am not subscribed to the mailing list.\n"},{"id":"299807","messageId":"20160821020537.eiyh4vnhmnuctgtz@x","threadId":"43896","inReplyTo":"CAB2VqoZh6zZV3Lk2B7dBPVE5h6H1LCwyxWrUo4yJFhqB=hAjig@mail.gmail.com","subject":"Re: Adding more namespace support to git","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-08-21T02:05:39Z","receivedAt":"2016-08-21T02:05:50Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Sat, Aug 20, 2016 at 08:07:00PM +0100, Richard wrote:\n> I work on a git server called gitano.\n> We've been using and recommending cgit for the web UI.\n> \n> I've been working on adding git namespace support to both,\n> so that we can separate administrative branches from code branches.\n> \n> Because git is not namespace aware for anything but git-upload-pack\n> and git-receive-pack,\n> I've had to implement namespace parsing in cgit\n> for listing branches, showing logs, displaying notes and commit decorations.\n> It might be more useful if this support was added to git itself,\n> so other git servers could make use of it so there's less duplicated code.\n> \n> I think the way to do this would be to make the low-level ref reading functions,\n> read_raw_ref, for_each_reflog_ent*, reflog_exists etc.,\n> interpret the ref they are passed as being relative to the current git\n> namespace.\n> \n> Since when upload-pack and receive-pack run hooks they leave GIT_NAMESPACE set\n> there are hook scripts that expect that the current namespace is ignored,\n> so commands that now want to be namespace aware would have to opt-in.\n\nThat seems really unfortunate.  While at the time we wanted to start\nwith namespace support in upload-pack and receive-pack (and\nhttp-backend) because those would allow using it as a server-side\nstorage format, I don't think we realized that leaving GIT_NAMESPACE in\nthe hook environment would completely prevent other git commands from\nautomatically handling namespaces.\n\nAnd conversely, we can't just have upload-pack and receive-pack start\nremoving it from the hook environment, because a hook might expect to\nread the current namespace from it (and then run git commands that the\nhook expects will ignore it).\n\n(This also affects libgit2; I recently added a function to libgit2 to\ninterpret various git environment variables, including GIT_NAMESPACE.\nIf git commands can't just use that automatically, that'll need to\nchange too, to avoid unexpected behavior in hooks.)\n\nFor that matter, someone could run \"GIT_NAMESPACE=foo git push\nremotename branchname\" or\n\"GIT_NAMESPACE=foo git clone remotename\", and based on the current\nbehavior, they'd expect to have the namespace apply to the remote end,\nbut not the local end.\n\nUnfortunately, I think at this point, GIT_NAMESPACE has to exclusively\nrefer to the namespace for the remote end, to avoid breakage.  Which\nmeans any automatic pervasive support for namespaces on the local side\nwould need to use a different mechanism.  (In addition to applying to\nref enumeration, this would also need to apply to the local end of\nrefspecs.)  And this new mechanism would need to not affect the remote\nend, to allow remapping the local end while accessing an un-namespaced\n(or differently namespaced) remote.\n"},{"id":"299814","messageId":"CAB2VqoZdj1rhdt-4gUhXTAMQjJTOqCuwXiO6yfB2Fzyy_qSWHg@mail.gmail.com","threadId":"43896","inReplyTo":"20160821020537.eiyh4vnhmnuctgtz@x","subject":"Re: Adding more namespace support to git","fromName":"Richard","fromEmail":"richard.maw@gmail.com","sentAt":"2016-08-21T11:30:16Z","receivedAt":"2016-08-21T11:30:22Z","isPatch":false,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"On 21 August 2016 at 03:05, Josh Triplett <josh@joshtriplett.org> wrote:\n> On Sat, Aug 20, 2016 at 08:07:00PM +0100, Richard wrote:\n>> Since when upload-pack and receive-pack run hooks they leave GIT_NAMESPACE set\n>> there are hook scripts that expect that the current namespace is ignored,\n>> so commands that now want to be namespace aware would have to opt-in.\n>\n> That seems really unfortunate.  While at the time we wanted to start\n> with namespace support in upload-pack and receive-pack (and\n> http-backend) because those would allow using it as a server-side\n> storage format, I don't think we realized that leaving GIT_NAMESPACE in\n> the hook environment would completely prevent other git commands from\n> automatically handling namespaces.\n>\n> And conversely, we can't just have upload-pack and receive-pack start\n> removing it from the hook environment, because a hook might expect to\n> read the current namespace from it (and then run git commands that the\n> hook expects will ignore it).\n\nThis is exactly what I've had to do for my proof of concept\nhttps://git.gitano.org.uk/cgit.git/commit/?h=richardmaw/namespaces&id=379124469a8a13208f976eb816375b00901ae77f\n\n> For that matter, someone could run \"GIT_NAMESPACE=foo git push\n> remotename branchname\" or\n> \"GIT_NAMESPACE=foo git clone remotename\", and based on the current\n> behavior, they'd expect to have the namespace apply to the remote end,\n> but not the local end.\n\nI'm fairly sure this isn't the case, at least from what I've tried.\nAt one point it appeared to be working,\nbut that was just because it started the upload-pack as a subprocess,\nwhich inherited the GIT_NAMESPACE environment variable rather than\nbeing passed it.\nI think this is why the test suite always sets up a remote with the ext:: helper\nso it can set --namespace=foo in the command.\n\nThis is one of the reasons why I have been working on namespace\nsupport in the git server,\nyou have to encode the namespace in the url somehow\nsince it isn't passed through the git protocol.\n\nWe were thinking of adding ssh://git@server/~username/repo/path.git syntax\nfor letting users have their own private namespace in a repository,\nand later extending the backend of the git server's repository storage\nso that other repositories could just be namespaces of a different repository\nso we could do something like repository forks\nprovided the repositories have the same availability.\n\n> Unfortunately, I think at this point, GIT_NAMESPACE has to exclusively\n> refer to the namespace for the remote end, to avoid breakage.  Which\n> means any automatic pervasive support for namespaces on the local side\n> would need to use a different mechanism.  (In addition to applying to\n> ref enumeration, this would also need to apply to the local end of\n> refspecs.)  And this new mechanism would need to not affect the remote\n> end, to allow remapping the local end while accessing an un-namespaced\n> (or differently namespaced) remote.\n\nThe problem for hooks is that it is implicitly inherited,\nso it could work if upload-pack receive-pack and http-backend work\nwith GIT_NAMESPACE set,\nbut everything else that wants to use a namespace has to set\n--namespace on the command-line.\n"},{"id":"299816","messageId":"20160821140710.qpbm37nhhvbkmxgz@x","threadId":"43896","inReplyTo":"CAB2VqoZdj1rhdt-4gUhXTAMQjJTOqCuwXiO6yfB2Fzyy_qSWHg@mail.gmail.com","subject":"Re: Adding more namespace support to git","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-08-21T14:07:10Z","receivedAt":"2016-08-21T14:07:21Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Sun, Aug 21, 2016 at 12:30:16PM +0100, Richard wrote:\n> On 21 August 2016 at 03:05, Josh Triplett <josh@joshtriplett.org> wrote:\n> > Unfortunately, I think at this point, GIT_NAMESPACE has to exclusively\n> > refer to the namespace for the remote end, to avoid breakage.  Which\n> > means any automatic pervasive support for namespaces on the local side\n> > would need to use a different mechanism.  (In addition to applying to\n> > ref enumeration, this would also need to apply to the local end of\n> > refspecs.)  And this new mechanism would need to not affect the remote\n> > end, to allow remapping the local end while accessing an un-namespaced\n> > (or differently namespaced) remote.\n> \n> The problem for hooks is that it is implicitly inherited,\n> so it could work if upload-pack receive-pack and http-backend work\n> with GIT_NAMESPACE set,\n> but everything else that wants to use a namespace has to set\n> --namespace on the command-line.\n\nI'd like to see it work more automatically than that.  Perhaps a\nseparate environment variable to set the client-side namespace?\n"},{"id":"299863","messageId":"20160822191622.6cok5tbbbc3ysu6f@x","threadId":"43896","inReplyTo":"CAB2Vqoa+1QKD-7zjNUJBuu2qNowgY7n9fLM77zUdzf8aJi4XaQ@mail.gmail.com","subject":"Re: Adding more namespace support to git","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-08-22T19:16:22Z","receivedAt":"2016-08-22T19:16:42Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Mon, Aug 22, 2016 at 07:36:31PM +0100, Richard wrote:\n> On 21 Aug 2016 15:07, \"Josh Triplett\" <josh@joshtriplett.org> wrote:\n> > I'd like to see it work more automatically than that.  Perhaps a\n> > separate environment variable to set the client-side namespace?\n> \n> How about a config option? That could be set globally, per repository, in\n> the environment or on the command line.\n\nThat might work, though you wouldn't normally want to set it globally or\nper-repository (since it affects access to a repository and you'd\ntypically want to use multiple different values or it wouldn't have much\npoint).\n\n- Josh Triplett\n"},{"id":"299865","messageId":"CAB2VqoaG=8hJ8=FG5K_rS6rtMQhk2rEWowdMD2t5kk77SXfV2A@mail.gmail.com","threadId":"43896","inReplyTo":"20160822191622.6cok5tbbbc3ysu6f@x","subject":"Re: Adding more namespace support to git","fromName":"Richard","fromEmail":"richard.maw@gmail.com","sentAt":"2016-08-22T19:36:44Z","receivedAt":"2016-08-22T19:36:49Z","isPatch":false,"sender":{"key":"richard.maw@gmail.com","avatar":"https://gravatar.com/avatar/ee59b32f2b88ec3ea1653e30e237c5952e6eaa8602906afd161831127745ff70?d=mp&s=160"},"body":"On 22 August 2016 at 20:16, Josh Triplett <josh@joshtriplett.org> wrote:\n> On Mon, Aug 22, 2016 at 07:36:31PM +0100, Richard wrote:\n>> On 21 Aug 2016 15:07, \"Josh Triplett\" <josh@joshtriplett.org> wrote:\n>> > I'd like to see it work more automatically than that.  Perhaps a\n>> > separate environment variable to set the client-side namespace?\n>>\n>> How about a config option? That could be set globally, per repository, in\n>> the environment or on the command line.\n>\n> That might work, though you wouldn't normally want to set it globally or\n> per-repository (since it affects access to a repository and you'd\n> typically want to use multiple different values or it wouldn't have much\n> point).\n\nGlobally is a bit contrived, but could be used to keep the top-level\nnamespace clean\nso you might opt to default to fetching into a namespace called \"main\"\nso that if you need to temporarily fetch into a different namespace it\nwouldn't be problematic.\nPerhaps it's a kernel tree from a vendor with a messy branch naming scheme\nso you don't want to fetch it into your primary namespace and make it\ndifficult to find your branches,\nbut you don't know which of their branches you need until you've got them all.\nSo you fetch into the different namespace rather than a fresh clone\nto avoid re-fetching everything (numerous alternative solutions exist)\nThen once you've found out which branch you need,\nyou make a note, switch back to the \"main\" namespace and re-fetch just\nthat branch.\n\nA per repository default namespace could also be useful\nif an upstream repository has multiple namespaces (code vs\ndocumentation maybe) you could fetch them all\nand then switch between them when you need to work on different parts,\nand if it's config rather than an environment variable it will persist\nbetween shell sessions easier.\n"},{"id":"300088","messageId":"20160824174956.ln4fpxorszuzpw7k@sigill.intra.peff.net","threadId":"43896","inReplyTo":"CAB2VqoZh6zZV3Lk2B7dBPVE5h6H1LCwyxWrUo4yJFhqB=hAjig@mail.gmail.com","subject":"Re: Adding more namespace support to git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-24T17:49:56Z","receivedAt":"2016-08-24T17:51:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 20, 2016 at 08:07:00PM +0100, Richard wrote:\n\n> Because git is not namespace aware for anything but git-upload-pack\n> and git-receive-pack, I've had to implement namespace parsing in cgit\n> for listing branches, showing logs, displaying notes and commit\n> decorations.  It might be more useful if this support was added to git\n> itself, so other git servers could make use of it so there's less\n> duplicated code.\n> \n> I think the way to do this would be to make the low-level ref reading\n> functions, read_raw_ref, for_each_reflog_ent*, reflog_exists etc.,\n> interpret the ref they are passed as being relative to the current git\n> namespace.\n\nAt GitHub, we store many forks for a single repository, and we\nconsidered using namespaces for our storage strategy. But like you, we\nran into the problem that you they only work for certain operations. :)\n\nOur solution is to use separate repositories, each with their own ref\nstorage, but pointing to a shared object store. It works, but there are\na lot of gotchas and performance issues with migrating objects around,\nrunning repacks.\n\nMichael Haggerty (cc'd) has picked up the pluggable ref-backend work\nstarted by others, and I know has some ideas on doing namespaces at that\nlevel. Basically, the concept of \"namespaces\" should be able to plug in\nbetween the actual storage backend and the rest of the git code. Git\ncode wouldn't have to care whether the namespace plugin was in use, and\nthe namespace plugin wouldn't have to care which storage backend was in\nuse (it would just silently translate \"refs/heads/foo\" into\n\"refs/namespaces/123/heads/foo\", and vice versa).\n\nThat's a very \"complete\" solution in the sense that the git code does\nnot know about the namespaces, and cannot even access refs outside of\nit. But I think in general it would do what you want. Most operations\nwould run in a certain namespace (i.e., pretend nothing outside of that\nnamespace exists, for fetches, diffs, etc), and others would want to\nlook at the whole namespace (e.g., repacking, pruning). I don't know of\nany operations that want to see both views in the same process.\n\n-Peff\n"}]}