{"thread":{"id":"50700","subject":"Git server side \"pre-receive\" hook to create new repositories","startedAt":"2019-03-09T10:47:22Z","lastAt":"2019-03-21T09:28:25Z","messageCount":5,"participants":["Dimitri Joukoff","Jeff King","Ævar Arnfjörð Bjarmason","Sitaram Chamarty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"371020","messageId":"SYXPR01MB095712C6765970605923A2FDDD4E0@SYXPR01MB0957.ausprd01.prod.outlook.com","threadId":"50700","inReplyTo":null,"subject":"Git server side \"pre-receive\" hook to create new repositories","fromName":"Dimitri Joukoff","fromEmail":"dimitri.joukoff@griffithuni.edu.au","sentAt":"2019-03-09T10:46:09Z","receivedAt":"2019-03-09T10:47:22Z","isPatch":false,"sender":{"key":"dimitri.joukoff@griffithuni.edu.au","avatar":null},"body":"Some time ago, Feb 2018, I put in a proposal at Swift.org , that would\nallow the Swift Package Manager to clone packages from the Internet to a\nlocal git server and use the local version without requiring updates to\nthe Swift package:\nhttps://forums.swift.org/t/proposal-to-enable-spm-to-use-a-local-repository-mirror/10291/5\n.  At the time, I also raised an issue at\nhttps://github.com/github/platform-samples/issues/171 asking for the\nfeature described below.  However, today I realised that that was not\nthe right place to make such a request, and hence I've written this email.\n\nThe above thread contains a solution to the local mirror problem whose\nonly drawback was that new repositories had to be MANUALLY initialised\nbare on the local server.  In all other respects, in my opinion, the\nsolution worked perfectly and saved me downloading huge amounts of data\neach time I performed a distribution clean on the project or switched to\nanother project that used the same package.  The solution published at\nSwift.org used a Ubuntu 16.04.2 server running Git 2.16.2, and the\nclient was a Macbook Pro running MacOS with its default git.\n\nThus, this feature request is asking that the 'pre-receive' hook\ntriggers when someone tries to push to a repository regardless of\nwhether the repository exists.  This would allow automatic creation of\nnew repositories and smooth the work-flow described above.  If the\nsemantics of the existing 'pre-receive' hook are such that it would not\nbe suitable for such a purpose, then an alternative way of providing the\ncall-back ability would be implemented.\n\nBest regards,\nDimitri.\n\n\n"},{"id":"371886","messageId":"20190319070629.GE31801@sigill.intra.peff.net","threadId":"50700","inReplyTo":"SYXPR01MB095712C6765970605923A2FDDD4E0@SYXPR01MB0957.ausprd01.prod.outlook.com","subject":"Re: Git server side \"pre-receive\" hook to create new repositories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-03-19T07:06:30Z","receivedAt":"2019-03-19T07:06:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:\n\n> Thus, this feature request is asking that the 'pre-receive' hook\n> triggers when someone tries to push to a repository regardless of\n> whether the repository exists.  This would allow automatic creation of\n> new repositories and smooth the work-flow described above.  If the\n> semantics of the existing 'pre-receive' hook are such that it would not\n> be suitable for such a purpose, then an alternative way of providing the\n> call-back ability would be implemented.\n\nThe pre-receive hook is a bit too late for this. It runs after the\nserver has told the client what it has in the repo, the client decides\nwhat to push, and the server has received the pack. So receive-pack\nwould have to know about this and fake having an empty repository. And\nthen figure out where to store the incoming packfile, since we have no\nrepo.\n\nSo I think it would have to be another hook that runs before the rest of\nreceive-pack. I.e., a system-level config option that says \"if you are\nasked to accept a push for a repo and it doesn't exist, run this instead\nand then run as usual\".\n\nIt does feel a little error-prone, though, if the client does not\npositively say \"I want you to create this if it doesn't exist\".\nOtherwise if I do \"git push server:my-misspelled-repo.git\", the result\nis going to be rather confusing. And retro-fitting that into the\nreceive-pack protocol is going to be tricky.\n\nIt would be much easier to have a separate endpoint for the client to\nsay \"please make this repo if it doesn't exist\". And then just run that\nbefore doing the push.\n\nFor an unrestricted client connecting over ssh, we already have that:\nyou can just run \"ssh $host git init /path/to/repo\". There isn't a\nsimilar thing that can be done over HTTP, though.\n\n-Peff\n"},{"id":"371893","messageId":"87d0mndxop.fsf@evledraar.gmail.com","threadId":"50700","inReplyTo":"20190319070629.GE31801@sigill.intra.peff.net","subject":"Re: Git server side \"pre-receive\" hook to create new repositories","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-19T09:24:06Z","receivedAt":"2019-03-19T09:24:11Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 19 2019, Jeff King wrote:\n\n> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:\n>\n>> Thus, this feature request is asking that the 'pre-receive' hook\n>> triggers when someone tries to push to a repository regardless of\n>> whether the repository exists.  This would allow automatic creation of\n>> new repositories and smooth the work-flow described above.  If the\n>> semantics of the existing 'pre-receive' hook are such that it would not\n>> be suitable for such a purpose, then an alternative way of providing the\n>> call-back ability would be implemented.\n>\n> The pre-receive hook is a bit too late for this. It runs after the\n> server has told the client what it has in the repo, the client decides\n> what to push, and the server has received the pack. So receive-pack\n> would have to know about this and fake having an empty repository. And\n> then figure out where to store the incoming packfile, since we have no\n> repo.\n>\n> So I think it would have to be another hook that runs before the rest of\n> receive-pack. I.e., a system-level config option that says \"if you are\n> asked to accept a push for a repo and it doesn't exist, run this instead\n> and then run as usual\".\n>\n> It does feel a little error-prone, though, if the client does not\n> positively say \"I want you to create this if it doesn't exist\".\n> Otherwise if I do \"git push server:my-misspelled-repo.git\", the result\n> is going to be rather confusing. And retro-fitting that into the\n> receive-pack protocol is going to be tricky.\n>\n> It would be much easier to have a separate endpoint for the client to\n> say \"please make this repo if it doesn't exist\". And then just run that\n> before doing the push.\n>\n> For an unrestricted client connecting over ssh, we already have that:\n> you can just run \"ssh $host git init /path/to/repo\". There isn't a\n> similar thing that can be done over HTTP, though.\n\nSounds simpler to just change the user's login shell to a wrapper that\nchecks if the repo exists, and if not create it before proceeding.\n\nThe same with http(s). I.e. in whatever webserver that's now pointing to\ngit-http-backend as a script point to the same wrapper script.\n"},{"id":"371902","messageId":"SYXPR01MB09570A4CBFC30A36150C90BCDD400@SYXPR01MB0957.ausprd01.prod.outlook.com","threadId":"50700","inReplyTo":"87d0mndxop.fsf@evledraar.gmail.com","subject":"Re: Git server side \"pre-receive\" hook to create new repositories","fromName":"Dimitri Joukoff","fromEmail":"dimitri.joukoff@griffithuni.edu.au","sentAt":"2019-03-19T12:16:07Z","receivedAt":"2019-03-19T12:16:15Z","isPatch":false,"sender":{"key":"dimitri.joukoff@griffithuni.edu.au","avatar":null},"body":"Thanks for the feedback.\n\nWhilst I understand the options that you've both proposed, the intent\nwas to enable the Swift Package Manager to mirror repositories in a\ntransparent way.  I'll look into whether these options can be\nimplemented inside the SPM.\n\nCheers,\nDimitri.\n\n\nOn 19/3/19 7:24 pm, Ævar Arnfjörð Bjarmason wrote:\n> On Tue, Mar 19 2019, Jeff King wrote:\n>\n>> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:\n>>\n>>> Thus, this feature request is asking that the 'pre-receive' hook\n>>> triggers when someone tries to push to a repository regardless of\n>>> whether the repository exists.  This would allow automatic creation of\n>>> new repositories and smooth the work-flow described above.  If the\n>>> semantics of the existing 'pre-receive' hook are such that it would not\n>>> be suitable for such a purpose, then an alternative way of providing the\n>>> call-back ability would be implemented.\n>> The pre-receive hook is a bit too late for this. It runs after the\n>> server has told the client what it has in the repo, the client decides\n>> what to push, and the server has received the pack. So receive-pack\n>> would have to know about this and fake having an empty repository. And\n>> then figure out where to store the incoming packfile, since we have no\n>> repo.\n>>\n>> So I think it would have to be another hook that runs before the rest of\n>> receive-pack. I.e., a system-level config option that says \"if you are\n>> asked to accept a push for a repo and it doesn't exist, run this instead\n>> and then run as usual\".\n>>\n>> It does feel a little error-prone, though, if the client does not\n>> positively say \"I want you to create this if it doesn't exist\".\n>> Otherwise if I do \"git push server:my-misspelled-repo.git\", the result\n>> is going to be rather confusing. And retro-fitting that into the\n>> receive-pack protocol is going to be tricky.\n>>\n>> It would be much easier to have a separate endpoint for the client to\n>> say \"please make this repo if it doesn't exist\". And then just run that\n>> before doing the push.\n>>\n>> For an unrestricted client connecting over ssh, we already have that:\n>> you can just run \"ssh $host git init /path/to/repo\". There isn't a\n>> similar thing that can be done over HTTP, though.\n> Sounds simpler to just change the user's login shell to a wrapper that\n> checks if the repo exists, and if not create it before proceeding.\n>\n> The same with http(s). I.e. in whatever webserver that's now pointing to\n> git-http-backend as a script point to the same wrapper script.\n>\n>\n\n\n\n"},{"id":"372078","messageId":"c9211cf1-7c27-1274-0433-bc1237bbe199@gmail.com","threadId":"50700","inReplyTo":"SYXPR01MB09570A4CBFC30A36150C90BCDD400@SYXPR01MB0957.ausprd01.prod.outlook.com","subject":"Re: Git server side \"pre-receive\" hook to create new repositories","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2019-03-21T09:28:21Z","receivedAt":"2019-03-21T09:28:25Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"\n\nOn 19/03/2019 17.46, Dimitri Joukoff wrote:\n> Thanks for the feedback.\n> \n> Whilst I understand the options that you've both proposed, the intent\n> was to enable the Swift Package Manager to mirror repositories in a\n> transparent way.  I'll look into whether these options can be\n> implemented inside the SPM.\n\nI don't know the Swift Package Manager, but looking at the preceding\ndiscussion reminded that gitolite allows this.\n\nI'm not saying that someone should install gitolite just to get this\nfeature, but rather that *any* script that traps the incoming request\ncan do this.  Almost trivial if the request is coming in via ssh,\nactually.  Just set the shell to a script that looks at and parses\nSSH_ORIGINAL_COMMAND and do whatever is needed before passing control to\ngit proper.\n\nsitaram\n\n> \n> Cheers,\n> Dimitri.\n> \n> \n> On 19/3/19 7:24 pm, Ævar Arnfjörð Bjarmason wrote:\n>> On Tue, Mar 19 2019, Jeff King wrote:\n>>\n>>> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:\n>>>\n>>>> Thus, this feature request is asking that the 'pre-receive' hook\n>>>> triggers when someone tries to push to a repository regardless of\n>>>> whether the repository exists.  This would allow automatic creation of\n>>>> new repositories and smooth the work-flow described above.  If the\n>>>> semantics of the existing 'pre-receive' hook are such that it would not\n>>>> be suitable for such a purpose, then an alternative way of providing the\n>>>> call-back ability would be implemented.\n>>> The pre-receive hook is a bit too late for this. It runs after the\n>>> server has told the client what it has in the repo, the client decides\n>>> what to push, and the server has received the pack. So receive-pack\n>>> would have to know about this and fake having an empty repository. And\n>>> then figure out where to store the incoming packfile, since we have no\n>>> repo.\n>>>\n>>> So I think it would have to be another hook that runs before the rest of\n>>> receive-pack. I.e., a system-level config option that says \"if you are\n>>> asked to accept a push for a repo and it doesn't exist, run this instead\n>>> and then run as usual\".\n>>>\n>>> It does feel a little error-prone, though, if the client does not\n>>> positively say \"I want you to create this if it doesn't exist\".\n>>> Otherwise if I do \"git push server:my-misspelled-repo.git\", the result\n>>> is going to be rather confusing. And retro-fitting that into the\n>>> receive-pack protocol is going to be tricky.\n>>>\n>>> It would be much easier to have a separate endpoint for the client to\n>>> say \"please make this repo if it doesn't exist\". And then just run that\n>>> before doing the push.\n>>>\n>>> For an unrestricted client connecting over ssh, we already have that:\n>>> you can just run \"ssh $host git init /path/to/repo\". There isn't a\n>>> similar thing that can be done over HTTP, though.\n>> Sounds simpler to just change the user's login shell to a wrapper that\n>> checks if the repo exists, and if not create it before proceeding.\n>>\n>> The same with http(s). I.e. in whatever webserver that's now pointing to\n>> git-http-backend as a script point to the same wrapper script.\n>>\n>>\n> \n> \n> \n\n-- \nsitaram\n"}]}