# Git server side "pre-receive" hook to create new repositories 5 messages from 2019-03-09 to 2019-03-21. Participants: Dimitri Joukoff, Jeff King, Ævar Arnfjörð Bjarmason, Sitaram Chamarty. Thread: https://gitlist.dev/t/50700 ## Dimitri Joukoff, 2019-03-09 10:46 Subject: Git server side "pre-receive" hook to create new repositories Message-ID: URL: https://gitlist.dev/e/SYXPR01MB095712C6765970605923A2FDDD4E0%40SYXPR01MB0957.ausprd01.prod.outlook.com ``` Some time ago, Feb 2018, I put in a proposal at Swift.org , that would allow the Swift Package Manager to clone packages from the Internet to a local git server and use the local version without requiring updates to the Swift package: https://forums.swift.org/t/proposal-to-enable-spm-to-use-a-local-repository-mirror/10291/5 . At the time, I also raised an issue at https://github.com/github/platform-samples/issues/171 asking for the feature described below. However, today I realised that that was not the right place to make such a request, and hence I've written this email. The above thread contains a solution to the local mirror problem whose only drawback was that new repositories had to be MANUALLY initialised bare on the local server. In all other respects, in my opinion, the solution worked perfectly and saved me downloading huge amounts of data each time I performed a distribution clean on the project or switched to another project that used the same package. The solution published at Swift.org used a Ubuntu 16.04.2 server running Git 2.16.2, and the client was a Macbook Pro running MacOS with its default git. Thus, this feature request is asking that the 'pre-receive' hook triggers when someone tries to push to a repository regardless of whether the repository exists. This would allow automatic creation of new repositories and smooth the work-flow described above. If the semantics of the existing 'pre-receive' hook are such that it would not be suitable for such a purpose, then an alternative way of providing the call-back ability would be implemented. Best regards, Dimitri. ``` ## Jeff King, 2019-03-19 07:06 Subject: Re: Git server side "pre-receive" hook to create new repositories Message-ID: <20190319070629.GE31801@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190319070629.GE31801%40sigill.intra.peff.net In-Reply-To: ``` On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote: > Thus, this feature request is asking that the 'pre-receive' hook > triggers when someone tries to push to a repository regardless of > whether the repository exists. This would allow automatic creation of > new repositories and smooth the work-flow described above. If the > semantics of the existing 'pre-receive' hook are such that it would not > be suitable for such a purpose, then an alternative way of providing the > call-back ability would be implemented. The pre-receive hook is a bit too late for this. It runs after the server has told the client what it has in the repo, the client decides what to push, and the server has received the pack. So receive-pack would have to know about this and fake having an empty repository. And then figure out where to store the incoming packfile, since we have no repo. So I think it would have to be another hook that runs before the rest of receive-pack. I.e., a system-level config option that says "if you are asked to accept a push for a repo and it doesn't exist, run this instead and then run as usual". It does feel a little error-prone, though, if the client does not positively say "I want you to create this if it doesn't exist". Otherwise if I do "git push server:my-misspelled-repo.git", the result is going to be rather confusing. And retro-fitting that into the receive-pack protocol is going to be tricky. It would be much easier to have a separate endpoint for the client to say "please make this repo if it doesn't exist". And then just run that before doing the push. For an unrestricted client connecting over ssh, we already have that: you can just run "ssh $host git init /path/to/repo". There isn't a similar thing that can be done over HTTP, though. -Peff ``` ## Ævar Arnfjörð Bjarmason, 2019-03-19 09:24 Subject: Re: Git server side "pre-receive" hook to create new repositories Message-ID: <87d0mndxop.fsf@evledraar.gmail.com> URL: https://gitlist.dev/e/87d0mndxop.fsf%40evledraar.gmail.com In-Reply-To: <20190319070629.GE31801@sigill.intra.peff.net> ``` On Tue, Mar 19 2019, Jeff King wrote: > On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote: > >> Thus, this feature request is asking that the 'pre-receive' hook >> triggers when someone tries to push to a repository regardless of >> whether the repository exists. This would allow automatic creation of >> new repositories and smooth the work-flow described above. If the >> semantics of the existing 'pre-receive' hook are such that it would not >> be suitable for such a purpose, then an alternative way of providing the >> call-back ability would be implemented. > > The pre-receive hook is a bit too late for this. It runs after the > server has told the client what it has in the repo, the client decides > what to push, and the server has received the pack. So receive-pack > would have to know about this and fake having an empty repository. And > then figure out where to store the incoming packfile, since we have no > repo. > > So I think it would have to be another hook that runs before the rest of > receive-pack. I.e., a system-level config option that says "if you are > asked to accept a push for a repo and it doesn't exist, run this instead > and then run as usual". > > It does feel a little error-prone, though, if the client does not > positively say "I want you to create this if it doesn't exist". > Otherwise if I do "git push server:my-misspelled-repo.git", the result > is going to be rather confusing. And retro-fitting that into the > receive-pack protocol is going to be tricky. > > It would be much easier to have a separate endpoint for the client to > say "please make this repo if it doesn't exist". And then just run that > before doing the push. > > For an unrestricted client connecting over ssh, we already have that: > you can just run "ssh $host git init /path/to/repo". There isn't a > similar thing that can be done over HTTP, though. Sounds simpler to just change the user's login shell to a wrapper that checks if the repo exists, and if not create it before proceeding. The same with http(s). I.e. in whatever webserver that's now pointing to git-http-backend as a script point to the same wrapper script. ``` ## Dimitri Joukoff, 2019-03-19 12:16 Subject: Re: Git server side "pre-receive" hook to create new repositories Message-ID: URL: https://gitlist.dev/e/SYXPR01MB09570A4CBFC30A36150C90BCDD400%40SYXPR01MB0957.ausprd01.prod.outlook.com In-Reply-To: <87d0mndxop.fsf@evledraar.gmail.com> ``` Thanks for the feedback. Whilst I understand the options that you've both proposed, the intent was to enable the Swift Package Manager to mirror repositories in a transparent way. I'll look into whether these options can be implemented inside the SPM. Cheers, Dimitri. On 19/3/19 7:24 pm, Ævar Arnfjörð Bjarmason wrote: > On Tue, Mar 19 2019, Jeff King wrote: > >> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote: >> >>> Thus, this feature request is asking that the 'pre-receive' hook >>> triggers when someone tries to push to a repository regardless of >>> whether the repository exists. This would allow automatic creation of >>> new repositories and smooth the work-flow described above. If the >>> semantics of the existing 'pre-receive' hook are such that it would not >>> be suitable for such a purpose, then an alternative way of providing the >>> call-back ability would be implemented. >> The pre-receive hook is a bit too late for this. It runs after the >> server has told the client what it has in the repo, the client decides >> what to push, and the server has received the pack. So receive-pack >> would have to know about this and fake having an empty repository. And >> then figure out where to store the incoming packfile, since we have no >> repo. >> >> So I think it would have to be another hook that runs before the rest of >> receive-pack. I.e., a system-level config option that says "if you are >> asked to accept a push for a repo and it doesn't exist, run this instead >> and then run as usual". >> >> It does feel a little error-prone, though, if the client does not >> positively say "I want you to create this if it doesn't exist". >> Otherwise if I do "git push server:my-misspelled-repo.git", the result >> is going to be rather confusing. And retro-fitting that into the >> receive-pack protocol is going to be tricky. >> >> It would be much easier to have a separate endpoint for the client to >> say "please make this repo if it doesn't exist". And then just run that >> before doing the push. >> >> For an unrestricted client connecting over ssh, we already have that: >> you can just run "ssh $host git init /path/to/repo". There isn't a >> similar thing that can be done over HTTP, though. > Sounds simpler to just change the user's login shell to a wrapper that > checks if the repo exists, and if not create it before proceeding. > > The same with http(s). I.e. in whatever webserver that's now pointing to > git-http-backend as a script point to the same wrapper script. > > ``` ## Sitaram Chamarty, 2019-03-21 09:28 Subject: Re: Git server side "pre-receive" hook to create new repositories Message-ID: URL: https://gitlist.dev/e/c9211cf1-7c27-1274-0433-bc1237bbe199%40gmail.com In-Reply-To: ``` On 19/03/2019 17.46, Dimitri Joukoff wrote: > Thanks for the feedback. > > Whilst I understand the options that you've both proposed, the intent > was to enable the Swift Package Manager to mirror repositories in a > transparent way. I'll look into whether these options can be > implemented inside the SPM. I don't know the Swift Package Manager, but looking at the preceding discussion reminded that gitolite allows this. I'm not saying that someone should install gitolite just to get this feature, but rather that *any* script that traps the incoming request can do this. Almost trivial if the request is coming in via ssh, actually. Just set the shell to a script that looks at and parses SSH_ORIGINAL_COMMAND and do whatever is needed before passing control to git proper. sitaram > > Cheers, > Dimitri. > > > On 19/3/19 7:24 pm, Ævar Arnfjörð Bjarmason wrote: >> On Tue, Mar 19 2019, Jeff King wrote: >> >>> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote: >>> >>>> Thus, this feature request is asking that the 'pre-receive' hook >>>> triggers when someone tries to push to a repository regardless of >>>> whether the repository exists. This would allow automatic creation of >>>> new repositories and smooth the work-flow described above. If the >>>> semantics of the existing 'pre-receive' hook are such that it would not >>>> be suitable for such a purpose, then an alternative way of providing the >>>> call-back ability would be implemented. >>> The pre-receive hook is a bit too late for this. It runs after the >>> server has told the client what it has in the repo, the client decides >>> what to push, and the server has received the pack. So receive-pack >>> would have to know about this and fake having an empty repository. And >>> then figure out where to store the incoming packfile, since we have no >>> repo. >>> >>> So I think it would have to be another hook that runs before the rest of >>> receive-pack. I.e., a system-level config option that says "if you are >>> asked to accept a push for a repo and it doesn't exist, run this instead >>> and then run as usual". >>> >>> It does feel a little error-prone, though, if the client does not >>> positively say "I want you to create this if it doesn't exist". >>> Otherwise if I do "git push server:my-misspelled-repo.git", the result >>> is going to be rather confusing. And retro-fitting that into the >>> receive-pack protocol is going to be tricky. >>> >>> It would be much easier to have a separate endpoint for the client to >>> say "please make this repo if it doesn't exist". And then just run that >>> before doing the push. >>> >>> For an unrestricted client connecting over ssh, we already have that: >>> you can just run "ssh $host git init /path/to/repo". There isn't a >>> similar thing that can be done over HTTP, though. >> Sounds simpler to just change the user's login shell to a wrapper that >> checks if the repo exists, and if not create it before proceeding. >> >> The same with http(s). I.e. in whatever webserver that's now pointing to >> git-http-backend as a script point to the same wrapper script. >> >> > > > -- sitaram ```