threads / discuss / 50700

Git server side "pre-receive" hook to create new repositories

Subject: Git server side "pre-receive" hook to create new repositories

## tl;dr

5 messages between Mar 9, 2019 and Mar 21, 2019.

replies: 4people: 4as markdown or json

Dimitri Joukoff· Mar 9, 2019, 10:46 UTC · lore

Some time ago, Feb 2018, I put in a proposal at Swift.org , that would allow the Swift Package Manager to clone packages from the Internet to a local git server and use the local version without requiring updates to the Swift package: https://forums.swift.org/t/proposal-to-enable-spm-to-use-a-local-repository-mirror/10291/5 . At the time, I also raised an issue at https://github.com/github/platform-samples/issues/171 asking for the feature described below. However, today I realised that that was not the right place to make such a request, and hence I've written this email.

The above thread contains a solution to the local mirror problem whose only drawback was that new repositories had to be MANUALLY initialised bare on the local server. In all other respects, in my opinion, the solution worked perfectly and saved me downloading huge amounts of data each time I performed a distribution clean on the project or switched to another project that used the same package. The solution published at Swift.org used a Ubuntu 16.04.2 server running Git 2.16.2, and the client was a Macbook Pro running MacOS with its default git.

Thus, this feature request is asking that the 'pre-receive' hook triggers when someone tries to push to a repository regardless of whether the repository exists. This would allow automatic creation of new repositories and smooth the work-flow described above. If the semantics of the existing 'pre-receive' hook are such that it would not be suitable for such a purpose, then an alternative way of providing the call-back ability would be implemented.

Best regards, Dimitri.

Jeff King· Mar 19, 2019, 07:06 UTC · re: Dimitri Joukoff · lore

Re: Git server side "pre-receive" hook to create new repositories

On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:
Show 7 quoted lines
> Thus, this feature request is asking that the 'pre-receive' hook
> triggers when someone tries to push to a repository regardless of
> whether the repository exists.  This would allow automatic creation of
> new repositories and smooth the work-flow described above.  If the
> semantics of the existing 'pre-receive' hook are such that it would not
> be suitable for such a purpose, then an alternative way of providing the
> call-back ability would be implemented.

The pre-receive hook is a bit too late for this. It runs after the server has told the client what it has in the repo, the client decides what to push, and the server has received the pack. So receive-pack would have to know about this and fake having an empty repository. And then figure out where to store the incoming packfile, since we have no repo.

So I think it would have to be another hook that runs before the rest of receive-pack. I.e., a system-level config option that says "if you are asked to accept a push for a repo and it doesn't exist, run this instead and then run as usual".

It does feel a little error-prone, though, if the client does not positively say "I want you to create this if it doesn't exist". Otherwise if I do "git push server:my-misspelled-repo.git", the result is going to be rather confusing. And retro-fitting that into the receive-pack protocol is going to be tricky.

It would be much easier to have a separate endpoint for the client to say "please make this repo if it doesn't exist". And then just run that before doing the push.

For an unrestricted client connecting over ssh, we already have that: you can just run "ssh $host git init /path/to/repo". There isn't a similar thing that can be done over HTTP, though.

-Peff
Ævar Arnfjörð Bjarmason· Mar 19, 2019, 09:24 UTC · re: Jeff King · lore

Re: Git server side "pre-receive" hook to create new repositories

On Tue, Mar 19 2019, Jeff King wrote:
Show 35 quoted lines
> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:
>
>> Thus, this feature request is asking that the 'pre-receive' hook
>> triggers when someone tries to push to a repository regardless of
>> whether the repository exists.  This would allow automatic creation of
>> new repositories and smooth the work-flow described above.  If the
>> semantics of the existing 'pre-receive' hook are such that it would not
>> be suitable for such a purpose, then an alternative way of providing the
>> call-back ability would be implemented.
>
> The pre-receive hook is a bit too late for this. It runs after the
> server has told the client what it has in the repo, the client decides
> what to push, and the server has received the pack. So receive-pack
> would have to know about this and fake having an empty repository. And
> then figure out where to store the incoming packfile, since we have no
> repo.
>
> So I think it would have to be another hook that runs before the rest of
> receive-pack. I.e., a system-level config option that says "if you are
> asked to accept a push for a repo and it doesn't exist, run this instead
> and then run as usual".
>
> It does feel a little error-prone, though, if the client does not
> positively say "I want you to create this if it doesn't exist".
> Otherwise if I do "git push server:my-misspelled-repo.git", the result
> is going to be rather confusing. And retro-fitting that into the
> receive-pack protocol is going to be tricky.
>
> It would be much easier to have a separate endpoint for the client to
> say "please make this repo if it doesn't exist". And then just run that
> before doing the push.
>
> For an unrestricted client connecting over ssh, we already have that:
> you can just run "ssh $host git init /path/to/repo". There isn't a
> similar thing that can be done over HTTP, though.

Sounds simpler to just change the user's login shell to a wrapper that checks if the repo exists, and if not create it before proceeding.

The same with http(s). I.e. in whatever webserver that's now pointing to git-http-backend as a script point to the same wrapper script.

Dimitri Joukoff· Mar 19, 2019, 12:16 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: Git server side "pre-receive" hook to create new repositories

Thanks for the feedback.

Whilst I understand the options that you've both proposed, the intent was to enable the Swift Package Manager to mirror repositories in a transparent way. I'll look into whether these options can be implemented inside the SPM.

Cheers, Dimitri.

On 19/3/19 7:24 pm, Ævar Arnfjörð Bjarmason wrote:
Show 43 quoted lines
> On Tue, Mar 19 2019, Jeff King wrote:
>
>> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:
>>
>>> Thus, this feature request is asking that the 'pre-receive' hook
>>> triggers when someone tries to push to a repository regardless of
>>> whether the repository exists.  This would allow automatic creation of
>>> new repositories and smooth the work-flow described above.  If the
>>> semantics of the existing 'pre-receive' hook are such that it would not
>>> be suitable for such a purpose, then an alternative way of providing the
>>> call-back ability would be implemented.
>> The pre-receive hook is a bit too late for this. It runs after the
>> server has told the client what it has in the repo, the client decides
>> what to push, and the server has received the pack. So receive-pack
>> would have to know about this and fake having an empty repository. And
>> then figure out where to store the incoming packfile, since we have no
>> repo.
>>
>> So I think it would have to be another hook that runs before the rest of
>> receive-pack. I.e., a system-level config option that says "if you are
>> asked to accept a push for a repo and it doesn't exist, run this instead
>> and then run as usual".
>>
>> It does feel a little error-prone, though, if the client does not
>> positively say "I want you to create this if it doesn't exist".
>> Otherwise if I do "git push server:my-misspelled-repo.git", the result
>> is going to be rather confusing. And retro-fitting that into the
>> receive-pack protocol is going to be tricky.
>>
>> It would be much easier to have a separate endpoint for the client to
>> say "please make this repo if it doesn't exist". And then just run that
>> before doing the push.
>>
>> For an unrestricted client connecting over ssh, we already have that:
>> you can just run "ssh $host git init /path/to/repo". There isn't a
>> similar thing that can be done over HTTP, though.
> Sounds simpler to just change the user's login shell to a wrapper that
> checks if the repo exists, and if not create it before proceeding.
>
> The same with http(s). I.e. in whatever webserver that's now pointing to
> git-http-backend as a script point to the same wrapper script.
>
>
Sitaram Chamarty· Mar 21, 2019, 09:28 UTC · re: Dimitri Joukoff · lore

Re: Git server side "pre-receive" hook to create new repositories

On 19/03/2019 17.46, Dimitri Joukoff wrote:
Show 6 quoted lines
> Thanks for the feedback.
> 
> Whilst I understand the options that you've both proposed, the intent
> was to enable the Swift Package Manager to mirror repositories in a
> transparent way.  I'll look into whether these options can be
> implemented inside the SPM.

I don't know the Swift Package Manager, but looking at the preceding discussion reminded that gitolite allows this.

I'm not saying that someone should install gitolite just to get this feature, but rather that *any* script that traps the incoming request can do this. Almost trivial if the request is coming in via ssh, actually. Just set the shell to a script that looks at and parses SSH_ORIGINAL_COMMAND and do whatever is needed before passing control to git proper.

sitaram
Show 52 quoted lines
> 
> Cheers,
> Dimitri.
> 
> 
> On 19/3/19 7:24 pm, Ævar Arnfjörð Bjarmason wrote:
>> On Tue, Mar 19 2019, Jeff King wrote:
>>
>>> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:
>>>
>>>> Thus, this feature request is asking that the 'pre-receive' hook
>>>> triggers when someone tries to push to a repository regardless of
>>>> whether the repository exists.  This would allow automatic creation of
>>>> new repositories and smooth the work-flow described above.  If the
>>>> semantics of the existing 'pre-receive' hook are such that it would not
>>>> be suitable for such a purpose, then an alternative way of providing the
>>>> call-back ability would be implemented.
>>> The pre-receive hook is a bit too late for this. It runs after the
>>> server has told the client what it has in the repo, the client decides
>>> what to push, and the server has received the pack. So receive-pack
>>> would have to know about this and fake having an empty repository. And
>>> then figure out where to store the incoming packfile, since we have no
>>> repo.
>>>
>>> So I think it would have to be another hook that runs before the rest of
>>> receive-pack. I.e., a system-level config option that says "if you are
>>> asked to accept a push for a repo and it doesn't exist, run this instead
>>> and then run as usual".
>>>
>>> It does feel a little error-prone, though, if the client does not
>>> positively say "I want you to create this if it doesn't exist".
>>> Otherwise if I do "git push server:my-misspelled-repo.git", the result
>>> is going to be rather confusing. And retro-fitting that into the
>>> receive-pack protocol is going to be tricky.
>>>
>>> It would be much easier to have a separate endpoint for the client to
>>> say "please make this repo if it doesn't exist". And then just run that
>>> before doing the push.
>>>
>>> For an unrestricted client connecting over ssh, we already have that:
>>> you can just run "ssh $host git init /path/to/repo". There isn't a
>>> similar thing that can be done over HTTP, though.
>> Sounds simpler to just change the user's login shell to a wrapper that
>> checks if the repo exists, and if not create it before proceeding.
>>
>> The same with http(s). I.e. in whatever webserver that's now pointing to
>> git-http-backend as a script point to the same wrapper script.
>>
>>
> 
> 
> 
-- 
sitaram

← back to recent threads