threads / discuss / 34919

Fwd: git-daemon access-hook race condition

Subject: Fwd: git-daemon access-hook race condition

## tl;dr

10 messages between Sep 12, 2013 and Sep 13, 2013.

replies: 9people: 2as markdown or json

Eugene Sajine· Sep 12, 2013, 18:51 UTC · lore
Hi,

We are serving repos in closed netwrok via git protocol. We are using git-daemon access hook (thank you very much for such a great feature) in order to create push notifications for Jenkins. I.e. upon the push the access-hook is called and then the curl command is created and executed. As we have several instances of Jenkins, that we need to notify (three), the execution of the access-hook can take some time.

Sometimes we have a situation when the whole chain works fine but Jenkins git plugin doesn't recognize the changes. I think it happens because we hit a kind of race condition:

1. Incoming push triggers access-hook
2. notify jenkins 1
3. notify jenkins 2
4. jenkins 1 polls repo but sees no changes
5. notify Jenkins 3
6. the push data transfer finishes - consequent pushes will find
changes w/o any problem
The question is:

Is there a way to avoid that? Is it possible to have access-hook to be executed after receive? Is it possible to introduce a parameter that would specify if it needs to be executed before receive or after?

Thanks, Eugene

Junio C Hamano· Sep 12, 2013, 19:15 UTC · re: Eugene Sajine · lore

Re: Fwd: git-daemon access-hook race condition

Eugene Sajine <euguess@gmail.com> writes:
> Is it possible to have access-hook to be executed after receive?

The whole point of access-hook is to allow it to decide whether the access is allowed or not, so that is a non-starter.

A notification _after_ successful push update is usually done via the post-receive hook in the receiving repository, I think.

Eugene Sajine· Sep 12, 2013, 20:30 UTC · lore

Fwd: Fwd: git-daemon access-hook race condition

On Thu, Sep 12, 2013 at 3:15 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 9 quoted lines
> Eugene Sajine <euguess@gmail.com> writes:
>
>> Is it possible to have access-hook to be executed after receive?
>
> The whole point of access-hook is to allow it to decide whether the
> access is allowed or not, so that is a non-starter.
>
> A notification _after_ successful push update is usually done via
> the post-receive hook in the receiving repository, I think.
Junio,
Thanks for the reply!

This is interesting: i always thought about the access-hook as something to be executed when the repo is accessed, not just verification if access is allowed - your definition is much more limiting.

we have about 1400 bare repos - so i would like to avoid the configuration of each one of them. I could probably find a way to automate it, but already having access-hook in current implementation makes me reluctant to go this way, because it is so much easier to use centralized manager.

So are you really sure that it is a non-starter to have --before-service/--after-service options for access-hook?

Thanks, Eugene

Junio C Hamano· Sep 12, 2013, 21:08 UTC · re: Eugene Sajine · lore

Re: Fwd: Fwd: git-daemon access-hook race condition

Eugene Sajine <euguess@gmail.com> writes:
> So are you really sure that it is a non-starter to have
> --before-service/--after-service options for access-hook?
Given the definition of "--access-hook" in "git help daemon":
    --access-hook=<path>::
            Every time a client connects, first run an external command
            specified by the <path> ... The external command can decide
            to decline the service by exiting with a non-zero status (or
            to allow it by exiting with a zero status)....

There is *NO* way in anywhere --after-service makes any sense (and by definition --before-service is redundant).

What you _could_ propose is to define a *new* hook that is run when the spawned service has returned, with the same information that is fed to the access hook (possibly with its exit status).

I do not offhand know if we retain the original service information that long after the main daemon process has spawned the service process, though. With the current system, the only thing it needs to know is the PID of the service processes that are to be culled by calls to waitpid(). So you may have to extend existing bookkeeping data structures a bit to keep those pieces of information around if you wanted to add such a new hook.

Eugene Sajine· Sep 12, 2013, 21:16 UTC · re: Junio C Hamano · lore

Re: Fwd: Fwd: git-daemon access-hook race condition

Junio,
Thanks for the clarification! Your solution does look better.

For now though i think i will have to delay the notification somehow and let the service finish first then notify the server.

Thanks again!
Eugene
On Thu, Sep 12, 2013 at 5:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 29 quoted lines
> Eugene Sajine <euguess@gmail.com> writes:
>
>> So are you really sure that it is a non-starter to have
>> --before-service/--after-service options for access-hook?
>
> Given the definition of "--access-hook" in "git help daemon":
>
>     --access-hook=<path>::
>             Every time a client connects, first run an external command
>             specified by the <path> ... The external command can decide
>             to decline the service by exiting with a non-zero status (or
>             to allow it by exiting with a zero status)....
>
> There is *NO* way in anywhere --after-service makes any sense (and
> by definition --before-service is redundant).
>
> What you _could_ propose is to define a *new* hook that is run when
> the spawned service has returned, with the same information that is
> fed to the access hook (possibly with its exit status).
>
> I do not offhand know if we retain the original service information
> that long after the main daemon process has spawned the service
> process, though.  With the current system, the only thing it needs
> to know is the PID of the service processes that are to be culled by
> calls to waitpid().  So you may have to extend existing bookkeeping
> data structures a bit to keep those pieces of information around if
> you wanted to add such a new hook.
>
>
Eugene Sajine· Sep 12, 2013, 22:01 UTC · re: Eugene Sajine · lore

Re: Fwd: Fwd: git-daemon access-hook race condition

On Thu, Sep 12, 2013 at 5:16 PM, Eugene Sajine <euguess@gmail.com> wrote:
Show 42 quoted lines
> Junio,
>
> Thanks for the clarification! Your solution does look better.
>
> For now though i think i will have to delay the notification somehow
> and let the service finish first then notify the server.
>
> Thanks again!
>
> Eugene
>
>
> On Thu, Sep 12, 2013 at 5:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Eugene Sajine <euguess@gmail.com> writes:
>>
>>> So are you really sure that it is a non-starter to have
>>> --before-service/--after-service options for access-hook?
>>
>> Given the definition of "--access-hook" in "git help daemon":
>>
>>     --access-hook=<path>::
>>             Every time a client connects, first run an external command
>>             specified by the <path> ... The external command can decide
>>             to decline the service by exiting with a non-zero status (or
>>             to allow it by exiting with a zero status)....
>>
>> There is *NO* way in anywhere --after-service makes any sense (and
>> by definition --before-service is redundant).
>>
>> What you _could_ propose is to define a *new* hook that is run when
>> the spawned service has returned, with the same information that is
>> fed to the access hook (possibly with its exit status).
>>
>> I do not offhand know if we retain the original service information
>> that long after the main daemon process has spawned the service
>> process, though.  With the current system, the only thing it needs
>> to know is the PID of the service processes that are to be culled by
>> calls to waitpid().  So you may have to extend existing bookkeeping
>> data structures a bit to keep those pieces of information around if
>> you wanted to add such a new hook.
>>
>>
For now I'm trying to do the following:
access-hook.bash has:
delayed-notify.bash $@ &
delayed-notify.bash has:

sleep 10 ... curl ...

I'm expecting access-hook to spawn new process and return without waiting for it to finish to let the service to do its job. But when i do push - it sleeps for 10 seconds anyway. Am i missing something obvious here?

Any help is much appreciated!

Thanks, Eugene

Eugene Sajine· Sep 13, 2013, 17:17 UTC · re: Eugene Sajine · lore

Re: Fwd: Fwd: git-daemon access-hook race condition

Show 22 quoted lines
>
> For now I'm trying to do the following:
>
> access-hook.bash has:
>
> delayed-notify.bash $@ &
>
> delayed-notify.bash has:
>
> sleep 10
> ...
> curl ...
>
> I'm expecting access-hook to spawn new process and return without
> waiting for it to finish to let the service to do its job. But when i
> do push - it sleeps for 10 seconds anyway. Am i missing something
> obvious here?
>
> Any help is much appreciated!
>
> Thanks,
> Eugene

I found a following solution to make it happen while waiting for somebody to be generous enough to take on the --post-service-hook (unfortunately i'm not a C guy):

It is using 'at' command. The access-hook script has:
echo "delayed-notify.bash $@" | at now
while delayed-notify.bash has:

sleep 10 curl ...

This is not perfect and in certain situations can still fail because the delay is not long enough but this will at least resolve 90% of issues.

I hope that might be helpful for someone.

Thanks, Eugene

Junio C Hamano· Sep 12, 2013, 22:20 UTC · re: Junio C Hamano · lore

Re: Fwd: Fwd: git-daemon access-hook race condition

Junio C Hamano <gitster@pobox.com> writes:
Show 19 quoted lines
> Eugene Sajine <euguess@gmail.com> writes:
>
>> So are you really sure that it is a non-starter to have
>> --before-service/--after-service options for access-hook?
>
> Given the definition of "--access-hook" in "git help daemon":
>
>     --access-hook=<path>::
>             Every time a client connects, first run an external command
>             specified by the <path> ... The external command can decide
>             to decline the service by exiting with a non-zero status (or
>             to allow it by exiting with a zero status)....
>
> There is *NO* way in anywhere --after-service makes any sense (and
> by definition --before-service is redundant).
>
> What you _could_ propose is to define a *new* hook that is run when
> the spawned service has returned, with the same information that is
> fed to the access hook (possibly with its exit status).
Scratch that "exit status" part, as I do not think it is useful.

To a receive-pack and a send-pack that is talking to it, if a push results in a failure, it is a failure. Likewise for upload-pack and fetch-pack for a transfer in the reverse direction.

And the way that failure is communicated from the receive-pack to the end-user via the send-pack is for the receive-pack to send a protocol message that tells the send-pack about the failure, and the send-pack showing the error message and signalling the failure with its exit status. Likewise for upload-pack and fetch-pack (hence "fetch", which is conceptually a thin wrapper around it).

Between the deamon and the receive-pack (or the fetch-pack) process, however, such a failed push (or fetch) is still a success. "I correctly diagnosed and successfully sent a rejection notice to the other end" is signalled by receive-pack to the daemon by exiting with success (i.e. 0) exit status.

So even if we feed the exit status of the service process to the hook script specified by the --post-service-hook, it does not tell the script if the service "succeeded" in that sense.

Eugene Sajine· Sep 12, 2013, 23:17 UTC · re: Junio C Hamano · lore

Re: Fwd: Fwd: git-daemon access-hook race condition

On Thu, Sep 12, 2013 at 6:20 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 44 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
>
>> Eugene Sajine <euguess@gmail.com> writes:
>>
>>> So are you really sure that it is a non-starter to have
>>> --before-service/--after-service options for access-hook?
>>
>> Given the definition of "--access-hook" in "git help daemon":
>>
>>     --access-hook=<path>::
>>             Every time a client connects, first run an external command
>>             specified by the <path> ... The external command can decide
>>             to decline the service by exiting with a non-zero status (or
>>             to allow it by exiting with a zero status)....
>>
>> There is *NO* way in anywhere --after-service makes any sense (and
>> by definition --before-service is redundant).
>>
>> What you _could_ propose is to define a *new* hook that is run when
>> the spawned service has returned, with the same information that is
>> fed to the access hook (possibly with its exit status).
>
> Scratch that "exit status" part, as I do not think it is useful.
>
> To a receive-pack and a send-pack that is talking to it, if a push
> results in a failure, it is a failure.  Likewise for upload-pack and
> fetch-pack for a transfer in the reverse direction.
>
> And the way that failure is communicated from the receive-pack to
> the end-user via the send-pack is for the receive-pack to send a
> protocol message that tells the send-pack about the failure, and the
> send-pack showing the error message and signalling the failure with
> its exit status.  Likewise for upload-pack and fetch-pack (hence
> "fetch", which is conceptually a thin wrapper around it).
>
> Between the deamon and the receive-pack (or the fetch-pack) process,
> however, such a failed push (or fetch) is still a success.  "I
> correctly diagnosed and successfully sent a rejection notice to the
> other end" is signalled by receive-pack to the daemon by exiting
> with success (i.e. 0) exit status.
>
> So even if we feed the exit status of the service process to the
> hook script specified by the --post-service-hook, it does not tell
> the script if the service "succeeded" in that sense.

I see what you're saying. In my particular use case I can work around that service status because even if it failed it will just trigger Jenkins to poll and in case of failure to transfer data there will be no new changes for Jenkins to work with. If we would want the --post-service-hook to know that data transfer succeeded or failed, then may be there should be some difference between "service status" and "service process status"? In this case the existing logic works with "service process status" while the --post-service-hook is fed with the "service status" (or name it "data transfer status")

Do i make any sense?
Junio C Hamano· Sep 12, 2013, 23:27 UTC · re: Eugene Sajine · lore

Re: Fwd: Fwd: git-daemon access-hook race condition

Eugene Sajine <euguess@gmail.com> writes:
Show 16 quoted lines
>> So even if we feed the exit status of the service process to the
>> hook script specified by the --post-service-hook, it does not tell
>> the script if the service "succeeded" in that sense.
>
> I see what you're saying.
> In my particular use case I can work around that service status
> because even if it failed it will just trigger Jenkins to poll and in
> case of failure to transfer data there will be no new changes for
> Jenkins to work with. If we would want the --post-service-hook to know
> that data transfer succeeded or failed, then may be there should be
> some difference between "service status" and "service process status"?
> In this case the existing logic works with "service process status"
> while the --post-service-hook is fed with the "service status" (or
> name it "data transfer status")
>
> Do i make any sense?

Almost; you missed that there is no channel to pass "data transfer status" from the service back to the daemon.

← back to recent threads