post-fetch, tweak-fetch hook

Hi!

I am writing a plugin which allows syncing of GitHub repositories with local Trac mirrored ones.

http://trac-hacks.org/wiki/GitHubSyncPlugin

In this configuration, I have a git clone --mirror local repository of GitHub repository and on each push to GitHub repository, GitHub does a POST notification to my Trac installation, where my Trac plugin receives that notification and calls fetch on local mirror.

The problem is that I also have to notify Trac of all new revisions (their hashes) the fetch retrieved. If this would be push, post-receive hook would be a place to get this information. But as I read, there is no post-fetch hook. The argument is that it is not needed. That locally run fetch can also run needed post-processing. But I have three counter arguments to this, based on my current experience:

Code reuse: having same interface for both post-fetch and post-receive hooks would mean easier post-processing. They are similar and I can imagine that there exist many scenarios where same script would be used for both hooks. Together with this is also maintainability: you could make a myfetch command doing custom post-processing, but interfacing this with a known API through hooks, all in one directory, would make things much easier to maintain. Not that one script is in hooks (post-receive) and the other is invoked in some other manner.

I am a git newbie, but after few hours or reading and searching I have not found a simple way to get a list of revisions retrieved by a fetch, so that I could call my custom post-processing. If this is really so simple that there is no need for post-fetch hook, I am all ears. FETCH_HEAD file contains only last revisions for each branch, not a range (old-new, like post-receive). Furthermore, even if there have been no new revisions retrieved, FETCH_HEAD file stays at its old state (not for example deleting it), so some additional logic would be needed. Parsing the output of git fetch also does not look like something easily parsed by a program.

Even if there is a way to reconstruct data passed to post-receive (to be given to post-fetch), I am concerned about race-condition of this. Because in post-receive this data is tightly connected to the push being done. Even if there is another push in process at the same time, git would take care post-receive is called with exactly those revisions. In case of reconstructing those revisions, it could happen, that two fetches overlap in such a manner that in my custom myfetch script I reconstruct same revisions for both fetches (for example, read same version of FETCH_HEAD file which has been updated twice by fetch, but I would get only one version), while there were of course different.

I have found some work from Joey Hess on tweak-fetch hook and I would really welcome such addition. Maybe my problems stem only from me being a git novice, but post-fetch (tweak-fetch) hook would really really make things simple and intuitive also for such users.

So I would kindly ask for some advice on how to get in a safe manner data similar to what is provided to post-receive.

Mitar

In message <CAKLmikNaqVRb=pGUhbvVQTX2tYWT0HSS2R6Ezmico3X0rMgvYQ@mail.gmail.com>, Mitar writes:

    I am a git newbie, but after few hours or reading and searching I have
    not found a simple way to get a list of revisions retrieved by a
    fetch

The output of fetch seems to do that, quite nicely.

----------------------------------------------------------------------

remote: Counting objects: 24155, done.
remote: Compressing objects: 100% (6651/6651), done.
remote: Total 21446 (delta 15831), reused 20146 (delta 14640)
Receiving objects: 100% (21446/21446), 6.78 MiB | 239 KiB/s, done.
Resolving deltas: 100% (15831/15831), completed with 574 local objects.
From git://git.kernel.org/pub/scm/git/git
   ea2c69e..edf1412  maint      -> origin/maint
   ae4479d..8275905  master     -> origin/master
 + b6b16ad...8a79d96 next       -> origin/next  (forced update)
 + 47db9a0...30b8c95 pu         -> origin/pu  (forced update)
   ce29fc8..3ca5cbc  todo       -> origin/todo
----------------------------------------------------------------------

OK, ignoring that output:

----------------------------------------------------------------------

edf1412 refs/remotes/origin/maint@{0}: fetch: fast-forward ea2c69e 8275905 refs/remotes/origin/master@{0}: fetch: fast-forward ae4479d 8a79d96 refs/remotes/origin/next@{0}: fetch: forced-update b6b16ad 30b8c95 refs/remotes/origin/pu@{0}: fetch: forced-update 47db9a0 3ca5cbc refs/remotes/origin/todo@{0}: fetch: fast-forward ----------------------------------------------------------------------

The reflog, of course, only gives you the latest change for each branch, which means that two fetches in a row will return the same output if no changes were received. Of course there is the classic:

---------------------------------------------------------------------- git for-each-ref | pcregrep 'commit\srefs/remotes/' > /tmp/old git fetch git for-each-ref | pcregrep 'commits\srefs/remotes/' > /tmp/new diff /tmp/old /tmp/new ----------------------------------------------------------------------

I'm in favor of more git hooks myself, but there is a solution to your needs without it.

    Even if there is a way to reconstruct data passed to post-receive (to
    be given to post-fetch), I am concerned about race-condition of this.

If you care about race conditions (and really, a lockfile(1) call can take care of that easily enough), then parse the output of fetch which will make it clear what *this* call did.

					-Seth Robertson

Hi!

Thank you for examples. They guide me in good direction. But still ...

On Mon, May 7, 2012 at 1:10 AM, Seth Robertson <in-gitvger@baka.org> wrote:

Of course there is a solution. Bash is turing-complete programming language. But the question is how easy and fast it is to use something.

If git would support post-fetch hook, I would get this for free and would not have to care about race conditions.

Yes, because it is really easy to parse it? There are so many different things it can output and I am not sure how to find which one should I take care of and which one should I ignore. Yes, I could learn much more about git and all possible things which can happen with fetch and its output, but I am trying to avoid this. Why? Because I believe it should be much easier.

I am sorry if I look a bit negative, but I am quite frustrated that for something so simple is so hard (in a time/knowledge meaning) to achieve. Of course there is a way and that if you know all git command this is maybe obvious, but why it couldn't be easier, if it is really a very simple patch to make it easier (and it was already posted to this mailing list few months ago).

Mitar

On Sun, May 06, 2012 at 07:10:17PM -0400, Seth Robertson wrote:

This output is human-consumable, and is not guaranteed to remain stable in future versions of git. Push has a --porcelain mode for this reason, but nobody has bothered to implement it for fetch.

Custom locking is not sufficient, as a push could modify refs behind your back. I guess you could get by with a pre-receive hook that also took the lock. But that is unnecessarily crappy; git does not have a whole repo lock, and there is no need for lock contention between pushes and fetches that are touching different refs.

I would say the "most git" thing would be to implement "fetch --porcelain", and use its output.

-Peff

Hi!

On Mon, May 7, 2012 at 9:29 AM, Jeff King <peff@peff.net> wrote:

Yes, that would be also useful. It still makes two different interfaces for probably same post-processing (after push and after fetch), but still better than nothing, what is current state.

Mitar

On Mon, May 07, 2012 at 11:11:30AM +0200, Mitar wrote:

There is nothing to say that the output from "git fetch" could not look exactly like the post-receive hook's input (in fact, that seems like a very simple and sensible format). Then you could reuse the code easily.

They would still differ in that one is a hook and one is not, of course. But at the same time, not being a hook leaves the caller of "git fetch" with much more flexibility about deciding when to call the hook and when not (whereas push does not have that luxury, because the code is running on the remote side).

-Peff