mirroring and development with three levels of repositories?

Apologies in advance for the long question. I'm obviously doing something wrong, but I'm not sure what and searching hasn't provided me with anything. If there is a git-users mailing list this would be more appropriate for, please tell me.

Short question: It it possible for a bare repository to have both local (company-wide) and remote (mirroring upstream) branches, as well as serving those branches to our devs? If so, how?

Long question:

We're working on a project that's a collaboration between several companies. The full tree for everyone is stored on a central server that, for firewall reasons, only one person can access in the company. So, that person creates and updates nightly a local mirror of the central server. We then have a few branches of our own that we do development on inside the company, and periodically merge those branches into the main ones and push them upstream.

In mercurial, we do the branching by cloning the local mirror on our internal server, having the devs push and pull from that repo, and then pushing those changes to our local mirror, which gets pushed to the central server.

Now the central server has some things using git, and the git repo has 6 branches within it, as opposed to individual repos. The setup looks like this:

+----------+
| upstream |  <--- contains tree with 6 branches
+----------+
    |
------------- <---- firewall
    |
+--------------+
| Local Mirror | <--- "git clone --mirror" from upstream.
+--------------+
    |
    |---------|
  +-----+     +-----+
  | dev |     | dev |
  +-----+     +-----+

I would like to create two company-wide branches here that remain local, one that branches from upstream's HEAD, and one that branches from one of the existing branches on upstream (call it branch-a). I then want the devs to be able to git clone our mirror, and then switch to our company-local branch and hack away, putting thier changes on the company-wide server.

I thought I'd do this in git by the following:

(on client/dev machine): $ git clone http://local-server/local-mirror.git

For later reference:
$ git remote show origin
* remote origin
  Fetch URL: http://local-server/local-mirror.git
  Push  URL: http://local-server/local-mirror.git
  HEAD branch: xxx/stable
  Remote branches:
    feature             tracked
    branch-a            tracked
    xxx/master          tracked
    xxx/feature1        tracked
    xxx/feature2        tracked
    xxx/stable          tracked
  Local branch configured for 'git pull':
    xxx/stable merges with remote xxx/stable
  Local ref configured for 'git push':
    xxx/stable pushes to xxx/stable (up to date)

Then, we create a new branch and push it to the server:

$ git branch company-name/test1 $ git push origin company-name/test1

So far, so good, gitweb shows the new beanch in the mirror, pointing to HEAD. Now lets create and push the other new company-wide branch:

$ git checkout -b company-name/branch-a origin/branch-a $ git push origin company-name/branch-a

Again, all seems well. gitweb shows the new company-name/* branches on the local server.

However, when you do a new clone of the local mirror, it now fails:

$ git clone http://local-server/local-mirror.git Initialized empty Git repository in /home/clemej/git/local-mirror/.git/ warning: remote HEAD refers to nonexistent ref, unable to checkout.

$ git branch -a remotes/origin/company-name/test1 remotes/origin/company-name/branch-a

Note the distinct absense of all the other branches in the repository.

$ git remote show origin
* remote origin
  Fetch URL: http://local-server/local-mirror.git
  Push  URL: http://lodql-server/local-mirror.git
  HEAD branch: (unknown)
  Remote branches:
    company-name/test1       tracked
    company-name/branch-a    tracked

Jumping over to the server, we see:

$ cd local-mirror.git/
$ git remote show origin
* remote origin
  Fetch URL: http://upstream/tree.git
  Push  URL: http://upstream/tree.git
  HEAD branch: xxx/stable
  Remote branches:
    company-name/test1       stale (use 'git remote prune' to remove)
    company-name/branch-a    stale (use 'git remote prune' to remove)
    feature             tracked
    branch-a            tracked
    xxx/master          tracked
    xxx/feature1        tracked
    xxx/feature2        tracked
    xxx/stable          tracked
  Local refs will be mirrored by 'git push'

So, I'm obviously not thinking about this the right way. It it possible for a bare repository to have both local (company-wide) and remote (mirroring upstream) branches, as well as serving those branches to our devs? If so, how?

Thanks for your time, john.c

* John Clemens <clemej@gmail.com> wrote:

Yes. Just set up the proper remotes in the config files (same as w/ non-bare repos). The main problem for you might be that the hub-repo's remotes wont show up as remotes in it's clones - that's because per default only remote's refs/heads/* namespace is mapped into local's refs/remotes/*.

Most convenient would probably doing this in the local hub repo. Add a proper fetch statement in the upstream remote's config section. Suppose the upstream's remote name is "origin":

[remote "origin"]
    url = ...
    fetch = +refs/heads/*:refs/heads/origin/*
    fetch = +refs/tags/*:refs/tags/origin/*

This will make the upstream's heads and tags as they were pushed to the hub repo directly, but with the "origin/" prefix.

What does the refs/HEAD file tell in the local mirror repo ?
 

cu

On Tue, Sep 28, 2010 at 1:37 PM, Enrico Weigelt <weigelt@metux.de> wrote:

Hmm.. I do actually see the branches on dev machine. When I clone from our local mirror, everything works fine UNTIL I create a new branch and push it to our local mirror. From that moment on, all new clones fail (more precisely, they do download all the blobs, but the refs only point to the newly created branches, and HEAD is messed up.

config on the local mirror is this:

[remote "origin"]
        fetch = +refs/*:refs/*
        mirror = true
        url = xxx

presumably, fetch is recursive, so this should catch everything from upstream?

So, the problem appears to be that creating a new branch in the local mirror repo (level 2 in the diagram), messes up all the upstream refs, even though they still exist in packed-refs. Any new clones clone to the dev machine (level 3) only see the new branches, and not any of the other in the main mirror.

On the local mirror, HEAD is: ref: refs/heads/xxx/stable

On any new clone, HEAD still points to refs/heads/master (presumably because the checkout failed).

Thanks, john.c

* John Clemens <clemej@gmail.com> wrote:

Ah, maybe it corrupts refs/HEAD somehow ? What does it point to ?

Perhaps you better don't fetch the whole refs/* but refs/heads/* and refs/tags/* namespaces separately (as described above).

Does that ref exist there ? What does it tell on the upstream (big central) repo ?

cu

Wait. I just set up some dummy repositories in a working directory that mirror what we're trying to do, and it all worked. I created a dummy "upstream" repo with some branches, then a local mirror repo with 'git clone --mirror'. I'm able to check out from the local mirror, add "local" branches and push them to the mirror repo, and able to then commit to those "local" branches, no problem.

Since it all seems to work locally, I suspect a problem with our webdav git server. I'll check that out tomorrow. Thanks for the help and sorry for the noise.

john.c

On Tue, Sep 28, 2010 at 2:41 PM, Enrico Weigelt <weigelt@metux.de> wrote: