{"thread":{"id":"25276","subject":"mirroring and development with three levels of repositories?","startedAt":"2010-09-28T17:25:40Z","lastAt":"2010-09-29T01:58:46Z","messageCount":5,"participants":["John Clemens","Enrico Weigelt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"151935","messageId":"AANLkTinFiOA6Grzk16W2D=k8Xt+EgTanrS7iryW2evop@mail.gmail.com","threadId":"25276","inReplyTo":null,"subject":"mirroring and development with three levels of repositories?","fromName":"John Clemens","fromEmail":"clemej@gmail.com","sentAt":"2010-09-28T17:25:40Z","receivedAt":"2010-09-28T17:25:40Z","isPatch":false,"sender":{"key":"clemej@gmail.com","avatar":"https://gravatar.com/avatar/c2302fb7ce5ea22bf8fec296d731c9d9150ac444430656a667b7b7dd34b7ecbe?d=mp&s=160"},"body":"Apologies in advance for the long question. I'm obviously doing something\nwrong, but I'm not sure what and searching hasn't provided me with anything.\nIf there is a git-users mailing list this would be more appropriate for,\nplease tell me.\n\nShort question:\nIt it possible for a bare repository to have both local (company-wide) and\nremote (mirroring upstream) branches, as well as serving those branches to our\ndevs? If so, how?\n\nLong question:\n\nWe're working on a project that's a collaboration between several companies.\nThe full tree for everyone is stored on a central server that, for firewall\nreasons, only one person can access in the company. So, that person creates\nand updates nightly a local mirror of the central server. We then have a few\nbranches of our own that we do development on inside the company, and\nperiodically merge those branches into the main ones and push them upstream.\n\nIn mercurial, we do the branching by cloning the local mirror on our internal\nserver, having the devs push and pull from that repo, and then pushing those\nchanges to our local mirror, which gets pushed to the central server.\n\nNow the central server has some things using git, and the git repo has\n6 branches within it, as opposed to individual repos. The setup looks\nlike this:\n\n+----------+\n| upstream | <--- contains tree with 6 branches\n+----------+\n |\n------------- <---- firewall\n |\n+--------------+\n| Local Mirror | <--- \"git clone --mirror\" from upstream.\n+--------------+\n |\n |---------|\n +-----+ +-----+\n | dev | | dev |\n +-----+ +-----+\n\nI would like to create two company-wide branches here that remain local,\none that branches from upstream's HEAD, and one that branches from one\nof the existing branches on upstream (call it branch-a). I then want the devs\nto be able to git clone our mirror, and then switch to our company-local branch\nand hack away, putting thier changes on the company-wide server.\n\nI thought I'd do this in git by the following:\n\n(on client/dev machine):\n$ git clone http://local-server/local-mirror.git\n\nFor later reference:\n$ git remote show origin\n* remote origin\n Fetch URL: http://local-server/local-mirror.git\n Push URL: http://local-server/local-mirror.git\n HEAD branch: xxx/stable\n Remote branches:\n feature tracked\n branch-a tracked\n xxx/master tracked\n xxx/feature1 tracked\n xxx/feature2 tracked\n xxx/stable tracked\n Local branch configured for 'git pull':\n xxx/stable merges with remote xxx/stable\n Local ref configured for 'git push':\n xxx/stable pushes to xxx/stable (up to date)\n\nThen, we create a new branch and push it to the server:\n\n$ git branch company-name/test1\n$ git push origin company-name/test1\n\nSo far, so good, gitweb shows the new beanch in the mirror, pointing to\nHEAD. Now lets create and push the other new company-wide branch:\n\n$ git checkout -b company-name/branch-a origin/branch-a\n$ git push origin company-name/branch-a\n\nAgain, all seems well. gitweb shows the new company-name/* branches on the\nlocal server.\n\nHowever, when you do a new clone of the local mirror, it now fails:\n\n$ git clone http://local-server/local-mirror.git\nInitialized empty Git repository in /home/clemej/git/local-mirror/.git/\nwarning: remote HEAD refers to nonexistent ref, unable to checkout.\n\n$ git branch -a\nremotes/origin/company-name/test1\nremotes/origin/company-name/branch-a\n\nNote the distinct absense of all the other branches in the repository.\n\n$ git remote show origin\n* remote origin\n Fetch URL: http://local-server/local-mirror.git\n Push URL: http://lodql-server/local-mirror.git\n HEAD branch: (unknown)\n Remote branches:\n company-name/test1 tracked\n company-name/branch-a tracked\n\nJumping over to the server, we see:\n\n$ cd local-mirror.git/\n$ git remote show origin\n* remote origin\n Fetch URL: http://upstream/tree.git\n Push URL: http://upstream/tree.git\n HEAD branch: xxx/stable\n Remote branches:\n company-name/test1 stale (use 'git remote prune' to remove)\n company-name/branch-a stale (use 'git remote prune' to remove)\n feature tracked\n branch-a tracked\n xxx/master tracked\n xxx/feature1 tracked\n xxx/feature2 tracked\n xxx/stable tracked\n Local refs will be mirrored by 'git push'\n\nSo, I'm obviously not thinking about this the right way. It it possible\nfor a bare repository to have both local (company-wide) and remote (mirroring\nupstream) branches, as well as serving those branches to our devs? If so, how?\n\nThanks for your time,\njohn.c\n\n-- \nJohn Clemens \n"},{"id":"151938","messageId":"20100928173710.GA31419@nibiru.local","threadId":"25276","inReplyTo":"AANLkTinFiOA6Grzk16W2D=k8Xt+EgTanrS7iryW2evop@mail.gmail.com","subject":"Re: mirroring and development with three levels of repositories?","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-09-28T17:37:11Z","receivedAt":"2010-09-28T17:37:11Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* John Clemens wrote:\n\n> Short question:\n> It it possible for a bare repository to have both local (company-wide) and\n> remote (mirroring upstream) branches, as well as serving those branches to our\n> devs? If so, how?\n\nYes. Just set up the proper remotes in the config files (same as w/\nnon-bare repos). The main problem for you might be that the hub-repo's\nremotes wont show up as remotes in it's clones - that's because per\ndefault only remote's refs/heads/* namespace is mapped into local's\nrefs/remotes/*.\n\nMost convenient would probably doing this in the local hub repo.\nAdd a proper fetch statement in the upstream remote's config section.\nSuppose the upstream's remote name is \"origin\":\n\n\n[remote \"origin\"]\n url = ...\n fetch = +refs/heads/*:refs/heads/origin/*\n fetch = +refs/tags/*:refs/tags/origin/*\n\nThis will make the upstream's heads and tags as they were pushed\nto the hub repo directly, but with the \"origin/\" prefix.\n\n> $ git clone http://local-server/local-mirror.git\n> Initialized empty Git repository in /home/clemej/git/local-mirror/.git/\n> warning: remote HEAD refers to nonexistent ref, unable to checkout.\n\nWhat does the refs/HEAD file tell in the local mirror repo ?\n \n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"151940","messageId":"AANLkTin2UPUPn5Sr=7dqT5bxFB7a-BYFzJhGV74r6taF@mail.gmail.com","threadId":"25276","inReplyTo":"20100928173710.GA31419@nibiru.local","subject":"Re: mirroring and development with three levels of repositories?","fromName":"John Clemens","fromEmail":"clemej@gmail.com","sentAt":"2010-09-28T18:27:29Z","receivedAt":"2010-09-28T18:27:29Z","isPatch":false,"sender":{"key":"clemej@gmail.com","avatar":"https://gravatar.com/avatar/c2302fb7ce5ea22bf8fec296d731c9d9150ac444430656a667b7b7dd34b7ecbe?d=mp&s=160"},"body":"On Tue, Sep 28, 2010 at 1:37 PM, Enrico Weigelt wrote:\n> * John Clemens wrote:\n>\n>> Short question:\n>> It it possible for a bare repository to have both local (company-wide) and\n>> remote (mirroring upstream) branches, as well as serving those branches to our\n>> devs? If so, how?\n>\n> Yes. Just set up the proper remotes in the config files (same as w/\n> non-bare repos). The main problem for you might be that the hub-repo's\n> remotes wont show up as remotes in it's clones - that's because per\n> default only remote's refs/heads/* namespace is mapped into local's\n> refs/remotes/*.\n>\n> Most convenient would probably doing this in the local hub repo.\n> Add a proper fetch statement in the upstream remote's config section.\n> Suppose the upstream's remote name is \"origin\":\n>\n>\n> [remote \"origin\"]\n>    url = ...\n>    fetch = +refs/heads/*:refs/heads/origin/*\n>    fetch = +refs/tags/*:refs/tags/origin/*\n\nHmm.. I do actually see the branches on dev machine. When I clone from our\nlocal mirror, everything works fine UNTIL I create a new branch and push it\nto our local mirror. From that moment on, all new clones fail (more precisely,\nthey do download all the blobs, but the refs only point to the newly created\nbranches, and HEAD is messed up.\n\nconfig on the local mirror is this:\n\n[remote \"origin\"]\n fetch = +refs/*:refs/*\n mirror = true\n url = xxx\n\npresumably, fetch is recursive, so this should catch everything from upstream?\n\nSo, the problem appears to be that creating a new branch in the local mirror\nrepo (level 2 in the diagram), messes up all the upstream refs, even though\nthey still exist in packed-refs. Any new clones clone to the dev machine\n(level 3) only see the new branches, and not any of the other in the\nmain mirror.\n\n> This will make the upstream's heads and tags as they were pushed\n> to the hub repo directly, but with the \"origin/\" prefix.\n>\n>> $ git clone http://local-server/local-mirror.git\n>> Initialized empty Git repository in /home/clemej/git/local-mirror/.git/\n>> warning: remote HEAD refers to nonexistent ref, unable to checkout.\n>\n> What does the refs/HEAD file tell in the local mirror repo ?\n\nOn the local mirror, HEAD is:\nref: refs/heads/xxx/stable\n\nOn any new clone, HEAD still points to refs/heads/master (presumably\nbecause the checkout failed).\n\nThanks,\njohn.c\n\n-- \nJohn Clemens \n"},{"id":"151946","messageId":"20100928184116.GA27401@nibiru.local","threadId":"25276","inReplyTo":"AANLkTin2UPUPn5Sr=7dqT5bxFB7a-BYFzJhGV74r6taF@mail.gmail.com","subject":"Re: mirroring and development with three levels of repositories?","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-09-28T18:41:18Z","receivedAt":"2010-09-28T18:41:18Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* John Clemens wrote:\n> > [remote \"origin\"]\n> >    url = ...\n> >    fetch = +refs/heads/*:refs/heads/origin/*\n> >    fetch = +refs/tags/*:refs/tags/origin/*\n> \n> Hmm.. I do actually see the branches on dev machine. When I clone from our\n> local mirror, everything works fine UNTIL I create a new branch and push it\n> to our local mirror. From that moment on, all new clones fail (more precisely,\n> they do download all the blobs, but the refs only point to the newly created\n> branches, and HEAD is messed up.\n> \n> config on the local mirror is this:\n> \n> [remote \"origin\"]\n> fetch = +refs/*:refs/*\n> mirror = true\n> url = xxx\n\nAh, maybe it corrupts refs/HEAD somehow ? What does it point to ?\n\nPerhaps you better don't fetch the whole refs/* but refs/heads/* and\nrefs/tags/* namespaces separately (as described above).\n\n> On the local mirror, HEAD is:\n> ref: refs/heads/xxx/stable\n\nDoes that ref exist there ? \nWhat does it tell on the upstream (big central) repo ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"152007","messageId":"AANLkTimg9NKbL033Et9CgKw_KUZ97oDSCy2Wh7mz2p5E@mail.gmail.com","threadId":"25276","inReplyTo":"20100928184116.GA27401@nibiru.local","subject":"Re: mirroring and development with three levels of repositories?","fromName":"John Clemens","fromEmail":"clemej@gmail.com","sentAt":"2010-09-29T01:58:46Z","receivedAt":"2010-09-29T01:58:46Z","isPatch":false,"sender":{"key":"clemej@gmail.com","avatar":"https://gravatar.com/avatar/c2302fb7ce5ea22bf8fec296d731c9d9150ac444430656a667b7b7dd34b7ecbe?d=mp&s=160"},"body":"Wait. I just set up some dummy repositories in a working directory\nthat mirror what we're trying to do, and it all worked. I created a\ndummy \"upstream\" repo with some branches, then a local mirror repo\nwith 'git clone --mirror'. I'm able to check out from the local\nmirror, add \"local\" branches and push them to the mirror repo, and\nable to then commit to those \"local\" branches, no problem.\n\nSince it all seems to work locally, I suspect a problem with our\nwebdav git server. I'll check that out tomorrow. Thanks for the help\nand sorry for the noise.\n\njohn.c\n\nOn Tue, Sep 28, 2010 at 2:41 PM, Enrico Weigelt wrote:\n> * John Clemens wrote:\n>> > [remote \"origin\"]\n>> >    url = ...\n>> >    fetch = +refs/heads/*:refs/heads/origin/*\n>> >    fetch = +refs/tags/*:refs/tags/origin/*\n>>\n>> Hmm.. I do actually see the branches on dev machine.  When I clone from our\n>> local mirror, everything works fine UNTIL I create a new branch and push it\n>> to our local mirror.  From that moment on, all new clones fail (more precisely,\n>> they do download all the blobs, but the refs only point to the newly created\n>> branches, and HEAD is messed up.\n>>\n>> config on the local mirror is this:\n>>\n>> [remote \"origin\"]\n>>         fetch = +refs/*:refs/*\n>>         mirror = true\n>>         url = xxx\n>\n> Ah, maybe it corrupts refs/HEAD somehow ? What does it point to ?\n>\n> Perhaps you better don't fetch the whole refs/* but refs/heads/* and\n> refs/tags/* namespaces separately (as described above).\n>\n>> On the local mirror, HEAD is:\n>> ref: refs/heads/xxx/stable\n>\n> Does that ref exist there ?\n> What does it tell on the upstream (big central) repo ?\n>\n>\n> cu\n> --\n> ----------------------------------------------------------------------\n>  Enrico Weigelt, metux IT service -- http://www.metux.de/\n>\n>  phone:  +49 36207 519931  email: weigelt@metux.de\n>  mobile: +49 151 27565287  icq:   210169427         skype: nekrad666\n> ----------------------------------------------------------------------\n>  Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n> ----------------------------------------------------------------------\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n\n\n\n-- \nJohn Clemens \n"}]}