sharing between local "work" and "nightly build" git repos

I'd like to have the following setup: a ~david/git directory, where I am free to work on things, and a ~david/nightly-git, where a cron job is going build and test a nightly "next" branch.

I'd like to share as much as possible between the two repos. My naive first attempt was to clone the local repo (~david/git) using -l and -s (which I admit I do not completely understand). This sort of worked, but one issue is that doing a "git pull" in nightly is going to pull from the *locally*-cloned repo, not from the main git. Another is that a checkout in nightly failed with the obscure error:

[david@tashtego ~/git-nightly]% git checkout -b nightly-next next git checkout: updating paths is incompatible with switching branches/forcing Did you intend to checkout 'next' which can not be resolved as commit?

I assume this is because too much state is being shared the repos, and something is unfinished in the "git" directory.

I'd love some pointers on how to:

* share as many objects as possible
* share as little state as possible
* make git pull pull from remote in both repos.

Cheers,

- David

"David Frech" <nimblemachines@gmail.com> writes:

You assumed wrong. "-l and -s" does not have to do anything with the above symptoms.

git-clone helps people by setting up the new repository to follow where you cloned from, but there is no rule that you cannot change it. Look at .git/config in the nightly repository and find [remote "origin"] section; update its URL to whichever repository you would want to track from and you are done.

"git checkout -b nightly-next next" is telling git to:

 - create a new branch nightly-next starting from 'next'
 - check it out

You most likely would want to fork off of "origin/next", not 'next' which probably does not exist in your repository.

If you are willing to redo the nightly repository from scratch, I would probably recommend using --reference option when cloning, like this:

 $ git clone --reference ~david/git git://git.kernel.org/... ~/nightly-git
 $ cd ~/nightly-git
 $ git checkout --track -b next origin/next

Then a nightly update would go like this:

 $ cd ~/nightly-git
 $ git pull origin next
 $ make clean
 $ make test || barf

On 7/12/07, Junio C Hamano <gitster@pobox.com> wrote:

This makes sense, including the --track, which I neglected to set.

Could you, for my sake and the list's (if others are as confused) be clearer about the distinctions among -l, -s, and --reference? Exactly what they do, and their orthogonality (or lack of) really isn't clear from reading the man page.

Thanks for your help! I'll set this up and hopefully start running it tonight.

If the build or test barfs, who should get the mail?

BTW, I figured out a sneaky way (using filters in Gmail) to send it thru my Gmail account, so I won't have any issues with sending via SMTP from a dynamic IP.

Cheers,

- David

On 7/12/07, David Frech <nimblemachines@gmail.com> wrote:

One more thing: would it make sense to do "make -k test" so that *all* failures (if >1) show up?

Cheers,
- David

David Frech, Fri, Jul 13, 2007 02:33:19 +0200:

Yes, definitely. You'll find you will see failures which you know are already fixed, just didn't reach the nightly-build yet.

BTW, do _not_ send the errors to this mailing list. It just too easy gets out of control. And consider saving the build log somewhere after doing a _completely_ clean build, i.e. like this:

    #!/bin/bash
    cd ~/nightly-git || exit
    echo 'Subject: Nightly build git next ' $(date) >/tmp/mail.msg
    echo >>/tmp/mail.msg
    {
	rm -rf * # it does not remove .git
	git reset --hard
	git pull origin next || exit 1
	make || exit 1
	make -k test|| exit 1
    } &>> /tmp/mail.msg

    $ nightly-build || sendmail local-user@localhost < /tmp/mail.msg

It is simplier to find out what went wrong if you know what state did you have before doing the test. And the state after rm -rf * is very simple to predict.

"David Frech" <nimblemachines@gmail.com> writes:

Clone "-l" makes the new repository share the initial set of objects from the original by hardlinking each and every file under .git/objects/. After "-l" clone, two repositories are truly independent, except for the fact that the new one obviously treats the old one as its "origin".

There are a few implications:

 - New objects created in the new repository is not visible in
   the original repository and vice versa.  If you do
   development in the new repository, and push the results back
   to the original, the push will involve object transfer.  If
   your original one is updated (perhaps with git-pull from the
   upstream), you would need to update the clone separately.

 - You can safely rewind the refs in the original repository and
   run git-prune there without worrying about corrupting the
   cloned repository.

Clone "-s" does not make hardlinked copy like "-l" makes. It creates an entry in .git/objects/info/alternates file in the new repository, that instructs git that any object missing in the new repository should be looked up in the cloned repository. New objects created in the original will be shared with the cloned repository because of this, but the sharing does not happen in the other direction. Also if you run "rebase" (or "reset --hard" to rewind the tip of branches) in the original repository and then run git-gc, it would cause some objects to become unneeded in the original repository and they can get discarded. If your cloned repository depends in them, you will be in trouble. So you would need to be a bit careful when you use "-s" --- the rule of thumb is not to rewind the branches, or if you do, make sure the cloned ones are kept in sync (i.e. rewind the commits lost in the original the same way in the clone).

The "--reference" option uses the same mechanism as "-s" uses to share objects from a local repository. It is useful if you have a repository of a different project that is related to the project you are cloning from outside. For example, if you have a copy of LInus's Linux 2.6 kernel repository locally, and if you want to clone 2.6.x.y maintenance "stable" repository, you know the two repositories are alike and share a lot of objects, so you would:

    $ git clone --reference ../linus-2.6/ git://.../linux-2.6.x.y/ stable.git

To this option, because it uses the same "alternate object store" mechanism as "-s" option does, the same "be careful rewinding the branches in the original" caveat applies. Because Linus never rewind his branches, the above clone is safe.

For the purpose of "keep rebuilding 'next' from git.git repository", because my 'next' will never rewind by social convention, you do not have to worry about the "original repository being rewound and losing objects" problem, either, as long as 'next' is the branch you rebuild in the "nightly" repository.