threads / discuss / 18566

Test that every revision builds before pushing changes?

Subject: Test that every revision builds before pushing changes?

## tl;dr

7 messages between Mar 26, 2009 and Mar 27, 2009.

replies: 6people: 4as markdown or json

Daniel Pittman· Mar 26, 2009, 06:29 UTC · lore
G'day.

I would like to ensure that my commits are fully bisectable before I commit them to an upstream repository, at least to the limits of an automatic tool for testing them.

'git bisect run' is similar: it can automatically locate the breaking in a test suite, for example, but that doesn't help me in the case of three commits, A (good), B (bad) and C (good, fixing B).

I would much rather, in this case, use rebase to fix B so that it, too, builds before I push the changes and pollute a public repository with a broken changeset — and make bisect that much harder to use in future.

Regards,
        Daniel
Andreas Ericsson· Mar 26, 2009, 08:16 UTC · re: Daniel Pittman · lore

Re: Test that every revision builds before pushing changes?

Daniel Pittman wrote:
Show 14 quoted lines
> G'day.
> 
> I would like to ensure that my commits are fully bisectable before I
> commit them to an upstream repository, at least to the limits of an
> automatic tool for testing them.
> 
> 'git bisect run' is similar: it can automatically locate the breaking in
> a test suite, for example, but that doesn't help me in the case of three
> commits, A (good), B (bad) and C (good, fixing B).
> 
> I would much rather, in this case, use rebase to fix B so that it, too,
> builds before I push the changes and pollute a public repository with a
> broken changeset — and make bisect that much harder to use in future.
> 

You can do that, but it requires manual work too. The trick is to make the release branch immutable on the public repository and use topic branches with per-developer namespaces. The per-developer namespace thing is actually important, as it leaves the freedom to rewind and recreate topics to the developers (which shared branches do not).

The manual step comes at merge-time; Someone has to be responsible for merging all the topics that are to be included in the release branch and make sure it builds and passes all tests after each merge.

This workflow is a bit cumbersome. NASA uses something like this but with an extra step for multiple peer reviews on every feature/fix for software they send to satellites. Or so I've heard anyways.

If you're thinking of a staging area that should queue all commits and lock the repo while testing is in progress, you need to think again, I'm afraid, as that locks up developer time in such huge amounts that it isn't really worth it. Without the locking, you get the problem of trying to automate merges (which may conflict and need a manual resolution). Although I guess a merge conflict could result in a delayed push error, so perhaps that could work too. If you manage to cook up a solution for this, please make a small writeup on this list how you went about achieving it.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.
Daniel Pittman· Mar 26, 2009, 09:10 UTC · re: Andreas Ericsson · lore

Re: Test that every revision builds before pushing changes?

Andreas Ericsson <ae@op5.se> writes:
Show 23 quoted lines
> Daniel Pittman wrote:
>>
>> I would like to ensure that my commits are fully bisectable before I
>> commit them to an upstream repository, at least to the limits of an
>> automatic tool for testing them.
>>
>> 'git bisect run' is similar: it can automatically locate the breaking in
>> a test suite, for example, but that doesn't help me in the case of three
>> commits, A (good), B (bad) and C (good, fixing B).
>>
>> I would much rather, in this case, use rebase to fix B so that it, too,
>> builds before I push the changes and pollute a public repository with a
>> broken changeset — and make bisect that much harder to use in future.
>
> You can do that, but it requires manual work too. The trick is to make
> the release branch immutable on the public repository and use topic
> branches with per-developer namespaces. The per-developer namespace
> thing is actually important, as it leaves the freedom to rewind and
> recreate topics to the developers (which shared branches do not).
>
> The manual step comes at merge-time; Someone has to be responsible for
> merging all the topics that are to be included in the release branch
> and make sure it builds and passes all tests after each merge.

Ah. You have not quite grasped what I was looking for: I was after a tool to help automate that step, rather than a workflow around it.

For example, the responsible person for that testing could use the hypothetical (until someone tells me where to find it):

    git test public..test make test
Which would then effectively wrap:
    for each revision between public and private:
         git checkout revision
         make test
         # report if that fails, allow fixing the commit or whatever
         # then 'git test continue' to carry on...

That turn the process from a manual one to an automated one: it runs that command for every revision until it fails, or until they all pass.

Regards,
        Daniel
Andreas Ericsson· Mar 26, 2009, 09:46 UTC · re: Daniel Pittman · lore

Re: Test that every revision builds before pushing changes?

Daniel Pittman wrote:
Show 26 quoted lines
> Andreas Ericsson <ae@op5.se> writes:
>> Daniel Pittman wrote:
>>> I would like to ensure that my commits are fully bisectable before I
>>> commit them to an upstream repository, at least to the limits of an
>>> automatic tool for testing them.
>>>
>>> 'git bisect run' is similar: it can automatically locate the breaking in
>>> a test suite, for example, but that doesn't help me in the case of three
>>> commits, A (good), B (bad) and C (good, fixing B).
>>>
>>> I would much rather, in this case, use rebase to fix B so that it, too,
>>> builds before I push the changes and pollute a public repository with a
>>> broken changeset — and make bisect that much harder to use in future.
>> You can do that, but it requires manual work too. The trick is to make
>> the release branch immutable on the public repository and use topic
>> branches with per-developer namespaces. The per-developer namespace
>> thing is actually important, as it leaves the freedom to rewind and
>> recreate topics to the developers (which shared branches do not).
>>
>> The manual step comes at merge-time; Someone has to be responsible for
>> merging all the topics that are to be included in the release branch
>> and make sure it builds and passes all tests after each merge.
> 
> Ah.  You have not quite grasped what I was looking for: I was after a
> tool to help automate that step, rather than a workflow around it.
> 

Oh right. Sorry, I'm stuck in continuous-integration land where people tend to want the server to take care of such things.

Show 16 quoted lines
> For example, the responsible person for that testing could use the
> hypothetical (until someone tells me where to find it):
> 
>     git test public..test make test
> 
> Which would then effectively wrap:
> 
>     for each revision between public and private:
>          git checkout revision
>          make test
>          # report if that fails, allow fixing the commit or whatever
>          # then 'git test continue' to carry on...
> 
> That turn the process from a manual one to an automated one: it runs
> that command for every revision until it fails, or until they all pass.
> 

Something like this? --%<--%<-- #!/bin/sh

git stash
revspec="$1"
shift
for rev in $(git rev-list "$revspec"); do
	git checkout $rev
	"$@" || break
done
--%<--%<--

Run it as such: ./git-test.sh public..test make test

Adding support for "continue" in there shouldn't be hard, since you'd just replace the first rev with the first parent of the last tested revision, although if you want to fix up a commit in the middle, you'd have to manually rebase the other ones onto the newly created commit (again, not very hard but I'll leave both as an exercise to you).

It doesn't handle merges very nicely, btw, but I guess this should be run prior to merging anyways.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.
Daniel Pittman· Mar 27, 2009, 01:30 UTC · re: Andreas Ericsson · lore

Re: Test that every revision builds before pushing changes?

Andreas Ericsson <ae@op5.se> writes:
Show 6 quoted lines
> Daniel Pittman wrote:
>> Andreas Ericsson <ae@op5.se> writes:
>>> Daniel Pittman wrote:
>>>> I would like to ensure that my commits are fully bisectable before I
>>>> commit them to an upstream repository, at least to the limits of an
>>>> automatic tool for testing them.
[...]
Show 9 quoted lines
>>> The manual step comes at merge-time; Someone has to be responsible for
>>> merging all the topics that are to be included in the release branch
>>> and make sure it builds and passes all tests after each merge.
>>
>> Ah.  You have not quite grasped what I was looking for: I was after a
>> tool to help automate that step, rather than a workflow around it.
>
> Oh right. Sorry, I'm stuck in continuous-integration land where people
> tend to want the server to take care of such things.

We have that also; I am primarily motivated by avoiding trivial breakage in the CI server, as well as bisection.

>> For example, the responsible person for that testing could use the
>> hypothetical (until someone tells me where to find it):
>>
>>     git test public..test make test
[...]
Show 15 quoted lines
> Something like this?
> --%<--%<--
> #!/bin/sh
>
> git stash
> revspec="$1"
> shift
> for rev in $(git rev-list "$revspec"); do
> 	git checkout $rev
> 	"$@" || break
> done
> --%<--%<--
>
> Run it as such:
> ./git-test.sh public..test make test

Thank you, that points me in the right direction, and I can obviously season the rest of it to taste — reverse that revision list, for example.

Thank you also to Wincent Colaiuta, who provided a similar script albiet with significantly more detail.

[...]
> It doesn't handle merges very nicely, btw, but I guess this should be
> run prior to merging anyways.

Once I have the framework I can quietly work my way through fixing nasty issues one way or another. Thanks.

Regards,
        Daniel
Jeff King· Mar 26, 2009, 09:49 UTC · re: Daniel Pittman · lore

Re: Test that every revision builds before pushing changes?

On Thu, Mar 26, 2009 at 08:10:52PM +1100, Daniel Pittman wrote:
Show 15 quoted lines
> For example, the responsible person for that testing could use the
> hypothetical (until someone tells me where to find it):
> 
>     git test public..test make test
> 
> Which would then effectively wrap:
> 
>     for each revision between public and private:
>          git checkout revision
>          make test
>          # report if that fails, allow fixing the commit or whatever
>          # then 'git test continue' to carry on...
> 
> That turn the process from a manual one to an automated one: it runs
> that command for every revision until it fails, or until they all pass.

I don't think such a script exists. There are continuous integration systems that support git, but I don't know if they actually check every commit.

-Peff
Wincent Colaiuta· Mar 26, 2009, 09:59 UTC · re: Jeff King · lore

Re: Test that every revision builds before pushing changes?

El 26/3/2009, a las 10:49, Jeff King escribió:
Show 23 quoted lines
> On Thu, Mar 26, 2009 at 08:10:52PM +1100, Daniel Pittman wrote:
>
>> For example, the responsible person for that testing could use the
>> hypothetical (until someone tells me where to find it):
>>
>>    git test public..test make test
>>
>> Which would then effectively wrap:
>>
>>    for each revision between public and private:
>>         git checkout revision
>>         make test
>>         # report if that fails, allow fixing the commit or whatever
>>         # then 'git test continue' to carry on...
>>
>> That turn the process from a manual one to an automated one: it runs
>> that command for every revision until it fails, or until they all  
>> pass.
>
> I don't think such a script exists. There are continuous integration
> systems that support git, but I don't know if they actually check  
> every
> commit.

It shouldn't be too hard to put together a script that does exactly what you want.

A couple of years ago I whipped up a simple script (see below) to test all the commits on a topic branch before submitting them upstream.

It's designed for that specific case, and I'm neither a shell nor a plumbing expert, but it worked for my simple workflow at least.

Cheers, Wincent

die () {
   echo "$1"
   exit 1
}

git diff --quiet || die "Unstaged changes; won't start" git diff --cached --quiet || die "Staged changes; won't start"

# parse HEAD (something like "ref: refs/heads/my_local_branch") to get  
remote and merge
GIT_DIR=$(git rev-parse --git-dir)
TOPIC="$GIT_DIR/HEAD"
test -f "$TOPIC" || die "No HEAD found at '$TOPIC'"
BRANCH_REF=$(< "$TOPIC")
if [ "${BRANCH_REF%%: */*}" != "ref" ]; then
   die "expected HEAD '$BRANCH_REF' to be of the form 'ref: .../...'"
fi
BRANCH=${BRANCH_REF##*/}
REMOTE=$(git config branch.$BRANCH.remote) || die "failed to get  
branch.$BRANCH.remote"
MERGE=$(git config branch.$BRANCH.merge) || die "failed to get branch. 
$BRANCH.merge"
MERGE=${MERGE##*/}

# remember which branch we were on prior to starting trap 'git checkout $BRANCH' exit

# will check all commits in topic branch not present in origin
# (note: may be more appropriate to use "git merge-base" here)
echo "On branch $BRANCH"
echo "Commits to be checked:"
git rev-list --pretty=oneline HEAD ^$REMOTE/$MERGE
for COMMIT in $(git rev-list HEAD ^$REMOTE/$MERGE); do
   echo "Checking $COMMIT"
   git checkout $COMMIT
   make clean
   make test || die "Commit $COMMIT is bad (make test failed)"
done
echo "All revisions passed!"

← back to recent threads