simple cvs-like git wrapper

hey all,

We've been trying out git here for a while now, and we've noticed two things that we both like and dislike: git's flexibility, and git's flexibility.

Git's flexibility is great in the sense that power users can basically bend git to their will, but its' flexibility is also causing workflow issues in our environment, where beginning users can get lost in all the options that it has, and this is causing communication issues for these folks with the rest of our team.

Hence, I was hoping that people could suggest ways of simplifying git, making a cvs-like frontend for people to use.

I was thinking of something like this:

gvs branch <branch name>: 

	creates a branch for people to start making edits on in their
	localized copy.

gvs commit:

	commits that branch to a centralized git repository.

gvs update:

    Takes the latest changes, from all branches, that everyone 
	else has committed into the centralized git repository, and merges 
	them onto the current branch. 

gvs list:

	lists all the branches that have been merged into the current
	workspace.

In other words, what I'm looking for is sort of 'cvs+'. Instead of working on one, synchronized branch as per cvs, we want to work on several, parallel, branches that synchronize on intervals.

We basically want this for managing related changesets - we want to be able to switch from one patch branch to another and commit them separately - but we don't want to sacrifice the automatic integration that you get from cvs by doing:

	cvs update

on a given branch.

Anyways, hope this makes sense. I'm not sure how feasible the above is - it's meant to be as simple it can be, with as much DWIM-ness as possible. Any feedback is appreciated.

Ed

( ps - We could just use CVS of course, but that's just too simple, with no easy way of managing which change goes along with which feature request... )

"Ed S. Peschko" <esp5@pge.com> writes:

[...]

[...]

One thing (besides horrible branching and even worse merging) which I hated in multi-user CVS is the "cvs update", namely the fact that if you want to commit changes, you _have_ to rebase them on top of current work. So when you are ready to commit, when you have tested everything, you are sometimes forced to resolve a merge to be able to commit... and have to test resolved merge... and perhaps again, and again.

Working on branches is much nicer IMVHO. And it allows to separate changes into series of small, incremental commits[*1*]. If you want to work in centralized or semi-centralized way, you probably would want to use rebase based workflow, with "git pull --rebase" (which just got implemented).

Footonotes: =========== [*1*] I'd like to point to LKML post about creating perfect patch *series*, but I have forgot to bookmark it, and canot find it again (IIRC somebody posted link some time ago here on git mailing list).

Yeah, I realize that it's not exactly the best solution for every project, but for projects tied to a piece of hardware (ie: a database, a particular box, etc), its much more important to be in sync, to have 'one true view' of the world rather than to have the freedom to have multiple views.

In our case, our code is tied to a database and a database instance. An environment equals attachment to a given oracle SID. If someone is out of sync with other people's changes, then that person's environment is wrong.

I agree with you, however, with the working on branches. We need the ability to do the small incremental commits, and then tie them back to SOX requirements (bleah).

Hence the hope for the automatic merging along with a given branch - that, when you do an 'gvs update', it takes all the outstanding deltas on all the branches that have been uploaded into the central repository, and applies them, one by one, to your local repository, and keeps the branch intact.

That it basically does the perfect patch series functionality you are talking about, but in an automatic way..

A couple of questions:

	1. How do you get a list - on a shared, remote, repository - of all the 
       branches that a shared repository contains, from the point of
	   view of a client? ie: git-branch shows local branches..

    2. Could the above 'gvs update' be implemented in terms of a series 
	   of 'git pull --rebase' or even 'git pull' merges from the
	   centralized repository based on the output from the command 
       above?

Anyways, I wouldn't mind it if 'gvs update' paused at the end of each merge - that you'd do a 'gvs update', it would show you exactly what was going to merge before it did it (maybe even via a vimdiff of old and new side by side), and would allow you to do a regression test after each patchset was applied..

After all, it's my wrapper so I'll implement it the way I like it.. ;-)

Thanks,

Ed

"Ed S. Peschko" <esp5@pge.com> wrote:

Surely not every single code change impacts the database schema and meaning of column values? If that were truely the case then I'd say you have bigger issues to tackle.

There's perfectly valid reasons a user (or several users) may go off on a branch for a little while. Like maybe they are working together to refactor a chunk of front end user interface, to better handle result sets >100 rows, something that wasn't anticipated would ever happen when the system was first built. It happens. There may not be any database changes involved on that but it may be distrubtive enough that you don't want to see their changes until its mostly done.

I've been there, done that with Oracle and CVS.  Having Git makes
it a whole lot easier to go off on a side branch when refactoring
a chunk of UI, as you don't have to impact all of your coworkers
right away.
 

So, uh, mention the requirement using some sort of unique requirement identifier in each commit message, and use a commit-msg hook to make sure you didn't forget to include the requirement number. We make links like that in our issue tracking system with "lpmNNNN" keywords, where NNNN is the issue tracker number.

If you ever want to know what changes were done for a particular
requirement, pop open gitk and search for contains "soxNNNN", or use
`git log --grep=soxNNN`, and there's your list.
 

I don't think you fully understand what git-merge does. It retains the full history of every individual change made along that branch, as well as who performed the merge, and when. Git also tracks if the merge makes changes that weren't on either parent branch, and correctly blames such changes to the individual who did the merge. Most older SCMs are simply not able to do such detailed reporting. Heck, some modern SCMs also can't do it.

Yea, git-merge is totally freaking automatic.  Except when there is
a conflict.  Then you have to fix up a few things.  But Git isn't
a mind reader (yet) so there will always be cases where it doesn't
quite do what you wanted it to without a little additional guidence.
 

I think you are talking about `git ls-remote --heads url`.
 

Yes, but why are you merging all of the available branches? Why aren't you merging a single specific branch? It seems very odd to me that you want to merge every branch available. I just cannot see how that gives you any benefit over just having a single branch called "master" that you rebase your changes onto before pushing them, thus enforcing a mostly linear history.

Only linear history has its downsides. It doesn't really quite show you what the commits were developed against. So it may be possible that after a rebase the end result compiles and works, but an earlier commit that used to work now doesn't, due to other changes.

If you are really worried about SOX requirements, I'd imagine you are also worried about SOX auditing. In which case I would think that for most changes that are going to be part of your permenant history don't want to use rebase, but instead merge, to show the original history of every change.

One of my favorite ways to regression test every commit in a series during a rebase is to use git-rebase -i:

	$ git rebase -i
	... editor opens ... which is vi ...
	:%s/^pick /edit /
	:wq

	... so now it stops between every patch ...
	$ while test -d .git/.dotest-merge; do \
	make clean test && git-rebase --continue || break; done

If anything breaks, you are right there on the broken change and you can use `git show` to see it, `git commit --amend` to fix it up, you can test the fix, and finally then continue the rebase to move onto the next change.

I've got a 67 patch series I'm in the middle of doing at day-job using that exact process. Takes a few minutes per commit as our build cycle is slow, but it works darn well. I'm letting it run overnight as I expect it to get about 70% through before it finds a problem, if it is going to find one.

On Tue, Jan 29, 2008 at 11:00:02PM -0500, Shawn O. Pearce wrote:

well, no, but I'd say 80-90% of the changes we have are ones that we want to instantly share with everybody. I was thinking that ones that we didn't would be prefixed, as in:

	git-branch exp-<change_name>

and those would need to be renamed explicitly to become 'mainline' branches before they were merged..

You've got some good points, and my original intent was to answer them point-by-point, but suffice to say:

	1. I was hoping to make each branch correspond to a work request,
	   that would be tracked for SOX. We also need to track the changes
	   in mercury interactive, not git, so I've got some challenges
	   there in making a wrapper to handle this.

	2. A single, linear history on the remote end wouldn't be easy for
	   reporting purposes.

    3. A single linear history on the remote end wouldn't support 
	   the rare cases where I *do* want a single change.

I guess my scheme's workability depends on how effective git is at doing merges from branch to branch, and how good it is at fixing conflicts in a way that is simple for the user. In CVS, I get:

    >>>>>
    ...
    =====
    ...
    <<<<<

when a conflict occurs, and you need to resolve that conflict before re-committing again. Does git do a similar thing?

Also, with git-ls-remote - is there a way to see more information about the remote branch rather than just its name, ie: can you say:

    git-ls-remote -l --heads origin

to get a list of changes in the order they were made? And is there a command that does what I want, ie:

	git pull origin --all 

Which pulls all branches from origin and merges them into the current branch in an intelligent way, ie: by order in which the branches were committed, or even:

	git pull origin --re: '^(?!exp)'

which pulls in all branches matching a given regular expression (in this case, not matching 'exp' at the beginning..

Ed

"Ed S. Peschko" <esp5@pge.com> wrote:

OK, that's sensible.
 

Yea, that's not a bad idea, because then if a dev wants to make multiple small changes before saying they are finished with a particular request they can.

I've got a script I use at final "intergration" point between developers and the testing team that scans through all of the commit messages for all changes since the last integration and updates our issue database automatically to mark those items in some way. Rather short Perl script, except for the 1300 lines of hideous C code to speak to the vendor's tracking software. We find it works well to hold off on updating external non-git records until near the very end.

You do realize that in Git once a branch has been merged you can delete that branch, and the history of all of those changes remains? Git itself has probably been built from thousands of individual branches over the years, yet I'm positive Junio only has a small handful (25-30) in his working repository that he uses to maintain Git. This branches are only the "live" ones that are still being actively worked on and aren't ready for release.

Yup, there were 1,756 branches that lead up to v1.5.4-rc5:

	$ git rev-list --parents v1.5.4-rc5 \
	  | perl -ne 'print if /.* .* .*/' \
	  | wc -l
    1756

that of course was since the beginning of time. Since v1.5.3 we have had about 240:

	$ git rev-list --parents v1.5.3..v1.5.4-rc5 \
	  | perl -ne 'print if /.* .* .*/' \
	  | wc -l
    1756

So by this it sounds like you don't want to ever use rebase? And instead encourage your team to always use git-merge or git-pull (without its rebase flag).

Yes. Git also tries harder to keep you from committing a file that has an unresolved conflict in it. Git's merging is a lot smarter than CVS's ever was, as Git does automatic merge base detection during each merge, so you don't have to keep track of this yourself.

Nope. To really see anything further you need access to the objects. That means either executing a tool like git-log directly on the server, using gitweb through your webbrowser, or fetching the objects down to your local repository with git-fetch, where you can then browse them with any git log viewing tool (git-log, gitk, rev-list, etc.).

Merging all branches on the remote named "origin" is simply not an
intelligent thing to do.  Nobody blindly merges everything available
from a remote, and nobody has ever asked for such a function before
in Git.  I still think its nuts, but I don't know all details of
your situation so I'll just shut up now and hope you know what you
are really asking for.
 

You can do something like this, but I *really* think this is a horribly bad idea:

	#!/bin/sh

	# Download all new branches, remove any now deleted branches.
	#
	git fetch || exit
	git remote prune origin || exit

	# Loop through all non-exp branches and merge them
	#
	for b in $(git for-each-ref --format=%(refname) refs/remotes/origin)
	do
		case $b in
		refs/remotes/origin/exp-*)
			: do not merge
			;;
		*)
			echo "==> Merging ${b##refs/remotes/origin/} ..."
			if git merge $b
			then
				: good merge
			else
				echo >&2
				echo >&2 error: Fix conflicts, commit, rerun $0
				exit 1
			fi
		esac
	done

This is going to be slow as you are running git-merge for each and every branch available to you. You can do a lot better by loading the branch DAG into memory in Perl/C/Python and doing a graph coloring algorithm to see if a merge is necessary or not, as if you are merging everything all of the time almost everything is going to be always merged to everything else. Which as I said earlier is nuts.

Ok, I'm not going to belabor the point, but for the most part, the requests in our particular domain are separate. A developer for the most part works only on files that other developers are not working on.

There are exceptions to this, but for the most part this is true.. Hence, most of the time there will be no conflicts.

Anyways, I'll try to improve on your script, but it looks like what I want to do.

hmm. Is there a simple method to get this graph? I'm assuming that you would have to get all the local commits and compare them to the remote commits, and only merge the branches that have commits not yet merged..

Ed

"Ed S. Peschko" <esp5@pge.com> wrote:

Something along these lines:

	%remotes = \
		git for-each-ref \
		--format='%(objname) %(refname)' \
		refs/remotes/origin;

	@lines = \
		git rev-list \
		keys %remotes \
		--not HEAD

	foreach $line in @lines {
		if $remotes contains $line
			git merge $remotes{$line}
	}

That gets you the graph. The %(objname) string coming back from for-each-ref is in $line in the loop. If you see $line inside that map you built from for-each-ref then that commit isn't yet in the current branch. So you'd then want to merge that commit.

"Shawn O. Pearce" <spearce@spearce.org> writes:

I guess you can ask "show-branch --independent" to cull branches that are pure subset of other branches.

But no matter how you do this, the resulting history would be less efficient to bisect if you make Octopus.

As you can ask "git log --no-merges" to omit merges from the listing, I am not sure if it is worth it to avoid two-side merges and insist on making an Octopus these days.

On Tue, 29 Jan 2008, Ed S. Peschko wrote:

Actually, it's a simpler concern: with CVS (and most version control systems), you can't make a permanent record of your state without resolving conflicts. With git, you make the permanent record first, and then resolve conflicts in order to get a state that includes your changes and can be pushed to the shared location. With git, you can still have "one true view" of the world; it's just that git really really doesn't want to lose your work, so it refuses to overwrite your files unless you've put them in its storage, where it can give them back if it does something wrong with your working tree.

Of course, your "gvs update" could do a local git commit of the current state, fetch the remote changes, and rebase the local commit onto them, so it's not necessarily a workflow change. But note that, if the user fails to do a manual merge, the user can try again, or try working on a different branch, or fork at that point, depending on how hopeless the situation is.

	-Daniel
*This .sig left intentionally blank*

I'm leaning towards agreement with Shawn that you're a little confused as to how syncing works or you're not adequately expressing what you're really hoping for and why git isn't doing it and we're misunderstanding you.

Regardless, if you're looking for something that will pull down all the updates from a central branch, merge them in, and then rebase your current changes on top of it you should check out how git-p4 does it in order to get the basic mechanics for your script.

~kate = masukomi http://weblog.masukomi.org

Hi,

On Fri, 1 Feb 2008, Kate Rhodes wrote:

Well, the thing is: git (plumbing) does not really care about syncing and how that works. The core of git is "just" a very clever object database, geared towards revision control.

So in theory, Ed could just go and write a porcelain for git, which does _exactly_ what he wants, yet would be mostly interoperable with other git porcelains.

I imagine that the "cvsish update" would boil down to fetching upstream, updating the index with what the user has in the working directory (something like "git add -u"), then write out a tree from that, and call merge-recursive on that tree and the upstream's current tree. Force fast forward master, but keep the index, keeping the error output of merge-recursive. Done.

However, you need to understand the internals of git enough to do that. Especially the conflict management, and its relation with the index.

And I cannot see how that understanding would not lead to abandoning that work-flow as suboptimal. But hey, that's just my private opinion.

Ciao, Dscho

On Wed, 30 Jan 2008, Ed S. Peschko wrote:

Please try to not remove attributions. TIA.

Err, if it is as bad as you say, and it is not possible to change environment to what is saved in commit [message] on git-checkout and on git-reset this just removes any possibility of parallel development. Not that CVS-like centralized development would do much better in such case...

It would be nice to save environment state somehow in the commit, or in worktree / commit tree. Versioning databases (or is it just database schema) is PITA, and I don't know any good solution. phpBB modules which need to modify database have some kind of diff-like thing, but...

[cut] ---- Below there is description of "cvs update" workflow and how it can be implemented in git, rebase based workflow which is very similar but allows to generate small incremental commits while retaining linear history, and default merge based workflow.

(Perhaps something like that should be added to cvs-migration.txt?)

Let's assume that time, or rather parentage/sequence of commits flow from left to right, so "A---B" means that B is child of commit A (is later revision), and that A is parent of commit B. Let us mark uncommited changes by '*'.

1. First, how 'cvs commit' and 'cvs update' works.
1.1. The case where there were no changes in the central (origin)
     repository

     before "cvs commit"

       A---B---C---* 

     after "cvs commit"

       A---B---C---D

1.2. The case where there were changes in the origin repo
     ("cvs commit" says: need to update, or something like that)

     before "cvs commit", and "cvs update"

       A---B---C---*

     after "cvs update"

       A---B---C---d---e---*'

     where *' means that the state might be modified wrt. *; you might
     have to resolve conflicts, while still not having you work saved
     under version control anywhere

     after "cvs commit" (and after resolving conficts, if there were any)

       A---B---C---d---e---F 

You can implement such workflow in git by stashing your changes, doing fasfforward-only pull (or fetch), then unstashing changes, resolving conflicts if there are any, and finally commiting changes. You would have to implement "needs update" pre-commit hook if you want to follow CVS workflow fully.

NOTE that while you are working on '*' somobody might have changed environment!

2. Rebase based workflow; the "git pull --rebase" needs new enough git
   version

   before your work (common base both on your local repository, and
   in the origin / central / distribution point one)

     A---B---C

   you create a few commits, splitting your work into small, self
   contained, easy to understand, bisectable commits

     A---B---C---1---2

   now you want to be up to date wrt central repository, to send your
   changes (push, publish) to central repo, via "git pull --rebase"

     A---B---C---1---2
              \
               \-d---e

     A---B---C---1---2
              \
               \-d---e---1'---2'

     A---B---C---d---e---1'---2'

   where 1' and 2' are your commits modified by the presence of
   'd' and 'e' in the commit chain. Note that in the process of moving
   (copying) your changes on top of fetched changes from central repo
   there can be conflicts.

3. Merge based workflow; ordinary "git pull"

   before your work (common base both on your local repository, and
   in the origin / central / distribution point one)

     A---B---C

   you create a few commits, splitting your work into small, self
   contained, easy to understand, bisectable commits

     A---B---C---1---2

   now you want to be up to date wrt central repository, to send your
   changes (push, publish) to central repo, via "git pull"

     A---B---C---1---2
              \
               \-d---e

     A---B---C---1---2---M
              \         /
               \-d---e-/

   where M is a merge commit; you might have to resolve conflicts here.

Hello,

I remember:

	http://groups.google.com/group/linux.kernel/browse_thread/thread/26f1247fd4a2acbf/ac9743b603e53bae?lnk=gst&q=perfect+patch#ac9743b603e53bae

.  Do you mean that?

Best regards Uwe

Uwe Kleine-König wrote:

Thanks a lot, but it is not what I was searching for. This is interesting article about what to put and what to not put in the cover letter [0/n], while I am trying to find post about how to divide/split change into series of commits implementing one separate part, so the whole is easy to understand and bisectable.