pull into dirty working tree

We have some CVS users who complain that they cannot do a pull into a dirty working tree, as they could under CVS. Here is their scenario: they make a few changes to their code and want to test it out; someone else pushes changes to the central repo that they then want to add to their working tree to test also; they then want to pull in these changes and test everything, as if they had done 'mv stuff stuff-; git pull; mv stuff- stuff'.

They would like an option (perhaps a config option) to do a "dirty pull".

The git-merge documentation states:

  You may have local modifications in the working tree files. In other
  words, git-diff is allowed to report changes. However, the merge uses
  your working tree as the working area, and in order to prevent the
  merge operation from losing such changes, it makes sure that they do
  not interfere with the merge. Those complex tables in read-tree
  documentation define what it means for a path to "interfere with the
  merge". And if your local modifications interfere with the merge,
  again, it stops before touching anything.

But my colleagues are still wondering: why can't git just do it as CVS does?

I know there are workarounds: I myself documented a set of commands to "put things on a shelf", but they still are whining.

I need a convincing argument: not a technical one, but one that is practical (e.g. where CVS would do harm that git is preventing).

So, any explanation that I can give them why we can't have a 'git pull --dirty' that moves things out of the way, then does the merge, then moves thing back, aside from that it is stupid?

Bill

On Wed, Jun 13, 2007 at 09:14:32AM -0500, Bill Lear wrote:

  I suppose the following way would work:

  $ git commit -a -m "temporary commit"  # save current work
  $ git branch -f dirty                  # ..in a separate branch
  $ git reset --hard HEAD~1              # unwind this commit
  $ git pull                             # perform a clean pull
  $ git rebase master dirty              # rewrite the work
  <you may have to fix some conficts here>
  $ git reset master                     # "undo" the commit

  So that's definitely doable.

  Though, in git, if you really work in a "pure" git environment, you
never pull until your work in your topic branch is ready for a merge.
It's a very bad habit to do otherwise: you don't _need_ to pull until
you have a clean slate.

On Wed, Jun 13, 2007 at 04:38:45PM +0200, Pierre Habouzit wrote:

  okay this is wrong because you would then "live" in the `dirty`
branch. So you'd have to do sth like:

   git checkout master
   git diff master..dirty | git apply

On Wed, Jun 13, 2007 at 04:43:11PM +0200, Pierre Habouzit wrote:

  Alternatively and definitely shorter:

  $ git commit -a -m "temporary commit"        # save the current work
  $ git checkout -f -b dirty HEAD~1            # have a dirty branch for the pull
  $ git pull                                   # perform the pull
  $ git rebase dirty master                    # rewrite the work
  <you may have to fix some conficts here>
  $ git reset HEAD~1                           # then unwind the commit

[Pierre writes:]

I know, but I can't throw git purity at them as an explanation, they won't understand. And they would disagree about the "need" to pull. That's for them to say: they WANT to pull without having to move aside the makefile that they modified to add the '-wingit' option to the compile line and just get on with their work without having to run 14 different git commands.

I'm not trying to justify their habits, but to try to see if there is any clinching reason why this habit is not only "bad", but positively harmful.

Bill

On Wed, Jun 13, 2007 at 09:45:28AM -0500, Bill Lear wrote:

  If it's because they have local modifications that match their use and
are not meant to be commited then I'd say that leaving it as "unclean"
work is a bad idea, because one day or the other they will have to
modify this Makefile to add a thing to commit for real. and then, 99
times over 100 they will commit their local modification too. _that_ is
harmful. And usually there is very soon a new commit to remove the local
change.

  I have a project where we had .htaccess that people had to customize
to have their local checkout work in the devel web server setup. It was
always commited (wrongly). We weren't using git.

  To solve those issues, there is many ways, not all are very handy, but
it works. The simplest way is to use a repository with a tool like
guilt, and each time you commit, you:

  guilt pop -a (remove your changes)
  ... do the stuff ...
  guilt push -a (push your changes again).

  Maybe you can also have a local branch where you store those local
changes. But that's still a bit awkward to use. Maybe someone will come
with a better workflow for this.

Bill> We have some CVS users who complain that they cannot do a pull Bill> into a dirty working tree, as they could under CVS. Here is Bill> their scenario: they make a few changes to their code and want Bill> to test it out; someone else pushes changes to the central repo Bill> that they then want to add to their working tree to test also; Bill> they then want to pull in these changes and test everything, as Bill> if they had done 'mv stuff stuff-; git pull; mv stuff- stuff'.

Bill> They would like an option (perhaps a config option) to do a "dirty Bill> pull".

Maybe this will do it, presuming they haven't published any of their local work, and they're on a topic branch "topic"

git-tag WIP # mark HEAD so we can come back git-commit -a -m WIP # commit current work so we can replay it git-fetch origin # grabs the upstream git-rebase origin # rebase current work-in-progress onto new upstream # might need to resolve and commit conflicts repeatedly git-reset --soft WIP # next commit will be on top of commit prior to rebase git-reset # mark all files as uncommitted as yet git-tag -d WIP # no more need for this tag

This effectively puts the upstream changes "under" (or "prior to") the current topic branch.

Randal L. Schwartz, Wed, Jun 13, 2007 17:01:14 +0200:

Wont work for new files (not yet known to git) which conflict with the same names from origin. Your method will not put them anywhere and their presence will break git-rebase. You _must_ do a merge.

Alex> Wont work for new files (not yet known to git) which conflict with the Alex> same names from origin. Your method will not put them anywhere and Alex> their presence will break git-rebase. You _must_ do a merge.

So I missed git-add for those. Then it'll work. I guess I implied that the git-commit initial step should capture all of the "interesting" state.

git-rebase *will* do the merge.  It must. :)

Randal L. Schwartz, Wed, Jun 13, 2007 21:32:35 +0200:

No, it wont. What files are you going to add? All, but *.o? All, but *.log? What if user hasn't setup .gitignore yet? What if he meant to add an ignored file anyway?

It won't merge anything which isn't known to git. Now, I come to think about it, no existing merge method will help you here (and very likely it shouldn't).

It is actually much simplier doing things right: all this stashing people keep talking about.

Alex> No, it wont. What files are you going to add?

Whatever you are working on. Whatever you want tracked. It's a commit. I'm not sure why you're having trouble following me. What did I leave out?

Alex> It won't merge anything which isn't known to git.

And that's irrelevant, because my first step *did* a commit so they are *known* to git.

Alex> Now, I come to think Alex> about it, no existing merge method will help you here (and very likely Alex> it shouldn't).

Yes, it should. You're merging your changes onto the upstream. This happens dozens of times a day with git repos all over the world. :)

Alex> It is actually much simplier doing things right: all this stashing Alex> people keep talking about.

Yes, and that's what I said too.

Randal L. Schwartz, Wed, Jun 13, 2007 22:52:35 +0200:

You left the process of figuring out what files should be temporarily added to the index. _Automatically_. Because that's what you need if you want "git pull" to just work. Otherwise it just fails.

It is not enough. It didn't add the "other" files to index, so the files are _not_ known to git.

Imagine you added a file to Makefile (which was known to git), wrote the file, put some code in it, played with it. Never called "git add" on it yet (you were just playing, as original poster suggested). Now someone from team calls and asks you to tests his changes (and imagine he added a file with exactly the same name as yours). You do a pull from his repo and it breaks. With your method it will break too, "git commit -a" wont add your file to index.

Other way around. You merge upstream in your branch. And it conflicts with work you haven't told git about just yet.

You (and others) just keep forgetting about this small detail with "other" files. Junio noticed it too. This "small" detail can make the whole stashing bussiness impossible except for maybe a common case. Which is not very helpful for a newbie who meets the uncommon, but his case. He'll learn his way though, of course. After saying out loud: "git sucks!"; which what they already do.

Alex> You left the process of figuring out what files should be temporarily Alex> added to the index. _Automatically_. Because that's what you need if Alex> you want "git pull" to just work. Otherwise it just fails.

I'd just commit *all* of it in my scenerio. There won't be any *other* files. And because of the final git-reset --soft and git-reset, it won't matter if I commited more than I'd ever commit on the final hit.

The result of my sequence (I believe) is that my topic commits will appear rebased on top of the new upstream, and that I'll have a dirty working directory that represents things as they were, ready for the next commit.

I *lose* the idea of what files were partially added before the last commit, but I can always reconstruct that by hand, because I should already be thinking about that before the next commit.

Using those famous trees, if I start with:

A--B--C--D origin
         --E--F--G my_topic
                 --(X) dirty tree

and upstream commits P Q R, the result will be:

A--B--C--D--P--Q--R origin
                  --E'--F'--G' (E, F, G rebased and merged) my_topic
                            --(X') dirty tree rebased

This works because I temporarily make H, which follows E, F, G and then rebase E-F-G-H onto origin, and then simply "uncommit" H'.

Oops. I see the problem. I don't need the "WIP" tag. I want to uncommit to G' not to G. Is that what you were referencing?

In which case, it's even simpler:

git-add . # add *everything* that's not .gitignored git-commit -a -m WIP # save for replay (H above) git-fetch origin # get upstream (P Q R) git-rebase origin # creates E' F' G' H' git-reset --soft HEAD^ # back up to G' git-reset # mark everything un-added as of G'

Yeah, this is far easier.

Randal L. Schwartz, Thu, Jun 14, 2007 00:01:22 +0200:

It is also wrong. You either add a lot of junk (because it is not ignored yet - who'll setup .gitignores just to check something?!) and spend a millenium hashing it (ever tried to add 4.7 DVD?) or it does not add really important files which just happen to be in .gitignore.

IOW, this recipe is dangerous. It'll only _mostly_ work.

Hi,

On Wed, 13 Jun 2007, Bill Lear wrote:

Two things you can do. First thing is: teach them to commit first. If they decide later that they did not want that change, they still can go back with "git reset HEAD@{2}".

The other thing, if you have to, is to put all dirty changes into the index before pull. Something like "git add $(git ls-files --modified)". You can even make that a global alias for your users. Although IIRC it does not work if the merge changes the same files as your dirty work tree touches, but I could very well be wrong there.

Hth, Dscho

On Wednesday 2007 June 13, Johannes Schindelin wrote:

Or the shiny new

 git add -u

which works a treat :-)

Andy

Hi,

On Wed, 13 Jun 2007, Andy Parkins wrote:

Yeah, completely forgot about that.

Another idea just hit me: you could add this to your "[alias]" section:

	stash = !git add -u && \
		tree=$(git-write-tree) && \
		commit=$(echo stash $(date) | \
			git-commit-tree $tree -p HEAD) && \
		git-update-ref refs/heads/stash $commit && \
		git-reset --hard

	up = !git stash && git pull && git cherry-pick -n stash

Or something like that (untested!). Then, your users could use "git up" to pull from origin, and reapply their changes on top, without committing.

This assumes that your users only have the remote "origin", but given their obvious "objections" to Git concepts, I think they do.

Ciao, Dscho

On Wednesday, June 13, 2007 at 16:40:18 (+0100) Andy Parkins writes:

Better.

I wonder, also, if there could be a way to alert users that their working tree is dirty before all the git pull blather comes out, scaring their poor little souls? So, instead of this:

% git pull
remote: Generating pack...
remote: Done counting 122 objects.
remote: Result has 90 objects.
remote: Deltifying 90 objects.
remote:  100% (90/90) done
Unpacking 90 objects
remote: Total 90 (delta 59), reused 41 (delta 10)
 100% (90/90) done
* refs/remotes/origin/master: fast forward to branch 'master' of
git://source/sc
  old..new: 171b65f..0be3472
* refs/remotes/origin/v1.0: fast forward to branch 'v1.0' of
git://source/sc
  old..new: a9de9dd..efa3a73
Updating 717d9f6..0be3472
src/fs/testsuite/fs.tst/gettest: needs update
src/nl/EocCompiler.cc: needs update
src/nl/EocCompiler.hh: needs update
src/nl/Nl.cc: needs update
fatal: Entry 'src/netlist/EocCompiler.cc' not uptodate. Cannot merge.

we could have:

% git pull Sorry, I can't pull, as you have a dirty working tree. Please commit your changes or move your files before you pull. These are the files that are preventing this:

    src/fs/testsuite/fs.tst/gettest
    src/nl/EocCompiler.cc
    src/nl/EocCompiler.hh
    src/nl/Nl.cc

Bill

Hi,

On Wed, 13 Jun 2007, Bill Lear wrote:

Well, it's their fault, isn't it?

Sorry, this is the first time Git can realize that the dirty working directory conflicts with the changes about to be applied.

For example, I run "git pull" very often with a modified Makefile. If the merge would not touch the Makefile, it would succeed. No need to do anything fancy.

If you do have to shut the (otherwise useful) messages up, you can always have an alias (using the advanced technique illustrated in another post in this thread).

As far as I can see, gettest is not responsible, so this would be wrong.

Ciao, Dscho

On Wednesday, June 13, 2007 at 17:07:20 (+0100) Johannes Schindelin writes:

Yes, it is.  But that's no reason not to try to produce a nicer warning.

Ah, this makes sense. It has to do all the other stuff first and only when it comes across one that won't "fit" does it complain. This makes more sense now...

Bill

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

I do not think that would change the issue in general, as the index needs to be clean (wrt HEAD) during a conflicting merge.

Incidentally, I was planning to start a "stash dirty state and later reapply on a different commit" soon (if I survive an event planned tomorrow, that is ;-).

When you are in the middle of something (and when you are not in the middle of a merge), you have two states you care about. What is in the index, and what is in the working tree. Conceptually, even though you do _not_ have commits for these two extra states, your "history" would look like this:

                 o <- working tree state = W
                /
               o <- index state = I
              /
      -------o <- HEAD

A dirty merge (whether it is done as "git checkout $another", "git merge $commit", or "git pull $somebodyelse") is to first stash away I and W, perform the merge proper to advance HEAD, and after all that is done, recreate the index and working tree state on top of the updated HEAD, to arrive at this:

                 o  -->  * <- new working tree state = W'
                /       /
               o  -->  * <- new index state = I'
              /       /
      -------o-------* <- merge = new HEAD
    old HEAD ^      /
                   /
      --------o---o <- other history

This can be internally done as four steps:

 * stash the dirty states (save I and W)

   - record I by "git commit" (no parameters to commit the index).

   - record W by "git add . && git commit -a".

   - remember HEAD as $STASH.

 * match the working tree and the index to original HEAD by
   "reset HEAD^ && reset --hard HEAD".  The first --mixed reset
   is to forget about new files added with "git add .", so that
   the second reset will not remove them from the working tree.

 * perform a merge in the resulting clean tree

 * unstash the dirty states

   - I' is the three-way merge between $STASH~1 and the new HEAD
     using $STASH~2 as the common ancestor.

   - W' is the three-way merge between $STASH and I' using
     $STASH~1 as the common ancestor.

Unstashing operation can potentially require two merge conflict resolutions. As we _care_ about distinction between the index state and the working tree state, this is unavoidable.

    Side note: as an implementation, it is a possibility not to
    record I and record only W as the direct child of the HEAD
    when stashing.  Unstashing would update only the working
    tree (i.e. after unstashing, the index and the HEAD exactly
    matches).  But it risks "forgetting" newly added files and I
    would say it is unacceptable for that reason, even if we
    declare that this feature is only for non-git people.

Combined with the potential conflict resolution for the merge proper, you have to resolve up to three merges in a row. One conflict resolution is something even CVS users must accept anyway, so we are talking about up to two _extra_ conflict resolutions here.

On Wed, 13 Jun 2007, Bill Lear wrote:

Where CVS would do harm that git is preventing is if they did something brilliant, forgot how they did it, got other people's changes from the central repository, and got complicated merge conflicts, and lost their change trying to resolve them. (Or, for that matter, if the merge algorithm screwed up the file without reporting conflicts.)

What git refuses to do is overwrite a file you've changed when you haven't committed it, because something could go wrong, and you'd lose the work.

It would be possible to tell git that you're okay with it accidentally losing your work, but people tend not to like this idea quite so much when it's phrased like that.

The git sequence for this situation is:

$ git commit -a $ git fetch $ git rebase origin

The operation they want to perform is "rebase", which puts the changes they made on top of other people's changes instead of where they were written. It also wants the changes committed, so that it doesn't have to worry about losing your work, but afterward you can use "git commit --amend" to add fixes and the rest of your changes, because your work is the top commit and hasn't been pushed out. Alternatively, "git reset HEAD^" at the end of the sequence will turn the commit into uncommitted changes.

	-Daniel
*This .sig left intentionally blank*

On Wed, 13 Jun 2007, Bill Lear wrote:

Well, a lot of people have told you that the answer is "don't do that", but I actually somewhat disagree.

I think it might be perfectly fine to allow for a *fast-forward* pull to do a three-way merge on the working tree, assuming the index is clean in the paths that got modified.

For a real merge (not just a fast-forward), we really *really* must not do it, for a very simple reason: we have no sane way to handle conflicts if we have both a merge from the pull itself _and_ a merge from the working tree. Don't get me wrong: I'm sure it's possible in theory, I just think that in practice it's such a total hairball that it's not worth it!

So I think we could actually try to allow "git pull" with a fast-forward pull and a dirty working tree.

(We obviously _already_ allow a working tree that is dirty in the paths that don't actually get changed at all! I use that all the time. So this is strictly limited to the "dirty state actually overlaps with what got pulled!)

It might make it a bit easier for CVS people to get used to the git model: keep your dirty working tree, and do "git pull" to update it, and fix up any conflicts in the working tree. That's how CVS works - it's a bad model, but it's a model that may be worth supporting just to get people more easily into the _good_ model.

		Linus

Linus Torvalds <torvalds@linux-foundation.org> writes:

If a bad model _is_ supported, what incentive is there for these people to move into the good model, I honestly wonder...

On Thu, 2007-06-14 at 00:49 -0700, Junio C Hamano wrote:

Is it a bad model for the normal development process? Probably.

Would it solve the problem of "I want to keep my debug-statements that I won't ever commit in the merged result"? ... yes

I think that's at least one valid usecase we should probably support. One I'd also like to use ;-)

Junio C Hamano wrote:

Presumably it's a good model because it's easier, more productive, more predictable, more reliable, or some combination of those things; that's the incentive. If it's none of those things for a given developer, then maybe it's not in fact a better model for them.

Of course, even if it is better for them, some people will never move -- but those are the people who won't willingly move to git anyway unless the bad model is supported. There are enough of them out there that people who *do* want to use the good model, but have to work in an environment where they're outnumbered by those other folks, find they can't sell the organization on git because it forces a change in work style on people who aren't interested in changing their work styles. (Not a purely hypothetical statement, sadly.)

You can view this in terms of being a leg up for people who *do* want to use git, but are in environments where they are unable to convince or force everyone else to adopt git-style workflows. I think it's telling that almost all the discussions about this kind of feature are of the form, "I'm trying to convince my team to use git, and they find it no good because of X." It's the person trying to sell git to the group, presumably so they can use it themselves without having to go through a CVS or Subversion or p4 gateway, that this stuff really helps. That the rest of the team will benefit down the road too is nice but probably not the immediate selfish personal goal of the people who are asking for this kind of feature.

-Steve

On Thu, 14 Jun 2007, Steven Grimm wrote:

Personally, I think there is a point where it isn't worth trying to convert the world. If people consider GIT bad and unwilling to use it because of X or Z then they probably better stay with CVS. There is a limit to how backward bending should GIT do to accomodate everyone, especially if it is about compromize in its usage model just to make life easier for people who want to preserve their inferior work flow.

This being said, I don't claim to have a particular opinion about the issue discussed in this thread. Simply that things should be decided on a technical basis and be justified with good arguments. Saying that "I can't convince my co-workers to use GIT if it doesn't do X" is _not_ a good argument.

Nicolas

On Wednesday, June 13, 2007 at 22:21:38 (-0700) Linus Torvalds writes:

I have now officially fallen out of my chair.

Exactly my desires. I think it could work reliably, and as they mature into git users, they will come to appreciate branches.

Bill

On Thu, 14 Jun 2007, Bill Lear wrote:

Well, the thing is, I actually pull into dirty trees all the time. So I can really see the point of wanting to have some dirty state (you're not ready to commit it yet), but still wanting to update your tree to some newer state..

Of course, in the kernel (where I do this - I do it to a much lesser degree in git too, but for the kernel it's "normal" for me to do it), we have very good modularization of source code, so I can do the "pull into a dirty tree" with _current_ git, just because there is almost never a clash (and if there is, nothing bad happens: the pull won't succeed, and I can decide to either stash away my diff or just undo it, and then re-pull afterwards).

But I can also well imagine that other projects aren't quite as modular as the kernel is. In fact, I pretty much know that for a fact.. We've spent years splitting things up, just because clashes are nasty.

So I don't think the "pull into a dirty tree" is necessarily a horribly bad workflow. It *can* be due to bad habits, but it can equally well be due to perfectly fine habits like having added some debugging code that you actually want to eventually throw away, but you haven't quite debugged it totally yet.

For example, maybe the reason you pull is because there's a potential fix in upstream - you want to keep your debugging code (to _verify_ the fix, or verify that it wasn't a fix at all).

The fact that some CVS users do it because they are used to it doesn't _automatically_ make it bad form. They probably have really bad reasons for doing it (namely the fact that under CVS, you cannot commit to your tree as aggressively as you can under git, since committing affects everybody else too), and *those* reasons may not be true under git, but the other ones (see above) are still what appear to be valid reasons for allowing this..

So the only reason I'm ambivalent is actually that I suspect it's just hard to do cleanly. For example, doing it for the fast-forward case is much easier, but then people will start *wanting* to do it for the more complex "real merge" case, and will complain when that doesn't work. And that one really _is_ fundamentally harder.

So it might be easier to take a "git stash ; git pull ; git unstash" approach instead of making "git pull" handle working tree conflicts itseld.

		Linus

On Thu, Jun 14, 2007 at 08:46:27AM -0700, Linus Torvalds wrote:

Isn't that "git add .; git commit; git fetch; git rebase <something>; git reset ^HEAD"? With the conflict resolution happening at rebase time.

  OG.

On Thu, 14 Jun 2007, Olivier Galibert wrote:

No.

The two workflows happen to co-incide *if* the "git pull" is a fast-forward, but not if you actually had previous commits that you wanted the "git pull" to merge.

So if you want things to actually work as a "git pull with dirty state merge", you really do need to do "git stash + git pull + git unstash".

		Linus

On 6/15/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:

Right now git merges/fforwards well with dirty state as long as the same path is not touched on both sides. But there are several situations where it could do better allowing those ops to go through if they don't result in any conflict.

- For Fast Forwards on a dirty path - attempt the merge on a temp file
and refuse to complete the FF there is a conflict.
- For merges on a dirty path, attempt the merge. If both the tree
merge _and_ the subsequent with the dirty state are clean, then there
is no problem updating the checkout.

In both cases, we can still go ahead in the case of a conflict against the local state and give the user the normal conflict markers (or separate files of the patch doesn't apply at all. The situation where I think it is valid to refuse to go ahead is in the "merge on dirty path" where the tree merge results in a conflict. Too many states to keep track of -- not for git but for the user.

cheers,

martin

On Fri, 15 Jun 2007, Martin Langhoff wrote:

I agree, but there is actually a practical implementation problem with doing that:

 - currently, we can decide *ahead* of time (by just looking at the index, 
   whether the index entry is clean, and the two branches) whether the 
   merge can go ahead or not.

 - so we actually do two passes: the first pass checks that we can do what 
   we want to do cleanly, and the second pass actually starts changing the 
   working tree!

Now, if you actually start doing the *merge* thing, the biggest practical problem ends up being that the natural place where you find out that "oops, we can't get a clean result" is in phase 2 - *after* you have potentially already done earlier merges in the working directory!

And that's unacceptable. A "git pull" needs to either fail early without making any modifications at all (telling people that the tree is dirty and cannot be merged), or it needs to complete but leave conflict markers.

But yeah, if you can check in stage 1 (_without_ changing the working tree) whether the merge will work, then everything is fine.

		Linus

On 6/15/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:

Aha- so at phase 1 we know
 - what paths are dirty in the checkout
 - what paths of the merge need an actuall diff3 merge

perhaps we can do those diff3 merges elsewhere (tempfiles). If they are trivial diff3 merges, then we can complete the merge operation without touching the checkout. After this is complete, we can then update the checkout...

cheers,

m

fredag 15 juni 2007 skrev Martin Langhoff:

Can't you treat this like git-am or git-rabase. Save the diff to .dottest. Then peform the pull just like you do normally involving the user if necessary. After that either automatically or after a --continue/--abort you apply the diff with it's own conflicts.

-- robin