threads / discuss / 15918

Re: Untracked working tree files

Subject: Re: Untracked working tree files

## tl;dr

24 messages between Oct 15, 2008 and Oct 16, 2008.

replies: 23people: 7as markdown or json

Andrew Morton· Oct 15, 2008, 18:56 UTC · lore

Untracked working tree files

I often get this (running git 1.5.6.rc0 presently):

y:/usr/src/git26> git-checkout linux-next error: Untracked working tree file 'arch/x86/kernel/apic.c' would be overwritten by merge.

which screws things up. I fix it by removing the offending file, which gets irritating because git bails out after the first such instance, so I need to rerun git-checkout once per file (there are sometimes tens of them).

Should this be happening? I don't know what causes it, really. All I've been doing in that directory is running `git-checkout' against various maintainers' trees. 95% of the time this works OK but eventually git seems to get all confused and the above happens.

Is there some way in which I can work around this with a single command rather than having to run git-checkout once per offending file? I suppose a good old `rm -rf *' would do it...

Thanks.
david@lang.hm· Oct 15, 2008, 19:09 UTC · re: Andrew Morton · lore
On Wed, 15 Oct 2008, Andrew Morton wrote:
Show 13 quoted lines
> Date: Wed, 15 Oct 2008 11:56:54 -0700
> From: Andrew Morton <akpm@linux-foundation.org>
> To: git@vger.kernel.org
> Subject: Untracked working tree files
> 
> I often get this (running git 1.5.6.rc0 presently):
>
> y:/usr/src/git26> git-checkout linux-next
> error: Untracked working tree file 'arch/x86/kernel/apic.c' would be overwritten by merge.
>
> which screws things up.  I fix it by removing the offending file, which
> gets irritating because git bails out after the first such instance, so
> I need to rerun git-checkout once per file (there are sometimes tens of them).

what I do when I run into this is "git reset --hard HEAD" which makes all files in the working directory match HEAD, and then I can do the other checkout.

David Lang
Show 15 quoted lines
> Should this be happening?  I don't know what causes it, really.  All
> I've been doing in that directory is running `git-checkout' against
> various maintainers' trees.  95% of the time this works OK but
> eventually git seems to get all confused and the above happens.
>
> Is there some way in which I can work around this with a single command
> rather than having to run git-checkout once per offending file?  I
> suppose a good old `rm -rf *' would do it...
>
> Thanks.
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
david@lang.hm· Oct 15, 2008, 19:14 UTC · re: david@lang.hm · lore
On Wed, 15 Oct 2008, david@lang.hm wrote:
Show 21 quoted lines
> On Wed, 15 Oct 2008, Andrew Morton wrote:
>
>> Date: Wed, 15 Oct 2008 11:56:54 -0700
>> From: Andrew Morton <akpm@linux-foundation.org>
>> To: git@vger.kernel.org
>> Subject: Untracked working tree files
>> 
>> I often get this (running git 1.5.6.rc0 presently):
>> 
>> y:/usr/src/git26> git-checkout linux-next
>> error: Untracked working tree file 'arch/x86/kernel/apic.c' would be 
>> overwritten by merge.
>> 
>> which screws things up.  I fix it by removing the offending file, which
>> gets irritating because git bails out after the first such instance, so
>> I need to rerun git-checkout once per file (there are sometimes tens of 
>> them).
>
> what I do when I run into this is "git reset --hard HEAD" which makes all 
> files in the working directory match HEAD, and then I can do the other 
> checkout.

I think you can also do git checkout -f head to force the checkout to overwrite all files

the fact that git will happily leave modified things in the working directory appears to be very helpful for some developers, but it's also a big land mine for others.

is there a way to disable this?
David Lang
Andrew Morton· Oct 15, 2008, 19:24 UTC · re: david@lang.hm · lore

On Wed, 15 Oct 2008 12:14:34 -0700 (PDT) david@lang.hm wrote:

Show 23 quoted lines
> On Wed, 15 Oct 2008, david@lang.hm wrote:
> 
> > On Wed, 15 Oct 2008, Andrew Morton wrote:
> >
> >> Date: Wed, 15 Oct 2008 11:56:54 -0700
> >> From: Andrew Morton <akpm@linux-foundation.org>
> >> To: git@vger.kernel.org
> >> Subject: Untracked working tree files
> >> 
> >> I often get this (running git 1.5.6.rc0 presently):
> >> 
> >> y:/usr/src/git26> git-checkout linux-next
> >> error: Untracked working tree file 'arch/x86/kernel/apic.c' would be 
> >> overwritten by merge.
> >> 
> >> which screws things up.  I fix it by removing the offending file, which
> >> gets irritating because git bails out after the first such instance, so
> >> I need to rerun git-checkout once per file (there are sometimes tens of 
> >> them).
> >
> > what I do when I run into this is "git reset --hard HEAD" which makes all 
> > files in the working directory match HEAD, and then I can do the other 
> > checkout.
I was using this but it seems it wasn't in the right place in the script.
> I think you can also do git checkout -f head to force the checkout to 
> overwrite all files
OK, I'll try that.
> the fact that git will happily leave modified things in the working 
> directory appears to be very helpful for some developers, but it's also a 
> big land mine for others.

These files weren't modified. By me, at least. git might have "modified" them, but it has all the info necessary to know that the file has no uncommitted changes.

> is there a way to disable this?
> 
> David Lang
Andrew Morton· Oct 15, 2008, 19:26 UTC · re: david@lang.hm · lore

On Wed, 15 Oct 2008 12:14:34 -0700 (PDT) david@lang.hm wrote:

Show 23 quoted lines
> On Wed, 15 Oct 2008, david@lang.hm wrote:
> 
> > On Wed, 15 Oct 2008, Andrew Morton wrote:
> >
> >> Date: Wed, 15 Oct 2008 11:56:54 -0700
> >> From: Andrew Morton <akpm@linux-foundation.org>
> >> To: git@vger.kernel.org
> >> Subject: Untracked working tree files
> >> 
> >> I often get this (running git 1.5.6.rc0 presently):
> >> 
> >> y:/usr/src/git26> git-checkout linux-next
> >> error: Untracked working tree file 'arch/x86/kernel/apic.c' would be 
> >> overwritten by merge.
> >> 
> >> which screws things up.  I fix it by removing the offending file, which
> >> gets irritating because git bails out after the first such instance, so
> >> I need to rerun git-checkout once per file (there are sometimes tens of 
> >> them).
> >
> > what I do when I run into this is "git reset --hard HEAD" which makes all 
> > files in the working directory match HEAD, and then I can do the other 
> > checkout.
I do
	git-reset --hard HEAD
	git-reset --hard linux-next
	git-checkout linux-next
and get

error: Untracked working tree file 'Next/SHA1s' would be overwritten by merge. y

grr.
> I think you can also do git checkout -f head to force the checkout to 
> overwrite all files
yup, that fixed it.  whee, thanks.
Show 7 quoted lines
> the fact that git will happily leave modified things in the working 
> directory appears to be very helpful for some developers, but it's also a 
> big land mine for others.
> 
> is there a way to disable this?
> 
> David Lang
Nicolas Pitre· Oct 15, 2008, 19:32 UTC · re: Andrew Morton · lore
On Wed, 15 Oct 2008, Andrew Morton wrote:
Show 12 quoted lines
> I do
> 
> 	git-reset --hard HEAD
> 	git-reset --hard linux-next
> 	git-checkout linux-next
> 
> and get
> 
> error: Untracked working tree file 'Next/SHA1s' would be overwritten by merge.
> y
> 
> grr.
What about simply:
	git-checkout -f linux-next
Nicolas
Nicolas Pitre· Oct 15, 2008, 19:34 UTC · re: Nicolas Pitre · lore
On Wed, 15 Oct 2008, Nicolas Pitre wrote:
Show 18 quoted lines
> On Wed, 15 Oct 2008, Andrew Morton wrote:
> 
> > I do
> > 
> > 	git-reset --hard HEAD
> > 	git-reset --hard linux-next
> > 	git-checkout linux-next
> > 
> > and get
> > 
> > error: Untracked working tree file 'Next/SHA1s' would be overwritten by merge.
> > y
> > 
> > grr.
> 
> What about simply:
> 
> 	git-checkout -f linux-next
Never mind -- you apparently did that already with success.
Nicolas
Linus Torvalds· Oct 15, 2008, 19:31 UTC · re: david@lang.hm · lore
On Wed, 15 Oct 2008, david@lang.hm wrote:
> 
> the fact that git will happily leave modified things in the working directory
> appears to be very helpful for some developers, but it's also a big land mine
> for others.

Hmm. It doesn't actually do that normally. If you switch between trees, git will (or _should_) remove the old files that it knows about. If you get a lot of left-over turds, there's something wrong.

It could be a git bug, of course. That said, especially considering the source of this, I wonder if it's just that Andrew ends up using all those non-git scripts on top of a git tree, and then that can result in git *not* knowing about a certain file, and then when switching between trees (with either git checkout or with git reset), the data that was created with non-git tools gets left behind and now git will be afraid to overwrite it.

So yes, there are ways to force it (both "git checkout -f" and "git reset --hard" having already been mentioned), but the need for that - especially if it's common - is a bit discouraging.

Especially since it's still possible that it's some particular mode of git usage that leaves those things around. Andrew - have you any clue what it is that triggers the behavior?

(By the filename, I realize it's a file that doesn't exist in one tree or the other, and which doesn't get removed at some point. But have you had merge failures, for example? Is it perhaps a file that was created during a non-clean merge, and then got left behind due to the merge being aborted? It would be interesting to know what led up to this..)

			Linus
david@lang.hm· Oct 15, 2008, 19:42 UTC · re: Linus Torvalds · lore
On Wed, 15 Oct 2008, Linus Torvalds wrote:
Show 25 quoted lines
> On Wed, 15 Oct 2008, david@lang.hm wrote:
>>
>> the fact that git will happily leave modified things in the working directory
>> appears to be very helpful for some developers, but it's also a big land mine
>> for others.
>
> Hmm. It doesn't actually do that normally. If you switch between trees,
> git will (or _should_) remove the old files that it knows about. If you
> get a lot of left-over turds, there's something wrong.
>
> It could be a git bug, of course. That said, especially considering the
> source of this, I wonder if it's just that Andrew ends up using all those
> non-git scripts on top of a git tree, and then that can result in git
> *not* knowing about a certain file, and then when switching between trees
> (with either git checkout or with git reset), the data that was created
> with non-git tools gets left behind and now git will be afraid to
> overwrite it.
>
> So yes, there are ways to force it (both "git checkout -f"  and "git reset
> --hard" having already been mentioned), but the need for that - especially
> if it's common - is a bit discouraging.
>
> Especially since it's still possible that it's some particular mode of git
> usage that leaves those things around. Andrew - have you any clue what it
> is that triggers the behavior?

I see it fairly frequently when switching between different branches of a project.

I also see it when I try applying a patch to a tree, then want to get up to date with that tree (in this case it really is different)

It could be that git is looking to see if the file is the same as the old tree had it before checking out the new tree. if it isn't for any reason it sounds the alert.

David Lang
Show 8 quoted lines
> (By the filename, I realize it's a file that doesn't exist in one tree or
> the other, and which doesn't get removed at some point. But have you had
> merge failures, for example? Is it perhaps a file that was created during
> a non-clean merge, and then got left behind due to the merge being
> aborted? It would be interesting to know what led up to this..)
>
> 			Linus
>
Linus Torvalds· Oct 15, 2008, 19:56 UTC · re: david@lang.hm · lore
On Wed, 15 Oct 2008, david@lang.hm wrote:
> 
> I see it fairly frequently when switching between different branches of a
> project.

So, at least for any normal switch, assuming file 'a' doesn't exist in the other branch, you really should have a few different cases:

 - you have a dirty file, and git should say something like
	error: You have local changes to 'file'; cannot switch branches.
   because it refuses to modify the file to match the other branch (which 
   includes removing it) if it doesn't match the index.
   So this case shouldn't leave anything behind.
 - You have that extra file, but it's not in the index.
   If it's in your current HEAD, we should still notice it with something 
   like:
	error: Untracked working tree file 'tree' would be removed by merge.
   because now it's untracked (not in the index), but the switching 
   between branches tries to essentially "apply" the difference between 
   your current HEAD and the new branch, and finds that the difference 
   involves removing a file that git isn't tracking.
See?
HOWEVER.

If you're used to doing "git checkout -f" or "git reset --hard", both of those checks are just ignored. After all, you asked for a forced switch.

And at least in the second case, what I think happens is that git won't remove the file it doesn't know about, so you'll have a "turd" left around.

So yes, you can certainly get these kinds of left-overs, but they really should be only happening if you "force" something. Do you do that often?

			Linus
david@lang.hm· Oct 15, 2008, 20:17 UTC · re: Linus Torvalds · lore
On Wed, 15 Oct 2008, Linus Torvalds wrote:
Show 28 quoted lines
> On Wed, 15 Oct 2008, david@lang.hm wrote:
>>
>> I see it fairly frequently when switching between different branches of a
>> project.
>
> So, at least for any normal switch, assuming file 'a' doesn't exist in the
> other branch, you really should have a few different cases:
>
> - you have a dirty file, and git should say something like
>
> 	error: You have local changes to 'file'; cannot switch branches.
>
>   because it refuses to modify the file to match the other branch (which
>   includes removing it) if it doesn't match the index.
>
>   So this case shouldn't leave anything behind.
>
> - You have that extra file, but it's not in the index.
>
>   If it's in your current HEAD, we should still notice it with something
>   like:
>
> 	error: Untracked working tree file 'tree' would be removed by merge.
>
>   because now it's untracked (not in the index), but the switching
>   between branches tries to essentially "apply" the difference between
>   your current HEAD and the new branch, and finds that the difference
>   involves removing a file that git isn't tracking.

one place that I know I've run into it frequently is in an internal project that I did not properly setup .gitignore and did "git add ." and "git commit -a" to. that projects repository contains the compiled binaries and I frequently get these errors when switching trees.

that sounds like the first case.

I've seen discussion of a new sequencer functionality, would it allow me to define a .gitignore file and re-create the repository as if that file had existed all along?

David Lang
Show 16 quoted lines
> See?
>
> HOWEVER.
>
> If you're used to doing "git checkout -f" or "git reset --hard", both of
> those checks are just ignored. After all, you asked for a forced switch.
>
> And at least in the second case, what I think happens is that git won't
> remove the file it doesn't know about, so you'll have a "turd" left
> around.
>
> So yes, you can certainly get these kinds of left-overs, but they really
> should be only happening if you "force" something. Do you do that often?
>
> 			Linus
>
Andrew Morton· Oct 15, 2008, 19:49 UTC · re: Linus Torvalds · lore

On Wed, 15 Oct 2008 12:31:40 -0700 (PDT) Linus Torvalds <torvalds@linux-foundation.org> wrote:

Show 19 quoted lines
> 
> 
> On Wed, 15 Oct 2008, david@lang.hm wrote:
> > 
> > the fact that git will happily leave modified things in the working directory
> > appears to be very helpful for some developers, but it's also a big land mine
> > for others.
> 
> Hmm. It doesn't actually do that normally. If you switch between trees, 
> git will (or _should_) remove the old files that it knows about. If you 
> get a lot of left-over turds, there's something wrong.
> 
> It could be a git bug, of course. That said, especially considering the 
> source of this, I wonder if it's just that Andrew ends up using all those 
> non-git scripts on top of a git tree, and then that can result in git 
> *not* knowing about a certain file, and then when switching between trees 
> (with either git checkout or with git reset), the data that was created 
> with non-git tools gets left behind and now git will be afraid to 
> overwrite it.

I treat my git directory as a read-only thing. I only ever modify it with git commands.

Show 7 quoted lines
> So yes, there are ways to force it (both "git checkout -f"  and "git reset 
> --hard" having already been mentioned), but the need for that - especially 
> if it's common - is a bit discouraging.
> 
> Especially since it's still possible that it's some particular mode of git 
> usage that leaves those things around. Andrew - have you any clue what it 
> is that triggers the behavior?
Sorry, no, I haven't seen a pattern.
Show 5 quoted lines
> (By the filename, I realize it's a file that doesn't exist in one tree or 
> the other, and which doesn't get removed at some point. But have you had 
> merge failures, for example? Is it perhaps a file that was created during 
> a non-clean merge, and then got left behind due to the merge being 
> aborted? It would be interesting to know what led up to this..)

That's certainly a possibility - I get a lot of merge failures. A real lot. And then quite a bit of rebasing goes on, especially in linux-next. And then there's all the other stuff which Stephen does on top of the underlying trees to get something releasable happening.

Linus Torvalds· Oct 15, 2008, 20:08 UTC · re: Andrew Morton · lore
On Wed, 15 Oct 2008, Andrew Morton wrote:
> 
> I treat my git directory as a read-only thing.  I only ever modify it
> with git commands.
Ok. 
Show 10 quoted lines
> > (By the filename, I realize it's a file that doesn't exist in one tree or 
> > the other, and which doesn't get removed at some point. But have you had 
> > merge failures, for example? Is it perhaps a file that was created during 
> > a non-clean merge, and then got left behind due to the merge being 
> > aborted? It would be interesting to know what led up to this..)
> 
> That's certainly a possibility - I get a lot of merge failures.  A real
> lot.  And then quite a bit of rebasing goes on, especially in
> linux-next.  And then there's all the other stuff which Stephen does on
> top of the underlying trees to get something releasable happening.
Is "git checkout -f" part of the scripting? Or "git reset --hard"?
So what I could imagine is happening is:
 - you have a lot of automated merging
 - a merge goes south with a data conflict, and since it's all automated, 
   you just want to throw it away. So you do "git reset --force" to do 
   that.
 - but what "git reset --hard" means is to basically ignore all error 
   cases, including any unmerged entries that it just basically ignores.
 - so it did set the tree back, but the whole point of "--hard" is that it 
   ignores error cases, and doesn't really touch them.

Now, I don't think we ever really deeply thought about what the error cases should do when they are ignored. Should the file that is in some state we don't like be removed? Or should we just ignore the error and return without removing the file? Generally git tries to avoid touching things it doesn't understand, but I do think this may explain some pain for you, and it may not be the right thing in this case.

(And when I say "this case", I don't really know whether you use "git checkout -f" or "git reset --hard" or something else, so I'm not even going to say I'm sure exactly _which_ case "this case" actually us :)

Of course, the cheesy way for you to fix this may be to just add a
	git clean -dqfx

to directly after whatever point where you decide to reset and revert to an earlier stage. That just says "force remove all files I don't know about, including any I might ignore". IOW, "git reset --hard" will guarantee that all _tracked_ files are reset, but if you worry about some other crud that could have happened due to a failed merge, that additional "git clean" may be called for.

Of course, it's going to read the whole directory tree and that's not really cheap, but especially if you only do this for error cases, it's probably not going to be any worse. And I'm assuming you're not compiling in that tree, so you probably don't want to save object files (you can remove the "x" part, but then you could still at least in theory get a filename clash with something that is ignored and thus didn't get cleaned up).

			Linus
Andrew Morton· Oct 15, 2008, 20:23 UTC · re: Linus Torvalds · lore

On Wed, 15 Oct 2008 13:08:36 -0700 (PDT) Linus Torvalds <torvalds@linux-foundation.org> wrote:

Show 21 quoted lines
> 
> 
> On Wed, 15 Oct 2008, Andrew Morton wrote:
> > 
> > I treat my git directory as a read-only thing.  I only ever modify it
> > with git commands.
> 
> Ok. 
> 
> > > (By the filename, I realize it's a file that doesn't exist in one tree or 
> > > the other, and which doesn't get removed at some point. But have you had 
> > > merge failures, for example? Is it perhaps a file that was created during 
> > > a non-clean merge, and then got left behind due to the merge being 
> > > aborted? It would be interesting to know what led up to this..)
> > 
> > That's certainly a possibility - I get a lot of merge failures.  A real
> > lot.  And then quite a bit of rebasing goes on, especially in
> > linux-next.  And then there's all the other stuff which Stephen does on
> > top of the underlying trees to get something releasable happening.
> 
> Is "git checkout -f" part of the scripting? Or "git reset --hard"?

well, this script has been hacked on so many times I'm not sure what it does any more.

Presently the main generate-a-diff function is
doit()
{
	tree=$1
	upstream=$2
	cd $GIT_TREE
	git checkout "$upstream"
	git reset --hard "$upstream"
	git fetch "$tree" || exit 1
	git merge --no-commit 'test merge' HEAD FETCH_HEAD > /dev/null
	{
		git_header "$tree"
		git log --no-merges ORIG_HEAD..FETCH_HEAD
		git diff --patch-with-stat ORIG_HEAD
	} >$PULL/$tree.patch
	{
		echo DESC
		echo $tree.patch
		echo EDESC
		git_header "$tree"
		git log --no-merges ORIG_HEAD..FETCH_HEAD
	} >$PULL/$tree.txt
	git reset --hard "$upstream"
}
usually invoked as

doit origin v2.6.27 doit origin linux-next

etc.
the above seemed fairly busted, so I'm now using
        git checkout -f "$upstream"
        git reset --hard "$upstream"
        git fetch "$tree" || exit 1

which seems a bit more sensible. Perhaps I should do the reset before the checkout, dunno.

That function has been through sooooooo many revisions and each time some scenario get fixed (more like "improved"), some other scenario gets busted (more like "worsened"). The above sorta mostly works, although it presently generates thirty-odd rejects against git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-tip.git#auto-latest, which is way above my fix-it-manually threshold. linux-next is still dead because it's taking Stephen over two days to fix the mess he's been fed so I'm madly rebasing everything on mainline over here.

Show 7 quoted lines
> So what I could imagine is happening is:
> 
>  - you have a lot of automated merging
> 
>  - a merge goes south with a data conflict, and since it's all automated, 
>    you just want to throw it away. So you do "git reset --force" to do 
>    that.
didn't know about --force.
Show 12 quoted lines
>  - but what "git reset --hard" means is to basically ignore all error 
>    cases, including any unmerged entries that it just basically ignores.
> 
>  - so it did set the tree back, but the whole point of "--hard" is that it 
>    ignores error cases, and doesn't really touch them.
> 
> Now, I don't think we ever really deeply thought about what the error 
> cases should do when they are ignored. Should the file that is in some 
> state we don't like be removed? Or should we just ignore the error and 
> return without removing the file? Generally git tries to avoid touching 
> things it doesn't understand, but I do think this may explain some pain 
> for you, and it may not be the right thing in this case.

Yeah, there's no easy solution here, and I suspect the real solution is "read programmer's mind". Providing a reliable override (like -f) is a sensible solution.

Show 14 quoted lines
> (And when I say "this case", I don't really know whether you use "git 
> checkout -f" or "git reset --hard" or something else, so I'm not even 
> going to say I'm sure exactly _which_ case "this case" actually us :)
> 
> Of course, the cheesy way for you to fix this may be to just add a
> 
> 	git clean -dqfx
> 
> to directly after whatever point where you decide to reset and revert to 
> an earlier stage. That just says "force remove all files I don't know 
> about, including any I might ignore". IOW, "git reset --hard" will 
> guarantee that all _tracked_ files are reset, but if you worry about some 
> other crud that could have happened due to a failed merge, that additional 
> "git clean" may be called for.
OK, I'll try git clean -dqfx if it blows up again.
Show 7 quoted lines
> Of course, it's going to read the whole directory tree and that's not 
> really cheap, but especially if you only do this for error cases, it's 
> probably not going to be any worse. And I'm assuming you're not compiling 
> in that tree, so you probably don't want to save object files (you can 
> remove the "x" part, but then you could still at least in theory get a 
> filename clash with something that is ignored and thus didn't get cleaned 
> up).
Paolo Ciarrocchi· Oct 16, 2008, 08:42 UTC · re: Andrew Morton · lore

On Wed, Oct 15, 2008 at 10:23 PM, Andrew Morton <akpm@linux-foundation.org> wrote:

> On Wed, 15 Oct 2008 13:08:36 -0700 (PDT)
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
>> On Wed, 15 Oct 2008, Andrew Morton wrote:
[...]
Show 7 quoted lines
>> Is "git checkout -f" part of the scripting? Or "git reset --hard"?
>
> well, this script has been hacked on so many times I'm not sure what
> it does any more.
>
> Presently the main generate-a-diff function is
>

Hi Andrew, I was wondering whether you could share the scripts you built on top of git, you might get some useful suggestions from this list and they could be inspiration for further improvement in GIT (it just happened with this thread ;-)

Thanks.
Ciao,
-- 
Paolo
http://paolo.ciarrocchi.googlepages.com/
Andrew Morton· Oct 16, 2008, 09:32 UTC · re: Paolo Ciarrocchi · lore
On Thu, 16 Oct 2008 10:42:13 +0200 "Paolo Ciarrocchi" <paolo.ciarrocchi@gmail.com> wrote:
Show 19 quoted lines
> On Wed, Oct 15, 2008 at 10:23 PM, Andrew Morton
> <akpm@linux-foundation.org> wrote:
> > On Wed, 15 Oct 2008 13:08:36 -0700 (PDT)
> > Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >> On Wed, 15 Oct 2008, Andrew Morton wrote:
> [...]
> >> Is "git checkout -f" part of the scripting? Or "git reset --hard"?
> >
> > well, this script has been hacked on so many times I'm not sure what
> > it does any more.
> >
> > Presently the main generate-a-diff function is
> >
> 
> Hi Andrew,
> I was wondering whether you could share the scripts you built on top of git,
> you might get some useful suggestions from this list and they could be
> inspiration for further improvement in GIT (it just happened with this
> thread ;-)

oh gee, you don't want to look. It should all be in http://userweb.kernel.org/~akpm/stuff/patch-scripts.tar.gz

But really it's just the one script, pull-git-patches, below. That thing's been hacked around so much that I daren't breathe on it.

Fortunately as long as Stephen Rothwell is producing linux-next I don't have much need for it any more.

#!/bin/sh

GIT_TREE=/usr/src/git26 PULL=/usr/src/pull

git_header()
{
	tree="$1"
	echo GIT $(cat .git/refs/heads/$tree) $(cat .git/branches/$tree)
	echo
}
# maybe use git clean -dqfx
doit()
{
	tree=$1
	upstream=$2
	cd $GIT_TREE
	git checkout -f "$upstream"
	git reset --hard "$upstream"
	git fetch "$tree" || exit 1
	git merge --no-commit 'test merge' HEAD FETCH_HEAD > /dev/null
	{
		git_header "$tree"
		git log --no-merges ORIG_HEAD..FETCH_HEAD
		git diff --patch-with-stat ORIG_HEAD
	} >$PULL/$tree.patch
	{
		echo DESC
		echo $tree.patch
		echo EDESC
		git_header "$tree"
		git log --no-merges ORIG_HEAD..FETCH_HEAD
	} >$PULL/$tree.txt
	git reset --hard "$upstream"
}
do_one()
{
	tree=$1
	upstream=$2
	if [ ! -e $PULL/$tree.patch ]
	then
		echo "*** doing $tree, based on $upstream"
		git branch -D $tree
		doit $tree $upstream
	else
		echo skipping $tree
	fi
}
mkdir -p $PULL
if [ $1"x" = "-x" ]
then
	exit
fi

cd $GIT_TREE git checkout -f master

cd /usr/src
if [ $# == 0 ]
then
	trees=/usr/src/git-trees
else
	trees="$1"
fi
if [ $# == 2 ]
then
	do_one $1 $2
else
	while read x
	do
		if echo $x | grep '^#.*' > /dev/null
		then
			true
		else
			do_one $x
		fi
	done < $trees
fi
Linus Torvalds· Oct 15, 2008, 20:23 UTC · re: Linus Torvalds · lore
On Wed, 15 Oct 2008, Linus Torvalds wrote:
> 
>  - a merge goes south with a data conflict, and since it's all automated, 
>    you just want to throw it away.

Actually, with your filename, I suspect the conflict would be not a real file content, but more of a "delete" conflicting with a modification to that file. IOW, I'm guessing that the thing you hit with arch/x86/kernel/apic.c was that some branch you pulled:

 - created that file
 - deleted arch/x86/kernel/apic_[32|64].c
 - the old file got marked as a rename source for the new apic.c and 
   there was a data conflict when trying to apply the changes.

as a result, your working tree would have that "apic.c" file in it, but with conflict markers, and marked as unmerged.

When you then do "git reset --hard", it will just ignore unmerged entries, and since the original tree (and the destination tree) match, and neither of them contain apic.c either, git will totally ignore that file and not even try to remove it (since it wasn't there originally).

> So you do "git reset --force" to do that.

It's "--hard", not "--force". Yeah, the git reset flags are insane. As is the default action, for that matter. It's one of the earliest interfaces, and it's stupid and reflects git internal implementations rather than what we ended up learning about using git later. Oh, well.

But 'git checkout -f' (which is nicer from a user interface standpoint) has the exact same logic and I think shares all the implementation. I think they both end up just calling "git read-tree --reset -u".

It's quite possible that we should remove unmerged entries. Except that's not how our internal 'read_cache_unmerged()' function works. It really just ignores them, and throws them on the floor. We _could_ try to just turn them into a (since) stage-0 entry.

Junio?
			Linus
Andrew Morton· Oct 15, 2008, 20:30 UTC · re: Linus Torvalds · lore

On Wed, 15 Oct 2008 13:23:50 -0700 (PDT) Linus Torvalds <torvalds@linux-foundation.org> wrote:

Show 21 quoted lines
> 
> 
> On Wed, 15 Oct 2008, Linus Torvalds wrote:
> > 
> >  - a merge goes south with a data conflict, and since it's all automated, 
> >    you just want to throw it away.
> 
> Actually, with your filename, I suspect the conflict would be not a real 
> file content, but more of a "delete" conflicting with a modification to 
> that file. IOW, I'm guessing that the thing you hit with 
> arch/x86/kernel/apic.c was that some branch you pulled:
> 
>  - created that file
> 
>  - deleted arch/x86/kernel/apic_[32|64].c
> 
>  - the old file got marked as a rename source for the new apic.c and 
>    there was a data conflict when trying to apply the changes.
> 
> as a result, your working tree would have that "apic.c" file in it, but 
> with conflict markers, and marked as unmerged.

That sounds likely. I suspect things were especially bad today because I accidentally pulled four-week-old linux-next, which had over 500 rejects in it.

Junio C Hamano· Oct 15, 2008, 22:06 UTC · re: Linus Torvalds · lore
Linus Torvalds <torvalds@linux-foundation.org> writes:
Show 8 quoted lines
> On Wed, 15 Oct 2008, Linus Torvalds wrote:
>> 
> It's quite possible that we should remove unmerged entries. Except that's 
> not how our internal 'read_cache_unmerged()' function works. It really 
> just ignores them, and throws them on the floor. We _could_ try to just 
> turn them into a (since) stage-0 entry.
>
> Junio?

I'd agree that dropping unmerged entries to stage-0 when we can would make sense. An conflicted existing path would get an stage-0 entry in the index, which is compared with the switched-to HEAD (which could be the same as the current one when "git reset --hard" is run without a rev), we notice that they are different and the index entry and the work tree path is overwritten by the version from the switched-to HEAD. For a new path that a failed merge tried to bring in, we notice that the switched-to HEAD does not have that path and happily remove it from the index and from the work tree. All will go a lot smoother than the current code.

I am not sure what should happen when we can't drop the unmerged entry down to stage-0 due to D/F conflicts, though. IIRC, read-tree proper would not touch the work tree in such a case, but merge-recursive creates our and their versions with funny suffixes, which will not be known to the index and will be left in the working tree.

Junio C Hamano· Oct 15, 2008, 23:00 UTC · re: Junio C Hamano · lore

[PATCH] reset --hard/read-tree --reset -u: remove unmerged new paths

When aborting a failed merge that has brought in a new path using "git reset --hard" or "git read-tree --reset -u", we used to first forget about the new path (via read_cache_unmerged) and then matched the working tree to what is recorded in the index, thus ending up leaving the new path in the work tree.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Junio C Hamano <gitster@pobox.com> writes:
 > Linus Torvalds <torvalds@linux-foundation.org> writes:
 >
 >> On Wed, 15 Oct 2008, Linus Torvalds wrote:
 >>> 
 >> It's quite possible that we should remove unmerged entries. Except that's 
 >> not how our internal 'read_cache_unmerged()' function works. It really 
 >> just ignores them, and throws them on the floor. We _could_ try to just 
 >> turn them into a (since) stage-0 entry.
 >>
 >> Junio?
 >
 > I am not sure what should happen when we can't drop the unmerged entry
 > down to stage-0 due to D/F conflicts, though.  IIRC, read-tree proper
 > would not touch the work tree in such a case, but merge-recursive creates
 > our and their versions with funny suffixes, which will not be known to the
 > index and will be left in the working tree.
 I am still unsure what we should do when we hit D/F conflicts; this one
 simply replaces but it may be safer to drop ADD_CACHE_OK_TO_REPLACE from
 the options to trigger an error in such a case.  I dunno.
 read-cache.c               |   32 +++++++++++++++++++-------------
 t/t1005-read-tree-reset.sh |   30 ++++++++++++++++++++++++++++++
 2 files changed, 49 insertions(+), 13 deletions(-)
diff --git a/read-cache.c b/read-cache.c
index c229fd4..efbab6a 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1489,25 +1489,31 @@ int write_index(const struct index_state *istate, int newfd)
 int read_index_unmerged(struct index_state *istate)
 {
 	int i;
-	struct cache_entry **dst;
-	struct cache_entry *last = NULL;
+	int unmerged = 0;
 
 	read_index(istate);
-	dst = istate->cache;
 	for (i = 0; i < istate->cache_nr; i++) {
 		struct cache_entry *ce = istate->cache[i];
-		if (ce_stage(ce)) {
-			remove_name_hash(ce);
-			if (last && !strcmp(ce->name, last->name))
-				continue;
-			cache_tree_invalidate_path(istate->cache_tree, ce->name);
-			last = ce;
+		struct cache_entry *new_ce;
+		int size, len, option;
+
+		if (!ce_stage(ce))
 			continue;
-		}
-		*dst++ = ce;
+		unmerged = 1;
+		len = strlen(ce->name);
+		size = cache_entry_size(len);
+		new_ce = xcalloc(1, size);
+		hashcpy(new_ce->sha1, ce->sha1);
+		memcpy(new_ce->name, ce->name, len);
+		new_ce->ce_flags = create_ce_flags(len, 0);
+		new_ce->ce_mode = ce->ce_mode;
+		option = ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE;
+		if (add_index_entry(istate, new_ce, option))
+			return error("%s: cannot drop to stage #0",
+				     ce->name);
+		i = index_name_pos(istate, new_ce->name, len);
 	}
-	istate->cache_nr = dst - istate->cache;
-	return !!last;
+	return unmerged;
 }
 
 struct update_callback_data
diff --git a/t/t1005-read-tree-reset.sh b/t/t1005-read-tree-reset.sh
index b0d31f5..0cd519c 100755
--- a/t/t1005-read-tree-reset.sh
+++ b/t/t1005-read-tree-reset.sh
@@ -27,4 +27,34 @@ test_expect_success 'reset should work' '
   test_cmp expect actual
 '
 
+test_expect_success 'reset should remove remnants from a failed merge' '
+  git read-tree --reset -u HEAD &&
+  git ls-files -s >expect &&
+  sha1=$(git rev-parse :new) &&
+  (
+	echo "100644 $sha1 1	old"
+	echo "100644 $sha1 3	old"
+  ) | git update-index --index-info &&
+  >old &&
+  git ls-files -s &&
+  git read-tree --reset -u HEAD &&
+  git ls-files -s >actual &&
+  ! test -f old
+'
+
+test_expect_success 'Porcelain reset should remove remnants too' '
+  git read-tree --reset -u HEAD &&
+  git ls-files -s >expect &&
+  sha1=$(git rev-parse :new) &&
+  (
+	echo "100644 $sha1 1	old"
+	echo "100644 $sha1 3	old"
+  ) | git update-index --index-info &&
+  >old &&
+  git ls-files -s &&
+  git reset --hard &&
+  git ls-files -s >actual &&
+  ! test -f old
+'
+
 test_done
-- 
1.6.0.2.717.gc6f0a
Linus Torvalds· Oct 15, 2008, 23:16 UTC · re: Junio C Hamano · lore

Re: [PATCH] reset --hard/read-tree --reset -u: remove unmerged new paths

On Wed, 15 Oct 2008, Junio C Hamano wrote:
Show 6 quoted lines
>
> When aborting a failed merge that has brought in a new path using "git
> reset --hard" or "git read-tree --reset -u", we used to first forget about
> the new path (via read_cache_unmerged) and then matched the working tree
> to what is recorded in the index, thus ending up leaving the new path in
> the work tree.

Looks good to me. And from my tests, I think "git checkout -f" didn't have this problem at all, because it ends up using not got read-tree, but doing its own "reset_tree()" that uses unpack_trees().

I do wonder if "git reset" should perhaps be written in those terms, instead of just being a wrapper around git read-tree. But the patch looks fine.

		Linus
Junio C Hamano· Oct 16, 2008, 06:27 UTC · re: Linus Torvalds · lore

Re: [PATCH] reset --hard/read-tree --reset -u: remove unmerged new paths

Linus Torvalds <torvalds@linux-foundation.org> writes:
Show 15 quoted lines
> On Wed, 15 Oct 2008, Junio C Hamano wrote:
>>
>> When aborting a failed merge that has brought in a new path using "git
>> reset --hard" or "git read-tree --reset -u", we used to first forget about
>> the new path (via read_cache_unmerged) and then matched the working tree
>> to what is recorded in the index, thus ending up leaving the new path in
>> the work tree.
>
> Looks good to me. And from my tests, I think "git checkout -f" didn't have 
> this problem at all, because it ends up using not got read-tree, but doing 
> its own "reset_tree()" that uses unpack_trees().
>
> I do wonder if "git reset" should perhaps be written in those terms, 
> instead of just being a wrapper around git read-tree. But the patch looks 
> fine.

Let's do this for 'maint' and I'll let others think about possible improvements, then ;-).

Ingo Molnar· Oct 16, 2008, 07:20 UTC · re: Junio C Hamano · lore

Re: [PATCH] reset --hard/read-tree --reset -u: remove unmerged new paths

* Junio C Hamano <gitster@pobox.com> wrote:
Show 5 quoted lines
> When aborting a failed merge that has brought in a new path using "git 
> reset --hard" or "git read-tree --reset -u", we used to first forget 
> about the new path (via read_cache_unmerged) and then matched the 
> working tree to what is recorded in the index, thus ending up leaving 
> the new path in the work tree.

i've met this problem in various variants in the past few months, and i always assumed that it's "as designed" - as Git's policy is to never lose information unless forced to do so. (which i find very nice in general, and which saved modification from getting lost a couple of times in the past)

the situations where i end up with a messed up working tree [using git-c427559 right now]:

 - doing a conflicted Octopus merge will leave the tree in some weird 
   half-merged state, with lots of untracked working tree files that not 
   even a hard reset will recover from. The routine thing i do to clean 
   up is:
      git reset --hard HEAD
      git checkout HEAD .
      git ls-files --others | xargs rm              # DANGEROUS
   doing git checkout -f alone is not enough, as there might be various 
   dangling files left around.
 - git auto-gc thinking that it needs to do another pass in the middle 
   of a random git operation, but i dont have 10 minutes to wait so i 
   decide to Ctrl-C it.
 - doing the wrong "git checkout" and then Ctlr-C-ing it can leave the
   working tree in limbo as well, needing fixups. If i'm stuck between
   two branches that rename/remove files it might need the full fixup
   sequence above.
 - if a testbox has a corrupted system clock, its git repo and the 
   kernel build can get confused. This is to be expected i think - but
   the full sequence above will recover the corrupted tree. Not much Git
   can do about this i guess.

Does your fix mean that all i have to do in the future is a hard reset back to HEAD, and that dangling files are not supposed to stay around?

	Ingo
Junio C Hamano· Oct 16, 2008, 14:49 UTC · re: Ingo Molnar · lore

Re: [PATCH] reset --hard/read-tree --reset -u: remove unmerged new paths

Ingo Molnar <mingo@elte.hu> writes:
> Does your fix mean that all i have to do in the future is a hard reset 
> back to HEAD, and that dangling files are not supposed to stay around?

As long as the index *somehow* knows about these new files, they are removed.

The situation is:
 (0) you start from a HEAD that does not have path xyzzy;
 (1) you attempt to merge a rev that has path xyzzy;
 (2) the merge conflicts, leaving higher staged index entries for the
     path.
 (3) you decide not to conclude the merge by saying "reset --hard".

The old logic for "reset" was to remove paths that exist in the index at stage #0 (i.e. cleanly merged) and not in HEAD. The patch changes the rule to remove paths that exist in the index at any stage (i.e. including the ones that have conflicted and not resolved yet) and not in HEAD.

← back to recent threads