file rename causes history to disappear

I moved a bunch of SATA drivers in the Linux kernel from drivers/scsi to drivers/ata.

When I tried to look at the past history of a file using git-whatchanged, post-rename, it only shows the history from HEAD to the point of rename. Everything prior to the rename is lost.

I also tried git-whatchanged on the old path, but that produces an error.

[jgarzik@pretzel libata-dev]$ rpm -q git-core git-core-1.4.1-1.fc5

Repository ("upstream" branch): git://git.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-dev.git

Jeff Garzik <jeff@garzik.org> wrote:

Try "git log -- old/path/...". Path limiting works without "--" only if the path exists.

On Wed, 6 Sep 2006, Jeff Garzik wrote:

For filenames that don't exist right now, you need to clearly separate the revision name from the filename (ie you need to use "--").

There were patches to do "--follow-rename" which I don't think got applied yet, but in the meantime, just do

	git whatchanged -M -- drivers/ata/filename.c drivers/scsi/filename.c

where the "-M" means "show diffs as renames if possible" (which is different from having the history actually _follow_ them), and the "--" is the filename separator to tell git that the nonexistent "drivers/ata/filename.c" file isn't a (currently) nonexistent revision name, it's a (currently) nonexistent _filename_.

		Linus

Linus Torvalds wrote:

Since I'm just interested in the log (ATM), even the lack of "-M" seems to produce useful results. Thanks.

IMO it is highly counter-intuitive that renames are -not- followed. I don't see the point of a "--follow-rename", it should Just Work(tm).

	Jeff

On Wed, 6 Sep 2006, Jeff Garzik wrote:

Sure, if you don't actually want the diff, the "-M" isn't worthwhile.

No, it should not. 

You haven't thought it through, and I excuse you, because even people who should know better (and design SCM's) often haven't thought it through.

There's a huge difference between "pathname" and "inode". And git operates on _pathnames_, not on inodes. So when you give a pathname specifier, that's _exactly_ what it is. It's a pathname specifier, _not_ an "inode" specifier.

And pathnames don't change. They're just names for paths to possibly _find_ a file/inode. They can't be "renamed". The data that is found behind a pathname may be moved to _another_ pathname (and we call that a rename), but that doesn't change the original pathname in any way, shape, or form.

Now, you can say "git shouldn't work with pathnames, it should work with inodes, and use the pathnames to look them up", but you'd be wrong. You'd be wrong for many reasons, so let me explain:

 - pathnames are actually often a hell of a lot more interesting that 
   "inodes". Doing thing by pathname means that you have sane and 
   well-defined semantics for something like

	git log -- drivers/scsi drivers/ata include/linux/ata.h

   even if (for example) some of those files or directories don't 
   necessarily even exist at one particular point in time. Exactly 
   _because_ a pathname is not actually affected by the contents of the 
   repository.

   So taking a filename-based approach is actually more _powerful_. You 
   can emulate the "follow a single file" behaviour on top of it, but you 
   can't sanely go the other way.

 - following inodes/files instead of following pathnames happens to also 
   be fundamentally ambiguous when you split or merge the file contents. 
   What happens? You simply _cannot_ describe that in the form of "files". 
   It's impossible. Really. Yet it's actually fairly common.

   In contrast, if you think of pathnames of _pathnames_ (rather than the 
   contents they point to), that particular sticky wicket simply doesn't 
   exist. It's a non-issue. File contents that get split? Big deal. We 
   don't care. We care about a particular set of pathnames, and if the 
   file content came from (or got split into) that set of pathnames, we 
   show it.

So thinking in terms of pathnames is not only fundamentally more powerful, it also very fundamentally avoids a confusing situation that you cannot avoid with a "inode" based model.

You just need to get used to the fact that the arguments you give to "git log" and friends really have _nothing_ to do with any particular "file" or "content at any particular time". They are immutable path specifiers. When you say

	git log -- drivers/scsi/libata.c

you're asking git "tell me what happened to this _pathname_". Not file. Not content (although if you ask for diffs it will show you the diff, but not for that "file", but simple AS IT PERTAINS TO THAT PATHNAME!)

It may take a bit of getting used to, but once you realize that git talks about immutable pathnames, and once you do get used to it, it's a hell of a powerful thing.

And then we can have "--follow-renames" when we are lazy and we _understand_ that git talks about pathnames, but we want git to show us the data _as_if_ it cared about how the inodes moved around.

			Linus

On Wed, 6 Sep 2006, Linus Torvalds wrote:

Side note: one thing that I wanted to do, but never got around to, is to allow wildcards in the tree-parsing code. It might be too expensive, but it's still occasionally something I'd like to do:

	git log -- 'mm/*.c'

to track every single C file in the VM (even if they don't exist right _now_).

Notice the difference between

	git log mm/*.c

and the above idea - the latter does actually work, but it only tracks the C files that exist right now under mm/. But it should be possible (and is potentially useful) to let the wildcard act over the history, rather than just a single point in time.

Because one additional advantage of thinking in terms of pathnames is exactly the fact that wildcards make sense in a way that they do _not_ make sense if you think of tracking "inodes". Exactly because "pathnames are forever", and a pathname has validity and exists regardless of whether a repository contains a _file_ with that name at any particular point in time.

So right now git does do the wildcard thing, but only for "git ls-files" (and through that, things like "git add", which used to be implemented in terms of ls-files). So you can do

	git add '*.c'

to add all C files (recursively - it's not the shell matcher).

			Linus

Linus Torvalds <torvalds@osdl.org> writes:

I am happy to see we are in agreement. I touched this in the ending note to

	http://article.gmane.org/gmane.comp.version-control.git/26432

The only people who will get burnt by this change are the ones with metacharacters in their pathnames, so it is relative safe change.

I think 'git grep' pathspec code is probably the best to reuse to convert diff-tree family. It knows how to match globs while traversing a tree down without descending into a subtree that would never match, which is what we need for them.

Junio> The only people who will get burnt by this change are the ones Junio> with metacharacters in their pathnames, so it is relative safe Junio> change.

But does that mean you'll provide the equivalent to "fgrep" for "grep", as in a switch that turns this off, or a seperate command?

I can think of times when I might be trying to track a file with a square bracket in the name.

merlyn@stonehenge.com (Randal L. Schwartz) writes:

If your path is "foo.c[1]" then "foo.c[1]" as fnmatch() pattern would not obviously match it, which is sad.

However, we do try to match the path literally before falling back to fnmatch() so in practice I do not think it is so bad.

$ git ls-files -s ;# everybody has "hello world". 100644 3b18e512dba79e4c8300dd08aeb37f8e728b8dad 0 foo.c 100644 3b18e512dba79e4c8300dd08aeb37f8e728b8dad 0 foo/bar[1]/baz/boa.c 100644 3b18e512dba79e4c8300dd08aeb37f8e728b8dad 0 foo/bar[2].c $ git grep hello -- 'foo/bar[1]' foo/bar[1]/baz/boa.c:hello world $ git grep hello -- 'foo/bar[[]*[]]*' foo/bar[1]/baz/boa.c:hello world foo/bar[2].c:hello world $ git grep hello -- 'fo*' foo.c:hello world foo/bar[1]/baz/boa.c:hello world foo/bar[2].c:hello world $ exit

Junio C Hamano, Wed, Sep 06, 2006 21:29:49 +0200:

May be make metacharacters the default behaviour, but provide a command-line option to disable it? It'll be seldom used, but would provide a way to disambiguate input for scripts and make possible (even if a bit harder) to use such filenames.

Linus Torvalds wrote:

So if/when git would have --follow option to git-log and git-diff-*, it would be rather --follow=<filename>, rather than --follow -- <paths>?

git-rev-list could then output hash with current set of <filenames>, which
were given <filename> at the beginning, i.e.
  <hash> -- <filename> [<filename>...]

On Wed, 6 Sep 2006, Jakub Narebski wrote:

That would probably be sensible, yes. Especially since "--follow" is fundamentally different from the "<paths>" thing in that you really should be able to only follow a single file.

(Following multiple files causes huge amounts of pain - it might be possible, but I don't think it's worth it).

I would argue that "--follow" would be incompatible with having other <paths> listed. But maybe there is some sensible rule for what the combination means (show the listed paths _and_ the file we're following?) I dunno.

		Linus

Linus Torvalds wrote:

I'm not that sure. The output could be changed to, for example
  <hash> SP <quoted-filename> [SP <quoted-filename> ...]
although I'm not sure if git can detect that two files were joined into one
(or, in reverse that one file was split into several; this doesn't matter
for following history of a file from top)

But --follow=<filename> with <pathspec> can be useful, e.g. when <pathspec> is a directory (or, perhaps in the future, glob), which would mean "follow the contents indicated in starting hash by <filename>, and stop following when it falls out outside given <pathspec>, in our case given directory".

As pathspecs doesn't change, there is no need to output them.

On Wed, 6 Sep 2006, Jakub Narebski wrote:

Yes, that would indeed make sense. The pathspec ends up being kept as a "limiter", and basically tells you what the "context" for following is allowed to be.

Color me convinced.

		Linus

Linus Torvalds <torvalds@osdl.org> writes:

Likewise.

Jakub Narebski wrote:

The "<hash> -- <filename> [<filename>...]" was to allow the followed 
<filename> to be pathspec for other command, for example 
   git-diff-tree --stdin
(if git-diff-tree accepts pathspec limiting on stdin, and not only
revisions, or pairs of revisions; according to 1.4.2 documentation
--stdin is for reading either one  <commit>  or a pair of <tree-ish>
separated with a single space from its input -- no pathspecs).