Proposal: tell git a file has been renamed

Yes, I know Linus specifically doesn't store file rename info in Git. The trouble is, every now and then, I'll come across a situation where Git doesn't successfully detect that I've renamed a file because I'm doing something like renaming a class at the same time. So I'll have a file OldClassNameTests.cs and a NewClassNameTests.cs but a bunch of lines in that file have also changed from OldClassName.DoThing() to NewClassName.DoThing(). I can clearly see that this is a rename, but Git sees enough changed content that it doesn't realize it, and puts it in as a delete/add, losing the content history.

The standard answer for this is to rename the file in one commit, then make the changes. That's fine if you know ahead of time you'll want to do this. However it's a total PITA if you have a bunch of changes and you realize that a rename has caused this problem. You now have to back out your changes to the renamed file, add the rename, commit it, then re-apply the changes.

Could a command be added to git that means you tell Git that counts as a file rename? Git would add a marker to the staging area that the file has been renamed, and upon commit, would first generate an additional commit for each rename before generating the main commit, ensuring the rename operation counts as an actual rename, and the content's history is maintained.

On Saturday, April 22, 2023 2:02 PM, Jeremy Morton wrote:

Would git mv work in your situation? You can stage changes to the original file, then use git mv. Or use git mv first. The rename shows as staged in any event. --Randall

I read that git mv is basically the equivalent to deleting the old file, creating the new file, and adding the changes. Isn't it? If so it's gonna have the same problem as I have now.

No, history is preserved in the rename.

https://stackoverflow.com/a/1094392/178757

says:

git mv oldname newname

is just shorthand for:

mv oldname newname git add newname git rm oldname

On Saturday, April 22, 2023 3:54 PM, Jeremy Morton wrote:

The above stackoverflow topic is from 2009. A lot has changed since then. My test follows. Please note the status and git log contents indicating the rename.

$ mkdir test2
$ cd test2
$ git init
$ echo "Initial" > file1
$ git add file1
$ commit -m "Commit 1"
$ git mv file1 file2
$ git status
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        renamed:    file1 -> file2
$ git commit -m "Rename"
$ git log --patch
commit 014068fcedaf361f45c356046cf513b79537f53f (HEAD -> master)
Author: Randall S. Becker <rsbecker@nexbridge.com>
Date:   Sat Apr 22 18:02:07 2023 -0400

    Rename

On Sat, Apr 22, 2023 at 9:57 PM Jeremy Morton <admin@game-point.net> wrote:

That's what I though but it's more like:
  git show :0:oldname > newname
  git add newname
  git rm --cached oldname
  mv oldname newname

Basically, a move but also preserving staged/unstaged contents. But yes. How git handles renames can sometimes be a bit of a PITA.

On Sun, Apr 23, 2023, 12:16 AM <rsbecker@nexbridge.com> wrote:

I think your point is better illustrated
  git add file1
  commit -m "Commit 1"
  echo 'Changed' > file1
  git mv file1 file2
  git status
Which yields:
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        renamed:    file1 -> file2

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   file2

Or perhaps even:
  echo Initial > file1
  seq 1 10 >> file1 # We need a larger file to detect rename with change
  git add file1
  git commit -m 'Commit 1'
  seq -i 's/Initial/Changed/' file1
  git add file1 # Stage changes
  seq 11 20 >> file1 # Add some unstaged changes
  git mv file1 file2

This will result in a staged rename, as well as the staged change 'Initial' -> 'Changed' with a bunch of additional unstaged lines of numbers 11 to 20. NB that this doesn't play nice unless the contents of file1 in HEAD and index are similar enough (at least 50%?)

On Sat, Apr 22, 2023, at 20:01, Jeremy Morton wrote:

I don’t see the (conceptual) problem with a modification of this as a history rewriting tool:

• Given a series of commits:
• Tool X modifies all the commits so that the default similarity index
  for tools like git-log(1) is triggered on intended file renames
• The user will be probably be prompted with a list of initial potential
  renames and then will
  • Keep the intended renames
  • Remove the not-intended renames
  • Add the additional renames

But this looks like something that a third-party tool could implement.

(By the way, apologies for the stuttering "gitt" typos that seem to keep creeping into my messages here. I'm not sure what's causing them -- I make a lot of typos but this is abnormal!)

On Sat, Apr 22, 2023 at 12:22 PM Jeremy Morton <admin@game-point.net> wrote:

I have long wondered if there was a way to improve this experience myself.

(I also note that most of the followup messages up to this point have missed the point. It's true that you can run `git mv` and `git commit`, but you've already said that this becomes particularly painful when you realize that it's appropriate *after the fact*, when you've already made intermediate commits and/or staged changes or whatever.)

The index *currently* has no room to store anything like this: it is, in effect, just the proposed next commit, stored as a flattened tree. There are, however, extra marker records that can be added. So:

If `git mv` (or a new command) had a flag to say "make a special index entry so that the next `git commit` does a double commit", we could in fact make this work. Alternatively, we could have a command -- similar to `git commit --only` in effect -- that uses the current (HEAD) commit to construct a renames-only commit, in which 100%-identical-file matching would (in general) find the desired renames -- and make it, perhaps also co-ordinating with `git mv` of existing files in the index. (I'd also like to have `git mv --after`, in the same vein as `hg mv --after`; I long ago wrote a cheesy script to achieve this, but it would be nice to have a proper command.)

On top of this, it might be nice to have a standardized commit message and/or other marker (in the commit header?) for a "rename- only" commit, which this kind of extra-rename-commit operation would use. Then `git log` and `git blame` and other commands could easily detect such commits and default to an automatic `--follow` style following, and `git log` might be allowed to omit the *display* of such a commit by default, by showing all the renames as renames in the subsequent commit (though this would presumably require an internal verification step to check for spoofed renames that are not in fact rename-only operations).

In any case, this *idea* is easy, like many ideas. It really comes down to implementation. If someone thinks this is a great idea, someone (perhaps me) should work on *implementing* it. :-)

Chris

On 24/04/2023 02:43, Chris Torek wrote:>> [renames are a problem]

I'm not sure what the utility of the --only thing would be - to detect renames that didn't have changed content so that all renames could be done in one pre-commit?

Huh, I just read the docs on that... does that mean hg already has this functionality of being able to store a "this was renamed" marker in its index?

I'm surprised --follow isn't the default, actually... isn't the whole point of detecting renames to allow content history to be tracked back through renames?

Another one that jumped to mind for me is bisect. As rename-only commits are liable to create broken builds, it should skip over them to the 'content' commit.

If you do please feel free to CC me in an email about it, it'd be good to know if this became available!

On Mon, Apr 24, 2023 at 3:15 AM Jeremy Morton <admin@game-point.net> wrote:

I mentioned `git commit --only` here just to point out that `git commit` already has the ability to make a commit without using the current index as the new commit's source. A "just do renames" commit operation (in spite of other changes already made in the current index) would need similar functionality.

Exactly how this might work, I haven't defined.

Mercurial does not have an index in the first place. The internal structure of the Mercurial database is an append-only series of changelongs, with files stored as deltas from the previous version of that file (with some exceptions). Files are listed in a manifest, and renaming a file preserves the file's identity despite the change of name.

(This internal format is very different from Git's. Git is a content-addressable file system, rather than an append-only changelog.)

Chris

On Mon, Apr 24, 2023 at 12:21 PM Jeremy Morton <admin@game-point.net> wrote:

You can set log to --follow by default
  If true, git log will act as if the --follow option was used when a
single <path> is given.
  This has the same limitations as --follow, i.e. it cannot be used to
follow multiple files
  and does not work well on non-linear history.
  https://git-scm.com/docs/git-log#Documentation/git-log.txt-logfollow

The reason it's not default is probably those limitations.

I always find this to be the main dilemma. I try to make commits as discrete changes but it's not always possible with renames. Sometimes, renaming a file changes it so much that the rename detection doesn't work by default. There are also other problems that arise when reordering commits and changes in a feature branch. I've found that the safest thing is to split renames out into discrete commits and only do 100% renames.

On 24/04/2023 11:49, Erik Cervin Edin wrote:

There's no getting away from the fact that this adds a lot of (IMHO unnecessary) work if you've already done a rename that git can't detect and have both that and a bunch of other changes sitting in the index. What feels like it would be a natural resolution in these cases, though, is a "no, this remove/add is actually a rename" command.

On Mon, Apr 24, 2023 at 1:17 PM Jeremy Morton <admin@game-point.net> wrote:

It can definitely be both arduous and non-obvious how to deal with this.

The problem is that such a command cannot exist atm. because renames don't exist, they are only interpreted. So the only way to achieve this is to revert enough of the contents staged to the index such that the rename is detected. The only way to do that in a foolproof manner is reverting all the staged changes except the path so that the moved file in the index is identical to the old file in HEAD.

If I understand you correctly, your point here is that it's non-trivial to go from a state where a file has been renamed, with enough changes staged into the index that the rename hasn't been detected, to a state where the rename is recorded in the index?

The most straightforward method I can think of to restore the staged
changes, do the rename without changes, commit and then go about the
rest
eg:
  git restore --staged file2
  mv file2 file1
  git mv file1 file2
  git commit -m rename

Yep, and that suggestion is rather arduous compared to something like:

git markasrenamed file2 file1 git commit

Erik Cervin Edin wrote:

I agree recording renames explicitely might be a good addition to git, but the real question is how are they going to be stored in the object storage.

My guess is that it can be added in the commit object after "committer", just add a "renames" field with all the renames, or one "rename" field per rename. It would be backwards compatible because any field can be added this way.

How to generate these fields is a separate issue: first things first.

On 4/24/2023 12:25 PM, Felipe Contreras wrote:

The other end of the solution space is to try to find ways to make the rename detection logic more accurate. We wouldn't need to store such a rename if the detection gave the correct answer in the first place.

You can already kind of do this by modifying the similarity values when searching for renames: the -M option takes a percentage for how similar files need to be in order for it to be a rename. The default is 50%, meaning that the files have to stay at least 50% the same in order to be considered a rename.

One potential in the case for renames which also must change class names, etc could be a mechanism to say "treat it as a rename if the content is the same after performing this replacement", where you could specify what string is being replaced in the rename somehow.

This could easily be done for reviewing a previous change, but specifically how to *store* this expression so you don't need to remember such values in the future is a bigger challenge.

Jacob Keller wrote:

We can make it more accurate, but it will never be 100%. Sometimes being explicit is the only way.

Cheers.

On 4/24/2023 12:25 PM, Felipe Contreras wrote:

I think such a renames field might work, but I know historically there was a lot of resistance to "baking in" such rename logic because you can't "fix" it later.

I know there are some big threads from early days of git discussing this, with many objections on baking in renames for various reasons.

I think the argument can be made that there is still value in a commit which says "I as the commiter know this is renaming the file, please show it as such and track it as such", and I think the other use cases (code moving from one file to another without a full rename, multiple copies of a file getting deleted, etc) can be handled in other ways or with an escape hatch to disable the commit rename indicator if necessary.

In the more complex cases where code is moved without a full rename, there are tools for handling this such as pickaxe or the proposed magic blame tool that was brought up in the discussions against static renames being coded into the object storage. I think those are interesting but separate problems from "I am renaming this file and some of its contents".

At commit basically is the only way this could be baked into the object storage, since blobs don't store filenames and trees don't store any method of difference individually. The commit is the only place to really do it.

The alternative would be something like Junio suggested where we have a separate rename storage database which is computed ahead of time (kind of like commit graph storage) and which could be overridden manually. This doesn't "bake" anything into the objects, but obviously requires manually keeping up to date and adds some difficulty in sharing such updates across remotes.

Jacob Keller <jacob.e.keller@intel.com> writes:

I think originally the recorded rename mechanism was dreamed up to be a cache. Unlike commit-graph that requires precomputation, it would be populated incrementally when you run "git log -M" in a part of the history that you haven't done so. Sharing it would just be as hard as sharing notes, once we point the rename cache with a ref, I would think.

I cannot take credit for the idea, though.

Another example of where the current implementation isn't working great

  A - B - C

commit A makes changes to foo commit B changes the name of foo to bar commit C symlinks foo => bar

Rebasing the commits and reordering them

  commit B changes the name of foo to bar
  commit C symlinks foo => bar
  commit A makes changes to foo

is going to result in merge conflicts. The best way to achieve this ordering is actually doing multiple rebases.

  commit B changes the name of foo to bar
  commit A makes changes to foo
  commit C symlinks foo => bar

followed by

  commit B changes the name of foo to bar
  commit C symlinks foo => bar
  commit A makes changes to foo

I guess the difficulty that arises from reordering commits like this isn't unique to renames but it arguably results in a more hostile user experience. It requires an unusual amount of discipline and understanding of what is going on to resolve as well as how to avoid it.

Jeremy Morton <admin@game-point.net> writes:

Telling Git is the easy part. How to make it honor what it was told earlier is harder.

I recall that there was an idea about recording similarity index between two blobs precomputed so that the same pair of blobs do not have to be computed over and over again, when you do "git log -M"?

An obvious way to implement this "I know HEAD^:path1 and HEAD:path2 are similar enough that they deserve to be called a rename" can be to invalidate and then contaminate such a similarity cache with end-user supplied data.

Jeremy Morton <admin@game-point.net> writes:

Oh, by the way, this is a pure myth that would unlikely be helpful in the bigger picture.

When you rename and heavily modify the resulting new path because you have to solve something, such a work would likely be done on the same topic branch. One step of it may be a pure rename, and other steps may involve heavily changing the renamed result, or you may update the contents in the original and the do a rename at the end, but either way, when you integrate the end result of the whole topic branch into the master history, what such a merge will see is that the original file has disappeared and a new file with contents not at all similar to the disappeared file has appeared. "pure rename with changes in separate commits" would have no effect when showing such a history with "git log --first-parent -p" for a birds-eye view.

But if you were doing, say, a git blame for a file, you'd be able to see the commits where its lines were actually last modified even through the rename, rather than a bunch of them being stopped at the "added this file (because it wasn't detected as a rename)" commit, no?