threads / discuss / 57271

A puzzle: reset --hard and hard links

Subject: A puzzle: reset --hard and hard links

## tl;dr

18 messages between Jan 19, 2022 and Jan 26, 2022.

replies: 17people: 6as markdown or json

Michael Herrmann· Jan 19, 2022, 20:37 UTC · lore
Hi all,

It seems `git reset --hard` recreates files that have external hard links. Is this intended?

The example below highlights the behavior. I have an unchanged Git repository. When I create an (outside) hard link to a file in the repository, then `git reset --hard` re-creates the file with a new modification time and inode. This occurs on Debian 11 with Git 2.30.2 but not on Debian 10 with Git 2.20.1

> git init
Initialized empty Git repository in .../git-test/.git/
> echo "test" > file.txt
> git add file.txt
> git commit -m "Import"
[main (root-commit) f73709f] Import
 1 file changed, 1 insertion(+)
 create mode 100644 file.txt
> stat -c '%y' file.txt
2022-01-19 18:43:52.147781748 +0100
> ls -i file.txt
74458979 file.txt
> ln file.txt ../file.txt
> git reset --hard
HEAD is now at f73709f Import
> stat -c '%y' file.txt
2022-01-19 18:44:47.013167127 +0100
> ls -i file.txt
74458995 file.txt
Can this behavior be avoided?

Best, Michael

brian m. carlson· Jan 19, 2022, 22:20 UTC · re: Michael Herrmann · lore

Re: A puzzle: reset --hard and hard links

On 2022-01-19 at 20:37:48, Michael Herrmann wrote:
Show 32 quoted lines
> Hi all,
> 
> It seems `git reset --hard` recreates files that have external hard
> links. Is this intended?
> 
> The example below highlights the behavior. I have an unchanged Git
> repository. When I create an (outside) hard link to a file in the
> repository, then `git reset --hard` re-creates the file with a new
> modification time and inode. This occurs on Debian 11 with Git 2.30.2
> but not on Debian 10 with Git 2.20.1
> 
> > git init
> Initialized empty Git repository in .../git-test/.git/
> > echo "test" > file.txt
> > git add file.txt
> > git commit -m "Import"
> [main (root-commit) f73709f] Import
>  1 file changed, 1 insertion(+)
>  create mode 100644 file.txt
> > stat -c '%y' file.txt
> 2022-01-19 18:43:52.147781748 +0100
> > ls -i file.txt
> 74458979 file.txt
> > ln file.txt ../file.txt
> > git reset --hard
> HEAD is now at f73709f Import
> > stat -c '%y' file.txt
> 2022-01-19 18:44:47.013167127 +0100
> > ls -i file.txt
> 74458995 file.txt
> 
> Can this behavior be avoided?

Git generally doesn't guarantee that it will preserve hard links in any particular situation. It can and does replace files rather than writing over the existing ones, so this behavior is expected in at least some circumstances.

Whether it happens in this particular case probably depends on what data is in the index and whether it's considered stale, since if the file is out of date, I believe a git reset --hard will replace it rather than try to determine whether it's up to date.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Junio C Hamano· Jan 19, 2022, 22:37 UTC · re: brian m. carlson · lore

Re: A puzzle: reset --hard and hard links

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
> Whether it happens in this particular case probably depends on what data
> is in the index and whether it's considered stale, since if the file is
> out of date, I believe a git reset --hard will replace it rather than
> try to determine whether it's up to date.

True, but I think it answers a different question, which was "to replace or to overwrite". I do not recall any codepath in git proper (I do not know about third-party tools and scripts around the fringe, though) that overwrites working tree files instead of writing into new files and replacing them.

If there is a hardlink to outside the working tree of a tracked path, and "git reset --hard [<committish>]" needs to modify the contents of that tracked path because it does not match what is recorded in the committish, I think it is the right thing to severe the link and leave the path outside the working tree intact. If we instead overwrote, we will damage "the other file" that shares the hardlink, which is outside the working tree hence outside our control.

Also, if two paths inside a working tree are made into hardlinks to each other, whenever Git needs to update either of them, we would severe the link, i.e. not ovewrite but replace, and it is the right thing to do, since Git trakcs these two paths as two separate things (i.e. the index has one entry for each of these paths).

brian m. carlson· Jan 20, 2022, 22:20 UTC · re: Michael Herrmann · lore

Re: A puzzle: reset --hard and hard links

On 2022-01-20 at 08:59:02, Michael Herrmann wrote:
> Thank you for your replies. Is there a way to tell Git not to sever
> hard links as highlighted by my example?

No, there isn't. If you need to deal with files that should be linked and stored in a Git repository, your best bet is symlinks.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Michael Herrmann· Jan 21, 2022, 12:50 UTC · re: brian m. carlson · lore

Re: A puzzle: reset --hard and hard links

On Thu, 20 Jan 2022 at 19:21, brian m. carlson <sandals@crustytoothpaste.net> wrote:

> No, there isn't.  If you need to deal with files that should be linked
> and stored in a Git repository, your best bet is symlinks.

Okay, thank you. I unfortunately do not have a choice over which kind of link (soft or hard) is used. I will find a workaround.

Michael Herrmann· Jan 24, 2022, 13:48 UTC · re: Michael Herrmann · lore

Re: A puzzle: reset --hard and hard links

I now believe this is a bug in Git because calling `git status` fixes the problem.

1) Create a hard link to a file in a Git repo.
2) Call `git status`.
3) Call `git reset --hard`.
Without 2), Git severs the hard link. With 2), the hard link is preserved.

Reproduction steps ("Test script") and further info: https://github.com/brave/brave-browser/issues/20316#issuecomment-1020054082

This occurs for me on Debian 10 & 11, as well as Git 2.30.2 and 2.20.1.
Junio C Hamano· Jan 24, 2022, 18:07 UTC · re: Michael Herrmann · lore

Re: A puzzle: reset --hard and hard links

Michael Herrmann <michael@herrmann.io> writes:
Show 6 quoted lines
> I now believe this is a bug in Git because calling `git status` fixes
> the problem.
>
> 1) Create a hard link to a file in a Git repo.
> 2) Call `git status`.
> 3) Call `git reset --hard`.

It is merely because you helped Git to realize that there is no need to change the contents of hte file with "reset --hard".

With another step 1.5 "append a line to the file in question", git should severe the link, I would think, as at that point, to revert the contents of the file in question to its pristine state, it needs to modify it.

Michael Herrmann· Jan 24, 2022, 18:16 UTC · re: Junio C Hamano · lore

Re: A puzzle: reset --hard and hard links

> It is merely because you helped Git to realize

I don't want to "help Git realize". I'm sorry but in my opinion `git status` should not have any effects on other commands. I don't understand how you can argue that calling `git status` is a valid fix to "help Git".

> With another step 1.5 "append a line to the file in question", git
> should severe the link,
I don't want to sever the hard link. I want to avoid that it gets severed.
On Mon, 24 Jan 2022 at 15:07, Junio C Hamano <gitster@pobox.com> wrote:
Show 23 quoted lines
>
> Michael Herrmann <michael@herrmann.io> writes:
>
> > I now believe this is a bug in Git because calling `git status` fixes
> > the problem.
> >
> > 1) Create a hard link to a file in a Git repo.
> > 2) Call `git status`.
> > 3) Call `git reset --hard`.
>
> It is merely because you helped Git to realize that there is no need
> to change the contents of hte file with "reset --hard".
>
> With another step 1.5 "append a line to the file in question", git
> should severe the link, I would think, as at that point, to revert
> the contents of the file in question to its pristine state, it needs
> to modify it.
>
>
>
>
>
>
Junio C Hamano· Jan 24, 2022, 21:19 UTC · re: Michael Herrmann · lore

Re: A puzzle: reset --hard and hard links

Michael Herrmann <michael@herrmann.io> writes:
Show 11 quoted lines
>> It is merely because you helped Git to realize
>
> I don't want to "help Git realize". I'm sorry but in my opinion `git
> status` should not have any effects on other commands. I don't
> understand how you can argue that calling `git status` is a valid fix
> to "help Git".
>
>> With another step 1.5 "append a line to the file in question", git
>> should severe the link,
>
> I don't want to sever the hard link. I want to avoid that it gets severed.

Sorry, if that is the case, what you want is not a version control, and it is certainly not Git. You want something else.

Think about this scenario.
    $ rm -fr one && git init one && cd one
    $ echo 0 >a; echo 0 >b; git add a b; git commit -m zero
We have two files, a and b, each of which has "0" in it.
    $ echo 1 >b; git add b; git commit -m one
Now they have "0" and "1" respectively.
    $ echo 2 >a; echo 2 >b; git commit -a -m two
    $ ln -f a b
    $ git diff
    $ git diff HEAD

Now they have "2". Since they have identical contents, "diff" would report no difference relative to the index or HEAD, even after we manually break the working tree by making one of them a hardlink to the other.

Now, what should this command do?
    $ git reset --hard HEAD^

What the user is asking is (1) to move the branch to point at the previous commit, which had 0 and 1 in a and b respectively, and (2) to make sure that the index and the working tree contents match what is recorded in the commit.

So for Git to be a usefully correct version control system, it is essential to make sure what it writes out would not affect any path other than the one it is writing out. When it writes "0" to "a", it MUST break the hardlink from elsewhere that points at "a" before it does so. Otherwise, the "0" it writes into "a" will also be seen elsewhere, which is not what the updated HEAD (i.e. commit "one") wants to see. The same for "b" when it is updated from "2" to "1" when this happens.

rsbecker@nexbridge.com· Jan 24, 2022, 22:18 UTC · re: Junio C Hamano · lore

RE: A puzzle: reset --hard and hard links

On January 24, 2022 4:19 PM, Junio wrote:
Show 13 quoted lines
> Michael Herrmann <michael@herrmann.io> writes:
> 
> >> It is merely because you helped Git to realize
> >
> > I don't want to "help Git realize". I'm sorry but in my opinion `git
> > status` should not have any effects on other commands. I don't
> > understand how you can argue that calling `git status` is a valid fix
> > to "help Git".
> >
> >> With another step 1.5 "append a line to the file in question", git
> >> should severe the link,
> >
> > I don't want to sever the hard link. I want to avoid that it gets
severed.
> 
> Sorry, if that is the case, what you want is not a version control, and it
is certainly
Show 19 quoted lines
> not Git.  You want something else.
> 
> Think about this scenario.
> 
>     $ rm -fr one && git init one && cd one
>     $ echo 0 >a; echo 0 >b; git add a b; git commit -m zero
> 
> We have two files, a and b, each of which has "0" in it.
> 
>     $ echo 1 >b; git add b; git commit -m one
> 
> Now they have "0" and "1" respectively.
> 
>     $ echo 2 >a; echo 2 >b; git commit -a -m two
>     $ ln -f a b
>     $ git diff
>     $ git diff HEAD
> 
> Now they have "2".  Since they have identical contents, "diff" would
report no
Show 8 quoted lines
> difference relative to the index or HEAD, even after we manually break the
> working tree by making one of them a hardlink to the other.
> 
> Now, what should this command do?
> 
>     $ git reset --hard HEAD^
> 
> What the user is asking is (1) to move the branch to point at the previous
commit,
> which had 0 and 1 in a and b respectively, and (2) to make sure that the
index and
> the working tree contents match what is recorded in the commit.
> 
> So for Git to be a usefully correct version control system, it is
essential to make
> sure what it writes out would not affect any path other than the one it is
writing
> out.  When it writes "0" to "a", it MUST break the hardlink from elsewhere
that
> points at "a" before it does so.  Otherwise, the "0" it writes into "a"
will also be
> seen elsewhere, which is not what the updated HEAD (i.e. commit "one")
wants to
> see.  The same for "b" when it is updated from "2" to "1"
> when this happens.

I think there are more use cases here than are apparent but also some serious question about why one would do this.

In a Linux/POSIX environment, one can do a hard link to a file inside a git repo, change the file using something like vim, and have git recognize that there is a change in git status. However, this only works on some platforms. Hard links do not have a 100% consistent semantic from one OS to another, one file system to another, or even between editors and scripts. As Junio pointed out, using a > operator on a hard link is likely going to replace the file instead of modifying the existing one. >> might correctly append and have git recognize it in a git status... on some platforms.

Making git consistent in this situation across every possible situation is not only impractical, it is likely impossible and cutting down what git is allowed to do so that only those common things are implemented might gut git badly.

If you are looking for doing external edits while keeping git notified, I would suggest wrapping the file modification in a script that is aware of hard links so that you get the results you want.

Alternatively, a soft link made externally to a physical git file location might do what you want - assuming your platform supports soft links. Your modification script/program/etc. would use the file directly in git instead of the hard link inode, so git is happy. A git status would see the change because the file really only exists in git. Other git operations, like restore, switch, etc., would cause the physical file to be modified correctly, and anything using the referencing soft link would see the change. Note: I am not suggest soft-linking from inside git to outside.

--Randall
Michael Herrmann· Jan 24, 2022, 21:50 UTC · re: Junio C Hamano · lore

Re: A puzzle: reset --hard and hard links

Thank you for your explanations Junio. This is the first part where I differ:
> $ ln -f a b

My hard link is outside the repo. In your example, it makes sense that Git has to sever the hard link to be able to give the files different contents. In my case and example, this complication is not present. And it does not address the main point:

My working tree is clean. `git reset --hard HEAD` (not HEAD^ like you had) should not do anything.

Finally, your (kind!) explanation does not give a reason why calling `git status` should change the behavior that Git unnecessarily severs the hard link.

My suspicion is that Git keeps a cache of the stat(...) result of files. An additional hard link increases the .st_nlink count of this struct. `git reset` compares the cached stat(...) values to the actual ones and sees that one has changed. `git status` does the same but is smart enough to realize that the additional hard link does not change anything. It writes this to the cache. `git reset` should also be smart!

Andreas Schwab· Jan 25, 2022, 08:49 UTC · re: Michael Herrmann · lore

Re: A puzzle: reset --hard and hard links

On Jan 24 2022, Michael Herrmann wrote:
Show 7 quoted lines
> My suspicion is that Git keeps a cache of the stat(...) result of
> files. An additional hard link increases the .st_nlink count of this
> struct. `git reset` compares the cached stat(...) values to the actual
> ones and sees that one has changed. `git status` does the same but is
> smart enough to realize that the additional hard link does not change
> anything. It writes this to the cache. `git reset` should also be
> smart!
See the core.trustctime config.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Ævar Arnfjörð Bjarmason· Jan 25, 2022, 11:33 UTC · re: Michael Herrmann · lore

Re: A puzzle: reset --hard and hard links

On Mon, Jan 24 2022, Michael Herrmann wrote:
Show 23 quoted lines
> Thank you for your explanations Junio. This is the first part where I differ:
>
>> $ ln -f a b
>
> My hard link is outside the repo. In your example, it makes sense that
> Git has to sever the hard link to be able to give the files different
> contents. In my case and example, this complication is not present.
> And it does not address the main point:
>
> My working tree is clean. `git reset --hard HEAD` (not HEAD^ like you
> had) should not do anything.
>
> Finally, your (kind!) explanation does not give a reason why calling
> `git status` should change the behavior that Git unnecessarily severs
> the hard link.
>
> My suspicion is that Git keeps a cache of the stat(...) result of
> files. An additional hard link increases the .st_nlink count of this
> struct. `git reset` compares the cached stat(...) values to the actual
> ones and sees that one has changed. `git status` does the same but is
> smart enough to realize that the additional hard link does not change
> anything. It writes this to the cache. `git reset` should also be
> smart!

What you're observing is that we tweak the index when various commands are run, some of that is documented, and others we consider purely implementation details. Whether we sever a hard link relationship is definitely on the "implementation detail" side of that.

I.e. that you can observe a behavior difference here doesn't mean that it's a bug, it means that you're poking at behavior that was never supposed to work this way, or be stable.

That being said I don't see a reason for why we shouldn't ever support what you're requesting here in some way. E.g. when we spin up different a different 'git worktree' on the same storage we could optionally hardlink to an existing checkout to save space.

This would be useful e.g. for spinning up a bunch of trees to run compilations on, where much of the checkout tree will be duplicated.

And this probably won't match your use-case, but I wonder how far you could get with the post-checkout hook, i.e. to have it run around after a checkout and fix up things that aren't hard links to be hardlinked appropriately.

I don't know of a tool to take two directories and hardlink things where possible, but it wouldn't be hard to write. I thought rsync could, but it appears just to support copying things as hardlink, not "fixing" files with the same content to be hardlinks after the fact (but maybe I've just missed a way to operate it).

Andreas Schwab· Jan 25, 2022, 13:29 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: A puzzle: reset --hard and hard links

On Jan 25 2022, Ævar Arnfjörð Bjarmason wrote:
> I don't know of a tool to take two directories and hardlink things where
> possible, but it wouldn't be hard to write.
https://github.com/adrianlopezroche/fdupes
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Michael Herrmann· Jan 25, 2022, 14:30 UTC · re: Andreas Schwab · lore

Re: A puzzle: reset --hard and hard links

Andreas Schwab wrote:
> See the core.trustctime config.

This sounds very promising. It also fixes the problem in my preliminary tests. Are there known drawbacks to changing this setting to false? I haven't yet noticed a performance impact.

Randall wrote:
> I think there are more use cases here than are apparent

The use case is Chromium's build process. It creates hard links from a src/... to a gen/... directory. Some actions do `git reset --hard` in src/. This updates the modification time because of the hard links - even when there are no changes. That in turn leads to unnecessary rebuilds. I have little control over the creation of the hard links.

Ævar, everything you wrote is very interesting and helpful. Thank you!
brian m. carlson· Jan 26, 2022, 02:14 UTC · re: Michael Herrmann · lore

Re: A puzzle: reset --hard and hard links

On 2022-01-25 at 14:30:11, Michael Herrmann wrote:
Show 7 quoted lines
> Andreas Schwab wrote:
> 
> > See the core.trustctime config.
> 
> This sounds very promising. It also fixes the problem in my
> preliminary tests. Are there known drawbacks to changing this setting
> to false? I haven't yet noticed a performance impact.

The index holds a bunch of information on files and checks that information to see whether the file has changed. If the stat information changes, then the file is marked dirty and may be re-read. The option above and core.checkStat control which information is included.

When you do "git reset --hard" and there's no change to the file in the index, Git happens not to rewrite it in the working tree. This is an implementation detail and isn't guaranteed, but that's why things happen to be working here.

The downside to restricting what's in the index is that Git can miss changes. For example, with core.trustctime turned off, a program that modifies a file without changing its size but resets it to have the same mtime would probably result in Git missing those changes. It shouldn't result in a performance difference, but it could theoretically result in a correctness difference. You may decide the tradeoff is worth it, however.

Show 9 quoted lines
> Randall wrote:
> 
> > I think there are more use cases here than are apparent
> 
> The use case is Chromium's build process. It creates hard links from a
> src/... to a gen/... directory. Some actions do `git reset --hard` in
> src/. This updates the modification time because of the hard links -
> even when there are no changes. That in turn leads to unnecessary
> rebuilds. I have little control over the creation of the hard links.

Thanks, this is helpful context and it explains why you'd want this behavior. If you're involved with the project, it may be helpful to point out to other project members that this occurs and suggest that the scripts avoid running "git reset --hard". For example, it may be easy to avoid if "git status --porcelain" produces empty output. I've heard stories about Chromium's build times and I'm sure such an optimization would be welcome.

-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA
Junio C Hamano· Jan 26, 2022, 18:46 UTC · re: brian m. carlson · lore

Re: A puzzle: reset --hard and hard links

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
Show 7 quoted lines
> Thanks, this is helpful context and it explains why you'd want this
> behavior.  If you're involved with the project, it may be helpful to
> point out to other project members that this occurs and suggest that the
> scripts avoid running "git reset --hard".  For example, it may be easy
> to avoid if "git status --porcelain" produces empty output.  I've heard
> stories about Chromium's build times and I'm sure such an optimization
> would be welcome.

I am not sure about that. If the ONLY problem is that hardlinks to UNMODIFIED paths are severed by "reset --hard" when it is not necessary in order to ensure that HEAD and the working tree matches in content without clobbering anything unrelated, then adding an internal call to refresh before "git reset --hard" would neatly solve it, and there should not be a need for end-user workaround like that.

But it does not change the fact that we try to avoid clobbering anything unrelated to the path we are updating when we need to update the contents of the working tree files, and the way we do so is to call checkout_entry(), which does unlink() followed by creat(). So even though you may be able to teach "git reset --hard" to refrain from severing extra hardlinks when it does not have to, it will do so when the contents of the path must be changed.

To be quite honest, I am not sure if the patch below is safe either. I doubt that the lack of "update-index --refresh" in the "reset --hard" command was a mistake; rather, I suspect that it was deliberately omitted to avoid some problems, which I do not offhand recall.

 builtin/reset.c | 1 +
 1 file changed, 1 insertion(+)
diff --git c/builtin/reset.c w/builtin/reset.c
index b97745ee94..8adc1be75b 100644
--- c/builtin/reset.c
+++ w/builtin/reset.c
@@ -83,6 +83,7 @@ static int reset_index(const char *ref, const struct object_id *oid, int reset_t
 	}
 
 	read_cache_unmerged();
+	refresh_cache(REFRESH_QUIET);
 
 	if (reset_type == KEEP) {
 		struct object_id head_oid;

← back to recent threads