threads / discuss / 33664

consistency problem on ZFS

Subject: consistency problem on ZFS

## tl;dr

9 messages between Apr 28, 2013 and Apr 29, 2013.

replies: 8people: 4as markdown or json

Yann Hodique· Apr 28, 2013, 19:11 UTC · lore
Hi,

I have a weird problem that seems to manifest itself only on ZFS (actually the Zevo distribution, on OSX). With git 1.8.2.1 by the way. I just switched to ZFS, so I can't blame that particular version of git.

"Sometimes" (I'd say something like 10-15% of the time, fairly reproducible anyway), "git diff-files" will see changes that don't exist for some time, then will catch up with the actual state of the file:

$ git checkout next; git diff-files; git checkout next; git diff-files Already on 'next' :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M magit.el Already on 'next' $

Here it was pretty quick, but sometimes I would see those phantom diffs for more than 5 seconds.

Interestingly, if I manage to run "git diff" during that time, this one reports no changes. See https://gist.github.com/sigma/5477827 for a more complex trace: "git merge" was complaining about changes, yet neither "git diff" nor "git status" did show any. That's the scenario I tried to minimize to the invocation above.

I'm willing to accept that the filesystem has some responsibility there, but the inconsistency bothers me. Anything I can provide to help finding the root cause of the issue ?

Thanks for any hint (and workarounds are highly welcome, as it breaks many of my scripts, and magit too :))

Yann.
-- 
A Duke must always take control of his household, for if he does not
rule those closest to him, he cannot hope to govern a planet.

  -- DUKE PAULUS ATREIDES
Thomas Rast· Apr 28, 2013, 19:33 UTC · re: Yann Hodique · lore

Re: consistency problem on ZFS

Yann Hodique <yann.hodique@gmail.com> writes:
Show 13 quoted lines
> I have a weird problem that seems to manifest itself only on ZFS
> (actually the Zevo distribution, on OSX). With git 1.8.2.1 by the way.
> I just switched to ZFS, so I can't blame that particular version of git.
>
> "Sometimes" (I'd say something like 10-15% of the time, fairly
> reproducible anyway), "git diff-files" will see changes that don't exist
> for some time, then will catch up with the actual state of the file:
>
> $ git checkout next; git diff-files; git checkout next; git diff-files
> Already on 'next'
> :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M	magit.el
> Already on 'next'
> $

git-diff-files doesn't refresh the index. Why are you using it? It's the plumbing version of 'git diff' (without args), which does the same but *does* refresh the index.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Yann Hodique· Apr 28, 2013, 19:44 UTC · re: Thomas Rast · lore

Re: consistency problem on ZFS

>>>>> "Thomas" == Thomas Rast <trast@inf.ethz.ch> writes:
Show 14 quoted lines
> Yann Hodique <yann.hodique@gmail.com> writes:
>> I have a weird problem that seems to manifest itself only on ZFS
>> (actually the Zevo distribution, on OSX). With git 1.8.2.1 by the way.
>> I just switched to ZFS, so I can't blame that particular version of git.
>> 
>> "Sometimes" (I'd say something like 10-15% of the time, fairly
>> reproducible anyway), "git diff-files" will see changes that don't exist
>> for some time, then will catch up with the actual state of the file:
>> 
>> $ git checkout next; git diff-files; git checkout next; git diff-files
>> Already on 'next'
>> :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M	magit.el
>> Already on 'next'
>> $
> git-diff-files doesn't refresh the index.  Why are you using it?  It's
> the plumbing version of 'git diff' (without args), which does the same
> but *does* refresh the index.

Well, as I said, I'm mostly trying to minimize the case here (which might or might not be successful, as it's essentially guesswork). But whatever git-diff-files does, it seems odd that it doesn't report the same thing twice, no ?

The actual real problem I have is the one that's exposed in the longer trace I posted: git merge kindly asks me to fix a problem that a) doesn't exist and b) isn't reported by porcelain commands (git diff, and git status), and then magically stops asking after a couple of seconds.

Thanks,
Yann.
-- 
Battle?  There's always a desire for breathing space motivating it somewhere.

  -- The Bashar Teg
John Szakmeister· Apr 28, 2013, 20:12 UTC · re: Yann Hodique · lore

Re: consistency problem on ZFS

On Sun, Apr 28, 2013 at 3:11 PM, Yann Hodique <yann.hodique@gmail.com> wrote:
Show 15 quoted lines
> Hi,
>
> I have a weird problem that seems to manifest itself only on ZFS
> (actually the Zevo distribution, on OSX). With git 1.8.2.1 by the way.
> I just switched to ZFS, so I can't blame that particular version of git.
>
> "Sometimes" (I'd say something like 10-15% of the time, fairly
> reproducible anyway), "git diff-files" will see changes that don't exist
> for some time, then will catch up with the actual state of the file:
>
> $ git checkout next; git diff-files; git checkout next; git diff-files
> Already on 'next'
> :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M      magit.el
> Already on 'next'
> $

Since you're running with Mac OS X, can I ask what version? Have you seen this with the regular file system (HFS) at all? It might be that you need to disable core.trustctime.

-John
PS  Sorry for the repeat Yann... I forgot to CC the list.
Yann Hodique· Apr 28, 2013, 20:21 UTC · re: John Szakmeister · lore

Re: consistency problem on ZFS

>>>>> "John" == John Szakmeister <john@szakmeister.net> writes:
Show 16 quoted lines
> On Sun, Apr 28, 2013 at 3:11 PM, Yann Hodique <yann.hodique@gmail.com> wrote:
>> Hi,
>> 
>> I have a weird problem that seems to manifest itself only on ZFS
>> (actually the Zevo distribution, on OSX). With git 1.8.2.1 by the way.
>> I just switched to ZFS, so I can't blame that particular version of git.
>> 
>> "Sometimes" (I'd say something like 10-15% of the time, fairly
>> reproducible anyway), "git diff-files" will see changes that don't exist
>> for some time, then will catch up with the actual state of the file:
>> 
>> $ git checkout next; git diff-files; git checkout next; git diff-files
>> Already on 'next'
>> :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M      magit.el
>> Already on 'next'
>> $
> Since you're running with Mac OS X, can I ask what version?  Have you
> seen this with the regular file system (HFS) at all?  It might be that
> you need to disable core.trustctime.

Sure, it's OS X 10.8.3 And no, I don't see this on HFS+ at all. I've just tried numerous times without success.

And setting core.trustctime seems to do the trick indeed. Many thanks, that's helping a lot in the short term !

Yann.
-- 
At the age of fifteen, he had already learned silence.

  -- from "A Child's History of Muad'Dib" by the Princess Irulan
Junio C Hamano· Apr 28, 2013, 21:33 UTC · re: Yann Hodique · lore

Re: consistency problem on ZFS

Yann Hodique <yann.hodique@gmail.com> writes:
Show 5 quoted lines
> $ git checkout next; git diff-files; git checkout next; git diff-files
> Already on 'next'
> :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M	magit.el
> Already on 'next'
> $

If you run "git update-index --refresh" between the first "checkout" and "diff-files", do you still see the issue?

Yann Hodique· Apr 29, 2013, 17:55 UTC · re: Junio C Hamano · lore

Re: consistency problem on ZFS

>>>>> "Junio" == Junio C Hamano <gitster@pobox.com> writes:
Show 6 quoted lines
> Yann Hodique <yann.hodique@gmail.com> writes:
>> $ git checkout next; git diff-files; git checkout next; git diff-files
>> Already on 'next'
>> :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M	magit.el
>> Already on 'next'
>> $
> If you run "git update-index --refresh" between the first "checkout"
> and "diff-files", do you still see the issue?

Yes, although *way* less often. I just ran a quick experiment and have seen the issue 2 times in 2500 tries.

Thanks,
Yann.
-- 
There exists no separation between gods and men:
one blends softly casual into the other.

  -- Proverbs of Muad'dib
Junio C Hamano· Apr 29, 2013, 18:01 UTC · re: Yann Hodique · lore

Re: consistency problem on ZFS

Yann Hodique <yann.hodique@gmail.com> writes:
Show 14 quoted lines
>>>>>> "Junio" == Junio C Hamano <gitster@pobox.com> writes:
>
>> Yann Hodique <yann.hodique@gmail.com> writes:
>>> $ git checkout next; git diff-files; git checkout next; git diff-files
>>> Already on 'next'
>>> :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M	magit.el
>>> Already on 'next'
>>> $
>
>> If you run "git update-index --refresh" between the first "checkout"
>> and "diff-files", do you still see the issue?
>
> Yes, although *way* less often. I just ran a quick experiment and have
> seen the issue 2 times in 2500 tries.

Hmph, that indicates somebody other than you or your Git is mucking with the inodes of your working tree files and perphas affecting ctime on them (we do not pay attention to atime because dirtying cached information due to somebody else reading from the file makes absolutely no sense, but we do notice ctime changes). Perhaps background virus scanner or something silly like that?

Yann Hodique· Apr 29, 2013, 18:16 UTC · re: Junio C Hamano · lore

Re: consistency problem on ZFS

>>>>> "Junio" == Junio C Hamano <gitster@pobox.com> writes:
Show 15 quoted lines
> Yann Hodique <yann.hodique@gmail.com> writes:
>>>>>>> "Junio" == Junio C Hamano <gitster@pobox.com> writes:
>> 
>>> Yann Hodique <yann.hodique@gmail.com> writes:
>>>> $ git checkout next; git diff-files; git checkout next; git diff-files
>>>> Already on 'next'
>>>> :100644 100644 bd774cccaa14e061c3c26996567ee28f4f77ec80 0000000000000000000000000000000000000000 M	magit.el
>>>> Already on 'next'
>>>> $
>> 
>>> If you run "git update-index --refresh" between the first "checkout"
>>> and "diff-files", do you still see the issue?
>> 
>> Yes, although *way* less often. I just ran a quick experiment and have
>> seen the issue 2 times in 2500 tries.
Show 6 quoted lines
> Hmph, that indicates somebody other than you or your Git is mucking
> with the inodes of your working tree files and perphas affecting
> ctime on them (we do not pay attention to atime because dirtying
> cached information due to somebody else reading from the file makes
> absolutely no sense, but we do notice ctime changes).  Perhaps
> background virus scanner or something silly like that?

Not that I know of. I certainly don't have any virus scanner, and I even deactivated spotlight. In any case, I can't think of anything that would affect that volume while leaving my HFS+ ones alone, except for zfs background processes themselves.

Yann.
-- 
What senses do we lack that we cannot see or hear another world all around us?

  -- The Orange Catholic Bible

← back to recent threads