git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3] t7063: work around FreeBSD's lazy mtime update feature

From
Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date
Aug 3, 2016, 17:45 UTC
Message-ID
<20160803174522.5571-1-pclouds@gmail.com>
In-Reply-To
<20160803160536.15596-1-pclouds@gmail.com>
Let's start with the commit message of [1] from freebsd.git [2]
    Sync timestamp changes for inodes of special files to disk as late
    as possible (when the inode is reclaimed).  Temporarily only do
    this if option UFS_LAZYMOD configured and softupdates aren't
    enabled.  UFS_LAZYMOD is intentionally left out of
    /sys/conf/options.
    This is mainly to avoid almost useless disk i/o on battery powered
    machines.  It's silly to write to disk (on the next sync or when
    the inode becomes inactive) just because someone hit a key or
    something wrote to the screen or /dev/null.
    PR:             5577 [3]

The short version of that, in the context of t7063, is that when a directory is updated, its mtime may be updated later, not immediately. This can be shown with a simple command sequence

    date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd .

One would expect that the date shown in `ls` would be one second from `date`, but it's 10 seconds later. If we put another `ls -lTd .` in front of `sleep 10`, then the date of the last `ls` comes as expected. The first `ls` somehow forces mtime to be updated.

t7063 is really sensitive to directory mtime. When mtime is too "new", git code suspects racy timestamps and will not trigger the shortcut in untracked cache, in t7063.24 and eventually be detected in t7063.27

We have two options thanks to this special FreeBSD feature:
1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely
   when running on FreeBSD
2) Work around this problem (using the same 'ls' trick) and continue
   to support untracked cache on FreeBSD

I initially wanted to go with 1) because I didn't know the exact nature of this feature and feared that it would make untracked cache work unreliably, using the cached version when it should not.

Since the behavior of this thing is clearer now. The picture is not that bad. If this indeed happens often, untracked cache would assume racy condition more often and _fall back_ to non-untracked cache code paths. Which means it may be less effective, but it will not show wrong things.

This patch goes with option 2.

PS. For those who want to look further in FreeBSD source code, this flag is now called IN_LAZYMOD. I can see it's effective in ext2 and ufs. zfs is not affected.

[1] 660e6408e6df99a20dacb070c5e7f9739efdf96d [2] git://github.com/freebsd/freebsd.git [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577

Reported-by: Eric Wong <e@80x24.org>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 v3 differs from v2 in one line
    --- a/t/t7063-status-untracked-cache.sh
    +++ b/t/t7063-status-untracked-cache.sh
    @@ -11,7 +11,7 @@ test_description='test untracked cache'
     # containing directory in sync with the reality after doing blah and
     # before checking the fast path behaviour
     sync_mtime () {
    -	find . -type d -exec ls -ld {} \; >/dev/null
    +	find . -type d -ls >/dev/null
     }
 
 avoid_racy() {
 t/t7063-status-untracked-cache.sh | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh
index a971884..d31d080 100755
--- a/t/t7063-status-untracked-cache.sh
+++ b/t/t7063-status-untracked-cache.sh
@@ -4,6 +4,16 @@ test_description='test untracked cache'
 
 . ./test-lib.sh
 
+# On some filesystems (e.g. FreeBSD's ext2 and ufs) this and that
+# happens when we do blah, which forces the untracked cache code to
+# take the slow path.  A test that wants to make sure the fast path
+# works correctly should call this helper to make mtime of the
+# containing directory in sync with the reality after doing blah and
+# before checking the fast path behaviour
+sync_mtime () {
+	find . -type d -ls >/dev/null
+}
+
 avoid_racy() {
 	sleep 1
 }
@@ -416,7 +426,8 @@ test_expect_success 'create/modify files, some of which are gitignored' '
 	echo four >done/four && # four is gitignored at a higher level
 	echo five >done/five && # five is not gitignored
 	echo test >base && #we need to ensure that the root dir is touched
-	rm base
+	rm base &&
+	sync_mtime
 '
 
 test_expect_success 'test sparse status with untracked cache' '
-- 
2.9.1.566.gbd532d4
Previous: Duy NguyenNext: Junio C Hamano
Message 21 of 24 in “t7063 failure on FreeBSD 10.3 i386/amd64”
  1. Eric WongJul 18, 2016
  2. Eric WongJul 18, 2016
  3. Duy NguyenJul 19, 2016
  4. Eric WongJul 20, 2016
  5. Duy NguyenJul 20, 2016
  6. Duy NguyenJul 27, 2016
  7. Duy NguyenJul 30, 2016
  8. Duy NguyenJul 30, 2016
  9. t7063: work around FreeBSD's lazy mtime update featureNguyễn Thái Ngọc Duy, Jul 30, 2016
  10. Eric WongJul 31, 2016
  11. Eric WongJul 31, 2016
  12. Duy NguyenJul 31, 2016
  13. Torstem BögershausenAug 1, 2016
  14. Duy NguyenAug 1, 2016
  15. Junio C HamanoAug 1, 2016
  16. Duy NguyenAug 2, 2016
  17. Junio C HamanoAug 2, 2016
  18. t7063: work around FreeBSD's lazy mtime update featureNguyễn Thái Ngọc Duy, Aug 3, 2016
  19. Junio C HamanoAug 3, 2016
  20. Duy NguyenAug 3, 2016
  21. t7063: work around FreeBSD's lazy mtime update featureNguyễn Thái Ngọc Duy, Aug 3, 2016
  22. Junio C HamanoAug 3, 2016
  23. Junio C HamanoAug 3, 2016
  24. Duy NguyenAug 4, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.