{"thread":{"id":"42874","subject":"t7063 failure on FreeBSD 10.3 i386/amd64","startedAt":"2016-07-18T22:30:43Z","lastAt":"2016-08-04T15:46:49Z","messageCount":24,"participants":["Eric Wong","Duy Nguyen","Nguyễn Thái Ngọc Duy","Torstem Bögershausen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"291692","messageId":"20160718223038.GA66056@plume","threadId":"42874","inReplyTo":null,"subject":"t7063 failure on FreeBSD 10.3 i386/amd64","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-18T22:30:38Z","receivedAt":"2016-07-18T22:30:43Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Not sure what's going on, below is the relevant output when\nrun with -i -v --tee (the rest succeeds without -i):\n\nok 26 - untracked cache correct after status\n\nexpecting success: \n\tavoid_racy &&\n\t: >../trace &&\n\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n\tgit status --porcelain >../status.actual &&\n\tcat >../status.expect <../trace.expect < wrote:\n> Not sure what's going on, below is the relevant output when\n> run with -i -v --tee (the rest succeeds without -i):\n> \n> ok 26 - untracked cache correct after status\n> \n> expecting success: \n> \tavoid_racy &&\n> \t: >../trace &&\n> \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n> \tgit status --porcelain >../status.actual &&\n> \tcat >../status.expect < M done/two\n> ?? .gitignore\n> ?? done/five\n> ?? dtwo/\n> EOF\n> \ttest_cmp ../status.expect ../status.actual &&\n> \tcat >../trace.expect < node creation: 0\n> gitignore invalidation: 0\n> directory invalidation: 0\n> opendir: 0\n> EOF\n> \ttest_cmp ../trace.expect ../trace\n> \n> --- ../trace.expect\t2016-07-18 22:23:28.679886000 +0000\n> +++ ../trace\t2016-07-18 22:23:28.677135000 +0000\n> @@ -1,4 +1,4 @@\n> node creation: 0\n> gitignore invalidation: 0\n> -directory invalidation: 0\n> -opendir: 0\n> +directory invalidation: 1\n> +opendir: 1\n> not ok 27 - test sparse status again with untracked cache\n"},{"id":"291728","messageId":"CACsJy8CRHsyT8YLPYoHZnxCuMvF1W=S5iayy2eoHZhbSe_qmDg@mail.gmail.com","threadId":"42874","inReplyTo":"20160718225424.GA813@plume","subject":"Re: t7063 failure on FreeBSD 10.3 i386/amd64","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-19T16:12:26Z","receivedAt":"2016-07-19T16:13:18Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 19, 2016 at 12:54 AM, Eric Wong wrote:\n> Oops, forgot to Cc some folks who worked on this :x\n>\n> Filesystem is ufs and it fails regardless of whether\n> soft-updates is enabled or not.\n\nNothing stands out to my eyes, so I'm going to install freebsd this\nweekend. I hope ufs does not have any nasty surprise for me. Stay\ntuned.\n-- \nDuy\n"},{"id":"291786","messageId":"20160720030226.GA7112@whir","threadId":"42874","inReplyTo":"CACsJy8CRHsyT8YLPYoHZnxCuMvF1W=S5iayy2eoHZhbSe_qmDg@mail.gmail.com","subject":"Re: t7063 failure on FreeBSD 10.3 i386/amd64","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-20T03:02:26Z","receivedAt":"2016-07-20T03:03:01Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Duy Nguyen wrote:\n> On Tue, Jul 19, 2016 at 12:54 AM, Eric Wong wrote:\n> > Oops, forgot to Cc some folks who worked on this :x\n> >\n> > Filesystem is ufs and it fails regardless of whether\n> > soft-updates is enabled or not.\n> \n> Nothing stands out to my eyes, so I'm going to install freebsd this\n> weekend. I hope ufs does not have any nasty surprise for me. Stay\n> tuned.\n\nThanks, this problem might be ufs-specific, tmpfs is fine.\nTested tmpfs with:\n\n\tkldload tmpfs\n\tmkdir /tmp/tmpfs\n\tmount -t tmpfs tmpfs /tmp/tmpfs\n\n(Documenting all this since much of this is new to me)\n\nI noticed FreeBSD now provides ready-to-run VM images along with\nnormal installation stuff, including qcow2 ones for QEMU users,\nso that saves some time.\n\nhttp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.3-RELEASE/amd64/Latest/FreeBSD-10.3-RELEASE-amd64.qcow2.xz\n\nNotes:\n\n* \"-net user\" because I'm lazy (ICMP ping won't work out-of-the-box)\n\n* kvm can be substituted for qemu-system-$ARCH for the KVM-less\n or users lacking write access to /dev/kvm\n\n* hostfwd=... allows me to ssh into port 22215 from my Linux host\n to hit port 22 in the guest\n\n* \"dhclient vtnet0 && pkg install git gettext gmake python libiconv\"\n should be enough to get started\n\nkvm -smp 8 -m 2048 \\\n\t-drive if=virtio,file=FreeBSD-10.3-RELEASE-amd64.qcow2 \\\n\t-net nic,model=virtio \\\n\t-net user,hostfwd=tcp:127.0.0.1:22215-:22 \\\n\t-display curses\n"},{"id":"291814","messageId":"CACsJy8CVT8Y2DBGUfABGJfyOqPrm9LHt5Kx8RxoKZ=7XvF0grg@mail.gmail.com","threadId":"42874","inReplyTo":"20160720030226.GA7112@whir","subject":"Re: t7063 failure on FreeBSD 10.3 i386/amd64","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-20T14:57:51Z","receivedAt":"2016-07-20T14:58:37Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 20, 2016 at 5:02 AM, Eric Wong wrote:\n> Duy Nguyen wrote:\n>> On Tue, Jul 19, 2016 at 12:54 AM, Eric Wong wrote:\n>> > Oops, forgot to Cc some folks who worked on this :x\n>> >\n>> > Filesystem is ufs and it fails regardless of whether\n>> > soft-updates is enabled or not.\n>>\n>> Nothing stands out to my eyes, so I'm going to install freebsd this\n>> weekend. I hope ufs does not have any nasty surprise for me. Stay\n>> tuned.\n>\n> Thanks, this problem might be ufs-specific, tmpfs is fine.\n> Tested tmpfs with:\n>\n> kldload tmpfs\n> mkdir /tmp/tmpfs\n> mount -t tmpfs tmpfs /tmp/tmpfs\n>\n> (Documenting all this since much of this is new to me)\n\nIf you can, try with some other \"real\" filesystems, not virtual ones\nlike tmpfs, e.g. zfs or ext2 (not sure if it's supported on fbsd).\nThis is just to make sure I didn't hit a bug specific to fbsd. Don't\nworry if you don't have time to do it. I'll get to it eventually.\n\n> I noticed FreeBSD now provides ready-to-run VM images along with\n> normal installation stuff, including qcow2 ones for QEMU users,\n> so that saves some time.\n>\n> http://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.3-RELEASE/amd64/Latest/FreeBSD-10.3-RELEASE-amd64.qcow2.xz\n\nYeah that's what I'm going to do, but not right now..\n-- \nDuy\n"},{"id":"292337","messageId":"CACsJy8D9Cy1bjzXddCTOOT7X3smBBB3xccEg7CHaOKAZruHVag@mail.gmail.com","threadId":"42874","inReplyTo":"CACsJy8CRHsyT8YLPYoHZnxCuMvF1W=S5iayy2eoHZhbSe_qmDg@mail.gmail.com","subject":"Re: t7063 failure on FreeBSD 10.3 i386/amd64","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-27T17:33:17Z","receivedAt":"2016-07-27T17:33:51Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 19, 2016 at 6:12 PM, Duy Nguyen wrote:\n> On Tue, Jul 19, 2016 at 12:54 AM, Eric Wong wrote:\n>> Oops, forgot to Cc some folks who worked on this :x\n>>\n>> Filesystem is ufs and it fails regardless of whether\n>> soft-updates is enabled or not.\n>\n> Nothing stands out to my eyes, so I'm going to install freebsd this\n> weekend. I hope ufs does not have any nasty surprise for me. Stay\n> tuned.\n\nThe good news is it looks like a false alarm due to a racy test (and\nit happens on ext2 too, zfs not tested), no big flaw or anything\n(phew!). The bad news is, I still have no idea what's happening and\nwhy is_racy_stat() returns true in this particular case. It'll take\nsome more time...\n-- \nDuy\n"},{"id":"292601","messageId":"20160730133102.GA4871@duynguyen","threadId":"42874","inReplyTo":"CACsJy8D9Cy1bjzXddCTOOT7X3smBBB3xccEg7CHaOKAZruHVag@mail.gmail.com","subject":"Re: t7063 failure on FreeBSD 10.3 i386/amd64","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-30T13:31:03Z","receivedAt":"2016-07-30T13:31:13Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 27, 2016 at 07:33:17PM +0200, Duy Nguyen wrote:\n> On Tue, Jul 19, 2016 at 6:12 PM, Duy Nguyen wrote:\n> > On Tue, Jul 19, 2016 at 12:54 AM, Eric Wong wrote:\n> >> Oops, forgot to Cc some folks who worked on this :x\n> >>\n> >> Filesystem is ufs and it fails regardless of whether\n> >> soft-updates is enabled or not.\n> >\n> > Nothing stands out to my eyes, so I'm going to install freebsd this\n> > weekend. I hope ufs does not have any nasty surprise for me. Stay\n> > tuned.\n> \n> The good news is it looks like a false alarm due to a racy test (and\n> it happens on ext2 too, zfs not tested), no big flaw or anything\n> (phew!). The bad news is, I still have no idea what's happening and\n> why is_racy_stat() returns true in this particular case. It'll take\n> some more time...\n\nI give up. FreeBSD behaves so weird in this case.\n\nThe code of interest is this\n\n test_expect_success 'test sparse status with untracked cache' '\n \t: >../trace &&\n \tavoid_racy &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../status.actual &&\n \t...\n '\n\nFor some reason mtime of current directory is always latest time when\ngit makes the stat call. Even if I make avoid_racy sleep longer,\nseveral seconds, mtime of \".\" will be the latest time.\n\nBut cwd stat info is _not_ supposed to change! We haven't touched it\nwhile avoid_racy is running. avoid_racy is just a wrapper of\nsleep. And sleep does not change cwd's mtime from the shell prompt. I\ntried running the script with bash too (suspecting problem with\ndefault shell) and replaced avoid_racy with /bin/sleep. Nothing.\n\nDoes our test framework run something in background??? No, it\ncan't be.\n\nIf a stat call is made before avoid_racy, then mtime is pinned down\nand does not change anymore. So a \"fix\" is something like this. I\ntried a minimal program that just does \"stat\" to make sure it's stat\nthat does it, not some side effect from 'ls'.\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 38b3890..fd199a0 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -421,6 +421,7 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n \n test_expect_success 'test sparse status with untracked cache' '\n : >../trace &&\n+ ls -d . &&\n avoid_racy &&\n GIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n git status --porcelain >../status.actual &&\n\n--\nDuy\n"},{"id":"292602","messageId":"CACsJy8Dof=KW1Y4h9NpQAAXBEAX4=v3jQSL0N3_CZm7hxFt9qg@mail.gmail.com","threadId":"42874","inReplyTo":"20160730133102.GA4871@duynguyen","subject":"Re: t7063 failure on FreeBSD 10.3 i386/amd64","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-30T13:54:19Z","receivedAt":"2016-07-30T13:54:54Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 30, 2016 at 3:31 PM, Duy Nguyen wrote:\n> I give up.\n\nI lied. Sleeping 30 seconds in that test case works, but there is no\nway I'm sending a patch to delay 30 seconds. FreeBSD must have some\ndelayed mtime update \"feature\" in its vfs layer somewhere.\n\nNow I give up.\n-- \nDuy\n"},{"id":"292656","messageId":"20160730182005.14426-1-pclouds@gmail.com","threadId":"42874","inReplyTo":"20160718223038.GA66056@plume","subject":"[PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-30T18:20:05Z","receivedAt":"2016-07-30T18:20:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Let's start with the commit message of [1] from freebsd.git [2]\n\n Sync timestamp changes for inodes of special files to disk as late\n as possible (when the inode is reclaimed). Temporarily only do\n this if option UFS_LAZYMOD configured and softupdates aren't\n enabled. UFS_LAZYMOD is intentionally left out of\n /sys/conf/options.\n\n This is mainly to avoid almost useless disk i/o on battery powered\n machines. It's silly to write to disk (on the next sync or when\n the inode becomes inactive) just because someone hit a key or\n something wrote to the screen or /dev/null.\n\n PR: 5577 [3]\n\nThe short version of that, in the context of t7063, is that when a\ndirectory is updated, its mtime may be updated later, not\nimmediately. This can be shown with a simple command sequence\n\n date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd .\n\nOne would expect that the date shown in `ls` would be one second from\n`date`, but it's 10 seconds later. If we put another `ls -lTd .` in\nfront of `sleep 10`, then the date of the last `ls` comes as\nexpected. The first `ls` somehow forces mtime to be updated.\n\nt7063 is really sensitive to directory mtime. When mtime is too \"new\",\ngit code suspects racy timestamps and will not trigger the shortcut in\nuntracked cache, in t7063.26 (or t7063.25 before this patch) and\neventually be detected in t7063.28\n\nWe have two options thanks to this special FreeBSD feature:\n\n1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely\n when running on FreeBSD\n\n2) Work around this problem (using the same 'ls' trick) and continue\n to support untracked cache on FreeBSD\n\nI initially wanted to go with 1) because I didn't know the exact\nnature of this feature and feared that it would make untracked cache\nwork unreliably, using the cached version when it should not.\n\nSince the behavior of this thing is clearer now. The picture is not\nthat bad. If this indeed happens often, untracked cache would assume\nracy condition more often and _fall back_ to non-untracked cache code\npaths. Which means it may be less effective, but it will not show\nwrong things.\n\nThis patch goes with option 2.\n\nPS. For those who want to look further in FreeBSD source code, this\nflag is now called IN_LAZYMOD. I can see it's effective in ext2 and\nufs. zfs is questionable.\n\n[1] 660e6408e6df99a20dacb070c5e7f9739efdf96d\n[2] git://github.com/freebsd/freebsd.git\n[3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577\n\nReported-by: Eric Wong \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n This is only of those commits that commit messages are more important\n than the patch itself. One of the good notes about directory mtime,\n if we start to use it in more places in git.\n\n t/t7063-status-untracked-cache.sh | 4 ++++\n t/test-lib.sh | 6 ++++++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex a971884..08fc586 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -419,6 +419,10 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n \trm base\n '\n \n+test_expect_success FREEBSD 'Work around lazy mtime update' '\n+\tls -ld . >/dev/null\n+'\n+\n test_expect_success 'test sparse status with untracked cache' '\n \t: >../trace &&\n \tavoid_racy &&\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 0055ebb..3c730a2 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -961,6 +961,12 @@ case $(uname -s) in\n \ttest_set_prereq SED_STRIPS_CR\n \ttest_set_prereq GREP_STRIPS_CR\n \t;;\n+*FreeBSD*)\n+\ttest_set_prereq FREEBSD\n+\ttest_set_prereq POSIXPERM\n+\ttest_set_prereq BSLASHPSPEC\n+\ttest_set_prereq EXECKEEPSPID\n+\t;;\n *)\n \ttest_set_prereq POSIXPERM\n \ttest_set_prereq BSLASHPSPEC\n-- \n2.9.1.566.gbd532d4\n\n"},{"id":"292667","messageId":"20160731001532.GA23146@starla","threadId":"42874","inReplyTo":"20160730182005.14426-1-pclouds@gmail.com","subject":"Re: [PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-31T00:15:32Z","receivedAt":"2016-07-31T00:15:39Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Nguyễn Thái Ngọc Duy wrote:\n> Let's start with the commit message of [1] from freebsd.git [2]\n> \n> Sync timestamp changes for inodes of special files to disk as late\n> as possible (when the inode is reclaimed). Temporarily only do\n> this if option UFS_LAZYMOD configured and softupdates aren't\n> enabled. UFS_LAZYMOD is intentionally left out of\n> /sys/conf/options.\n> \n> This is mainly to avoid almost useless disk i/o on battery powered\n> machines. It's silly to write to disk (on the next sync or when\n> the inode becomes inactive) just because someone hit a key or\n> something wrote to the screen or /dev/null.\n> \n> PR: 5577 [3]\n> \n> The short version of that, in the context of t7063, is that when a\n> directory is updated, its mtime may be updated later, not\n> immediately. This can be shown with a simple command sequence\n> \n> date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd .\n\nYikes. I guess FreeBSD doesn't have an in-memory inode cache it\ncan keep up-to-date without flushing to disk?\n\n> One would expect that the date shown in `ls` would be one second from\n> `date`, but it's 10 seconds later. If we put another `ls -lTd .` in\n> front of `sleep 10`, then the date of the last `ls` comes as\n> expected. The first `ls` somehow forces mtime to be updated.\n\nFwiw, `stat .` seems to have the same effect as `ls -lTd .`...\n\n> t7063 is really sensitive to directory mtime. When mtime is too \"new\",\n> git code suspects racy timestamps and will not trigger the shortcut in\n> untracked cache, in t7063.26 (or t7063.25 before this patch) and\n> eventually be detected in t7063.28\n> \n> We have two options thanks to this special FreeBSD feature:\n> \n> 1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely\n> when running on FreeBSD\n> \n> 2) Work around this problem (using the same 'ls' trick) and continue\n> to support untracked cache on FreeBSD\n> \n> I initially wanted to go with 1) because I didn't know the exact\n> nature of this feature and feared that it would make untracked cache\n> work unreliably, using the cached version when it should not.\n> \n> Since the behavior of this thing is clearer now. The picture is not\n> that bad. If this indeed happens often, untracked cache would assume\n> racy condition more often and _fall back_ to non-untracked cache code\n> paths. Which means it may be less effective, but it will not show\n> wrong things.\n> \n> This patch goes with option 2.\n> \n> PS. For those who want to look further in FreeBSD source code, this\n> flag is now called IN_LAZYMOD. I can see it's effective in ext2 and\n> ufs. zfs is questionable.\n> \n> [1] 660e6408e6df99a20dacb070c5e7f9739efdf96d\n> [2] git://github.com/freebsd/freebsd.git\n> [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577\n> \n> Reported-by: Eric Wong \n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> This is only of those commits that commit messages are more important\n> than the patch itself. One of the good notes about directory mtime,\n> if we start to use it in more places in git.\n\nThanks, Tested-by: Eric Wong \n\n> t/t7063-status-untracked-cache.sh | 4 ++++\n> t/test-lib.sh | 6 ++++++\n> 2 files changed, 10 insertions(+)\n> \n> diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\n> index a971884..08fc586 100755\n> --- a/t/t7063-status-untracked-cache.sh\n> +++ b/t/t7063-status-untracked-cache.sh\n> @@ -419,6 +419,10 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n> \trm base\n> '\n> \n> +test_expect_success FREEBSD 'Work around lazy mtime update' '\n> +\tls -ld . >/dev/null\n> +'\n\n\tstat . >/dev/null\n\nwould be more to the point of what is going on, here. But I\nalso wonder if untracked cache itself could/should be doing this\ninternally.\n\n(I'm not familiar with that code, of course)\n\nThanks again for looking into this.\n"},{"id":"292668","messageId":"20160731010732.GA31840@whir","threadId":"42874","inReplyTo":"20160731001532.GA23146@starla","subject":"Re: [PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-31T01:07:32Z","receivedAt":"2016-07-31T01:07:37Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Eric Wong wrote:\n> Nguyễn Thái Ngọc Duy wrote:\n> > +test_expect_success FREEBSD 'Work around lazy mtime update' '\n> > +\tls -ld . >/dev/null\n> > +'\n> \n> \tstat . >/dev/null\n\nIf there's some older FreeBSD w/o stat(1); \"test -x .\"\nought to work, too, and it's faster being a shell builtin.\n\nI suspect some shell might be clever about optimizing away\na more-obvious \"test -d .\", so I choose \"test -x .\"\n\n> would be more to the point of what is going on, here. But I\n> also wonder if untracked cache itself could/should be doing this\n> internally.\n\nStill wondering :>\n\n> (I'm not familiar with that code, of course)\n> \n> Thanks again for looking into this.\n"},{"id":"292685","messageId":"CACsJy8CfLYPT5hJRT1q6nCtjy0K6+nZWGV5G+d7u=ymuEZ85tg@mail.gmail.com","threadId":"42874","inReplyTo":"20160731010732.GA31840@whir","subject":"Re: [PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-31T14:30:34Z","receivedAt":"2016-07-31T14:34:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jul 31, 2016 at 3:07 AM, Eric Wong wrote:\n>> would be more to the point of what is going on, here. But I\n>> also wonder if untracked cache itself could/should be doing this\n>> internally.\n>\n> Still wondering :>\n\nThere's nothing we can do besides maybe run a cron job executing\n'sync' every second or so. We need to force mtime to be written down\nclose to.. you know.. mtime. By the time git is executed, it's already\nlate. You can execute 'sync' inside git then wait a couple seconds..\nbut that's just stupid. And removing the racy check is even more\ndangerous, now you can get false output.\n-- \nDuy\n"},{"id":"292699","messageId":"6955746D-E47E-4BB8-AB0E-44D461E67AD6@web.de","threadId":"42874","inReplyTo":"20160730182005.14426-1-pclouds@gmail.com","subject":"Re: [PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Torstem Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-08-01T01:37:28Z","receivedAt":"2016-08-01T01:37:48Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\n\n\n> Am 30.07.2016 um 15:20 schrieb Nguyễn Thái Ngọc Duy :\n> \n> Let's start with the commit message of [1] from freebsd.git [2]\n> \n> Sync timestamp changes for inodes of special files to disk as late\n> as possible (when the inode is reclaimed). Temporarily only do\n> this if option UFS_LAZYMOD configured and softupdates aren't\n> enabled. UFS_LAZYMOD is intentionally left out of\n> /sys/conf/options.\n> \n> This is mainly to avoid almost useless disk i/o on battery powered\n> machines. It's silly to write to disk (on the next sync or when\n> the inode becomes inactive) just because someone hit a key or\n> something wrote to the screen or /dev/null.\n> \n> PR: 5577 [3]\n> \n> The short version of that, in the context of t7063, is that when a\n> directory is updated, its mtime may be updated later, not\n> immediately. This can be shown with a simple command sequence\n> \n> date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd .\n> \n> One would expect that the date shown in `ls` would be one second from\n> `date`, but it's 10 seconds later. If we put another `ls -lTd .` in\n> front of `sleep 10`, then the date of the last `ls` comes as\n> expected. The first `ls` somehow forces mtime to be updated.\n> \n> t7063 is really sensitive to directory mtime. When mtime is too \"new\",\n> git code suspects racy timestamps and will not trigger the shortcut in\n> untracked cache, in t7063.26 (or t7063.25 before this patch) and\n> eventually be detected in t7063.28\n> \n> We have two options thanks to this special FreeBSD feature:\n> \n> 1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely\n> when running on FreeBSD\n> \n> 2) Work around this problem (using the same 'ls' trick) and continue\n> to support untracked cache on FreeBSD\n> \n> I initially wanted to go with 1) because I didn't know the exact\n> nature of this feature and feared that it would make untracked cache\n> work unreliably, using the cached version when it should not.\n> \n> Since the behavior of this thing is clearer now. The picture is not\n> that bad. If this indeed happens often, untracked cache would assume\n> racy condition more often and _fall back_ to non-untracked cache code\n> paths. Which means it may be less effective, but it will not show\n> wrong things.\n> \n> This patch goes with option 2.\n> \n> PS. For those who want to look further in FreeBSD source code, this\n> flag is now called IN_LAZYMOD. I can see it's effective in ext2 and\n> ufs. zfs is questionable.\n> \n> [1] 660e6408e6df99a20dacb070c5e7f9739efdf96d\n> [2] git://github.com/freebsd/freebsd.git\n> [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577\n> \n> Reported-by: Eric Wong \n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> This is only of those commits that commit messages are more important\n> than the patch itself. One of the good notes about directory mtime,\n> if we start to use it in more places in git.\n> \n> t/t7063-status-untracked-cache.sh | 4 ++++\n> t/test-lib.sh | 6 ++++++\n> 2 files changed, 10 insertions(+)\n> \n> diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\n> index a971884..08fc586 100755\n> --- a/t/t7063-status-untracked-cache.sh\n> +++ b/t/t7063-status-untracked-cache.sh\n> @@ -419,6 +419,10 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n> rm base\n> '\n> \n> +test_expect_success FREEBSD 'Work around lazy mtime update' '\n> + ls -ld . >/dev/null\n> +'\n> +\n\nthe term FREEBSD may be too generic to point out a single feature\nin an OS distributution.\nFollowing your investigations, it may even be possible that\nother systems adapt this \"feature\"?\n\nHow about\nLAZY_DIR_MTIME_UPDATE\n(or similar)\n\n"},{"id":"292751","messageId":"CACsJy8D=dZeE1tLFRaCefkkNX8dHQfTL134Nv--5=BXvnUm1ZQ@mail.gmail.com","threadId":"42874","inReplyTo":"6955746D-E47E-4BB8-AB0E-44D461E67AD6@web.de","subject":"Re: [PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-08-01T17:10:28Z","receivedAt":"2016-08-01T17:12:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Aug 1, 2016 at 3:37 AM, Torstem Bögershausen wrote:\n> the term FREEBSD may be too generic to point out a single feature\n> in an OS distributution.\n> Following your investigations, it may even be possible that\n> other systems adapt this \"feature\"?\n>\n> How about\n> LAZY_DIR_MTIME_UPDATE\n> (or similar)\n\nThis feature was added in 1998, so yes there's a chance it has spread\nto a few fbsd derivatives (not sure if openbsd or netbsd is close\nenough and they ever exchange changes). But I'd rather wait for the\nsecond OS to expose the same feature before renaming it.\n-- \nDuy\n"},{"id":"292792","messageId":"xmqq4m74i4k3.fsf@gitster.mtv.corp.google.com","threadId":"42874","inReplyTo":"CACsJy8D=dZeE1tLFRaCefkkNX8dHQfTL134Nv--5=BXvnUm1ZQ@mail.gmail.com","subject":"Re: [PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-01T21:04:44Z","receivedAt":"2016-08-01T21:11:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Mon, Aug 1, 2016 at 3:37 AM, Torstem Bögershausen wrote:\n>> the term FREEBSD may be too generic to point out a single feature\n>> in an OS distributution.\n>> Following your investigations, it may even be possible that\n>> other systems adapt this \"feature\"?\n>>\n>> How about\n>> LAZY_DIR_MTIME_UPDATE\n>> (or similar)\n>\n> This feature was added in 1998, so yes there's a chance it has spread\n> to a few fbsd derivatives (not sure if openbsd or netbsd is close\n> enough and they ever exchange changes). But I'd rather wait for the\n> second OS to expose the same feature before renaming it.\n\nI think a name based on the observed behaviour (\"feature\") would be\nmore appropriate because I'd be more worried about us finding other\nglitches we see (initially) only on FBSD. People who need to adjust\ntests that use the same FBSD prereq would have to wonder which\nprereq-skip is due to which glitch.\n"},{"id":"292840","messageId":"20160802153715.GA25286@duynguyen","threadId":"42874","inReplyTo":"xmqq4m74i4k3.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-08-02T15:37:15Z","receivedAt":"2016-08-02T15:38:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Aug 01, 2016 at 02:04:44PM -0700, Junio C Hamano wrote:\n> Duy Nguyen writes:\n> \n> > On Mon, Aug 1, 2016 at 3:37 AM, Torstem Bögershausen wrote:\n> >> the term FREEBSD may be too generic to point out a single feature\n> >> in an OS distributution.\n> >> Following your investigations, it may even be possible that\n> >> other systems adapt this \"feature\"?\n> >>\n> >> How about\n> >> LAZY_DIR_MTIME_UPDATE\n> >> (or similar)\n> >\n> > This feature was added in 1998, so yes there's a chance it has spread\n> > to a few fbsd derivatives (not sure if openbsd or netbsd is close\n> > enough and they ever exchange changes). But I'd rather wait for the\n> > second OS to expose the same feature before renaming it.\n> \n> I think a name based on the observed behaviour (\"feature\") would be\n> more appropriate because I'd be more worried about us finding other\n> glitches we see (initially) only on FBSD. People who need to adjust\n> tests that use the same FBSD prereq would have to wonder which\n> prereq-skip is due to which glitch.\n\nOK how about this squashed in? The name was taken from fbsd definition\nIN_LAZYMOD.\n\nOff topic. Since I found this macro defined twice, in ext2 and ufs,\nbut not in zfs (found its source!), I assume zfs does not have this\nparticular feature (but I didn't test it). Untracked cache may be more\neffecient there.\n\n-- 8< --\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 08fc586..8bb048a 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -419,7 +419,7 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n \trm base\n '\n \n-test_expect_success FREEBSD 'Work around lazy mtime update' '\n+test_expect_success LAZYMOD 'Work around lazy mtime update' '\n \tls -ld . >/dev/null\n '\n \ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 3c730a2..1fc5266 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -962,7 +962,7 @@ case $(uname -s) in\n \ttest_set_prereq GREP_STRIPS_CR\n \t;;\n *FreeBSD*)\n-\ttest_set_prereq FREEBSD\n+\ttest_set_prereq LAZYMOD\n \ttest_set_prereq POSIXPERM\n \ttest_set_prereq BSLASHPSPEC\n \ttest_set_prereq EXECKEEPSPID\n-- 8< --\n\n--\nDuy\n"},{"id":"292847","messageId":"xmqqfuqnf5uj.fsf@gitster.mtv.corp.google.com","threadId":"42874","inReplyTo":"20160802153715.GA25286@duynguyen","subject":"Re: [PATCH] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-02T17:17:24Z","receivedAt":"2016-08-02T17:22:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> OK how about this squashed in? The name was taken from fbsd definition\n> IN_LAZYMOD.\n\nI am sorry that I didn't spot this possiblity earlier, but do we\nneed anything conditional? Either FREEBSD or LAZYMOD prerequisite\ntells very little what the \"Work around lazy mtime update\" is and\nwhere it triggers (I think the conclusion of your investigation was\nthat the timestamp on the containing directory, but that is ONLY\nrecorded in the log message, i.e. readers need to run 'git blame'\nto find out).\n\nIt might be a better approach to have a helper function with\ndescriptive name and comment early in t7063, e.g.\n\n\t# On some filesystems (e.g. FreeBSD's ext2 and ufs) this\n # and that happens when we do blah, which forces the\n # untracked cache code to take the slow path. A test\n # that wants to make sure the fast path works correctly\n # should call this helper to make mtime of the containing\n # directory in sync with the reality after doing blah and\n # before checking the fast path behaviour\n\ttest_sync_directory_mtime () {\n\t\tls -ld . >/dev/null\n }\n\nand then call it at strategic places without any prerequisite.\n\nThe helper may turn out to be useful outside the context of 7063\nlater, at which time we can move it to test-lib-functions, but that\nis a separate step.\n\n> Off topic. Since I found this macro defined twice, in ext2 and ufs,\n> but not in zfs (found its source!), I assume zfs does not have this\n> particular feature (but I didn't test it). Untracked cache may be more\n> effecient there.\n\n\n> -- 8< --\n> diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\n> index 08fc586..8bb048a 100755\n> --- a/t/t7063-status-untracked-cache.sh\n> +++ b/t/t7063-status-untracked-cache.sh\n> @@ -419,7 +419,7 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n> \trm base\n> '\n> \n> -test_expect_success FREEBSD 'Work around lazy mtime update' '\n> +test_expect_success LAZYMOD 'Work around lazy mtime update' '\n> \tls -ld . >/dev/null\n> '\n> \n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 3c730a2..1fc5266 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -962,7 +962,7 @@ case $(uname -s) in\n> \ttest_set_prereq GREP_STRIPS_CR\n> \t;;\n> *FreeBSD*)\n> -\ttest_set_prereq FREEBSD\n> +\ttest_set_prereq LAZYMOD\n> \ttest_set_prereq POSIXPERM\n> \ttest_set_prereq BSLASHPSPEC\n> \ttest_set_prereq EXECKEEPSPID\n> -- 8< --\n>\n> --\n> Duy\n"},{"id":"292903","messageId":"20160803160536.15596-1-pclouds@gmail.com","threadId":"42874","inReplyTo":"20160730182005.14426-1-pclouds@gmail.com","subject":"[PATCH v2] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-08-03T16:05:36Z","receivedAt":"2016-08-03T17:04:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Let's start with the commit message of [1] from freebsd.git [2]\n\n Sync timestamp changes for inodes of special files to disk as late\n as possible (when the inode is reclaimed). Temporarily only do\n this if option UFS_LAZYMOD configured and softupdates aren't\n enabled. UFS_LAZYMOD is intentionally left out of\n /sys/conf/options.\n\n This is mainly to avoid almost useless disk i/o on battery powered\n machines. It's silly to write to disk (on the next sync or when\n the inode becomes inactive) just because someone hit a key or\n something wrote to the screen or /dev/null.\n\n PR: 5577 [3]\n\nThe short version of that, in the context of t7063, is that when a\ndirectory is updated, its mtime may be updated later, not\nimmediately. This can be shown with a simple command sequence\n\n date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd .\n\nOne would expect that the date shown in `ls` would be one second from\n`date`, but it's 10 seconds later. If we put another `ls -lTd .` in\nfront of `sleep 10`, then the date of the last `ls` comes as\nexpected. The first `ls` somehow forces mtime to be updated.\n\nt7063 is really sensitive to directory mtime. When mtime is too \"new\",\ngit code suspects racy timestamps and will not trigger the shortcut in\nuntracked cache, in t7063.24 and eventually be detected in t7063.27\n\nWe have two options thanks to this special FreeBSD feature:\n\n1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely\n when running on FreeBSD\n\n2) Work around this problem (using the same 'ls' trick) and continue\n to support untracked cache on FreeBSD\n\nI initially wanted to go with 1) because I didn't know the exact\nnature of this feature and feared that it would make untracked cache\nwork unreliably, using the cached version when it should not.\n\nSince the behavior of this thing is clearer now. The picture is not\nthat bad. If this indeed happens often, untracked cache would assume\nracy condition more often and _fall back_ to non-untracked cache code\npaths. Which means it may be less effective, but it will not show\nwrong things.\n\nThis patch goes with option 2.\n\nPS. For those who want to look further in FreeBSD source code, this\nflag is now called IN_LAZYMOD. I can see it's effective in ext2 and\nufs. zfs is not affected.\n\n[1] 660e6408e6df99a20dacb070c5e7f9739efdf96d\n[2] git://github.com/freebsd/freebsd.git\n[3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577\n\nReported-by: Eric Wong \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n v2 goes with Junio's suggestion (good one!). And since there is an\n intention to reuse this new function, I make sure all directories are\n stat'd to pin down their mtime.\n\n t/t7063-status-untracked-cache.sh | 13 ++++++++++++-\n 1 file changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex a971884..7dd4de0 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -4,6 +4,16 @@ test_description='test untracked cache'\n \n . ./test-lib.sh\n \n+# On some filesystems (e.g. FreeBSD's ext2 and ufs) this and that\n+# happens when we do blah, which forces the untracked cache code to\n+# take the slow path. A test that wants to make sure the fast path\n+# works correctly should call this helper to make mtime of the\n+# containing directory in sync with the reality after doing blah and\n+# before checking the fast path behaviour\n+sync_mtime () {\n+\tfind . -type d -exec ls -ld {} \\; >/dev/null\n+}\n+\n avoid_racy() {\n \tsleep 1\n }\n@@ -416,7 +426,8 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n \techo four >done/four && # four is gitignored at a higher level\n \techo five >done/five && # five is not gitignored\n \techo test >base && #we need to ensure that the root dir is touched\n-\trm base\n+\trm base &&\n+\tsync_mtime\n '\n \n test_expect_success 'test sparse status with untracked cache' '\n-- \n2.9.1.566.gbd532d4\n\n"},{"id":"292928","messageId":"xmqqmvktakvt.fsf@gitster.mtv.corp.google.com","threadId":"42874","inReplyTo":"20160803160536.15596-1-pclouds@gmail.com","subject":"Re: [PATCH v2] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-03T16:16:06Z","receivedAt":"2016-08-03T17:05:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> v2 goes with Junio's suggestion (good one!).\n\nEhh, have you even read what you copied and pasted? \"this and that\"\nand \"blah\" are meant to be placeholders for you to fill in.\n\nI am not sure if \"-exec ls -ld\" is a good idea. Doesn't \"find\" by\nitself does enough lstat(2) call to work already? Even if it were\nnecessary to trigger a separate stat(2) call, wouldn't \"-ls\" be\nsufficient?\n\n> +# On some filesystems (e.g. FreeBSD's ext2 and ufs) this and that\n> +# happens when we do blah, which forces the untracked cache code to\n> +# take the slow path. A test that wants to make sure the fast path\n> +# works correctly should call this helper to make mtime of the\n> +# containing directory in sync with the reality after doing blah and\n> +# before checking the fast path behaviour\n> +sync_mtime () {\n> +\tfind . -type d -exec ls -ld {} \\; >/dev/null\n> +}\n> +\n> avoid_racy() {\n> \tsleep 1\n> }\n> @@ -416,7 +426,8 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n> \techo four >done/four && # four is gitignored at a higher level\n> \techo five >done/five && # five is not gitignored\n> \techo test >base && #we need to ensure that the root dir is touched\n> -\trm base\n> +\trm base &&\n> +\tsync_mtime\n> '\n> \n> test_expect_success 'test sparse status with untracked cache' '\n"},{"id":"292932","messageId":"CACsJy8C3xuCj-2=8B18qpDhbRtW=S2UQ1zFLTaeqRqgXYSeyBw@mail.gmail.com","threadId":"42874","inReplyTo":"xmqqmvktakvt.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-08-03T16:25:09Z","receivedAt":"2016-08-03T17:05:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Aug 3, 2016 at 6:16 PM, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> v2 goes with Junio's suggestion (good one!).\n>\n> Ehh, have you even read what you copied and pasted? \"this and that\"\n> and \"blah\" are meant to be placeholders for you to fill in.\n\nI did. But after a bit of consideration, I think the confusion caused\nby \"this and that\" would force the reader to blame and read the entire\ncommit message.\n\n> I am not sure if \"-exec ls -ld\" is a good idea. Doesn't \"find\" by\n> itself does enough lstat(2) call to work already?\n\nHmm.. because in some cases we could have dtype from readdir()?\n\n> Even if it were\n> necessary to trigger a separate stat(2) call, wouldn't \"-ls\" be\n> sufficient?\n\nYep. Let me run some tests before sending v3.\n-- \nDuy\n"},{"id":"292950","messageId":"20160803174522.5571-1-pclouds@gmail.com","threadId":"42874","inReplyTo":"20160803160536.15596-1-pclouds@gmail.com","subject":"[PATCH v3] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-08-03T17:45:22Z","receivedAt":"2016-08-03T17:46:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Let's start with the commit message of [1] from freebsd.git [2]\n\n Sync timestamp changes for inodes of special files to disk as late\n as possible (when the inode is reclaimed). Temporarily only do\n this if option UFS_LAZYMOD configured and softupdates aren't\n enabled. UFS_LAZYMOD is intentionally left out of\n /sys/conf/options.\n\n This is mainly to avoid almost useless disk i/o on battery powered\n machines. It's silly to write to disk (on the next sync or when\n the inode becomes inactive) just because someone hit a key or\n something wrote to the screen or /dev/null.\n\n PR: 5577 [3]\n\nThe short version of that, in the context of t7063, is that when a\ndirectory is updated, its mtime may be updated later, not\nimmediately. This can be shown with a simple command sequence\n\n date; sleep 1; touch abc; rm abc; sleep 10; ls -lTd .\n\nOne would expect that the date shown in `ls` would be one second from\n`date`, but it's 10 seconds later. If we put another `ls -lTd .` in\nfront of `sleep 10`, then the date of the last `ls` comes as\nexpected. The first `ls` somehow forces mtime to be updated.\n\nt7063 is really sensitive to directory mtime. When mtime is too \"new\",\ngit code suspects racy timestamps and will not trigger the shortcut in\nuntracked cache, in t7063.24 and eventually be detected in t7063.27\n\nWe have two options thanks to this special FreeBSD feature:\n\n1) Stop supporting untracked cache on FreeBSD. Skip t7063 entirely\n when running on FreeBSD\n\n2) Work around this problem (using the same 'ls' trick) and continue\n to support untracked cache on FreeBSD\n\nI initially wanted to go with 1) because I didn't know the exact\nnature of this feature and feared that it would make untracked cache\nwork unreliably, using the cached version when it should not.\n\nSince the behavior of this thing is clearer now. The picture is not\nthat bad. If this indeed happens often, untracked cache would assume\nracy condition more often and _fall back_ to non-untracked cache code\npaths. Which means it may be less effective, but it will not show\nwrong things.\n\nThis patch goes with option 2.\n\nPS. For those who want to look further in FreeBSD source code, this\nflag is now called IN_LAZYMOD. I can see it's effective in ext2 and\nufs. zfs is not affected.\n\n[1] 660e6408e6df99a20dacb070c5e7f9739efdf96d\n[2] git://github.com/freebsd/freebsd.git\n[3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=5577\n\nReported-by: Eric Wong \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n v3 differs from v2 in one line\n\n --- a/t/t7063-status-untracked-cache.sh\n +++ b/t/t7063-status-untracked-cache.sh\n @@ -11,7 +11,7 @@ test_description='test untracked cache'\n # containing directory in sync with the reality after doing blah and\n # before checking the fast path behaviour\n sync_mtime () {\n -\tfind . -type d -exec ls -ld {} \\; >/dev/null\n +\tfind . -type d -ls >/dev/null\n }\n \n avoid_racy() {\n t/t7063-status-untracked-cache.sh | 13 ++++++++++++-\n 1 file changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex a971884..d31d080 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -4,6 +4,16 @@ test_description='test untracked cache'\n \n . ./test-lib.sh\n \n+# On some filesystems (e.g. FreeBSD's ext2 and ufs) this and that\n+# happens when we do blah, which forces the untracked cache code to\n+# take the slow path. A test that wants to make sure the fast path\n+# works correctly should call this helper to make mtime of the\n+# containing directory in sync with the reality after doing blah and\n+# before checking the fast path behaviour\n+sync_mtime () {\n+\tfind . -type d -ls >/dev/null\n+}\n+\n avoid_racy() {\n \tsleep 1\n }\n@@ -416,7 +426,8 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n \techo four >done/four && # four is gitignored at a higher level\n \techo five >done/five && # five is not gitignored\n \techo test >base && #we need to ensure that the root dir is touched\n-\trm base\n+\trm base &&\n+\tsync_mtime\n '\n \n test_expect_success 'test sparse status with untracked cache' '\n-- \n2.9.1.566.gbd532d4\n\n"},{"id":"292958","messageId":"CAPc5daU49iVYw14sYeshXDbHGE_nW4vWyM=2XSz-buACVLFwoA@mail.gmail.com","threadId":"42874","inReplyTo":"20160803174522.5571-1-pclouds@gmail.com","subject":"Re: [PATCH v3] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-03T17:52:04Z","receivedAt":"2016-08-03T18:06:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Wed, Aug 3, 2016 at 10:45 AM, Nguyễn Thái Ngọc Duy wrote:\n> v3 differs from v2 in one line\n>\n> --- a/t/t7063-status-untracked-cache.sh\n> +++ b/t/t7063-status-untracked-cache.sh\n> @@ -11,7 +11,7 @@ test_description='test untracked cache'\n> # containing directory in sync with the reality after doing blah and\n> # before checking the fast path behaviour\n> sync_mtime () {\n> - find . -type d -exec ls -ld {} \\; >/dev/null\n> + find . -type d -ls >/dev/null\n> }\n\n\"this and that\" and \"blah\" are still there.\n\nIf you mean to tell the user \"I won't describe it in detail, if you\nreally want to know,\ngo run blame yourself\", spell it out like so. I was hoping that you\ncan summarize\nin-line there to help the readers here.\n\n> +# On some filesystems (e.g. FreeBSD's ext2 and ufs) this and that\n> +# happens when we do blah, which forces the untracked cache code to\n> +# take the slow path. A test that wants to make sure the fast path\n> +# works correctly should call this helper to make mtime of the\n> +# containing directory in sync with the reality after doing blah and\n> +# before checking the fast path behaviour\n> +sync_mtime () {\n> + find . -type d -ls >/dev/null\n> +}\n> +\n> avoid_racy() {\n> sleep 1\n> }\n> @@ -416,7 +426,8 @@ test_expect_success 'create/modify files, some of which are gitignored' '\n> echo four >done/four && # four is gitignored at a higher level\n> echo five >done/five && # five is not gitignored\n> echo test >base && #we need to ensure that the root dir is touched\n> - rm base\n> + rm base &&\n> + sync_mtime\n> '\n>\n> test_expect_success 'test sparse status with untracked cache' '\n> --\n> 2.9.1.566.gbd532d4\n>\n"},{"id":"292967","messageId":"xmqqpopp8ycx.fsf@gitster.mtv.corp.google.com","threadId":"42874","inReplyTo":"CAPc5daU49iVYw14sYeshXDbHGE_nW4vWyM=2XSz-buACVLFwoA@mail.gmail.com","subject":"Re: [PATCH v3] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-03T19:07:58Z","receivedAt":"2016-08-03T19:20:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> If you mean to tell the user \"I won't describe it in detail, if you\n> really want to know,\n> go run blame yourself\", spell it out like so. I was hoping that you\n> can summarize\n> in-line there to help the readers here.\n\nHere is a proposed fixup.\n\n t/t7063-status-untracked-cache.sh | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex d31d080..e0a8228 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -4,12 +4,16 @@ test_description='test untracked cache'\n \n . ./test-lib.sh\n \n-# On some filesystems (e.g. FreeBSD's ext2 and ufs) this and that\n-# happens when we do blah, which forces the untracked cache code to\n-# take the slow path. A test that wants to make sure the fast path\n-# works correctly should call this helper to make mtime of the\n-# containing directory in sync with the reality after doing blah and\n-# before checking the fast path behaviour\n+# On some filesystems (e.g. FreeBSD's ext2 and ufs) directory mtime\n+# is updated lazily after contents in the directory changes, which\n+# forces the untracked cache code to take the slow path. A test\n+# that wants to make sure that the fast path works correctly should\n+# call this helper to make mtime of the containing directory in sync\n+# with the reality before checking the fast path behaviour.\n+#\n+# See <20160803174522.5571-1-pclouds@gmail.com> if you want to know\n+# more.\n+\n sync_mtime () {\n \tfind . -type d -ls >/dev/null\n }\n"},{"id":"293089","messageId":"CACsJy8AL9+uGni8huURb9LOktEXdX3HWi_GzdDQzTWFTVWn6_w@mail.gmail.com","threadId":"42874","inReplyTo":"xmqqpopp8ycx.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3] t7063: work around FreeBSD's lazy mtime update feature","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-08-04T15:46:12Z","receivedAt":"2016-08-04T15:46:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Aug 3, 2016 at 9:07 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n>> If you mean to tell the user \"I won't describe it in detail, if you\n>> really want to know,\n>> go run blame yourself\", spell it out like so. I was hoping that you\n>> can summarize\n>> in-line there to help the readers here.\n>\n> Here is a proposed fixup.\n\nGreat! Sorry I only have one or two hours these days and could not\npropose something else quicker.\n\n> t/t7063-status-untracked-cache.sh | 16 ++++++++++------\n> 1 file changed, 10 insertions(+), 6 deletions(-)\n>\n> diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\n> index d31d080..e0a8228 100755\n> --- a/t/t7063-status-untracked-cache.sh\n> +++ b/t/t7063-status-untracked-cache.sh\n> @@ -4,12 +4,16 @@ test_description='test untracked cache'\n>\n> . ./test-lib.sh\n>\n> -# On some filesystems (e.g. FreeBSD's ext2 and ufs) this and that\n> -# happens when we do blah, which forces the untracked cache code to\n> -# take the slow path. A test that wants to make sure the fast path\n> -# works correctly should call this helper to make mtime of the\n> -# containing directory in sync with the reality after doing blah and\n> -# before checking the fast path behaviour\n> +# On some filesystems (e.g. FreeBSD's ext2 and ufs) directory mtime\n> +# is updated lazily after contents in the directory changes, which\n> +# forces the untracked cache code to take the slow path. A test\n> +# that wants to make sure that the fast path works correctly should\n> +# call this helper to make mtime of the containing directory in sync\n> +# with the reality before checking the fast path behaviour.\n> +#\n> +# See <20160803174522.5571-1-pclouds@gmail.com> if you want to know\n> +# more.\n> +\n> sync_mtime () {\n> find . -type d -ls >/dev/null\n> }\n\n\n\n-- \nDuy\n"}]}