threads / patch / 53153

patchcommit-graph: fix buggy --expire-time option

Subject: [PATCH] commit-graph: fix buggy --expire-time option

## tl;dr

20 messages between Apr 1, 2020 and Apr 2, 2020. Diffs are folded; open one to read it.

replies: 19people: 6as markdown or json

Derrick Stolee via GitGitGadget· Apr 1, 2020, 18:11 UTC · lore
From: Derrick Stolee <dstolee@microsoft.com>

The commit-graph builtin has an --expire-time option that takes a datetime using OPT_EXPIRY_DATE(). However, the implementation inside expire_commit_graphs() was treating a non-zero value as a number of seconds to subtract from "now".

Update t5323-split-commit-graph.sh to demonstrate the correct value of the --expire-time option by actually creating a crud .graph file with mtime earlier than the expire time. Instead of using a super- early time (1980) we need to use a recent time or else the old logic actually passes by accident. This test will start passing again on the old logic in 40 years or so.

I noticed this when inspecting some Scalar repos that had an excess number of commit-graph files. In Scalar, we were using this second interpretation by using "--expire-time=3600" to mean "delete graphs older than one hour ago" to avoid deleting a commit-graph that a foreground process may be trying to load.

Also I noticed that the help text was copied from the --max-commits option. Fix that help text.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
    commit-graph: fix buggy --expire-time option
    
    This is embarassing. I should have noticed this when writing it the
    first time, or when integrating the feature into Scalar and VFS for Git.
    Sorry!
    
    Thanks, -Stolee
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-596%2Fderrickstolee%2Fcommit-graph-expire-fix-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-596/derrickstolee/commit-graph-expire-fix-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/596
 builtin/commit-graph.c        | 2 +-
 commit-graph.c                | 2 +-
 t/t5324-split-commit-graph.sh | 4 +++-
 3 files changed, 5 insertions(+), 3 deletions(-)
Show changes to 3 files +5 −3

builtin/commit-graph.c, commit-graph.c, t/t5324-split-commit-graph.sh

diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c
index 4a70b33fb5f..8000ff0d2ee 100644
--- a/builtin/commit-graph.c
+++ b/builtin/commit-graph.c
@@ -140,7 +140,7 @@ static int graph_write(int argc, const char **argv)
 		OPT_INTEGER(0, "size-multiple", &split_opts.size_multiple,
 			N_("maximum ratio between two levels of a split commit-graph")),
 		OPT_EXPIRY_DATE(0, "expire-time", &split_opts.expire_time,
-			N_("maximum number of commits in a non-base split commit-graph")),
+			N_("do not expire files newer than a number of seconds before now")),
 		OPT_END(),
 	};
 
diff --git a/commit-graph.c b/commit-graph.c
index f013a84e294..0d0d37787a0 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1707,7 +1707,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)
 	timestamp_t expire_time = time(NULL);
 
 	if (ctx->split_opts && ctx->split_opts->expire_time)
-		expire_time -= ctx->split_opts->expire_time;
+		expire_time = ctx->split_opts->expire_time;
 	if (!ctx->split) {
 		char *chain_file_name = get_chain_filename(ctx->odb);
 		unlink(chain_file_name);
diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
index 53b2e6b4555..4e4efcaff22 100755
--- a/t/t5324-split-commit-graph.sh
+++ b/t/t5324-split-commit-graph.sh
@@ -210,8 +210,10 @@ test_expect_success 'test merge stragety constants' '
 		git config core.commitGraph true &&
 		test_line_count = 2 $graphdir/commit-graph-chain &&
 		test_commit 15 &&
-		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
+		touch -m -t 201801010000.00 $graphdir/extra.graph &&
+		git commit-graph write --reachable --split --size-multiple=10 --expire-time=2019-01-01 &&
 		test_line_count = 1 $graphdir/commit-graph-chain &&
+		test_path_is_missing $graphdir/extra.graph &&
 		ls $graphdir/graph-*.graph >graph-files &&
 		test_line_count = 3 graph-files
 	) &&

base-commit: 274b9cc25322d9ee79aa8e6d4e86f0ffe5ced925
-- 
gitgitgadget
Derrick Stolee· Apr 1, 2020, 18:17 UTC · re: Derrick Stolee via GitGitGadget · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

On 4/1/2020 2:11 PM, Derrick Stolee via GitGitGadget wrote:
> Also I noticed that the help text was copied from the --max-commits
> option. Fix that help text.
...
Show 10 quoted lines
> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c
> index 4a70b33fb5f..8000ff0d2ee 100644
> --- a/builtin/commit-graph.c
> +++ b/builtin/commit-graph.c
> @@ -140,7 +140,7 @@ static int graph_write(int argc, const char **argv)
>  		OPT_INTEGER(0, "size-multiple", &split_opts.size_multiple,
>  			N_("maximum ratio between two levels of a split commit-graph")),
>  		OPT_EXPIRY_DATE(0, "expire-time", &split_opts.expire_time,
> -			N_("maximum number of commits in a non-base split commit-graph")),
> +			N_("do not expire files newer than a number of seconds before now")),
and of course I messed this up even now.  Should be:
> +			N_("do not expire files newer than a given date-time")),
-Stolee
Taylor Blau· Apr 1, 2020, 18:56 UTC · re: Derrick Stolee · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

On Wed, Apr 01, 2020 at 02:17:50PM -0400, Derrick Stolee wrote:
Show 18 quoted lines
> On 4/1/2020 2:11 PM, Derrick Stolee via GitGitGadget wrote:
> > Also I noticed that the help text was copied from the --max-commits
> > option. Fix that help text.
> ...
> > diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c
> > index 4a70b33fb5f..8000ff0d2ee 100644
> > --- a/builtin/commit-graph.c
> > +++ b/builtin/commit-graph.c
> > @@ -140,7 +140,7 @@ static int graph_write(int argc, const char **argv)
> >  		OPT_INTEGER(0, "size-multiple", &split_opts.size_multiple,
> >  			N_("maximum ratio between two levels of a split commit-graph")),
> >  		OPT_EXPIRY_DATE(0, "expire-time", &split_opts.expire_time,
> > -			N_("maximum number of commits in a non-base split commit-graph")),
> > +			N_("do not expire files newer than a number of seconds before now")),
>
> and of course I messed this up even now.  Should be:
>
> > +			N_("do not expire files newer than a given date-time")),

I wonder if the double-negative can be avoided. Perhaps this should be instead:

  expire files older than the given date-time
or similar.
> -Stolee

Thanks, Taylor

Taylor Blau· Apr 1, 2020, 19:27 UTC · re: Derrick Stolee via GitGitGadget · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

On Wed, Apr 01, 2020 at 06:11:35PM +0000, Derrick Stolee via GitGitGadget wrote:
Show 6 quoted lines
> From: Derrick Stolee <dstolee@microsoft.com>
>
> The commit-graph builtin has an --expire-time option that takes a
> datetime using OPT_EXPIRY_DATE(). However, the implementation inside
> expire_commit_graphs() was treating a non-zero value as a number of
> seconds to subtract from "now".

Ah, nice catch. The fix below seems straightforward. Since we're using 'OPT_EXPIRY_DATE', parse-options accepts either, but gives us back a date-time, so there's no need to subtract it from the current instant or anything like that.

This patch below looks fine (I left a minor note here, and in the documentation fixup that you sent shortly after this message, but otherwise I think that this is good to go, at least from my point of view).

Show 72 quoted lines
> Update t5323-split-commit-graph.sh to demonstrate the correct value
> of the --expire-time option by actually creating a crud .graph file
> with mtime earlier than the expire time. Instead of using a super-
> early time (1980) we need to use a recent time or else the old
> logic actually passes by accident. This test will start passing
> again on the old logic in 40 years or so.
>
> I noticed this when inspecting some Scalar repos that had an excess
> number of commit-graph files. In Scalar, we were using this second
> interpretation by using "--expire-time=3600" to mean "delete graphs
> older than one hour ago" to avoid deleting a commit-graph that a
> foreground process may be trying to load.
>
> Also I noticed that the help text was copied from the --max-commits
> option. Fix that help text.
>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>     commit-graph: fix buggy --expire-time option
>
>     This is embarassing. I should have noticed this when writing it the
>     first time, or when integrating the feature into Scalar and VFS for Git.
>     Sorry!
>
>     Thanks, -Stolee
>
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-596%2Fderrickstolee%2Fcommit-graph-expire-fix-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-596/derrickstolee/commit-graph-expire-fix-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/596
>
>  builtin/commit-graph.c        | 2 +-
>  commit-graph.c                | 2 +-
>  t/t5324-split-commit-graph.sh | 4 +++-
>  3 files changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c
> index 4a70b33fb5f..8000ff0d2ee 100644
> --- a/builtin/commit-graph.c
> +++ b/builtin/commit-graph.c
> @@ -140,7 +140,7 @@ static int graph_write(int argc, const char **argv)
>  		OPT_INTEGER(0, "size-multiple", &split_opts.size_multiple,
>  			N_("maximum ratio between two levels of a split commit-graph")),
>  		OPT_EXPIRY_DATE(0, "expire-time", &split_opts.expire_time,
> -			N_("maximum number of commits in a non-base split commit-graph")),
> +			N_("do not expire files newer than a number of seconds before now")),
>  		OPT_END(),
>  	};
>
> diff --git a/commit-graph.c b/commit-graph.c
> index f013a84e294..0d0d37787a0 100644
> --- a/commit-graph.c
> +++ b/commit-graph.c
> @@ -1707,7 +1707,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)
>  	timestamp_t expire_time = time(NULL);
>
>  	if (ctx->split_opts && ctx->split_opts->expire_time)
> -		expire_time -= ctx->split_opts->expire_time;
> +		expire_time = ctx->split_opts->expire_time;
>  	if (!ctx->split) {
>  		char *chain_file_name = get_chain_filename(ctx->odb);
>  		unlink(chain_file_name);
> diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
> index 53b2e6b4555..4e4efcaff22 100755
> --- a/t/t5324-split-commit-graph.sh
> +++ b/t/t5324-split-commit-graph.sh
> @@ -210,8 +210,10 @@ test_expect_success 'test merge stragety constants' '
>  		git config core.commitGraph true &&
>  		test_line_count = 2 $graphdir/commit-graph-chain &&
>  		test_commit 15 &&
> -		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
> +		touch -m -t 201801010000.00 $graphdir/extra.graph &&
> +		git commit-graph write --reachable --split --size-multiple=10 --expire-time=2019-01-01 &&
Could this be written instead as:
  touch -m -t $(date -r $test_tick +"%Y%m%d%H%M.%S") $graphdir/extra.graph &&
  test_tick &&
  git commit-graph write --reachable --split --size-multiple=10 --expire-time=now &&
to avoid breaking in 40 years?
Show 9 quoted lines
>  		test_line_count = 1 $graphdir/commit-graph-chain &&
> +		test_path_is_missing $graphdir/extra.graph &&
>  		ls $graphdir/graph-*.graph >graph-files &&
>  		test_line_count = 3 graph-files
>  	) &&
>
> base-commit: 274b9cc25322d9ee79aa8e6d4e86f0ffe5ced925
> --
> gitgitgadget

Thanks, Taylor

Eric Sunshine· Apr 1, 2020, 19:36 UTC · re: Taylor Blau · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

On Wed, Apr 1, 2020 at 3:27 PM Taylor Blau <me@ttaylorr.com> wrote:
Show 9 quoted lines
> On Wed, Apr 01, 2020 at 06:11:35PM +0000, Derrick Stolee via GitGitGadget wrote:
> > -             git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
> > +             touch -m -t 201801010000.00 $graphdir/extra.graph &&
> > +             git commit-graph write --reachable --split --size-multiple=10 --expire-time=2019-01-01 &&
>
> Could this be written instead as:
>
>   touch -m -t $(date -r $test_tick +"%Y%m%d%H%M.%S") $graphdir/extra.graph &&
>   test_tick &&

This won't work with GNU 'date' command which doesn't understand "-r<number>" (it understands only "-r<reference-file>").

SZEDER Gábor· Apr 1, 2020, 19:47 UTC · re: Derrick Stolee via GitGitGadget · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

On Wed, Apr 01, 2020 at 06:11:35PM +0000, Derrick Stolee via GitGitGadget wrote:
Show 78 quoted lines
> From: Derrick Stolee <dstolee@microsoft.com>
> 
> The commit-graph builtin has an --expire-time option that takes a
> datetime using OPT_EXPIRY_DATE(). However, the implementation inside
> expire_commit_graphs() was treating a non-zero value as a number of
> seconds to subtract from "now".
> 
> Update t5323-split-commit-graph.sh to demonstrate the correct value
> of the --expire-time option by actually creating a crud .graph file
> with mtime earlier than the expire time. Instead of using a super-
> early time (1980) we need to use a recent time or else the old
> logic actually passes by accident. This test will start passing
> again on the old logic in 40 years or so.
> 
> I noticed this when inspecting some Scalar repos that had an excess
> number of commit-graph files. In Scalar, we were using this second
> interpretation by using "--expire-time=3600" to mean "delete graphs
> older than one hour ago" to avoid deleting a commit-graph that a
> foreground process may be trying to load.
> 
> Also I noticed that the help text was copied from the --max-commits
> option. Fix that help text.
> 
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>     commit-graph: fix buggy --expire-time option
>     
>     This is embarassing. I should have noticed this when writing it the
>     first time, or when integrating the feature into Scalar and VFS for Git.
>     Sorry!
>     
>     Thanks, -Stolee
> 
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-596%2Fderrickstolee%2Fcommit-graph-expire-fix-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-596/derrickstolee/commit-graph-expire-fix-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/596
> 
>  builtin/commit-graph.c        | 2 +-
>  commit-graph.c                | 2 +-
>  t/t5324-split-commit-graph.sh | 4 +++-
>  3 files changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c
> index 4a70b33fb5f..8000ff0d2ee 100644
> --- a/builtin/commit-graph.c
> +++ b/builtin/commit-graph.c
> @@ -140,7 +140,7 @@ static int graph_write(int argc, const char **argv)
>  		OPT_INTEGER(0, "size-multiple", &split_opts.size_multiple,
>  			N_("maximum ratio between two levels of a split commit-graph")),
>  		OPT_EXPIRY_DATE(0, "expire-time", &split_opts.expire_time,
> -			N_("maximum number of commits in a non-base split commit-graph")),
> +			N_("do not expire files newer than a number of seconds before now")),
>  		OPT_END(),
>  	};
>  
> diff --git a/commit-graph.c b/commit-graph.c
> index f013a84e294..0d0d37787a0 100644
> --- a/commit-graph.c
> +++ b/commit-graph.c
> @@ -1707,7 +1707,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)
>  	timestamp_t expire_time = time(NULL);
>  
>  	if (ctx->split_opts && ctx->split_opts->expire_time)
> -		expire_time -= ctx->split_opts->expire_time;
> +		expire_time = ctx->split_opts->expire_time;
>  	if (!ctx->split) {
>  		char *chain_file_name = get_chain_filename(ctx->odb);
>  		unlink(chain_file_name);
> diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
> index 53b2e6b4555..4e4efcaff22 100755
> --- a/t/t5324-split-commit-graph.sh
> +++ b/t/t5324-split-commit-graph.sh
> @@ -210,8 +210,10 @@ test_expect_success 'test merge stragety constants' '
>  		git config core.commitGraph true &&
>  		test_line_count = 2 $graphdir/commit-graph-chain &&
>  		test_commit 15 &&
> -		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
> +		touch -m -t 201801010000.00 $graphdir/extra.graph &&
You can set a mtime relative to the current time with:
  test-tool chmtime =-3600 $graphdir/extra.graph
Show 10 quoted lines
> +		git commit-graph write --reachable --split --size-multiple=10 --expire-time=2019-01-01 &&
>  		test_line_count = 1 $graphdir/commit-graph-chain &&
> +		test_path_is_missing $graphdir/extra.graph &&
>  		ls $graphdir/graph-*.graph >graph-files &&
>  		test_line_count = 3 graph-files
>  	) &&
> 
> base-commit: 274b9cc25322d9ee79aa8e6d4e86f0ffe5ced925
> -- 
> gitgitgadget
Junio C Hamano· Apr 1, 2020, 19:49 UTC · re: Derrick Stolee via GitGitGadget · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
Show 10 quoted lines
> diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
> index 53b2e6b4555..4e4efcaff22 100755
> --- a/t/t5324-split-commit-graph.sh
> +++ b/t/t5324-split-commit-graph.sh
> @@ -210,8 +210,10 @@ test_expect_success 'test merge stragety constants' '
>  		git config core.commitGraph true &&
>  		test_line_count = 2 $graphdir/commit-graph-chain &&
>  		test_commit 15 &&
> -		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
> +		touch -m -t 201801010000.00 $graphdir/extra.graph &&

We have "test-tool chmtime" since 17e48368 (Add test-chmtime: a utility to change mtime on files, 2007-02-24) and refrained from using "touch -t" anywhere in our tests. Can we use it here, too?

Especially with its "relative" form, can't we make the test to stay correct not just for 40 years but forever ;-)?

Jeff King· Apr 1, 2020, 19:57 UTC · re: Junio C Hamano · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

On Wed, Apr 01, 2020 at 12:49:25PM -0700, Junio C Hamano wrote:
Show 16 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
> > diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
> > index 53b2e6b4555..4e4efcaff22 100755
> > --- a/t/t5324-split-commit-graph.sh
> > +++ b/t/t5324-split-commit-graph.sh
> > @@ -210,8 +210,10 @@ test_expect_success 'test merge stragety constants' '
> >  		git config core.commitGraph true &&
> >  		test_line_count = 2 $graphdir/commit-graph-chain &&
> >  		test_commit 15 &&
> > -		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
> > +		touch -m -t 201801010000.00 $graphdir/extra.graph &&
> 
> We have "test-tool chmtime" since 17e48368 (Add test-chmtime: a
> utility to change mtime on files, 2007-02-24) and refrained from
> using "touch -t" anywhere in our tests.  Can we use it here, too?

There are a couple new ones added last year in t5319. Nobody has complained yet, but I wonder if it's a matter of time.

-Peff
Junio C Hamano· Apr 1, 2020, 20:33 UTC · re: Jeff King · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

Jeff King <peff@peff.net> writes:
Show 21 quoted lines
> On Wed, Apr 01, 2020 at 12:49:25PM -0700, Junio C Hamano wrote:
>
>> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>> 
>> > diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
>> > index 53b2e6b4555..4e4efcaff22 100755
>> > --- a/t/t5324-split-commit-graph.sh
>> > +++ b/t/t5324-split-commit-graph.sh
>> > @@ -210,8 +210,10 @@ test_expect_success 'test merge stragety constants' '
>> >  		git config core.commitGraph true &&
>> >  		test_line_count = 2 $graphdir/commit-graph-chain &&
>> >  		test_commit 15 &&
>> > -		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
>> > +		touch -m -t 201801010000.00 $graphdir/extra.graph &&
>> 
>> We have "test-tool chmtime" since 17e48368 (Add test-chmtime: a
>> utility to change mtime on files, 2007-02-24) and refrained from
>> using "touch -t" anywhere in our tests.  Can we use it here, too?
>
> There are a couple new ones added last year in t5319. Nobody has
> complained yet, but I wonder if it's a matter of time.
Indeed.  We should fix them (#leftoverbits).
Derrick Stolee· Apr 1, 2020, 20:51 UTC · re: Junio C Hamano · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

On 4/1/2020 4:33 PM, Junio C Hamano wrote:
Show 25 quoted lines
> Jeff King <peff@peff.net> writes:
> 
>> On Wed, Apr 01, 2020 at 12:49:25PM -0700, Junio C Hamano wrote:
>>
>>> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>>>
>>>> diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
>>>> index 53b2e6b4555..4e4efcaff22 100755
>>>> --- a/t/t5324-split-commit-graph.sh
>>>> +++ b/t/t5324-split-commit-graph.sh
>>>> @@ -210,8 +210,10 @@ test_expect_success 'test merge stragety constants' '
>>>>  		git config core.commitGraph true &&
>>>>  		test_line_count = 2 $graphdir/commit-graph-chain &&
>>>>  		test_commit 15 &&
>>>> -		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
>>>> +		touch -m -t 201801010000.00 $graphdir/extra.graph &&
>>>
>>> We have "test-tool chmtime" since 17e48368 (Add test-chmtime: a
>>> utility to change mtime on files, 2007-02-24) and refrained from
>>> using "touch -t" anywhere in our tests.  Can we use it here, too?
>>
>> There are a couple new ones added last year in t5319. Nobody has
>> complained yet, but I wonder if it's a matter of time.
> 
> Indeed.  We should fix them (#leftoverbits).
I'm adding a patch to fix that now.
-Stolee
Derrick Stolee· Apr 1, 2020, 20:14 UTC · re: Junio C Hamano · lore

Re: [PATCH] commit-graph: fix buggy --expire-time option

On 4/1/2020 3:49 PM, Junio C Hamano wrote:
Show 16 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
>> index 53b2e6b4555..4e4efcaff22 100755
>> --- a/t/t5324-split-commit-graph.sh
>> +++ b/t/t5324-split-commit-graph.sh
>> @@ -210,8 +210,10 @@ test_expect_success 'test merge stragety constants' '
>>  		git config core.commitGraph true &&
>>  		test_line_count = 2 $graphdir/commit-graph-chain &&
>>  		test_commit 15 &&
>> -		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
>> +		touch -m -t 201801010000.00 $graphdir/extra.graph &&
> 
> We have "test-tool chmtime" since 17e48368 (Add test-chmtime: a
> utility to change mtime on files, 2007-02-24) and refrained from
> using "touch -t" anywhere in our tests.  Can we use it here, too?
I will definitely do the more portable thing here.
 
> Especially with its "relative" form, can't we make the test to
> stay correct not just for 40 years but forever ;-)?

I should clarify: the test won't break in 40 years. It will "fail to notice the bug" that was previously present. But the relative form is likely to continue catching this bug.

Thanks, -Stolee

Derrick Stolee via GitGitGadget· Apr 1, 2020, 21:00 UTC · re: Derrick Stolee via GitGitGadget · lore

[PATCH v2 0/2] commit-graph: fix buggy --expire-time option

This is embarassing. I should have noticed this when writing it the first time, or when integrating the feature into Scalar and VFS for Git. Sorry!

Update in V2:
 * Updated the helptext. Thanks, Taylor!
 * Added a patch for the "test-tool chmtime" in t5319
 * Used an absolute time with "test-tool chmtime" in t5324 instead of "touch
   -m"
 * Added a file that should be kept on the other side of the expire time to
   protect against off-by-one errors and future date errors.
Thanks, -Stolee
Derrick Stolee (2):
  t5319: replace 'touch -m' with 'test-tool chmtime'
  commit-graph: fix buggy --expire-time option
 builtin/commit-graph.c        | 2 +-
 commit-graph.c                | 2 +-
 t/t5319-multi-pack-index.sh   | 8 ++++----
 t/t5324-split-commit-graph.sh | 8 +++++++-
 4 files changed, 13 insertions(+), 7 deletions(-)
base-commit: 274b9cc25322d9ee79aa8e6d4e86f0ffe5ced925
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-596%2Fderrickstolee%2Fcommit-graph-expire-fix-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-596/derrickstolee/commit-graph-expire-fix-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/596
Range-diff vs v1:
 -:  ----------- > 1:  24e26ecda63 t5319: replace 'touch -m' with 'test-tool chmtime'
 1:  32c55c0fc21 ! 2:  56a312695fe commit-graph: fix buggy --expire-time option
     @@ Commit message
          Update t5323-split-commit-graph.sh to demonstrate the correct value
          of the --expire-time option by actually creating a crud .graph file
          with mtime earlier than the expire time. Instead of using a super-
     -    early time (1980) we need to use a recent time or else the old
     -    logic actually passes by accident. This test will start passing
     -    again on the old logic in 40 years or so.
     +    early time (1980) we use an explicit, and recent, time. Using
     +    test-tool chmtime to create two files on either end of an exact
     +    second, we create a test that catches this failure no matter the
     +    current time. Using a fixed date is more portable than trying to
     +    format a relative date string into the --expiry-date input.
      
          I noticed this when inspecting some Scalar repos that had an excess
          number of commit-graph files. In Scalar, we were using this second
     @@ builtin/commit-graph.c: static int graph_write(int argc, const char **argv)
       			N_("maximum ratio between two levels of a split commit-graph")),
       		OPT_EXPIRY_DATE(0, "expire-time", &split_opts.expire_time,
      -			N_("maximum number of commits in a non-base split commit-graph")),
     -+			N_("do not expire files newer than a number of seconds before now")),
     ++			N_("only expire files older than a given date-time")),
       		OPT_END(),
       	};
       
     @@ t/t5324-split-commit-graph.sh: test_expect_success 'test merge stragety constant
       		test_line_count = 2 $graphdir/commit-graph-chain &&
       		test_commit 15 &&
      -		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
     -+		touch -m -t 201801010000.00 $graphdir/extra.graph &&
     -+		git commit-graph write --reachable --split --size-multiple=10 --expire-time=2019-01-01 &&
     ++		touch $graphdir/to-delete.graph $graphdir/to-keep.graph &&
     ++		test-tool chmtime =1546362000 $graphdir/to-delete.graph &&
     ++		test-tool chmtime =1546362001 $graphdir/to-keep.graph &&
     ++		git commit-graph write --reachable --split --size-multiple=10 \
     ++			--expire-time="2019-01-01 12:00 -05:00" &&
       		test_line_count = 1 $graphdir/commit-graph-chain &&
     -+		test_path_is_missing $graphdir/extra.graph &&
     ++		test_path_is_missing $graphdir/to-delete.graph &&
     ++		test_path_is_file $graphdir/to-keep.graph &&
       		ls $graphdir/graph-*.graph >graph-files &&
       		test_line_count = 3 graph-files
       	) &&
-- 
gitgitgadget
Derrick Stolee via GitGitGadget· Apr 1, 2020, 21:00 UTC · re: Derrick Stolee via GitGitGadget · lore

[PATCH v2 2/2] commit-graph: fix buggy --expire-time option

From: Derrick Stolee <dstolee@microsoft.com>

The commit-graph builtin has an --expire-time option that takes a datetime using OPT_EXPIRY_DATE(). However, the implementation inside expire_commit_graphs() was treating a non-zero value as a number of seconds to subtract from "now".

Update t5323-split-commit-graph.sh to demonstrate the correct value of the --expire-time option by actually creating a crud .graph file with mtime earlier than the expire time. Instead of using a super- early time (1980) we use an explicit, and recent, time. Using test-tool chmtime to create two files on either end of an exact second, we create a test that catches this failure no matter the current time. Using a fixed date is more portable than trying to format a relative date string into the --expiry-date input.

I noticed this when inspecting some Scalar repos that had an excess number of commit-graph files. In Scalar, we were using this second interpretation by using "--expire-time=3600" to mean "delete graphs older than one hour ago" to avoid deleting a commit-graph that a foreground process may be trying to load.

Also I noticed that the help text was copied from the --max-commits option. Fix that help text.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 builtin/commit-graph.c        | 2 +-
 commit-graph.c                | 2 +-
 t/t5324-split-commit-graph.sh | 8 +++++++-
 3 files changed, 9 insertions(+), 3 deletions(-)
Show changes to 3 files +9 −3

builtin/commit-graph.c, commit-graph.c, t/t5324-split-commit-graph.sh

diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c
index 4a70b33fb5f..5bfba972498 100644
--- a/builtin/commit-graph.c
+++ b/builtin/commit-graph.c
@@ -140,7 +140,7 @@ static int graph_write(int argc, const char **argv)
 		OPT_INTEGER(0, "size-multiple", &split_opts.size_multiple,
 			N_("maximum ratio between two levels of a split commit-graph")),
 		OPT_EXPIRY_DATE(0, "expire-time", &split_opts.expire_time,
-			N_("maximum number of commits in a non-base split commit-graph")),
+			N_("only expire files older than a given date-time")),
 		OPT_END(),
 	};
 
diff --git a/commit-graph.c b/commit-graph.c
index f013a84e294..0d0d37787a0 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1707,7 +1707,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)
 	timestamp_t expire_time = time(NULL);
 
 	if (ctx->split_opts && ctx->split_opts->expire_time)
-		expire_time -= ctx->split_opts->expire_time;
+		expire_time = ctx->split_opts->expire_time;
 	if (!ctx->split) {
 		char *chain_file_name = get_chain_filename(ctx->odb);
 		unlink(chain_file_name);
diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
index 53b2e6b4555..b8b208fc3da 100755
--- a/t/t5324-split-commit-graph.sh
+++ b/t/t5324-split-commit-graph.sh
@@ -210,8 +210,14 @@ test_expect_success 'test merge stragety constants' '
 		git config core.commitGraph true &&
 		test_line_count = 2 $graphdir/commit-graph-chain &&
 		test_commit 15 &&
-		git commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&
+		touch $graphdir/to-delete.graph $graphdir/to-keep.graph &&
+		test-tool chmtime =1546362000 $graphdir/to-delete.graph &&
+		test-tool chmtime =1546362001 $graphdir/to-keep.graph &&
+		git commit-graph write --reachable --split --size-multiple=10 \
+			--expire-time="2019-01-01 12:00 -05:00" &&
 		test_line_count = 1 $graphdir/commit-graph-chain &&
+		test_path_is_missing $graphdir/to-delete.graph &&
+		test_path_is_file $graphdir/to-keep.graph &&
 		ls $graphdir/graph-*.graph >graph-files &&
 		test_line_count = 3 graph-files
 	) &&
-- 
gitgitgadget
Derrick Stolee via GitGitGadget· Apr 1, 2020, 21:00 UTC · re: Derrick Stolee via GitGitGadget · lore

[PATCH v2 1/2] t5319: replace 'touch -m' with 'test-tool chmtime'

From: Derrick Stolee <dstolee@microsoft.com>

The use of 'touch -m' to modify a file's mtime is slightly less portable than using our own 'test-tool chmtime'. The important thing is that these pack-files are ordered in a special way to ensure the multi-pack-index selects some as the "newer" pack-files when resolving duplicate objects.

Reported-by: Jeff King <peff@peff.net>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
 t/t5319-multi-pack-index.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
Show changes to t/t5319-multi-pack-index.sh +4 −4
diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index 43a7a66c9d1..6b20e60314d 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -521,10 +521,10 @@ test_expect_success 'repack with minimum size does not alter existing packs' '
 		cd dup &&
 		rm -rf .git/objects/pack &&
 		mv .git/objects/pack-backup .git/objects/pack &&
-		touch -m -t 201901010000 .git/objects/pack/pack-D* &&
-		touch -m -t 201901010001 .git/objects/pack/pack-C* &&
-		touch -m -t 201901010002 .git/objects/pack/pack-B* &&
-		touch -m -t 201901010003 .git/objects/pack/pack-A* &&
+		test-tool chmtime =-5 .git/objects/pack/pack-D* &&
+		test-tool chmtime =-4 .git/objects/pack/pack-C* &&
+		test-tool chmtime =-3 .git/objects/pack/pack-B* &&
+		test-tool chmtime =-2 .git/objects/pack/pack-A* &&
 		ls .git/objects/pack >expect &&
 		MINSIZE=$(test-tool path-utils file-size .git/objects/pack/*pack | sort -n | head -n 1) &&
 		git multi-pack-index repack --batch-size=$MINSIZE &&
-- 
gitgitgadget
Junio C Hamano· Apr 1, 2020, 21:35 UTC · re: Derrick Stolee via GitGitGadget · lore

Re: [PATCH v2 1/2] t5319: replace 'touch -m' with 'test-tool chmtime'

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> From: Derrick Stolee <dstolee@microsoft.com>
>
> The use of 'touch -m' to modify a file's mtime is slightly less
> portable than using our own 'test-tool chmtime'.

Portability aside, the relative form would also be resistant against skews between filesystem time and wallclock time and is preferrable when we can use it.

> The important
> thing is that these pack-files are ordered in a special way to
> ensure the multi-pack-index selects some as the "newer" pack-files
> when resolving duplicate objects.
This note is very much appreciated.
Show 10 quoted lines
>  		rm -rf .git/objects/pack &&
>  		mv .git/objects/pack-backup .git/objects/pack &&
> -		touch -m -t 201901010000 .git/objects/pack/pack-D* &&
> -		touch -m -t 201901010001 .git/objects/pack/pack-C* &&
> -		touch -m -t 201901010002 .git/objects/pack/pack-B* &&
> -		touch -m -t 201901010003 .git/objects/pack/pack-A* &&
> +		test-tool chmtime =-5 .git/objects/pack/pack-D* &&
> +		test-tool chmtime =-4 .git/objects/pack/pack-C* &&
> +		test-tool chmtime =-3 .git/objects/pack/pack-B* &&
> +		test-tool chmtime =-2 .git/objects/pack/pack-A* &&

The original wants D to be the oldest and A to be the newest, and the updated would want the same ordering.

When created, we know A gets created before B which gets created before C and so on, in the "setup expire tests" part. If each step takes too much time (e.g. the VM is heavily loaded), wouldn't the adjustment above become insufficient?

In other words, would we want to flip the order these packs get created in the "setup" part, in addition to the use of chmtime (which reads the existing file timestamp using stat(2) and then updates the file timestamp relative to the original timestamp) we see here?

Also, in the best case (i.e. original timestamp of A/B/C/D are the same), the above seems to assume that the filesystem has at least 1 second file timestamp granularity. Do we want to make them at least 2 seconds apart, or am I worried too much about ancient filesystems that no longer metter?

Thanks.
>  		ls .git/objects/pack >expect &&
>  		MINSIZE=$(test-tool path-utils file-size .git/objects/pack/*pack | sort -n | head -n 1) &&
>  		git multi-pack-index repack --batch-size=$MINSIZE &&
Derrick Stolee· Apr 2, 2020, 00:06 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 1/2] t5319: replace 'touch -m' with 'test-tool chmtime'

On 4/1/2020 5:35 PM, Junio C Hamano wrote:
Show 48 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: Derrick Stolee <dstolee@microsoft.com>
>>
>> The use of 'touch -m' to modify a file's mtime is slightly less
>> portable than using our own 'test-tool chmtime'.
> 
> Portability aside, the relative form would also be resistant against
> skews between filesystem time and wallclock time and is preferrable
> when we can use it.
> 
>> The important
>> thing is that these pack-files are ordered in a special way to
>> ensure the multi-pack-index selects some as the "newer" pack-files
>> when resolving duplicate objects.
> 
> This note is very much appreciated.
> 
>>  		rm -rf .git/objects/pack &&
>>  		mv .git/objects/pack-backup .git/objects/pack &&
>> -		touch -m -t 201901010000 .git/objects/pack/pack-D* &&
>> -		touch -m -t 201901010001 .git/objects/pack/pack-C* &&
>> -		touch -m -t 201901010002 .git/objects/pack/pack-B* &&
>> -		touch -m -t 201901010003 .git/objects/pack/pack-A* &&
>> +		test-tool chmtime =-5 .git/objects/pack/pack-D* &&
>> +		test-tool chmtime =-4 .git/objects/pack/pack-C* &&
>> +		test-tool chmtime =-3 .git/objects/pack/pack-B* &&
>> +		test-tool chmtime =-2 .git/objects/pack/pack-A* &&
> 
> The original wants D to be the oldest and A to be the newest, and
> the updated would want the same ordering.
> 
> When created, we know A gets created before B which gets created
> before C and so on, in the "setup expire tests" part.  If each step
> takes too much time (e.g. the VM is heavily loaded), wouldn't the
> adjustment above become insufficient?
> 
> In other words, would we want to flip the order these packs get
> created in the "setup" part, in addition to the use of chmtime
> (which reads the existing file timestamp using stat(2) and then
> updates the file timestamp relative to the original timestamp)
> we see here?
> 
> Also, in the best case (i.e. original timestamp of A/B/C/D are the
> same), the above seems to assume that the filesystem has at least 1
> second file timestamp granularity.  Do we want to make them at least
> 2 seconds apart, or am I worried too much about ancient filesystems
> that no longer metter?

The old test relied on one-second granularity, so that hasn't changed. I could easily space it out a bit more without issue.

Your concern about the original timestamps getting skewed shouldn't be an issue because "test-tool chmtime =-<seconds>" subtracts the <seconds> from the current system time, not the file's mtime. This is subtle: without the "=" it would modify it from the file's mtime.

Since we are assigning the offset values in the proper order (D to A) there isn't an issue if time ticks forward between these steps.

Thanks, -Stolee

Jeff King· Apr 2, 2020, 12:51 UTC · re: Derrick Stolee · lore

Re: [PATCH v2 1/2] t5319: replace 'touch -m' with 'test-tool chmtime'

On Wed, Apr 01, 2020 at 08:06:47PM -0400, Derrick Stolee wrote:
Show 16 quoted lines
> > Also, in the best case (i.e. original timestamp of A/B/C/D are the
> > same), the above seems to assume that the filesystem has at least 1
> > second file timestamp granularity.  Do we want to make them at least
> > 2 seconds apart, or am I worried too much about ancient filesystems
> > that no longer metter?
> 
> The old test relied on one-second granularity, so that hasn't changed.
> I could easily space it out a bit more without issue.
> 
> Your concern about the original timestamps getting skewed shouldn't
> be an issue because "test-tool chmtime =-<seconds>" subtracts the
> <seconds> from the current system time, not the file's mtime. This
> is subtle: without the "=" it would modify it from the file's mtime.
> 
> Since we are assigning the offset values in the proper order (D to A)
> there isn't an issue if time ticks forward between these steps.

I wonder if that would run afoul of the same "mtime and system clock are not quite the same" issue we saw recently in [1].

I think it might not because we're only comparing mtimes set through the same mechanism (find the system time, subtract from it, and assign to mtime). If system time is monotonically increasing at any rate, that would produce the desired effect.

That said, it seems so easy to just give it a 10 second (or even 10 minute) spread to avoid any possibility of confusion.

-Peff
[1] https://lore.kernel.org/git/pull.725.git.git.1584125875550.gitgitgadget@gmail.com/
Junio C Hamano· Apr 2, 2020, 16:36 UTC · re: Jeff King · lore

Re: [PATCH v2 1/2] t5319: replace 'touch -m' with 'test-tool chmtime'

Jeff King <peff@peff.net> writes:
> On Wed, Apr 01, 2020 at 08:06:47PM -0400, Derrick Stolee wrote:
>
> I wonder if that would run afoul of the same "mtime and system clock are
> not quite the same" issue we saw recently in [1].

I had the exact thing in mind, which was why I suggested "chmtime" in the first place (the second reason was that it was easy to do relative time using the tool than "date + touch"), but I missed that the tool has a mode that uses timestamp relative to system clock.

> I think it might not because we're only comparing mtimes set
> through the same mechanism (find the system time, subtract from
> it, and assign to mtime). If system time is monotonically
> increasing at any rate, that would produce the desired effect.

But in this particular case, I think, even though setting the filesystem timestamp based on something that was obtained from the system clock feels as if we are asking for trouble, the values we later read back from the filesystem for executing the codepath being tested (namely, expiration logic) get compared with another value that is derived from from the system clock (i.e. expiration cutoff), so in that sense you might be able to argue that it is even more correct ;-)

Junio C Hamano· Apr 1, 2020, 21:05 UTC · re: Derrick Stolee via GitGitGadget · lore

Re: [PATCH v2 0/2] commit-graph: fix buggy --expire-time option

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>  * Added a file that should be kept on the other side of the expire time to
>    protect against off-by-one errors and future date errors.

Thanks for being extra careful. It is always a pleasure to read patches from you ;-)

Derrick Stolee· Apr 1, 2020, 23:33 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 0/2] commit-graph: fix buggy --expire-time option

On 4/1/2020 5:05 PM, Junio C Hamano wrote:
Show 7 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>>  * Added a file that should be kept on the other side of the expire time to
>>    protect against off-by-one errors and future date errors.
> 
> Thanks for being extra careful.  It is always a pleasure to read
> patches from you ;-)
I appreciate that! Credit for this suggestion goes to Kevin Willford.
-Stolee

← back to recent threads