git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 04/15] reftable/stack: gracefully handle failed auto-compaction due to locks

From
Karthik Nayak <karthik.188@gmail.com>
Date
Mar 20, 2024, 22:22 UTC
Message-ID
<CAOLa=ZRxt7rwmU4UJAZra605zyqk5QJD5OD1oM_BidRpJbZP1w@mail.gmail.com>
In-Reply-To
<37a18b91ca998ca5ad27d17e86d286040b6e6ee1.1710706118.git.ps@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
Show 5 quoted lines
> Whenever we commit a new table to the reftable stack we will end up
> invoking auto-compaction of the stack to keep the total number of tables
> at bay. This auto-compaction may fail though in case at least one of the
> tables which we are about to compact is locked. This is indicated by the
> compaction function returning a positive value. We do not handle this
We no longer return a positive value, do we?
Show 29 quoted lines
> case though, and thus bubble that return value up the calling chain,
> which will ultimately cause a failure.
>
> Fix this bug by handling positive return values.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  reftable/stack.c           | 13 ++++++++++++-
>  t/t0610-reftable-basics.sh | 20 ++++++++++++++++++++
>  2 files changed, 32 insertions(+), 1 deletion(-)
>
> diff --git a/reftable/stack.c b/reftable/stack.c
> index ba15c48ddd..dd5160d751 100644
> --- a/reftable/stack.c
> +++ b/reftable/stack.c
> @@ -680,8 +680,19 @@ int reftable_addition_commit(struct reftable_addition *add)
>  	if (err)
>  		goto done;
>
> -	if (!add->stack->disable_auto_compact)
> +	if (!add->stack->disable_auto_compact) {
> +		/*
> +		 * Auto-compact the stack to keep the number of tables in
> +		 * control. Note that we explicitly ignore locking issues which
> +		 * may indicate that a concurrent process is already trying to
> +		 * compact tables. This is fine, so we simply ignore that error
> +		 * condition.
> +		 */
>
Nit: The last two sentences are somewhat the same, it'd be perhaps nicer
to explain why "this is fine".
Show 35 quoted lines
>  		err = reftable_stack_auto_compact(add->stack);
> +		if (err < 0 && err != REFTABLE_LOCK_ERROR)
> +			goto done;
> +		err = 0;
> +	}
>
>  done:
>  	reftable_addition_close(add);
> diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh
> index 686781192e..5f2f9baa9b 100755
> --- a/t/t0610-reftable-basics.sh
> +++ b/t/t0610-reftable-basics.sh
> @@ -340,6 +340,26 @@ test_expect_success 'ref transaction: empty transaction in empty repo' '
>  	EOF
>  '
>
> +test_expect_success 'ref transaction: fails gracefully when auto compaction fails' '
> +	test_when_finished "rm -rf repo" &&
> +	git init repo &&
> +	(
> +		cd repo &&
> +
> +		test_commit A &&
> +		for i in $(test_seq 10)
> +		do
> +			git branch branch-$i &&
> +			for table in .git/reftable/*.ref
> +			do
> +				touch "$table.lock" || exit 1
> +			done ||
> +			exit 1
> +		done &&
> +		test_line_count = 13 .git/reftable/tables.list
> +	)
> +'

I'm not sure about the testing setup for reftables, but maybe if we moved this to the unit tests, we could've also checked the `reftable_stack_compaction_stats(st)->failures` value. This is fine, but it doesn't really tell us if the compaction was even attempted.

Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 10 of 49 in “refs: introduce `--auto` to pack refs as needed”
  1. 00/15 refs: introduce `--auto` to pack refs as neededPatrick Steinhardt, Mar 18, 2024
  2. 01/15 reftable/stack: fix error handling in `reftable_stack_init_addition()`Patrick Steinhardt, Mar 18, 2024
  3. Karthik NayakMar 20, 2024
  4. Patrick SteinhardtMar 25, 2024
  5. 02/15 reftable/error: discern locked/outdated errorsPatrick Steinhardt, Mar 18, 2024
  6. 03/15 reftable/stack: use error codes when locking fails during compactionPatrick Steinhardt, Mar 18, 2024
  7. Karthik NayakMar 20, 2024
  8. Patrick SteinhardtMar 25, 2024
  9. 04/15 reftable/stack: gracefully handle failed auto-compaction due to locksPatrick Steinhardt, Mar 18, 2024
  10. Karthik NayakMar 20, 2024
  11. Patrick SteinhardtMar 25, 2024
  12. 05/15 refs/reftable: print errors on compaction failurePatrick Steinhardt, Mar 18, 2024
  13. 06/15 t/helper: drop pack-refs wrapperPatrick Steinhardt, Mar 18, 2024
  14. 07/15 refs: move `struct pack_refs_opts` to where it's usedPatrick Steinhardt, Mar 18, 2024
  15. 08/15 refs: remove `PACK_REFS_ALL` flagPatrick Steinhardt, Mar 18, 2024
  16. 09/15 refs/reftable: expose auto compaction via new flagPatrick Steinhardt, Mar 18, 2024
  17. 10/15 builtin/pack-refs: release allocated memoryPatrick Steinhardt, Mar 18, 2024
  18. Karthik NayakMar 20, 2024
  19. Patrick SteinhardtMar 25, 2024
  20. 11/15 builtin/pack-refs: introduce new "--auto" flagPatrick Steinhardt, Mar 18, 2024
  21. 12/15 builtin/gc: move `struct maintenance_run_opts`Patrick Steinhardt, Mar 18, 2024
  22. 13/15 t6500: extract objects with "17" prefixPatrick Steinhardt, Mar 18, 2024
  23. 14/15 builtin/gc: forward git-gc(1)'s `--auto` flag when packing refsPatrick Steinhardt, Mar 18, 2024
  24. Karthik NayakMar 20, 2024
  25. Patrick SteinhardtMar 25, 2024
  26. 15/15 builtin/gc: pack refs when using `git maintenance run --auto`Patrick Steinhardt, Mar 18, 2024
  27. Karthik NayakMar 20, 2024
  28. Patrick SteinhardtMar 25, 2024
  29. Karthik NayakMar 20, 2024
  30. Patrick SteinhardtMar 25, 2024
  31. 00/15 refs: introduce `--auto` to pack refs as neededPatrick Steinhardt, Mar 25, 2024
  32. 01/15 reftable/stack: fix error handling in `reftable_stack_init_addition()`Patrick Steinhardt, Mar 25, 2024
  33. 02/15 reftable/error: discern locked/outdated errorsPatrick Steinhardt, Mar 25, 2024
  34. 03/15 reftable/stack: use error codes when locking fails during compactionPatrick Steinhardt, Mar 25, 2024
  35. 04/15 reftable/stack: gracefully handle failed auto-compaction due to locksPatrick Steinhardt, Mar 25, 2024
  36. Patrick SteinhardtMar 25, 2024
  37. 05/15 refs/reftable: print errors on compaction failurePatrick Steinhardt, Mar 25, 2024
  38. 06/15 t/helper: drop pack-refs wrapperPatrick Steinhardt, Mar 25, 2024
  39. 07/15 refs: move `struct pack_refs_opts` to where it's usedPatrick Steinhardt, Mar 25, 2024
  40. 08/15 refs: remove `PACK_REFS_ALL` flagPatrick Steinhardt, Mar 25, 2024
  41. 09/15 refs/reftable: expose auto compaction via new flagPatrick Steinhardt, Mar 25, 2024
  42. 10/15 builtin/pack-refs: release allocated memoryPatrick Steinhardt, Mar 25, 2024
  43. 11/15 builtin/pack-refs: introduce new "--auto" flagPatrick Steinhardt, Mar 25, 2024
  44. 12/15 builtin/gc: move `struct maintenance_run_opts`Patrick Steinhardt, Mar 25, 2024
  45. 13/15 t6500: extract objects with "17" prefixPatrick Steinhardt, Mar 25, 2024
  46. 14/15 builtin/gc: forward git-gc(1)'s `--auto` flag when packing refsPatrick Steinhardt, Mar 25, 2024
  47. 15/15 builtin/gc: pack refs when using `git maintenance run --auto`Patrick Steinhardt, Mar 25, 2024
  48. Karthik NayakMar 25, 2024
  49. Junio C HamanoMar 25, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.