git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations

From
Jonathan Nieder <jrnieder@gmail.com>
Date
Oct 20, 2020, 01:35 UTC
Message-ID
<20201020013558.GA15198@google.com>
In-Reply-To
<64b41d537e68a45f2bb0a0c3078f2cd314b5a57d.1600814153.git.matheus.bernardino@usp.br>
Hi,
Matheus Tavares wrote:
> Add tests to populate the working tree during clone and checkout using
> the sequential and parallel modes, to confirm that they produce
> identical results. Also test basic checkout mechanics, such as checking
> for symlinks in the leading directories and the abidance to --force.

Thanks for implementing parallel checkout! I'm excited about the feature. And thanks for including these tests.

[...]
> --- /dev/null
> +++ b/t/lib-parallel-checkout.sh
> @@ -0,0 +1,39 @@
[...]
> +# Runs `git -c checkout.workers=$1 -c checkout.thesholdForParallelism=$2 ${@:4}`
> +# and checks that the number of workers spawned is equal to $3.
> +git_pc()

nit: what does git_pc mean? Can this spell it out more verbosely, or could callers take on more of the burden? (Perhaps it would make sense to use a helper that uses test_config to set the relevant configuration, and then the caller can use plain "git clone"?)

[...]
Show 5 quoted lines
> +	GIT_TRACE2="$(pwd)/trace" git \
> +		-c checkout.workers=$workers \
> +		-c checkout.thresholdForParallelism=$threshold \
> +		-c advice.detachedHead=0 \
> +		$@ &&

$@ needs to be quoted, or else it will act like $* (and in particular it won't handle parameters with embedded spaces).

Show 7 quoted lines
> +
> +	# Check that the expected number of workers has been used. Note that it
> +	# can be different than the requested number in two cases: when the
> +	# quantity of entries to be checked out is less than the number of
> +	# workers; and when the threshold has not been reached.
> +	#
> +	local workers_in_trace=$(grep "child_start\[.\+\] git checkout--helper" trace | wc -l) &&

Do we use grep's \+ operator in other tests? I thought we preferred to use the more portable *, but it may be that I'm out of date.

[...]
Show 8 quoted lines
> +# Verify that both the working tree and the index were created correctly
> +verify_checkout()
> +{
> +	git -C $1 diff-index --quiet HEAD -- &&
> +	git -C $1 diff-index --quiet --cached HEAD -- &&
> +	git -C $1 status --porcelain >$1.status &&
> +	test_must_be_empty $1.status
> +}
Like git_pc, this is not easy to take in at a glance.
"$1" needs to be quoted if we are to handle paths with spaces.
[...]
Show 32 quoted lines
> --- /dev/null
> +++ b/t/t2080-parallel-checkout-basics.sh
> @@ -0,0 +1,197 @@
> +#!/bin/sh
> +
> +test_description='parallel-checkout basics
> +
> +Ensure that parallel-checkout basically works on clone and checkout, spawning
> +the required number of workers and correctly populating both the index and
> +working tree.
> +'
> +
> +TEST_NO_CREATE_REPO=1
> +. ./test-lib.sh
> +. "$TEST_DIRECTORY/lib-parallel-checkout.sh"
> +
> +# NEEDSWORK: cloning a SHA1 repo with GIT_TEST_DEFAULT_HASH set to "sha256"
> +# currently produces a wrong result (See
> +# https://lore.kernel.org/git/20200911151717.43475-1-matheus.bernardino@usp.br/).
> +# So we skip the "parallel-checkout during clone" tests when this test flag is
> +# set to "sha256". Remove this when the bug is fixed.
> +#
> +if test "$GIT_TEST_DEFAULT_HASH" = "sha256"
> +then
> +	skip_all="t2080 currently don't work with GIT_TEST_DEFAULT_HASH=sha256"
> +	test_done
> +fi
> +
> +R_BASE=$GIT_BUILD_DIR
> +
> +test_expect_success 'sequential clone' '
> +	git_pc 1 0 0 clone --quiet -- $R_BASE r_sequential &&

This fails when I run it when building from a tarball, which is presenting me from releasing this patch series to Debian experimental.

Can we use an artificial repo instead of git.git? Using git.git as test data seems like a recipe for hard-to-reproduce test failures.

Thanks and hope that helps, Jonathan

Previous: Matheus TavaresNext: Taylor Blau
Message 51 of 61 in “[RFC] Parallel checkout”
  1. 00/21 [RFC] Parallel checkoutMatheus Tavares, Aug 10, 2020
  2. 01/21 convert: make convert_attrs() and convert structs publicMatheus Tavares, Aug 10, 2020
  3. 02/21 convert: add [async_]convert_to_working_tree_ca() variantsMatheus Tavares, Aug 10, 2020
  4. 03/21 convert: add get_stream_filter_ca() variantMatheus Tavares, Aug 10, 2020
  5. 04/21 convert: add conv_attrs classificationMatheus Tavares, Aug 10, 2020
  6. 05/21 entry: extract a header file for entry.c functionsMatheus Tavares, Aug 10, 2020
  7. 06/21 entry: make fstat_output() and read_blob_entry() publicMatheus Tavares, Aug 10, 2020
  8. 07/21 entry: extract cache_entry update from write_entry()Matheus Tavares, Aug 10, 2020
  9. 08/21 entry: move conv_attrs lookup up to checkout_entry()Matheus Tavares, Aug 10, 2020
  10. 09/21 entry: add checkout_entry_ca() which takes preloaded conv_attrsMatheus Tavares, Aug 10, 2020
  11. 10/21 unpack-trees: add basic support for parallel checkoutMatheus Tavares, Aug 10, 2020
  12. 11/21 parallel-checkout: make it truly parallelMatheus Tavares, Aug 10, 2020
  13. Jeff HostetlerAug 19, 2020
  14. Matheus Tavares BernardinoAug 20, 2020
  15. Jeff HostetlerAug 20, 2020
  16. 12/21 parallel-checkout: add configuration optionsMatheus Tavares, Aug 10, 2020
  17. 13/21 parallel-checkout: support progress displayingMatheus Tavares, Aug 10, 2020
  18. 14/21 make_transient_cache_entry(): optionally alloc from mem_poolMatheus Tavares, Aug 10, 2020
  19. 15/21 builtin/checkout.c: complete parallel checkout supportMatheus Tavares, Aug 10, 2020
  20. 16/21 checkout-index: add parallel checkout supportMatheus Tavares, Aug 10, 2020
  21. 17/21 parallel-checkout: avoid stat() calls in workersMatheus Tavares, Aug 10, 2020
  22. 18/21 entry: use is_dir_sep() when checking leading dirsMatheus Tavares, Aug 10, 2020
  23. 19/21 symlinks: make has_dirs_only_path() track FL_NOENTMatheus Tavares, Aug 10, 2020
  24. 20/21 parallel-checkout: create leading dirs in workersMatheus Tavares, Aug 10, 2020
  25. 21/21 parallel-checkout: skip checking the working tree on cloneMatheus Tavares, Aug 10, 2020
  26. Jeff HostetlerAug 12, 2020
  27. 00/19 Parallel Checkout (part I)Matheus Tavares, Sep 22, 2020
  28. 01/19 convert: make convert_attrs() and convert structs publicMatheus Tavares, Sep 22, 2020
  29. 03/19 convert: add get_stream_filter_ca() variantMatheus Tavares, Sep 22, 2020
  30. 02/19 convert: add [async_]convert_to_working_tree_ca() variantsMatheus Tavares, Sep 22, 2020
  31. 04/19 convert: add conv_attrs classificationMatheus Tavares, Sep 22, 2020
  32. 05/19 entry: extract a header file for entry.c functionsMatheus Tavares, Sep 22, 2020
  33. 06/19 entry: make fstat_output() and read_blob_entry() publicMatheus Tavares, Sep 22, 2020
  34. 07/19 entry: extract cache_entry update from write_entry()Matheus Tavares, Sep 22, 2020
  35. 08/19 entry: move conv_attrs lookup up to checkout_entry()Matheus Tavares, Sep 22, 2020
  36. Jeff HostetlerOct 1, 2020
  37. Jeff HostetlerOct 1, 2020
  38. 09/19 entry: add checkout_entry_ca() which takes preloaded conv_attrsMatheus Tavares, Sep 22, 2020
  39. 10/19 unpack-trees: add basic support for parallel checkoutMatheus Tavares, Sep 22, 2020
  40. parallel-checkout: drop unused checkout state parameterJeff King, Oct 5, 2020
  41. Matheus Tavares BernardinoOct 5, 2020
  42. Jeff KingOct 5, 2020
  43. 11/19 parallel-checkout: make it truly parallelMatheus Tavares, Sep 22, 2020
  44. Martin ÅgrenSep 29, 2020
  45. Matheus Tavares BernardinoSep 30, 2020
  46. 12/19 parallel-checkout: support progress displayingMatheus Tavares, Sep 22, 2020
  47. 13/19 make_transient_cache_entry(): optionally alloc from mem_poolMatheus Tavares, Sep 22, 2020
  48. 14/19 builtin/checkout.c: complete parallel checkout supportMatheus Tavares, Sep 22, 2020
  49. 15/19 checkout-index: add parallel checkout supportMatheus Tavares, Sep 22, 2020
  50. 16/19 parallel-checkout: add tests for basic operationsMatheus Tavares, Sep 22, 2020
  51. Jonathan NiederOct 20, 2020
  52. Taylor BlauOct 20, 2020
  53. Matheus Tavares BernardinoOct 20, 2020
  54. Junio C HamanoOct 20, 2020
  55. Matheus Tavares BernardinoOct 20, 2020
  56. Jonathan NiederOct 20, 2020
  57. Junio C HamanoOct 20, 2020
  58. 17/19 parallel-checkout: add tests related to clone collisionsMatheus Tavares, Sep 22, 2020
  59. 18/19 parallel-checkout: add tests related to .gitattributesMatheus Tavares, Sep 22, 2020
  60. 19/19 ci: run test round with parallel-checkout enabledMatheus Tavares, Sep 22, 2020
  61. Jeff HostetlerOct 1, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.