git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations

From
Matheus Tavares Bernardino <matheus.bernardino@usp.br>
Date
Oct 20, 2020, 03:18 UTC
Message-ID
<CAHd-oW7=Bd+StX_t+6iuaas0SWzEdbQCj5aWgasoOYT2kfVw7g@mail.gmail.com>
In-Reply-To
<20201020013558.GA15198@google.com>
Hi, Jonathan
On Mon, Oct 19, 2020 at 10:36 PM Jonathan Nieder <jrnieder@gmail.com> wrote:
Show 12 quoted lines
>
> Hi,
>
> Matheus Tavares wrote:
>
> > Add tests to populate the working tree during clone and checkout using
> > the sequential and parallel modes, to confirm that they produce
> > identical results. Also test basic checkout mechanics, such as checking
> > for symlinks in the leading directories and the abidance to --force.
>
> Thanks for implementing parallel checkout!  I'm excited about the
> feature.  And thanks for including these tests.
Thanks for the comments and feedback :)
Show 10 quoted lines
> [...]
> > --- /dev/null
> > +++ b/t/lib-parallel-checkout.sh
> > @@ -0,0 +1,39 @@
> [...]
> > +# Runs `git -c checkout.workers=$1 -c checkout.thesholdForParallelism=$2 ${@:4}`
> > +# and checks that the number of workers spawned is equal to $3.
> > +git_pc()
>
> nit: what does git_pc mean?

The idea was "git w/ parallel-checkout". But I realize it may have gotten too abbreviated...

> Can this spell it out more verbosely, or
> could callers take on more of the burden?  (Perhaps it would make sense
> to use a helper that uses test_config to set the relevant configuration,
> and then the caller can use plain "git clone"?)

Hmm, it's possible, but I think we might end up with quite a lot of repetition (to always check that checkout spawned the right number of workers).

Show 9 quoted lines
> [...]
> > +     GIT_TRACE2="$(pwd)/trace" git \
> > +             -c checkout.workers=$workers \
> > +             -c checkout.thresholdForParallelism=$threshold \
> > +             -c advice.detachedHead=0 \
> > +             $@ &&
>
> $@ needs to be quoted, or else it will act like $* (and in particular it
> won't handle parameters with embedded spaces).
Nice catch, thanks! I will send a patch for this tomorrow.
Show 10 quoted lines
> > +
> > +     # Check that the expected number of workers has been used. Note that it
> > +     # can be different than the requested number in two cases: when the
> > +     # quantity of entries to be checked out is less than the number of
> > +     # workers; and when the threshold has not been reached.
> > +     #
> > +     local workers_in_trace=$(grep "child_start\[.\+\] git checkout--helper" trace | wc -l) &&
>
> Do we use grep's \+ operator in other tests?  I thought we preferred to
> use the more portable *, but it may be that I'm out of date.

Oh, I didn't know about the portability issue with \+. This is already in `next`, but I guess it's worth sending a follow-up patch to fix it, right? (I see we have a second \+ occurrence in t7508, which could be changed in the same patch.)

Show 13 quoted lines
> [...]
> > +# Verify that both the working tree and the index were created correctly
> > +verify_checkout()
> > +{
> > +     git -C $1 diff-index --quiet HEAD -- &&
> > +     git -C $1 diff-index --quiet --cached HEAD -- &&
> > +     git -C $1 status --porcelain >$1.status &&
> > +     test_must_be_empty $1.status
> > +}
>
> Like git_pc, this is not easy to take in at a glance.
>
> "$1" needs to be quoted if we are to handle paths with spaces.

Thanks, again :) Currently, this function doesn't get paths with spaces, but I agree that it's better to be cautious here.

Show 36 quoted lines
> [...]
> > --- /dev/null
> > +++ b/t/t2080-parallel-checkout-basics.sh
> > @@ -0,0 +1,197 @@
> > +#!/bin/sh
> > +
> > +test_description='parallel-checkout basics
> > +
> > +Ensure that parallel-checkout basically works on clone and checkout, spawning
> > +the required number of workers and correctly populating both the index and
> > +working tree.
> > +'
> > +
> > +TEST_NO_CREATE_REPO=1
> > +. ./test-lib.sh
> > +. "$TEST_DIRECTORY/lib-parallel-checkout.sh"
> > +
> > +# NEEDSWORK: cloning a SHA1 repo with GIT_TEST_DEFAULT_HASH set to "sha256"
> > +# currently produces a wrong result (See
> > +# https://lore.kernel.org/git/20200911151717.43475-1-matheus.bernardino@usp.br/).
> > +# So we skip the "parallel-checkout during clone" tests when this test flag is
> > +# set to "sha256". Remove this when the bug is fixed.
> > +#
> > +if test "$GIT_TEST_DEFAULT_HASH" = "sha256"
> > +then
> > +     skip_all="t2080 currently don't work with GIT_TEST_DEFAULT_HASH=sha256"
> > +     test_done
> > +fi
> > +
> > +R_BASE=$GIT_BUILD_DIR
> > +
> > +test_expect_success 'sequential clone' '
> > +     git_pc 1 0 0 clone --quiet -- $R_BASE r_sequential &&
>
> This fails when I run it when building from a tarball, which is
> presenting me from releasing this patch series to Debian experimental.

Sorry for the trouble :( It didn't occur to me, while writing the test, that it could also be run from the tarball.

> Can we use an artificial repo instead of git.git?  Using git.git as
> test data seems like a recipe for hard-to-reproduce test failures.

I think we could maybe drop these tests. There are already some similar tests below these, which use an artificial repository. The goal of using git.git in this section was to test parallel-checkout with a real-world repo, and hopefully catch errors that we might not see with small artificial ones. But you have a very valid concern, as well. Hmm, I'm not sure what is the best solution to this case. What do you think?

Previous: Junio C HamanoNext: Jonathan Nieder
Message 55 of 61 in “[RFC] Parallel checkout”
  1. 00/21 [RFC] Parallel checkoutMatheus Tavares, Aug 10, 2020
  2. 01/21 convert: make convert_attrs() and convert structs publicMatheus Tavares, Aug 10, 2020
  3. 02/21 convert: add [async_]convert_to_working_tree_ca() variantsMatheus Tavares, Aug 10, 2020
  4. 03/21 convert: add get_stream_filter_ca() variantMatheus Tavares, Aug 10, 2020
  5. 04/21 convert: add conv_attrs classificationMatheus Tavares, Aug 10, 2020
  6. 05/21 entry: extract a header file for entry.c functionsMatheus Tavares, Aug 10, 2020
  7. 06/21 entry: make fstat_output() and read_blob_entry() publicMatheus Tavares, Aug 10, 2020
  8. 07/21 entry: extract cache_entry update from write_entry()Matheus Tavares, Aug 10, 2020
  9. 08/21 entry: move conv_attrs lookup up to checkout_entry()Matheus Tavares, Aug 10, 2020
  10. 09/21 entry: add checkout_entry_ca() which takes preloaded conv_attrsMatheus Tavares, Aug 10, 2020
  11. 10/21 unpack-trees: add basic support for parallel checkoutMatheus Tavares, Aug 10, 2020
  12. 11/21 parallel-checkout: make it truly parallelMatheus Tavares, Aug 10, 2020
  13. Jeff HostetlerAug 19, 2020
  14. Matheus Tavares BernardinoAug 20, 2020
  15. Jeff HostetlerAug 20, 2020
  16. 12/21 parallel-checkout: add configuration optionsMatheus Tavares, Aug 10, 2020
  17. 13/21 parallel-checkout: support progress displayingMatheus Tavares, Aug 10, 2020
  18. 14/21 make_transient_cache_entry(): optionally alloc from mem_poolMatheus Tavares, Aug 10, 2020
  19. 15/21 builtin/checkout.c: complete parallel checkout supportMatheus Tavares, Aug 10, 2020
  20. 16/21 checkout-index: add parallel checkout supportMatheus Tavares, Aug 10, 2020
  21. 17/21 parallel-checkout: avoid stat() calls in workersMatheus Tavares, Aug 10, 2020
  22. 18/21 entry: use is_dir_sep() when checking leading dirsMatheus Tavares, Aug 10, 2020
  23. 19/21 symlinks: make has_dirs_only_path() track FL_NOENTMatheus Tavares, Aug 10, 2020
  24. 20/21 parallel-checkout: create leading dirs in workersMatheus Tavares, Aug 10, 2020
  25. 21/21 parallel-checkout: skip checking the working tree on cloneMatheus Tavares, Aug 10, 2020
  26. Jeff HostetlerAug 12, 2020
  27. 00/19 Parallel Checkout (part I)Matheus Tavares, Sep 22, 2020
  28. 01/19 convert: make convert_attrs() and convert structs publicMatheus Tavares, Sep 22, 2020
  29. 03/19 convert: add get_stream_filter_ca() variantMatheus Tavares, Sep 22, 2020
  30. 02/19 convert: add [async_]convert_to_working_tree_ca() variantsMatheus Tavares, Sep 22, 2020
  31. 04/19 convert: add conv_attrs classificationMatheus Tavares, Sep 22, 2020
  32. 05/19 entry: extract a header file for entry.c functionsMatheus Tavares, Sep 22, 2020
  33. 06/19 entry: make fstat_output() and read_blob_entry() publicMatheus Tavares, Sep 22, 2020
  34. 07/19 entry: extract cache_entry update from write_entry()Matheus Tavares, Sep 22, 2020
  35. 08/19 entry: move conv_attrs lookup up to checkout_entry()Matheus Tavares, Sep 22, 2020
  36. Jeff HostetlerOct 1, 2020
  37. Jeff HostetlerOct 1, 2020
  38. 09/19 entry: add checkout_entry_ca() which takes preloaded conv_attrsMatheus Tavares, Sep 22, 2020
  39. 10/19 unpack-trees: add basic support for parallel checkoutMatheus Tavares, Sep 22, 2020
  40. parallel-checkout: drop unused checkout state parameterJeff King, Oct 5, 2020
  41. Matheus Tavares BernardinoOct 5, 2020
  42. Jeff KingOct 5, 2020
  43. 11/19 parallel-checkout: make it truly parallelMatheus Tavares, Sep 22, 2020
  44. Martin ÅgrenSep 29, 2020
  45. Matheus Tavares BernardinoSep 30, 2020
  46. 12/19 parallel-checkout: support progress displayingMatheus Tavares, Sep 22, 2020
  47. 13/19 make_transient_cache_entry(): optionally alloc from mem_poolMatheus Tavares, Sep 22, 2020
  48. 14/19 builtin/checkout.c: complete parallel checkout supportMatheus Tavares, Sep 22, 2020
  49. 15/19 checkout-index: add parallel checkout supportMatheus Tavares, Sep 22, 2020
  50. 16/19 parallel-checkout: add tests for basic operationsMatheus Tavares, Sep 22, 2020
  51. Jonathan NiederOct 20, 2020
  52. Taylor BlauOct 20, 2020
  53. Matheus Tavares BernardinoOct 20, 2020
  54. Junio C HamanoOct 20, 2020
  55. Matheus Tavares BernardinoOct 20, 2020
  56. Jonathan NiederOct 20, 2020
  57. Junio C HamanoOct 20, 2020
  58. 17/19 parallel-checkout: add tests related to clone collisionsMatheus Tavares, Sep 22, 2020
  59. 18/19 parallel-checkout: add tests related to .gitattributesMatheus Tavares, Sep 22, 2020
  60. 19/19 ci: run test round with parallel-checkout enabledMatheus Tavares, Sep 22, 2020
  61. Jeff HostetlerOct 1, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.