{"thread":{"id":"54015","subject":"[RFC PATCH 00/21] [RFC] Parallel checkout","startedAt":"2020-08-10T21:34:17Z","lastAt":"2020-10-20T19:15:04Z","messageCount":61,"participants":["Matheus Tavares","Jeff Hostetler","Matheus Tavares Bernardino","Martin Ågren","Jeff King","Jonathan Nieder","Taylor Blau","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":21},"messages":[{"id":"403283","messageId":"cover.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":null,"subject":"[RFC PATCH 00/21] [RFC] Parallel checkout","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:08Z","receivedAt":"2020-08-10T21:34:17Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This series adds parallel workers to the checkout machinery. The cache\nentries are distributed among helper processes which are responsible for\nreading, filtering and writing the blobs to the working tree. This\nshould benefit all commands that call unpack_trees() or check_updates(),\nsuch as: checkout, clone, sparse-checkout, checkout-index, etc.\n\nThis proposal is based on two previous ones, by Duy [1] and Jeff [2]. It\nuses some of the patches from these two series, with additional changes.\nThe final parallel version was benchmarked during three operations with\ncold cache in the linux repo: cloning v5.8, checking out v5.8 from\nv2.6.15 and checking out v5.8 from v5.7. The three tables below show the\nmean run times and standard deviations for 5 runs in: a local file\nsystem, a Linux NFS server and Amazon EFS. The number of workers was\nchosen based on what produces the best result for each case.\n\nLocal:\n\n Clone Checkout I Checkout II\nSequential 8.180 s ± 0.021 s 6.936 s ± 0.030 s 2.585 s ± 0.005 s\n10 workers 3.406 s ± 0.187 s 2.164 s ± 0.033 s 1.050 s ± 0.021 s\nSpeedup 2.40 ± 0.13 3.21 ± 0.05 2.46 ± 0.05\n\nLinux NFS server (v4.1, on EBS, single availability zone):\n\n Clone Checkout I Checkout II\nSequential 208.069 s ± 2.522 s 198.610 s ± 1.979 s 54.376 s ± 1.333 s\n32 workers 58.170 s ± 0.648 s 56.471 s ± 0.093 s 22.311 s ± 0.220 s\nSpeedup 3.58 ± 0.06 3.52 ± 0.04 2.44 ± 0.06\n\nEFS (v4.1, replicated over multiple availability zones):\n\n Clone Checkout I Checkout II\nSequential 1143.655 s ± 11.819 s 1277.891 s ± 10.481 s 396.891 s ± 7.505 s\n64 workers 94.778 s ± 4.984 s 201.674 s ± 2.286 s 149.951 s ± 12.895 s\nSpeedup 12.07 ± 0.65 6.34 ± 0.09 2.65 ± 0.23\n\n\nI also repeated the local benchmark tests including pc-p4-core [2], to\nmake sure the new proposal doesn't have performance regressions:\n\n Clone Checkout I Checkout II\npc-p4-core 3.746 s ± 0.044 s 3.158 s ± 0.041 s 1.597 s ± 0.019 s\n10 workers 3.595 s ± 0.111 s 2.263 s ± 0.027 s 1.098 s ± 0.023 s\nSpeedup 1.04 ± 0.03 1.40 ± 0.02 1.45 ± 0.04\n\n\nThe series is divided in three blocks:\n\n- The first 9 patches are preparatory steps in convert.c and entry.c.\n- The middle 7 actually implement parallel checkout.\n- The last 5 are ideas for further optimization of the parallel version.\n They don't bring a huge difference in local file systems (e.g. linux\n clone is only 1.04x faster than the previous parallel code), but in\n distributed file systems, there is a significant difference: 1.15x\n faster in NFS and 1.83x faster in Amazon EFS. (For comparison, the\n timings before these additional patches can be seen in the commit\n message of patch 11.)\n\nThe first 4 patches come from [2]. I couldn't get in touch with Jeff yet\nand ask for his approval on then, so I didn't include his Signed-off-by,\nfor the time being.\n\nNote: we probably want to add some extra validation and perf tests. But,\nfor now, parallel checkout is enabled by default in this series (with no\nthreshold on the minimum number of entries), so the test base is already\nexercising the parallel code. (see [3])\n\nThere are some additional optimization possibilities I want to\nexperiment with later, such as:\n- Work stealing, to better re-distribute tasks in case of non-uniform\n work loads. Duy already proposed a way to implement this in his\n original series.\n- Add a --stat option to checkout--helper, to avoid calling stat() when\n state.refresh_cache is false.\n- Try to detect when a repository is in NFS/EFS to automatically use a\n higher number of workers, as this showed out to be very effective in\n distributed file systems.\n\n[1]: https://gitlab.com/pclouds/git/-/commits/parallel-checkout\n[2]: https://github.com/jeffhostetler/git/commits/pc-p4-core\n[3]: https://github.com/matheustavares/git/actions/runs/203036951 \n\n----\nNotes on the benchmarks:\n\nLocal tests were executed in an i7-7700HQ (4 cores with hyper-threading)\nrunning Manjaro Linux, with SSD. NFS and EFS tests were executed in an\nAmazon EC2 c5n.large instance, with 2 vCPUs. The Linux NFS server was\nrunning on a m6g.large instance with 1 TB, EBS GP2 volume. For\npc-p4-core tests, I used the set of parameters that resulted in the\nfasted mean execution (of 5 runs) on my machine, which was:\n- For clone: async mode, 22 helpers, 2 writers, 10 preloading slots\n- For checkout I: async mode, 20 helpers, 2 writers, 20 preloading slots\n- For checkout II: sync mode, 4 helpers, 2 writers, 30 preloading slots\n\n\nJeff Hostetler (4):\n convert: make convert_attrs() and convert structs public\n convert: add [async_]convert_to_working_tree_ca() variants\n convert: add get_stream_filter_ca() variant\n convert: add conv_attrs classification\n\nMatheus Tavares (17):\n entry: extract a header file for entry.c functions\n entry: make fstat_output() and read_blob_entry() public\n entry: extract cache_entry update from write_entry()\n entry: move conv_attrs lookup up to checkout_entry()\n entry: add checkout_entry_ca() which takes preloaded conv_attrs\n unpack-trees: add basic support for parallel checkout\n parallel-checkout: make it truly parallel\n parallel-checkout: add configuration options\n parallel-checkout: support progress displaying\n make_transient_cache_entry(): optionally alloc from mem_pool\n builtin/checkout.c: complete parallel checkout support\n checkout-index: add parallel checkout support\n parallel-checkout: avoid stat() calls in workers\n entry: use is_dir_sep() when checking leading dirs\n symlinks: make has_dirs_only_path() track FL_NOENT\n parallel-checkout: create leading dirs in workers\n parallel-checkout: skip checking the working tree on clone\n\n .gitignore | 1 +\n Documentation/config/checkout.txt | 16 +\n Makefile | 2 +\n apply.c | 1 +\n builtin.h | 1 +\n builtin/checkout--helper.c | 135 +++++++\n builtin/checkout-index.c | 17 +\n builtin/checkout.c | 21 +-\n builtin/difftool.c | 3 +-\n cache.h | 35 +-\n convert.c | 121 +++---\n convert.h | 68 ++++\n entry.c | 180 +++++++--\n entry.h | 54 +++\n git.c | 2 +\n parallel-checkout.c | 611 ++++++++++++++++++++++++++++++\n parallel-checkout.h | 103 +++++\n read-cache.c | 12 +-\n symlinks.c | 42 +-\n unpack-trees.c | 24 +-\n 20 files changed, 1292 insertions(+), 157 deletions(-)\n create mode 100644 builtin/checkout--helper.c\n create mode 100644 entry.h\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\n-- \n2.27.0\n\n"},{"id":"403284","messageId":"b13165f10859b97412cb94828ea139136cdccfb2.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 01/21] convert: make convert_attrs() and convert structs public","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:09Z","receivedAt":"2020-08-10T21:34:27Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Jeff Hostetler \n\nMove convert_attrs() declaration from convert.c to convert.h, together\nwith the conv_attrs struct and the crlf_action enum. This function and\nthe data structures will be used outside convert.c in the upcoming\nparallel checkout implementation.\n\n[matheus.bernardino: squash and reword msg]\nSigned-off-by: Matheus Tavares \n---\n convert.c | 23 ++---------------------\n convert.h | 24 ++++++++++++++++++++++++\n 2 files changed, 26 insertions(+), 21 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 572449825c..9710d770dc 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -24,17 +24,6 @@\n #define CONVERT_STAT_BITS_TXT_CRLF 0x2\n #define CONVERT_STAT_BITS_BIN 0x4\n \n-enum crlf_action {\n-\tCRLF_UNDEFINED,\n-\tCRLF_BINARY,\n-\tCRLF_TEXT,\n-\tCRLF_TEXT_INPUT,\n-\tCRLF_TEXT_CRLF,\n-\tCRLF_AUTO,\n-\tCRLF_AUTO_INPUT,\n-\tCRLF_AUTO_CRLF\n-};\n-\n struct text_stat {\n \t/* NUL, CR, LF and CRLF counts */\n \tunsigned nul, lonecr, lonelf, crlf;\n@@ -1300,18 +1289,10 @@ static int git_path_check_ident(struct attr_check_item *check)\n \treturn !!ATTR_TRUE(value);\n }\n \n-struct conv_attrs {\n-\tstruct convert_driver *drv;\n-\tenum crlf_action attr_action; /* What attr says */\n-\tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n-\tint ident;\n-\tconst char *working_tree_encoding; /* Supported encoding or default encoding if NULL */\n-};\n-\n static struct attr_check *check;\n \n-static void convert_attrs(const struct index_state *istate,\n-\t\t\t struct conv_attrs *ca, const char *path)\n+void convert_attrs(const struct index_state *istate,\n+\t\t struct conv_attrs *ca, const char *path)\n {\n \tstruct attr_check_item *ccheck = NULL;\n \ndiff --git a/convert.h b/convert.h\nindex e29d1026a6..aeb4a1be9a 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -37,6 +37,27 @@ enum eol {\n #endif\n };\n \n+enum crlf_action {\n+\tCRLF_UNDEFINED,\n+\tCRLF_BINARY,\n+\tCRLF_TEXT,\n+\tCRLF_TEXT_INPUT,\n+\tCRLF_TEXT_CRLF,\n+\tCRLF_AUTO,\n+\tCRLF_AUTO_INPUT,\n+\tCRLF_AUTO_CRLF\n+};\n+\n+struct convert_driver;\n+\n+struct conv_attrs {\n+\tstruct convert_driver *drv;\n+\tenum crlf_action attr_action; /* What attr says */\n+\tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n+\tint ident;\n+\tconst char *working_tree_encoding; /* Supported encoding or default encoding if NULL */\n+};\n+\n enum ce_delay_state {\n \tCE_NO_DELAY = 0,\n \tCE_CAN_DELAY = 1,\n@@ -102,6 +123,9 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n int would_convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t\t const char *path);\n \n+void convert_attrs(const struct index_state *istate,\n+\t\t struct conv_attrs *ca, const char *path);\n+\n /*\n * Initialize the checkout metadata with the given values. Any argument may be\n * NULL if it is not applicable. The treeish should be a commit if that is\n-- \n2.27.0\n\n"},{"id":"403285","messageId":"b5cef40e2ac256b9001d3bf74e2122cbb4cbd375.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 02/21] convert: add [async_]convert_to_working_tree_ca() variants","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:10Z","receivedAt":"2020-08-10T21:34:36Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Jeff Hostetler \n\nSeparate the attribute gathering from the actual conversion by adding\n_ca() variants of the conversion functions. These variants receive a\nprecomputed 'struct conv_attrs', not relying, thus, on a index state.\nThey will be used in a future patch adding parallel checkout support,\nfor two reasons:\n\n- We will already load the conversion attributes in checkout_entry(),\n before conversion, to decide whether a path is eligible for parallel\n checkout. Therefore, it would be wasteful to load them again later,\n for the actual conversion.\n\n- The parallel workers will be responsible for reading, converting and\n writing blobs to the working tree. They won't have access to the main\n process' index state, so they cannot load the attributes. Instead,\n they will receive the preloaded ones and call the _ca() variant of\n the conversion functions. Furthermore, the attributes machinery is\n optimized to handle paths in sequential order, so it's better to leave\n it for the main process, anyway.\n\n[matheus.bernardino: squash, remove one function definition and reword]\nSigned-off-by: Matheus Tavares \n---\n convert.c | 50 ++++++++++++++++++++++++++++++++++++--------------\n convert.h | 9 +++++++++\n 2 files changed, 45 insertions(+), 14 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 9710d770dc..757dc2585c 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -1450,7 +1450,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tident_to_git(dst->buf, dst->len, dst, ca.ident);\n }\n \n-static int convert_to_working_tree_internal(const struct index_state *istate,\n+static int convert_to_working_tree_internal(const struct conv_attrs *ca,\n \t\t\t\t\t const char *path, const char *src,\n \t\t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t\t int normalizing,\n@@ -1458,11 +1458,8 @@ static int convert_to_working_tree_internal(const struct index_state *istate,\n \t\t\t\t\t struct delayed_checkout *dco)\n {\n \tint ret = 0, ret_filter = 0;\n-\tstruct conv_attrs ca;\n-\n-\tconvert_attrs(istate, &ca, path);\n \n-\tret |= ident_to_worktree(src, len, dst, ca.ident);\n+\tret |= ident_to_worktree(src, len, dst, ca->ident);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n@@ -1472,24 +1469,24 @@ static int convert_to_working_tree_internal(const struct index_state *istate,\n \t * is a smudge or process filter (even if the process filter doesn't\n \t * support smudge). The filters might expect CRLFs.\n \t */\n-\tif ((ca.drv && (ca.drv->smudge || ca.drv->process)) || !normalizing) {\n-\t\tret |= crlf_to_worktree(src, len, dst, ca.crlf_action);\n+\tif ((ca->drv && (ca->drv->smudge || ca->drv->process)) || !normalizing) {\n+\t\tret |= crlf_to_worktree(src, len, dst, ca->crlf_action);\n \t\tif (ret) {\n \t\t\tsrc = dst->buf;\n \t\t\tlen = dst->len;\n \t\t}\n \t}\n \n-\tret |= encode_to_worktree(path, src, len, dst, ca.working_tree_encoding);\n+\tret |= encode_to_worktree(path, src, len, dst, ca->working_tree_encoding);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n \n \tret_filter = apply_filter(\n-\t\tpath, src, len, -1, dst, ca.drv, CAP_SMUDGE, meta, dco);\n-\tif (!ret_filter && ca.drv && ca.drv->required)\n-\t\tdie(_(\"%s: smudge filter %s failed\"), path, ca.drv->name);\n+\t\tpath, src, len, -1, dst, ca->drv, CAP_SMUDGE, meta, dco);\n+\tif (!ret_filter && ca->drv && ca->drv->required)\n+\t\tdie(_(\"%s: smudge filter %s failed\"), path, ca->drv->name);\n \n \treturn ret | ret_filter;\n }\n@@ -1500,7 +1497,9 @@ int async_convert_to_working_tree(const struct index_state *istate,\n \t\t\t\t const struct checkout_metadata *meta,\n \t\t\t\t void *dco)\n {\n-\treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, dco);\n+\tstruct conv_attrs ca;\n+\tconvert_attrs(istate, &ca, path);\n+\treturn convert_to_working_tree_internal(&ca, path, src, len, dst, 0, meta, dco);\n }\n \n int convert_to_working_tree(const struct index_state *istate,\n@@ -1508,13 +1507,36 @@ int convert_to_working_tree(const struct index_state *istate,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta)\n {\n-\treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, NULL);\n+\tstruct conv_attrs ca;\n+\tconvert_attrs(istate, &ca, path);\n+\treturn convert_to_working_tree_internal(&ca, path, src, len, dst, 0, meta, NULL);\n+}\n+\n+int async_convert_to_working_tree_ca(const struct conv_attrs *ca,\n+\t\t\t\t const char *path, const char *src,\n+\t\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t\t const struct checkout_metadata *meta,\n+\t\t\t\t void *dco)\n+{\n+\treturn convert_to_working_tree_internal(ca, path, src, len, dst, 0, meta, dco);\n+}\n+\n+int convert_to_working_tree_ca(const struct conv_attrs *ca,\n+\t\t\t const char *path, const char *src,\n+\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t const struct checkout_metadata *meta)\n+{\n+\treturn convert_to_working_tree_internal(ca, path, src, len, dst, 0, meta, NULL);\n }\n \n int renormalize_buffer(const struct index_state *istate, const char *path,\n \t\t const char *src, size_t len, struct strbuf *dst)\n {\n-\tint ret = convert_to_working_tree_internal(istate, path, src, len, dst, 1, NULL, NULL);\n+\tstruct conv_attrs ca;\n+\tint ret;\n+\n+\tconvert_attrs(istate, &ca, path);\n+\tret = convert_to_working_tree_internal(&ca, path, src, len, dst, 1, NULL, NULL);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\ndiff --git a/convert.h b/convert.h\nindex aeb4a1be9a..46d537d1ae 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -100,11 +100,20 @@ int convert_to_working_tree(const struct index_state *istate,\n \t\t\t const char *path, const char *src,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta);\n+int convert_to_working_tree_ca(const struct conv_attrs *ca,\n+\t\t\t const char *path, const char *src,\n+\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t const struct checkout_metadata *meta);\n int async_convert_to_working_tree(const struct index_state *istate,\n \t\t\t\t const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t const struct checkout_metadata *meta,\n \t\t\t\t void *dco);\n+int async_convert_to_working_tree_ca(const struct conv_attrs *ca,\n+\t\t\t\t const char *path, const char *src,\n+\t\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t\t const struct checkout_metadata *meta,\n+\t\t\t\t void *dco);\n int async_query_available_blobs(const char *cmd,\n \t\t\t\tstruct string_list *available_paths);\n int renormalize_buffer(const struct index_state *istate,\n-- \n2.27.0\n\n"},{"id":"403286","messageId":"437d31f3b1d6c333b95096c999b9549cab25768e.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 03/21] convert: add get_stream_filter_ca() variant","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:11Z","receivedAt":"2020-08-10T21:34:42Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Jeff Hostetler \n\nLike the previous patch, we will also need to call get_stream_filter()\nwith a precomputed `struct conv_attrs`, when we add support for parallel\ncheckout workers. So add the _ca() variant which takes the conversion\nattributes struct as a parameter.\n\n[matheus.bernardino: move header comment to ca() variant and reword msg]\nSigned-off-by: Matheus Tavares \n---\n convert.c | 28 +++++++++++++++++-----------\n convert.h | 2 ++\n 2 files changed, 19 insertions(+), 11 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 757dc2585c..8e995b39c3 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -1963,34 +1963,31 @@ static struct stream_filter *ident_filter(const struct object_id *oid)\n }\n \n /*\n- * Return an appropriately constructed filter for the path, or NULL if\n+ * Return an appropriately constructed filter for the given ca, or NULL if\n * the contents cannot be filtered without reading the whole thing\n * in-core.\n *\n * Note that you would be crazy to set CRLF, smudge/clean or ident to a\n * large binary blob you would want us not to slurp into the memory!\n */\n-struct stream_filter *get_stream_filter(const struct index_state *istate,\n-\t\t\t\t\tconst char *path,\n-\t\t\t\t\tconst struct object_id *oid)\n+struct stream_filter *get_stream_filter_ca(const struct conv_attrs *ca,\n+\t\t\t\t\t const struct object_id *oid)\n {\n-\tstruct conv_attrs ca;\n \tstruct stream_filter *filter = NULL;\n \n-\tconvert_attrs(istate, &ca, path);\n-\tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n+\tif (ca->drv && (ca->drv->process || ca->drv->smudge || ca->drv->clean))\n \t\treturn NULL;\n \n-\tif (ca.working_tree_encoding)\n+\tif (ca->working_tree_encoding)\n \t\treturn NULL;\n \n-\tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\n+\tif (ca->crlf_action == CRLF_AUTO || ca->crlf_action == CRLF_AUTO_CRLF)\n \t\treturn NULL;\n \n-\tif (ca.ident)\n+\tif (ca->ident)\n \t\tfilter = ident_filter(oid);\n \n-\tif (output_eol(ca.crlf_action) == EOL_CRLF)\n+\tif (output_eol(ca->crlf_action) == EOL_CRLF)\n \t\tfilter = cascade_filter(filter, lf_to_crlf_filter());\n \telse\n \t\tfilter = cascade_filter(filter, &null_filter_singleton);\n@@ -1998,6 +1995,15 @@ struct stream_filter *get_stream_filter(const struct index_state *istate,\n \treturn filter;\n }\n \n+struct stream_filter *get_stream_filter(const struct index_state *istate,\n+\t\t\t\t\tconst char *path,\n+\t\t\t\t\tconst struct object_id *oid)\n+{\n+\tstruct conv_attrs ca;\n+\tconvert_attrs(istate, &ca, path);\n+\treturn get_stream_filter_ca(&ca, oid);\n+}\n+\n void free_stream_filter(struct stream_filter *filter)\n {\n \tfilter->vtbl->free(filter);\ndiff --git a/convert.h b/convert.h\nindex 46d537d1ae..262c1a1d46 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -169,6 +169,8 @@ struct stream_filter; /* opaque */\n struct stream_filter *get_stream_filter(const struct index_state *istate,\n \t\t\t\t\tconst char *path,\n \t\t\t\t\tconst struct object_id *);\n+struct stream_filter *get_stream_filter_ca(const struct conv_attrs *ca,\n+\t\t\t\t\t const struct object_id *oid);\n void free_stream_filter(struct stream_filter *);\n int is_null_stream_filter(struct stream_filter *);\n \n-- \n2.27.0\n\n"},{"id":"403287","messageId":"a4a4456d7c464b3a6f6eb8fad8c5f9bcdf295709.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 04/21] convert: add conv_attrs classification","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:12Z","receivedAt":"2020-08-10T21:34:45Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Jeff Hostetler \n\nCreate `enum conv_attrs_classification` to express the different ways\nthat attributes are handled for a blob during checkout.\n\nThis will be used in a later commit when deciding whether to add a file\nto the parallel or delayed queue during checkout. For now, we can also\nuse it in get_stream_filter_ca() to simplify the function (as the\nclassifying logic is the same).\n\n[matheus.bernardino: use classification in get_stream_filter_ca()]\nSigned-off-by: Matheus Tavares \n---\n convert.c | 26 +++++++++++++++++++-------\n convert.h | 33 +++++++++++++++++++++++++++++++++\n 2 files changed, 52 insertions(+), 7 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 8e995b39c3..c037bb99eb 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -1975,13 +1975,7 @@ struct stream_filter *get_stream_filter_ca(const struct conv_attrs *ca,\n {\n \tstruct stream_filter *filter = NULL;\n \n-\tif (ca->drv && (ca->drv->process || ca->drv->smudge || ca->drv->clean))\n-\t\treturn NULL;\n-\n-\tif (ca->working_tree_encoding)\n-\t\treturn NULL;\n-\n-\tif (ca->crlf_action == CRLF_AUTO || ca->crlf_action == CRLF_AUTO_CRLF)\n+\tif (classify_conv_attrs(ca) != CA_CLASS_STREAMABLE)\n \t\treturn NULL;\n \n \tif (ca->ident)\n@@ -2037,3 +2031,21 @@ void clone_checkout_metadata(struct checkout_metadata *dst,\n \tif (blob)\n \t\toidcpy(&dst->blob, blob);\n }\n+\n+enum conv_attrs_classification classify_conv_attrs(const struct conv_attrs *ca)\n+{\n+\tif (ca->drv) {\n+\t\tif (ca->drv->process)\n+\t\t\treturn CA_CLASS_INCORE_PROCESS;\n+\t\tif (ca->drv->smudge || ca->drv->clean)\n+\t\t\treturn CA_CLASS_INCORE_FILTER;\n+\t}\n+\n+\tif (ca->working_tree_encoding)\n+\t\treturn CA_CLASS_INCORE;\n+\n+\tif (ca->crlf_action == CRLF_AUTO || ca->crlf_action == CRLF_AUTO_CRLF)\n+\t\treturn CA_CLASS_INCORE;\n+\n+\treturn CA_CLASS_STREAMABLE;\n+}\ndiff --git a/convert.h b/convert.h\nindex 262c1a1d46..523ba9b140 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -190,4 +190,37 @@ int stream_filter(struct stream_filter *,\n \t\t const char *input, size_t *isize_p,\n \t\t char *output, size_t *osize_p);\n \n+enum conv_attrs_classification {\n+\t/*\n+\t * The blob must be loaded into a buffer before it can be\n+\t * smudged. All smudging is done in-proc.\n+\t */\n+\tCA_CLASS_INCORE,\n+\n+\t/*\n+\t * The blob must be loaded into a buffer, but uses a\n+\t * single-file driver filter, such as rot13.\n+\t */\n+\tCA_CLASS_INCORE_FILTER,\n+\n+\t/*\n+\t * The blob must be loaded into a buffer, but uses a\n+\t * long-running driver process, such as LFS. This might or\n+\t * might not use delayed operations. (The important thing is\n+\t * that there is a single subordinate long-running process\n+\t * handling all associated blobs and in case of delayed\n+\t * operations, may hold per-blob state.)\n+\t */\n+\tCA_CLASS_INCORE_PROCESS,\n+\n+\t/*\n+\t * The blob can be streamed and smudged without needing to\n+\t * completely read it into a buffer.\n+\t */\n+\tCA_CLASS_STREAMABLE,\n+};\n+\n+enum conv_attrs_classification classify_conv_attrs(\n+\tconst struct conv_attrs *ca);\n+\n #endif /* CONVERT_H */\n-- \n2.27.0\n\n"},{"id":"403288","messageId":"a8f05a2471389fd4b4103223678f259c261a335f.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 05/21] entry: extract a header file for entry.c functions","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:13Z","receivedAt":"2020-08-10T21:34:50Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The declarations of entry.c's public functions and structures currently\nreside in cache.h. Although not many, they contribute to the size of\ncache.h and, when changed, cause the unnecessary recompilation of\nmodules that don't really use these functions. So let's move them to a\nnew entry.h header.\n\nOriginal-patch-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n apply.c | 1 +\n builtin/checkout-index.c | 1 +\n builtin/checkout.c | 1 +\n builtin/difftool.c | 1 +\n cache.h | 24 -----------------------\n entry.c | 9 +--------\n entry.h | 41 ++++++++++++++++++++++++++++++++++++++++\n unpack-trees.c | 1 +\n 8 files changed, 47 insertions(+), 32 deletions(-)\n create mode 100644 entry.h\n\ndiff --git a/apply.c b/apply.c\nindex 8bff604dbe..1443c307a4 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -21,6 +21,7 @@\n #include \"quote.h\"\n #include \"rerere.h\"\n #include \"apply.h\"\n+#include \"entry.h\"\n \n struct gitdiff_data {\n \tstruct strbuf *root;\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex a854fd16e7..0f1ff73129 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -11,6 +11,7 @@\n #include \"quote.h\"\n #include \"cache-tree.h\"\n #include \"parse-options.h\"\n+#include \"entry.h\"\n \n #define CHECKOUT_ALL 4\n static int nul_term_line;\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 2837195491..3e09b29cfe 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -26,6 +26,7 @@\n #include \"unpack-trees.h\"\n #include \"wt-status.h\"\n #include \"xdiff-interface.h\"\n+#include \"entry.h\"\n \n static const char * const checkout_usage[] = {\n \tN_(\"git checkout [] \"),\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex 7ac432b881..dfa22b67eb 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -23,6 +23,7 @@\n #include \"lockfile.h\"\n #include \"object-store.h\"\n #include \"dir.h\"\n+#include \"entry.h\"\n \n static int trust_exit_code;\n \ndiff --git a/cache.h b/cache.h\nindex 0290849c19..e6963cf8fe 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1695,30 +1695,6 @@ const char *show_ident_date(const struct ident_split *id,\n */\n int ident_cmp(const struct ident_split *, const struct ident_split *);\n \n-struct checkout {\n-\tstruct index_state *istate;\n-\tconst char *base_dir;\n-\tint base_dir_len;\n-\tstruct delayed_checkout *delayed_checkout;\n-\tstruct checkout_metadata meta;\n-\tunsigned force:1,\n-\t\t quiet:1,\n-\t\t not_new:1,\n-\t\t clone:1,\n-\t\t refresh_cache:1;\n-};\n-#define CHECKOUT_INIT { NULL, \"\" }\n-\n-#define TEMPORARY_FILENAME_LENGTH 25\n-int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath, int *nr_checkouts);\n-void enable_delayed_checkout(struct checkout *state);\n-int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n-/*\n- * Unlink the last component and schedule the leading directories for\n- * removal, such that empty directories get removed.\n- */\n-void unlink_entry(const struct cache_entry *ce);\n-\n struct cache_def {\n \tstruct strbuf path;\n \tint flags;\ndiff --git a/entry.c b/entry.c\nindex 449bd32dee..f46c06e831 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -6,6 +6,7 @@\n #include \"submodule.h\"\n #include \"progress.h\"\n #include \"fsmonitor.h\"\n+#include \"entry.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -429,14 +430,6 @@ static void mark_colliding_entries(const struct checkout *state,\n \t}\n }\n \n-/*\n- * Write the contents from ce out to the working tree.\n- *\n- * When topath[] is not NULL, instead of writing to the working tree\n- * file named by ce, a temporary file is created by this function and\n- * its name is returned in topath[], which must be able to hold at\n- * least TEMPORARY_FILENAME_LENGTH bytes long.\n- */\n int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \t\t char *topath, int *nr_checkouts)\n {\ndiff --git a/entry.h b/entry.h\nnew file mode 100644\nindex 0000000000..2d69185448\n--- /dev/null\n+++ b/entry.h\n@@ -0,0 +1,41 @@\n+#ifndef ENTRY_H\n+#define ENTRY_H\n+\n+#include \"cache.h\"\n+#include \"convert.h\"\n+\n+struct checkout {\n+\tstruct index_state *istate;\n+\tconst char *base_dir;\n+\tint base_dir_len;\n+\tstruct delayed_checkout *delayed_checkout;\n+\tstruct checkout_metadata meta;\n+\tunsigned force:1,\n+\t\t quiet:1,\n+\t\t not_new:1,\n+\t\t clone:1,\n+\t\t refresh_cache:1;\n+};\n+#define CHECKOUT_INIT { NULL, \"\" }\n+\n+#define TEMPORARY_FILENAME_LENGTH 25\n+\n+/*\n+ * Write the contents from ce out to the working tree.\n+ *\n+ * When topath[] is not NULL, instead of writing to the working tree\n+ * file named by ce, a temporary file is created by this function and\n+ * its name is returned in topath[], which must be able to hold at\n+ * least TEMPORARY_FILENAME_LENGTH bytes long.\n+ */\n+int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n+\t\t char *topath, int *nr_checkouts);\n+void enable_delayed_checkout(struct checkout *state);\n+int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n+/*\n+ * Unlink the last component and schedule the leading directories for\n+ * removal, such that empty directories get removed.\n+ */\n+void unlink_entry(const struct cache_entry *ce);\n+\n+#endif /* ENTRY_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 323280dd48..a511fadd89 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -16,6 +16,7 @@\n #include \"fsmonitor.h\"\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n+#include \"entry.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n-- \n2.27.0\n\n"},{"id":"403289","messageId":"4f86b585b7cd1ae60ba5f90a74f0cca9af59929c.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 06/21] entry: make fstat_output() and read_blob_entry() public","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:14Z","receivedAt":"2020-08-10T21:34:55Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"These two functions will be used by the parallel checkout code, so let's\nmake them public. Note: fstat_output() is renamed to\nfstat_checkout_output(), now that it has become public, seeking to avoid\nfuture name collisions.\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 8 ++++----\n entry.h | 2 ++\n 2 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex f46c06e831..cc27564473 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -84,7 +84,7 @@ static int create_file(const char *path, unsigned int mode)\n \treturn open(path, O_WRONLY | O_CREAT | O_EXCL, mode);\n }\n \n-static void *read_blob_entry(const struct cache_entry *ce, unsigned long *size)\n+void *read_blob_entry(const struct cache_entry *ce, unsigned long *size)\n {\n \tenum object_type type;\n \tvoid *blob_data = read_object_file(&ce->oid, &type, size);\n@@ -109,7 +109,7 @@ static int open_output_fd(char *path, const struct cache_entry *ce, int to_tempf\n \t}\n }\n \n-static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n+int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st)\n {\n \t/* use fstat() only when path == ce->name */\n \tif (fstat_is_reliable() &&\n@@ -132,7 +132,7 @@ static int streaming_write_entry(const struct cache_entry *ce, char *path,\n \t\treturn -1;\n \n \tresult |= stream_blob_to_fd(fd, &ce->oid, filter, 1);\n-\t*fstat_done = fstat_output(fd, state, statbuf);\n+\t*fstat_done = fstat_checkout_output(fd, state, statbuf);\n \tresult |= close(fd);\n \n \tif (result)\n@@ -346,7 +346,7 @@ static int write_entry(struct cache_entry *ce,\n \n \t\twrote = write_in_full(fd, new_blob, size);\n \t\tif (!to_tempfile)\n-\t\t\tfstat_done = fstat_output(fd, state, &st);\n+\t\t\tfstat_done = fstat_checkout_output(fd, state, &st);\n \t\tclose(fd);\n \t\tfree(new_blob);\n \t\tif (wrote < 0)\ndiff --git a/entry.h b/entry.h\nindex 2d69185448..f860e60846 100644\n--- a/entry.h\n+++ b/entry.h\n@@ -37,5 +37,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n * removal, such that empty directories get removed.\n */\n void unlink_entry(const struct cache_entry *ce);\n+void *read_blob_entry(const struct cache_entry *ce, unsigned long *size);\n+int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st);\n \n #endif /* ENTRY_H */\n-- \n2.27.0\n\n"},{"id":"403290","messageId":"4b9e130278495aa7819e5cfd450b0b01d8b58b6d.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 07/21] entry: extract cache_entry update from write_entry()","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:15Z","receivedAt":"2020-08-10T21:34:59Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This code will be used by the parallel checkout functions, outside\nentry.c, so extract it to a public function.\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 26 +++++++++++++++++---------\n entry.h | 2 ++\n 2 files changed, 19 insertions(+), 9 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex cc27564473..837629a804 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -251,6 +251,19 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \treturn errs;\n }\n \n+void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,\n+\t\t\t struct stat *st)\n+{\n+\tif (state->refresh_cache) {\n+\t\tassert(state->istate);\n+\t\tfill_stat_cache_info(state->istate, ce, st);\n+\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(state->istate, ce);\n+\t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n+\t}\n+}\n+\n+\n static int write_entry(struct cache_entry *ce,\n \t\t char *path, const struct checkout *state, int to_tempfile)\n {\n@@ -371,15 +384,10 @@ static int write_entry(struct cache_entry *ce,\n \n finish:\n \tif (state->refresh_cache) {\n-\t\tassert(state->istate);\n-\t\tif (!fstat_done)\n-\t\t\tif (lstat(ce->name, &st) < 0)\n-\t\t\t\treturn error_errno(\"unable to stat just-written file %s\",\n-\t\t\t\t\t\t ce->name);\n-\t\tfill_stat_cache_info(state->istate, ce, &st);\n-\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\tmark_fsmonitor_invalid(state->istate, ce);\n-\t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n+\t\tif (!fstat_done && lstat(ce->name, &st) < 0)\n+\t\t\treturn error_errno(\"unable to stat just-written file %s\",\n+\t\t\t\t\t ce->name);\n+\t\tupdate_ce_after_write(state, ce , &st);\n \t}\n delayed:\n \treturn 0;\ndiff --git a/entry.h b/entry.h\nindex f860e60846..664aed1576 100644\n--- a/entry.h\n+++ b/entry.h\n@@ -39,5 +39,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n void unlink_entry(const struct cache_entry *ce);\n void *read_blob_entry(const struct cache_entry *ce, unsigned long *size);\n int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st);\n+void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,\n+\t\t\t struct stat *st);\n \n #endif /* ENTRY_H */\n-- \n2.27.0\n\n"},{"id":"403291","messageId":"934b025526d8ced39e7cb5b7a161bf80bc2ef99a.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 08/21] entry: move conv_attrs lookup up to checkout_entry()","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:16Z","receivedAt":"2020-08-10T21:35:03Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"In a following patch, checkout_entry() will use conv_attrs to decide\nwhether an entry should be enqueued for parallel checkout or not. But\nthe attributes lookup only happens lower in this call stack. To avoid\nthe unnecessary work of loading the attributes twice, let's move it up\nto checkout_entry(), and pass the loaded struct down to write_entry().\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 39 +++++++++++++++++++++++++++------------\n 1 file changed, 27 insertions(+), 12 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 837629a804..59d5335ff1 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -263,9 +263,9 @@ void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,\n \t}\n }\n \n-\n-static int write_entry(struct cache_entry *ce,\n-\t\t char *path, const struct checkout *state, int to_tempfile)\n+/* Note: ca is used (and required) iff the entry refers to a regular file. */\n+static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca,\n+\t\t const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n \tstruct delayed_checkout *dco = state->delayed_checkout;\n@@ -282,8 +282,7 @@ static int write_entry(struct cache_entry *ce,\n \tclone_checkout_metadata(&meta, &state->meta, &ce->oid);\n \n \tif (ce_mode_s_ifmt == S_IFREG) {\n-\t\tstruct stream_filter *filter = get_stream_filter(state->istate, ce->name,\n-\t\t\t\t\t\t\t\t &ce->oid);\n+\t\tstruct stream_filter *filter = get_stream_filter_ca(ca, &ce->oid);\n \t\tif (filter &&\n \t\t !streaming_write_entry(ce, path, filter,\n \t\t\t\t\t state, to_tempfile,\n@@ -330,14 +329,17 @@ static int write_entry(struct cache_entry *ce,\n \t\t * Convert from git internal format to working tree format\n \t\t */\n \t\tif (dco && dco->state != CE_NO_DELAY) {\n-\t\t\tret = async_convert_to_working_tree(state->istate, ce->name, new_blob,\n-\t\t\t\t\t\t\t size, &buf, &meta, dco);\n+\t\t\tret = async_convert_to_working_tree_ca(ca, ce->name,\n+\t\t\t\t\t\t\t new_blob, size,\n+\t\t\t\t\t\t\t &buf, &meta, dco);\n \t\t\tif (ret && string_list_has_string(&dco->paths, ce->name)) {\n \t\t\t\tfree(new_blob);\n \t\t\t\tgoto delayed;\n \t\t\t}\n-\t\t} else\n-\t\t\tret = convert_to_working_tree(state->istate, ce->name, new_blob, size, &buf, &meta);\n+\t\t} else {\n+\t\t\tret = convert_to_working_tree_ca(ca, ce->name, new_blob,\n+\t\t\t\t\t\t\t size, &buf, &meta);\n+\t\t}\n \n \t\tif (ret) {\n \t\t\tfree(new_blob);\n@@ -443,6 +445,7 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n {\n \tstatic struct strbuf path = STRBUF_INIT;\n \tstruct stat st;\n+\tstruct conv_attrs ca;\n \n \tif (ce->ce_flags & CE_WT_REMOVE) {\n \t\tif (topath)\n@@ -455,8 +458,13 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \t\treturn 0;\n \t}\n \n-\tif (topath)\n-\t\treturn write_entry(ce, topath, state, 1);\n+\tif (topath) {\n+\t\tif (S_ISREG(ce->ce_mode)) {\n+\t\t\tconvert_attrs(state->istate, &ca, ce->name);\n+\t\t\treturn write_entry(ce, topath, &ca, state, 1);\n+\t\t}\n+\t\treturn write_entry(ce, topath, NULL, state, 1);\n+\t}\n \n \tstrbuf_reset(&path);\n \tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n@@ -520,9 +528,16 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \t\treturn 0;\n \n \tcreate_directories(path.buf, path.len, state);\n+\n \tif (nr_checkouts)\n \t\t(*nr_checkouts)++;\n-\treturn write_entry(ce, path.buf, state, 0);\n+\n+\tif (S_ISREG(ce->ce_mode)) {\n+\t\tconvert_attrs(state->istate, &ca, ce->name);\n+\t\treturn write_entry(ce, path.buf, &ca, state, 0);\n+\t}\n+\n+\treturn write_entry(ce, path.buf, NULL, state, 0);\n }\n \n void unlink_entry(const struct cache_entry *ce)\n-- \n2.27.0\n\n"},{"id":"403292","messageId":"6ff36c853294f2449f781cfa1baaa63e691fa3d9.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 09/21] entry: add checkout_entry_ca() which takes preloaded conv_attrs","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:17Z","receivedAt":"2020-08-10T21:35:06Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The parallel checkout machinery will call checkout_entry() for entries\nthat could not be written in parallel due to path collisions. At this\npoint, we will already be holding the conversion attributes for each\nentry, and it would be wasteful to let checkout_entry() load these\nagain. Instead, let's add the checkout_entry_ca() variant, which\noptionally takes a preloaded conv_attrs struct.\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 23 ++++++++++++-----------\n entry.h | 13 +++++++++++--\n 2 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 59d5335ff1..f9835afba3 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -440,12 +440,13 @@ static void mark_colliding_entries(const struct checkout *state,\n \t}\n }\n \n-int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n-\t\t char *topath, int *nr_checkouts)\n+int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n+\t\t const struct checkout *state, char *topath,\n+\t\t int *nr_checkouts)\n {\n \tstatic struct strbuf path = STRBUF_INIT;\n \tstruct stat st;\n-\tstruct conv_attrs ca;\n+\tstruct conv_attrs ca_buf;\n \n \tif (ce->ce_flags & CE_WT_REMOVE) {\n \t\tif (topath)\n@@ -459,11 +460,11 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \t}\n \n \tif (topath) {\n-\t\tif (S_ISREG(ce->ce_mode)) {\n-\t\t\tconvert_attrs(state->istate, &ca, ce->name);\n-\t\t\treturn write_entry(ce, topath, &ca, state, 1);\n+\t\tif (S_ISREG(ce->ce_mode) && !ca) {\n+\t\t\tconvert_attrs(state->istate, &ca_buf, ce->name);\n+\t\t\tca = &ca_buf;\n \t\t}\n-\t\treturn write_entry(ce, topath, NULL, state, 1);\n+\t\treturn write_entry(ce, topath, ca, state, 1);\n \t}\n \n \tstrbuf_reset(&path);\n@@ -532,12 +533,12 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \tif (nr_checkouts)\n \t\t(*nr_checkouts)++;\n \n-\tif (S_ISREG(ce->ce_mode)) {\n-\t\tconvert_attrs(state->istate, &ca, ce->name);\n-\t\treturn write_entry(ce, path.buf, &ca, state, 0);\n+\tif (S_ISREG(ce->ce_mode) && !ca) {\n+\t\tconvert_attrs(state->istate, &ca_buf, ce->name);\n+\t\tca = &ca_buf;\n \t}\n \n-\treturn write_entry(ce, path.buf, NULL, state, 0);\n+\treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \n void unlink_entry(const struct cache_entry *ce)\ndiff --git a/entry.h b/entry.h\nindex 664aed1576..2081fbbbab 100644\n--- a/entry.h\n+++ b/entry.h\n@@ -27,9 +27,18 @@ struct checkout {\n * file named by ce, a temporary file is created by this function and\n * its name is returned in topath[], which must be able to hold at\n * least TEMPORARY_FILENAME_LENGTH bytes long.\n+ *\n+ * With checkout_entry_ca(), callers can optionally pass a preloaded\n+ * conv_attrs struct (to avoid reloading it), when ce refers to a\n+ * regular file. If ca is NULL, the attributes will be loaded\n+ * internally when (and if) needed.\n */\n-int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n-\t\t char *topath, int *nr_checkouts);\n+#define checkout_entry(ce, state, topath, nr_checkouts) \\\n+\t\tcheckout_entry_ca(ce, NULL, state, topath, nr_checkouts)\n+int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n+\t\t const struct checkout *state, char *topath,\n+\t\t int *nr_checkouts);\n+\n void enable_delayed_checkout(struct checkout *state);\n int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n /*\n-- \n2.27.0\n\n"},{"id":"403293","messageId":"1b39a4099a69f2c42211d46d615055c783703fea.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 10/21] unpack-trees: add basic support for parallel checkout","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:18Z","receivedAt":"2020-08-10T21:35:12Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This new interface allows us to enqueue some of the entries being\nchecked out to later call write_entry() for them in parallel. For now,\nthe parallel checkout machinery is enabled by default and there is no\nuser configuration, but run_parallel_checkout() just writes the queued\nentries in sequence (without spawning additional workers). The next\npatch will actually implement the parallelism and, later, we will make\nit configurable.\n\nWhen there are path collisions among the entries being written (which\ncan happen e.g. with case-sensitive files in case-insensitive file\nsystems), the parallel checkout code detects the problem and mark the\ncheckout_item with CI_RETRY. Later, these items are sequentially feed to\ncheckout_entry() again. This is similar to the way the sequential code\ndeals with collisions, overwriting the previously checked out entries\nwith the subsequent ones. The only difference is that, once we start\nwriting the entries in parallel, we will no longer be able to determine\nwhich of the colliding entries will survive on disk (for the sequential\nalgorithm, it is always the last one). Finally, just like the sequential\ncode, there is no additional overhead when there are no collisions.\n\nNote: we continue the loop of write_checkout_item() even if the previous\ncall returned an error. This is how checkout_entry() is called in\nbuiltin/checkout.c:checkout_paths() and unpack-trees.c:check-updates().\nIn the case of fatal errors, die() aborts the loop.\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n\nFor consistency, the parallel code replicates the sequential behavior of\noverwriting colliding entries. However, during parallel checkout it's\npossible to distinguish a path collision from the case where a path was\nalready present in the working tree before checkout. So, in the event of\na collision, we could chose to write a single entry and skip overwriting\nit with the next ones. Does that sounds reasonable, or are there other\nproblems in not writing the extra colliding entries?\n\n Makefile | 1 +\n entry.c | 4 +\n parallel-checkout.c | 340 ++++++++++++++++++++++++++++++++++++++++++++\n parallel-checkout.h | 20 +++\n unpack-trees.c | 6 +-\n 5 files changed, 370 insertions(+), 1 deletion(-)\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\ndiff --git a/Makefile b/Makefile\nindex 65f8cfb236..caab8e6401 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -933,6 +933,7 @@ LIB_OBJS += pack-revindex.o\n LIB_OBJS += pack-write.o\n LIB_OBJS += packfile.o\n LIB_OBJS += pager.o\n+LIB_OBJS += parallel-checkout.o\n LIB_OBJS += parse-options-cb.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\ndiff --git a/entry.c b/entry.c\nindex f9835afba3..47c2c20d5a 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -7,6 +7,7 @@\n #include \"progress.h\"\n #include \"fsmonitor.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -538,6 +539,9 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\tca = &ca_buf;\n \t}\n \n+\tif (!enqueue_checkout(ce, ca))\n+\t\treturn 0;\n+\n \treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nnew file mode 100644\nindex 0000000000..e3b44eeb34\n--- /dev/null\n+++ b/parallel-checkout.c\n@@ -0,0 +1,340 @@\n+#include \"cache.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"streaming.h\"\n+\n+enum ci_status {\n+\tCI_PENDING = 0,\n+\tCI_SUCCESS,\n+\tCI_RETRY,\n+\tCI_FAILED,\n+};\n+\n+struct checkout_item {\n+\t/* pointer to a istate->cache[] entry. Not owned by us. */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tstruct stat st;\n+\tenum ci_status status;\n+};\n+\n+struct parallel_checkout {\n+\tstruct checkout_item *items;\n+\tsize_t nr, alloc;\n+};\n+\n+static struct parallel_checkout *parallel_checkout = NULL;\n+\n+enum pc_status {\n+\tPC_UNINITIALIZED = 0,\n+\tPC_ACCEPTING_ENTRIES,\n+\tPC_RUNNING,\n+\tPC_HANDLING_RESULTS,\n+};\n+\n+static enum pc_status pc_status = PC_UNINITIALIZED;\n+\n+void init_parallel_checkout(void)\n+{\n+\tif (parallel_checkout)\n+\t\tBUG(\"parallel checkout already initialized\");\n+\n+\tparallel_checkout = xcalloc(1, sizeof(*parallel_checkout));\n+\tpc_status = PC_ACCEPTING_ENTRIES;\n+}\n+\n+static void finish_parallel_checkout(void)\n+{\n+\tif (!parallel_checkout)\n+\t\tBUG(\"cannot finish parallel checkout: not initialized yet\");\n+\n+\tfree(parallel_checkout->items);\n+\tFREE_AND_NULL(parallel_checkout);\n+\tpc_status = PC_UNINITIALIZED;\n+}\n+\n+static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n+\t\t\t\t\t const struct conv_attrs *ca)\n+{\n+\tenum conv_attrs_classification c;\n+\n+\tif (!S_ISREG(ce->ce_mode))\n+\t\treturn 0;\n+\n+\tc = classify_conv_attrs(ca);\n+\tswitch (c) {\n+\tcase CA_CLASS_INCORE:\n+\t\treturn 1;\n+\n+\tcase CA_CLASS_INCORE_FILTER:\n+\t\t/*\n+\t\t * It would be safe to allow concurrent instances of\n+\t\t * single-file smudge filters, like rot13, but we should not\n+\t\t * assume that all filters are parallel-process safe. So we\n+\t\t * don't allow this.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_INCORE_PROCESS:\n+\t\t/*\n+\t\t * The parallel queue and the delayed queue are not compatible,\n+\t\t * so they must be kept completely separated. And we can't tell\n+\t\t * if a long-running process will delay its response without\n+\t\t * actually asking it to perform the filtering. Therefore, this\n+\t\t * type of filter is not allowed in parallel checkout.\n+\t\t *\n+\t\t * Furthermore, there should only be one instance of the\n+\t\t * long-running process filter as we don't know how it is\n+\t\t * managing its own concurrency. So, spreading the entries that\n+\t\t * requisite such a filter among the parallel workers would\n+\t\t * require a lot more inter-process communication. We would\n+\t\t * probably have to designate a single process to interact with\n+\t\t * the filter and send all the necessary data to it, for each\n+\t\t * entry.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_STREAMABLE:\n+\t\treturn 1;\n+\n+\tdefault:\n+\t\tBUG(\"unsupported conv_attrs classification '%d'\", c);\n+\t}\n+}\n+\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n+{\n+\tstruct checkout_item *ci;\n+\n+\tif (!parallel_checkout || pc_status != PC_ACCEPTING_ENTRIES ||\n+\t !is_eligible_for_parallel_checkout(ce, ca))\n+\t\treturn -1;\n+\n+\tALLOC_GROW(parallel_checkout->items, parallel_checkout->nr + 1,\n+\t\t parallel_checkout->alloc);\n+\n+\tci = ¶llel_checkout->items[parallel_checkout->nr++];\n+\tci->ce = ce;\n+\tmemcpy(&ci->ca, ca, sizeof(ci->ca));\n+\n+\treturn 0;\n+}\n+\n+static int handle_results(struct checkout *state)\n+{\n+\tint ret = 0;\n+\tsize_t i;\n+\n+\tpc_status = PC_HANDLING_RESULTS;\n+\n+\tfor (i = 0; i < parallel_checkout->nr; ++i) {\n+\t\tstruct checkout_item *ci = ¶llel_checkout->items[i];\n+\t\tstruct stat *st = &ci->st;\n+\n+\t\tswitch(ci->status) {\n+\t\tcase CI_SUCCESS:\n+\t\t\tupdate_ce_after_write(state, ci->ce, st);\n+\t\t\tbreak;\n+\t\tcase CI_RETRY:\n+\t\t\t/*\n+\t\t\t * The fails for which we set CI_RETRY are the ones\n+\t\t\t * that might have been caused by a path collision. So\n+\t\t\t * we let checkout_entry_ca() retry writing, as it will\n+\t\t\t * properly handle collisions and the creation of\n+\t\t\t * leading dirs in the entry's path.\n+\t\t\t */\n+\t\t\tret |= checkout_entry_ca(ci->ce, &ci->ca, state, NULL, NULL);\n+\t\t\tbreak;\n+\t\tcase CI_FAILED:\n+\t\t\tret = -1;\n+\t\t\tbreak;\n+\t\tcase CI_PENDING:\n+\t\t\tBUG(\"parallel checkout finished with pending entries\");\n+\t\tdefault:\n+\t\t\tBUG(\"unknown checkout item status in parallel checkout\");\n+\t\t}\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int reset_fd(int fd, const char *path)\n+{\n+\tif (lseek(fd, 0, SEEK_SET) != 0)\n+\t\treturn error_errno(\"failed to rewind descriptor of %s\", path);\n+\tif (ftruncate(fd, 0))\n+\t\treturn error_errno(\"failed to truncate file %s\", path);\n+\treturn 0;\n+}\n+\n+static int write_checkout_item_to_fd(int fd, struct checkout *state,\n+\t\t\t\t struct checkout_item *ci, const char *path)\n+{\n+\tint ret;\n+\tstruct stream_filter *filter;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *new_blob;\n+\tunsigned long size;\n+\tsize_t newsize = 0;\n+\tssize_t wrote;\n+\n+\t/* Sanity check */\n+\tassert(is_eligible_for_parallel_checkout(ci->ce, &ci->ca));\n+\n+\tfilter = get_stream_filter_ca(&ci->ca, &ci->ce->oid);\n+\tif (filter) {\n+\t\tif (stream_blob_to_fd(fd, &ci->ce->oid, filter, 1)) {\n+\t\t\t/* On error, reset fd to try writing without streaming */\n+\t\t\tif (reset_fd(fd, path))\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tnew_blob = read_blob_entry(ci->ce, &size);\n+\tif (!new_blob)\n+\t\treturn error(\"unable to read sha1 file of %s (%s)\", path,\n+\t\t\t oid_to_hex(&ci->ce->oid));\n+\n+\t/*\n+\t * checkout metadata is used to give context for external process\n+\t * filters. Files requiring such filters are not eligible for parallel\n+\t * checkout, so pass NULL.\n+\t */\n+\tret = convert_to_working_tree_ca(&ci->ca, ci->ce->name, new_blob, size,\n+\t\t\t\t\t &buf, NULL);\n+\n+\tif (ret) {\n+\t\tfree(new_blob);\n+\t\tnew_blob = strbuf_detach(&buf, &newsize);\n+\t\tsize = newsize;\n+\t}\n+\n+\twrote = write_in_full(fd, new_blob, size);\n+\tfree(new_blob);\n+\tif (wrote < 0)\n+\t\treturn error(\"unable to write file %s\", path);\n+\n+\treturn 0;\n+}\n+\n+static int close_and_clear(int *fd)\n+{\n+\tint ret = 0;\n+\n+\tif (*fd >= 0) {\n+\t\tret = close(*fd);\n+\t\t*fd = -1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int check_leading_dirs(const char *path, int len, int prefix_len)\n+{\n+\tconst char *slash = path + len;\n+\n+\twhile (slash > path && *slash != '/')\n+\t\tslash--;\n+\n+\treturn has_dirs_only_path(path, slash - path, prefix_len);\n+}\n+\n+static void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n+{\n+\tunsigned int mode = (ci->ce->ce_mode & 0100) ? 0777 : 0666;\n+\tint fd = -1, fstat_done = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n+\tstrbuf_add(&path, ci->ce->name, ce_namelen(ci->ce));\n+\n+\t/*\n+\t * At this point, leading dirs should have already been created. But if\n+\t * a symlink being checked out has collided with one of the dirs, due to\n+\t * file system folding rules, it's possible that the dirs are no longer\n+\t * present. So we have to check again, and report any path collisions.\n+\t */\n+\tif (!check_leading_dirs(path.buf, path.len, state->base_dir_len)) {\n+\t\tci->status = CI_RETRY;\n+\t\tgoto out;\n+\t}\n+\n+\tfd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, mode);\n+\n+\tif (fd < 0) {\n+\t\tif (errno == EEXIST || errno == EISDIR) {\n+\t\t\t/*\n+\t\t\t * Errors which probably represent a path collision.\n+\t\t\t * Suppress the error message and mark the ci to be\n+\t\t\t * retried later, sequentially. ENOTDIR and ENOENT are\n+\t\t\t * also interesting, but check_leading_dirs() should\n+\t\t\t * have already caught these cases.\n+\t\t\t */\n+\t\t\tci->status = CI_RETRY;\n+\t\t} else {\n+\t\t\terror_errno(\"failed to open file %s\", path.buf);\n+\t\t\tci->status = CI_FAILED;\n+\t\t}\n+\t\tgoto out;\n+\t}\n+\n+\tif (write_checkout_item_to_fd(fd, state, ci, path.buf)) {\n+\t\t/* Error was already reported. */\n+\t\tci->status = CI_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tfstat_done = fstat_checkout_output(fd, state, &ci->st);\n+\n+\tif (close_and_clear(&fd)) {\n+\t\terror_errno(\"unable to close file %s\", path.buf);\n+\t\tci->status = CI_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tif (state->refresh_cache && !fstat_done && lstat(path.buf, &ci->st) < 0) {\n+\t\terror_errno(\"unable to stat just-written file %s\", path.buf);\n+\t\tci->status = CI_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tci->status = CI_SUCCESS;\n+\n+out:\n+\t/*\n+\t * No need to check close() return. At this point, either fd is already\n+\t * closed, or we are on an error path, that has already been reported.\n+\t */\n+\tclose_and_clear(&fd);\n+\tstrbuf_release(&path);\n+}\n+\n+static int run_checkout_sequentially(struct checkout *state)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < parallel_checkout->nr; ++i) {\n+\t\tstruct checkout_item *ci = ¶llel_checkout->items[i];\n+\t\twrite_checkout_item(state, ci);\n+\t}\n+\n+\treturn handle_results(state);\n+}\n+\n+\n+int run_parallel_checkout(struct checkout *state)\n+{\n+\tint ret;\n+\n+\tif (!parallel_checkout)\n+\t\tBUG(\"cannot run parallel checkout: not initialized yet\");\n+\n+\tpc_status = PC_RUNNING;\n+\n+\tret = run_checkout_sequentially(state);\n+\n+\tfinish_parallel_checkout();\n+\treturn ret;\n+}\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nnew file mode 100644\nindex 0000000000..8eef59ffcd\n--- /dev/null\n+++ b/parallel-checkout.h\n@@ -0,0 +1,20 @@\n+#ifndef PARALLEL_CHECKOUT_H\n+#define PARALLEL_CHECKOUT_H\n+\n+struct cache_entry;\n+struct checkout;\n+struct conv_attrs;\n+\n+void init_parallel_checkout(void);\n+\n+/*\n+ * Return -1 if parallel checkout is currently not enabled or if the entry is\n+ * not eligible for parallel checkout. Otherwise, enqueue the entry for later\n+ * write and return 0.\n+ */\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+\n+/* Write all the queued entries, returning 0 on success.*/\n+int run_parallel_checkout(struct checkout *state);\n+\n+#endif /* PARALLEL_CHECKOUT_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a511fadd89..1b1da7485a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -17,6 +17,7 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -438,7 +439,6 @@ static int check_updates(struct unpack_trees_options *o,\n \tif (should_update_submodules())\n \t\tload_gitmodules_file(index, &state);\n \n-\tenable_delayed_checkout(&state);\n \tif (has_promisor_remote()) {\n \t\t/*\n \t\t * Prefetch the objects that are to be checked out in the loop\n@@ -461,6 +461,9 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t\t\t\t to_fetch.oid, to_fetch.nr);\n \t\toid_array_clear(&to_fetch);\n \t}\n+\n+\tenable_delayed_checkout(&state);\n+\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -474,6 +477,7 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n+\terrs |= run_parallel_checkout(&state);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.27.0\n\n"},{"id":"403294","messageId":"7e7527ef3e8a9e71a012f1623e9642c47f7f741c.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 11/21] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:19Z","receivedAt":"2020-08-10T21:35:18Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Use multiple worker processes to distribute the queued entries and call\nwrite_checkout_item() in parallel for them. The items are distributed\nuniformly in contiguous chunks. This minimizes the chances of two\nworkers writing to the same directory simultaneously, which could\naffect performance due to lock contention in the kernel. Work stealing\n(or any other format of re-distribution) is not implemented yet.\n\nFor now, the number of workers is equal to the number of logical cores\navailable. But the next patch will add settings to configure this.\nDistributed file systems, such as NFS and EFS, can benefit from using\nmore workers than the actual number of cores (see timings below).\n\nThe parallel version was benchmarked during three operations in the\nlinux repo, with cold cache: cloning v5.8, checking out v5.8 from\nv2.6.15 (checkout I) and checking out v5.8 from v5.7 (checkout II). The\nthree tables below show the mean run times and standard deviations for\n5 runs in a local file system, a Linux NFS server and Amazon EFS. The\nnumbers of workers were chosen based on what produces the best result\nfor each case.\n\nLocal:\n\n Clone Checkout I Checkout II\nSequential 8.180 s ± 0.021 s 6.936 s ± 0.030 s 2.585 s ± 0.005 s\n10 workers 3.633 s ± 0.040 s 2.288 s ± 0.026 s 1.058 s ± 0.015 s\nSpeedup 2.25 ± 0.03 3.03 ± 0.04 2.44 ± 0.03\n\nLinux NFS server (v4.1, on EBS, single availability zone):\n\n Clone Checkout I Checkout II\nSequential 208.069 s ± 2.522 s 198.610 s ± 1.979 s 54.376 s ± 1.333 s\n32 workers 67.078 s ± 0.878 s 64.828 s ± 0.387 s 22.993 s ± 0.252 s\nSpeedup 3.10 ± 0.06 3.06 ± 0.04 2.36 ± 0.06\n\nEFS (v4.1, replicated over multiple availability zones):\n\n Clone Checkout I Checkout II\nSequential 1143.655 s ± 11.819 s 1277.891 s ± 10.481 s 396.891 s ± 7.505 s\n64 workers 173.242 s ± 1.484 s 282.421 s ± 1.521 s 165.424 s ± 9.564 s\nSpeedup 6.60 ± 0.09 4.52 ± 0.04 2.40 ± 0.15\n\nLocal tests were executed in an i7-7700HQ (4 cores with hyper-threading)\nrunning Manjaro Linux, with SSD. NFS and EFS tests were executed in an\nAmazon EC2 c5n.large instance, with 2 vCPUs. The Linux NFS server was\nrunning on a m6g.large instance with 1 TB, EBS GP2 volume. Before each\ntiming, the linux repository was removed (or checked out back), and\n`sync && sysctl vm.drop_caches=3` was executed.\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/checkout--helper.c | 135 +++++++++++++++++++++\n entry.c | 13 +-\n git.c | 2 +\n parallel-checkout.c | 237 +++++++++++++++++++++++++++++++------\n parallel-checkout.h | 74 +++++++++++-\n 8 files changed, 425 insertions(+), 39 deletions(-)\n create mode 100644 builtin/checkout--helper.c\n\ndiff --git a/.gitignore b/.gitignore\nindex ee509a2ad2..6c01f0a58c 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -33,6 +33,7 @@\n /git-check-mailmap\n /git-check-ref-format\n /git-checkout\n+/git-checkout--helper\n /git-checkout-index\n /git-cherry\n /git-cherry-pick\ndiff --git a/Makefile b/Makefile\nindex caab8e6401..926473d484 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1049,6 +1049,7 @@ BUILTIN_OBJS += builtin/check-attr.o\n BUILTIN_OBJS += builtin/check-ignore.o\n BUILTIN_OBJS += builtin/check-mailmap.o\n BUILTIN_OBJS += builtin/check-ref-format.o\n+BUILTIN_OBJS += builtin/checkout--helper.o\n BUILTIN_OBJS += builtin/checkout-index.o\n BUILTIN_OBJS += builtin/checkout.o\n BUILTIN_OBJS += builtin/clean.o\ndiff --git a/builtin.h b/builtin.h\nindex a5ae15bfe5..5790c68750 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -122,6 +122,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix);\n int cmd_bundle(int argc, const char **argv, const char *prefix);\n int cmd_cat_file(int argc, const char **argv, const char *prefix);\n int cmd_checkout(int argc, const char **argv, const char *prefix);\n+int cmd_checkout__helper(int argc, const char **argv, const char *prefix);\n int cmd_checkout_index(int argc, const char **argv, const char *prefix);\n int cmd_check_attr(int argc, const char **argv, const char *prefix);\n int cmd_check_ignore(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/checkout--helper.c b/builtin/checkout--helper.c\nnew file mode 100644\nindex 0000000000..269cf02feb\n--- /dev/null\n+++ b/builtin/checkout--helper.c\n@@ -0,0 +1,135 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"parse-options.h\"\n+#include \"pkt-line.h\"\n+\n+static void packet_to_ci(char *line, int len, struct checkout_item *ci)\n+{\n+\tstruct ci_fixed_portion *fixed_portion;\n+\tchar *encoding, *variant;\n+\n+\tif (len < sizeof(struct ci_fixed_portion))\n+\t\tBUG(\"checkout worker received too short item (got %d, exp %d)\",\n+\t\t len, (int)sizeof(struct ci_fixed_portion));\n+\n+\tfixed_portion = (struct ci_fixed_portion *)line;\n+\n+\tif (len - sizeof(struct ci_fixed_portion) !=\n+\t\tfixed_portion->name_len + fixed_portion->working_tree_encoding_len)\n+\t\tBUG(\"checkout worker received corrupted item\");\n+\n+\tvariant = line + sizeof(struct ci_fixed_portion);\n+\tif (fixed_portion->working_tree_encoding_len) {\n+\t\tencoding = xmemdupz(variant,\n+\t\t\t\t fixed_portion->working_tree_encoding_len);\n+\t\tvariant += fixed_portion->working_tree_encoding_len;\n+\t} else {\n+\t\tencoding = NULL;\n+\t}\n+\n+\tmemset(ci, 0, sizeof(*ci));\n+\tci->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n+\tci->ce->ce_namelen = fixed_portion->name_len;\n+\tci->ce->ce_mode = fixed_portion->ce_mode;\n+\tmemcpy(ci->ce->name, variant, ci->ce->ce_namelen);\n+\toidcpy(&ci->ce->oid, &fixed_portion->oid);\n+\n+\tci->id = fixed_portion->id;\n+\tci->ca.attr_action = fixed_portion->attr_action;\n+\tci->ca.crlf_action = fixed_portion->crlf_action;\n+\tci->ca.ident = fixed_portion->ident;\n+\tci->ca.working_tree_encoding = encoding;\n+}\n+\n+static void report_result(struct checkout_item *ci)\n+{\n+\tstruct ci_result res = { 0 };\n+\tsize_t size;\n+\n+\tres.id = ci->id;\n+\tres.status = ci->status;\n+\n+\tif (ci->status == CI_SUCCESS) {\n+\t\tres.st = ci->st;\n+\t\tsize = sizeof(res);\n+\t} else {\n+\t\tsize = ci_result_base_size();\n+\t}\n+\n+\tpacket_write(1, (const char *)&res, size);\n+}\n+\n+/* Free the worker-side malloced data, but not the ci itself. */\n+static void release_checkout_item_data(struct checkout_item *ci)\n+{\n+\tfree((char *)ci->ca.working_tree_encoding);\n+\tdiscard_cache_entry(ci->ce);\n+}\n+\n+static void worker_loop(struct checkout *state)\n+{\n+\tstruct checkout_item *items = NULL;\n+\tsize_t i, nr = 0, alloc = 0;\n+\n+\twhile (1) {\n+\t\tint len;\n+\t\tchar *line = packet_read_line(0, &len);\n+\n+\t\tif (!line)\n+\t\t\tbreak;\n+\n+\t\tALLOC_GROW(items, nr + 1, alloc);\n+\t\tpacket_to_ci(line, len, &items[nr++]);\n+\t}\n+\n+\tfor (i = 0; i < nr; ++i) {\n+\t\tstruct checkout_item *ci = &items[i];\n+\t\twrite_checkout_item(state, ci);\n+\t\treport_result(ci);\n+\t\trelease_checkout_item_data(ci);\n+\t}\n+\n+\tpacket_flush(1);\n+\n+\tfree(items);\n+}\n+\n+static const char * const checkout_helper_usage[] = {\n+\tN_(\"git checkout--helper []\"),\n+\tNULL\n+};\n+\n+int cmd_checkout__helper(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct checkout state = CHECKOUT_INIT;\n+\tstruct option checkout_helper_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &state.base_dir, N_(\"string\"),\n+\t\t\tN_(\"when creating files, prepend \")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(checkout_helper_usage,\n+\t\t\t\t checkout_helper_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix, checkout_helper_options,\n+\t\t\t checkout_helper_usage, 0);\n+\tif (argc > 0)\n+\t\tusage_with_options(checkout_helper_usage, checkout_helper_options);\n+\n+\tif (state.base_dir)\n+\t\tstate.base_dir_len = strlen(state.base_dir);\n+\n+\t/*\n+\t * Setting this on worker won't actually update the index. We just need\n+\t * to pretend so to induce the checkout machinery to stat() the written\n+\t * entries.\n+\t */\n+\tstate.refresh_cache = 1;\n+\n+\tworker_loop(&state);\n+\treturn 0;\n+}\ndiff --git a/entry.c b/entry.c\nindex 47c2c20d5a..b6c808dffa 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -427,8 +427,17 @@ static void mark_colliding_entries(const struct checkout *state,\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-\t\tif (dup == ce)\n-\t\t\tbreak;\n+\t\tif (dup == ce) {\n+\t\t\t/*\n+\t\t\t * Parallel checkout creates the files in a racy order.\n+\t\t\t * So the other side of the collision may appear after\n+\t\t\t * the given cache_entry in the array.\n+\t\t\t */\n+\t\t\tif (parallel_checkout_status() == PC_HANDLING_RESULTS)\n+\t\t\t\tcontinue;\n+\t\t\telse\n+\t\t\t\tbreak;\n+\t\t}\n \n \t\tif (dup->ce_flags & (CE_MATCHED | CE_VALID | CE_SKIP_WORKTREE))\n \t\t\tcontinue;\ndiff --git a/git.c b/git.c\nindex 8bd1d7551d..78c7bd412c 100644\n--- a/git.c\n+++ b/git.c\n@@ -486,6 +486,8 @@ static struct cmd_struct commands[] = {\n \t{ \"check-mailmap\", cmd_check_mailmap, RUN_SETUP },\n \t{ \"check-ref-format\", cmd_check_ref_format, NO_PARSEOPT },\n \t{ \"checkout\", cmd_checkout, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"checkout--helper\", cmd_checkout__helper,\n+\t\tRUN_SETUP | NEED_WORK_TREE | SUPPORT_SUPER_PREFIX },\n \t{ \"checkout-index\", cmd_checkout_index,\n \t\tRUN_SETUP | NEED_WORK_TREE},\n \t{ \"cherry\", cmd_cherry, RUN_SETUP },\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex e3b44eeb34..ec42342bc8 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,39 +1,23 @@\n #include \"cache.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n+#include \"pkt-line.h\"\n+#include \"run-command.h\"\n #include \"streaming.h\"\n \n-enum ci_status {\n-\tCI_PENDING = 0,\n-\tCI_SUCCESS,\n-\tCI_RETRY,\n-\tCI_FAILED,\n-};\n-\n-struct checkout_item {\n-\t/* pointer to a istate->cache[] entry. Not owned by us. */\n-\tstruct cache_entry *ce;\n-\tstruct conv_attrs ca;\n-\tstruct stat st;\n-\tenum ci_status status;\n-};\n-\n struct parallel_checkout {\n \tstruct checkout_item *items;\n \tsize_t nr, alloc;\n };\n \n static struct parallel_checkout *parallel_checkout = NULL;\n-\n-enum pc_status {\n-\tPC_UNINITIALIZED = 0,\n-\tPC_ACCEPTING_ENTRIES,\n-\tPC_RUNNING,\n-\tPC_HANDLING_RESULTS,\n-};\n-\n static enum pc_status pc_status = PC_UNINITIALIZED;\n \n+enum pc_status parallel_checkout_status(void)\n+{\n+\treturn pc_status;\n+}\n+\n void init_parallel_checkout(void)\n {\n \tif (parallel_checkout)\n@@ -113,9 +97,11 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \tALLOC_GROW(parallel_checkout->items, parallel_checkout->nr + 1,\n \t\t parallel_checkout->alloc);\n \n-\tci = ¶llel_checkout->items[parallel_checkout->nr++];\n+\tci = ¶llel_checkout->items[parallel_checkout->nr];\n \tci->ce = ce;\n \tmemcpy(&ci->ca, ca, sizeof(ci->ca));\n+\tci->id = parallel_checkout->nr;\n+\tparallel_checkout->nr++;\n \n \treturn 0;\n }\n@@ -200,7 +186,8 @@ static int write_checkout_item_to_fd(int fd, struct checkout *state,\n \t/*\n \t * checkout metadata is used to give context for external process\n \t * filters. Files requiring such filters are not eligible for parallel\n-\t * checkout, so pass NULL.\n+\t * checkout, so pass NULL. Note: if that changes, the metadata must also\n+\t * be passed from the main process to the workers.\n \t */\n \tret = convert_to_working_tree_ca(&ci->ca, ci->ce->name, new_blob, size,\n \t\t\t\t\t &buf, NULL);\n@@ -241,14 +228,14 @@ static int check_leading_dirs(const char *path, int len, int prefix_len)\n \treturn has_dirs_only_path(path, slash - path, prefix_len);\n }\n \n-static void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n+void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n {\n \tunsigned int mode = (ci->ce->ce_mode & 0100) ? 0777 : 0666;\n \tint fd = -1, fstat_done = 0;\n \tstruct strbuf path = STRBUF_INIT;\n \n \tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n-\tstrbuf_add(&path, ci->ce->name, ce_namelen(ci->ce));\n+\tstrbuf_add(&path, ci->ce->name, ci->ce->ce_namelen);\n \n \t/*\n \t * At this point, leading dirs should have already been created. But if\n@@ -311,30 +298,214 @@ static void write_checkout_item(struct checkout *state, struct checkout_item *ci\n \tstrbuf_release(&path);\n }\n \n-static int run_checkout_sequentially(struct checkout *state)\n+static void send_one_item(int fd, struct checkout_item *ci)\n+{\n+\tsize_t len_data;\n+\tchar *data, *variant;\n+\tstruct ci_fixed_portion *fixed_portion;\n+\tconst char *working_tree_encoding = ci->ca.working_tree_encoding;\n+\tsize_t name_len = ci->ce->ce_namelen;\n+\tsize_t working_tree_encoding_len = working_tree_encoding ?\n+\t\t\t\t\t strlen(working_tree_encoding) : 0;\n+\n+\tlen_data = sizeof(struct ci_fixed_portion) + name_len +\n+\t\t working_tree_encoding_len;\n+\n+\tdata = xcalloc(1, len_data);\n+\n+\tfixed_portion = (struct ci_fixed_portion *)data;\n+\tfixed_portion->id = ci->id;\n+\toidcpy(&fixed_portion->oid, &ci->ce->oid);\n+\tfixed_portion->ce_mode = ci->ce->ce_mode;\n+\tfixed_portion->attr_action = ci->ca.attr_action;\n+\tfixed_portion->crlf_action = ci->ca.crlf_action;\n+\tfixed_portion->ident = ci->ca.ident;\n+\tfixed_portion->name_len = name_len;\n+\tfixed_portion->working_tree_encoding_len = working_tree_encoding_len;\n+\n+\tvariant = data + sizeof(*fixed_portion);\n+\tif (working_tree_encoding_len) {\n+\t\tmemcpy(variant, working_tree_encoding, working_tree_encoding_len);\n+\t\tvariant += working_tree_encoding_len;\n+\t}\n+\tmemcpy(variant, ci->ce->name, name_len);\n+\n+\tpacket_write(fd, data, len_data);\n+\n+\tfree(data);\n+}\n+\n+static void send_batch(int fd, size_t start, size_t nr)\n {\n \tsize_t i;\n+\tfor (i = 0; i < nr; ++i)\n+\t\tsend_one_item(fd, ¶llel_checkout->items[start + i]);\n+\tpacket_flush(fd);\n+}\n \n-\tfor (i = 0; i < parallel_checkout->nr; ++i) {\n-\t\tstruct checkout_item *ci = ¶llel_checkout->items[i];\n-\t\twrite_checkout_item(state, ci);\n+static struct child_process *setup_workers(struct checkout *state, int num_workers)\n+{\n+\tstruct child_process *workers;\n+\tint i, workers_with_one_extra_item;\n+\tsize_t base_batch_size, next_to_assign = 0;\n+\n+\tbase_batch_size = parallel_checkout->nr / num_workers;\n+\tworkers_with_one_extra_item = parallel_checkout->nr % num_workers;\n+\tALLOC_ARRAY(workers, num_workers);\n+\n+\tfor (i = 0; i < num_workers; ++i) {\n+\t\tstruct child_process *cp = &workers[i];\n+\t\tsize_t batch_size = base_batch_size;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->git_cmd = 1;\n+\t\tcp->in = -1;\n+\t\tcp->out = -1;\n+\t\tstrvec_push(&cp->args, \"checkout--helper\");\n+\t\tif (state->base_dir_len)\n+\t\t\tstrvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n+\t\tif (start_command(cp))\n+\t\t\tdie(_(\"failed to spawn checkout worker\"));\n+\n+\t\t/* distribute the extra work evenly */\n+\t\tif (i < workers_with_one_extra_item)\n+\t\t\tbatch_size++;\n+\n+\t\tsend_batch(cp->in, next_to_assign, batch_size);\n+\t\tnext_to_assign += batch_size;\n \t}\n \n+\treturn workers;\n+}\n+\n+static void finish_workers(struct child_process *workers, int num_workers)\n+{\n+\tint i;\n+\tfor (i = 0; i < num_workers; ++i) {\n+\t\tstruct child_process *w = &workers[i];\n+\t\tif (w->in >= 0)\n+\t\t\tclose(w->in);\n+\t\tif (w->out >= 0)\n+\t\t\tclose(w->out);\n+\t\tif (finish_command(w))\n+\t\t\tdie(_(\"checkout worker finished with error\"));\n+\t}\n+\tfree(workers);\n+}\n+\n+static void parse_and_save_result(const char *line, int len)\n+{\n+\tstruct ci_result *res;\n+\tstruct checkout_item *ci;\n+\n+\t/*\n+\t * Worker should send either the full result struct or just the base\n+\t * (i.e. no stat data).\n+\t */\n+\tif (len != ci_result_base_size() && len != sizeof(struct ci_result))\n+\t\tBUG(\"received corrupted item from checkout worker\");\n+\n+\tres = (struct ci_result *)line;\n+\n+\tif (res->id > parallel_checkout->nr)\n+\t\tBUG(\"checkout worker sent unknown item id\");\n+\n+\tci = ¶llel_checkout->items[res->id];\n+\tci->status = res->status;\n+\n+\t/*\n+\t * Worker only sends stat data on success. Otherwise, we *cannot* access\n+\t * res->st as that will be an invalid address.\n+\t */\n+\tif (res->status == CI_SUCCESS)\n+\t\tci->st = res->st;\n+}\n+\n+static void gather_results_from_workers(struct child_process *workers,\n+\t\t\t\t\tint num_workers)\n+{\n+\tint i, active_workers = num_workers;\n+\tstruct pollfd *pfds;\n+\n+\tCALLOC_ARRAY(pfds, num_workers);\n+\tfor (i = 0; i < num_workers; ++i) {\n+\t\tpfds[i].fd = workers[i].out;\n+\t\tpfds[i].events = POLLIN;\n+\t}\n+\n+\twhile (active_workers) {\n+\t\tint nr = poll(pfds, num_workers, -1);\n+\n+\t\tif (nr < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tdie_errno(\"failed to poll checkout workers\");\n+\t\t}\n+\n+\t\tfor (i = 0; i < num_workers && nr > 0; ++i) {\n+\t\t\tstruct pollfd *pfd = &pfds[i];\n+\n+\t\t\tif (!pfd->revents)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (pfd->revents & POLLIN) {\n+\t\t\t\tint len;\n+\t\t\t\tconst char *line = packet_read_line(pfd->fd, &len);\n+\n+\t\t\t\tif (!line) {\n+\t\t\t\t\tpfd->fd = -1;\n+\t\t\t\t\tactive_workers--;\n+\t\t\t\t} else {\n+\t\t\t\t\tparse_and_save_result(line, len);\n+\t\t\t\t}\n+\t\t\t} else if (pfd->revents & POLLHUP) {\n+\t\t\t\tpfd->fd = -1;\n+\t\t\t\tactive_workers--;\n+\t\t\t} else if (pfd->revents & (POLLNVAL | POLLERR)) {\n+\t\t\t\tdie(_(\"error polling from checkout worker\"));\n+\t\t\t}\n+\n+\t\t\tnr--;\n+\t\t}\n+\t}\n+\n+\tfree(pfds);\n+}\n+\n+static int run_checkout_sequentially(struct checkout *state)\n+{\n+\tsize_t i;\n+\tfor (i = 0; i < parallel_checkout->nr; ++i)\n+\t\twrite_checkout_item(state, ¶llel_checkout->items[i]);\n \treturn handle_results(state);\n }\n \n+static const int workers_threshold = 0;\n \n int run_parallel_checkout(struct checkout *state)\n {\n-\tint ret;\n+\tint num_workers = online_cpus();\n+\tint ret = 0;\n+\tstruct child_process *workers;\n \n \tif (!parallel_checkout)\n \t\tBUG(\"cannot run parallel checkout: not initialized yet\");\n \n \tpc_status = PC_RUNNING;\n \n-\tret = run_checkout_sequentially(state);\n+\tif (parallel_checkout->nr == 0) {\n+\t\tgoto done;\n+\t} else if (parallel_checkout->nr < workers_threshold || num_workers == 1) {\n+\t\tret = run_checkout_sequentially(state);\n+\t\tgoto done;\n+\t}\n+\n+\tworkers = setup_workers(state, num_workers);\n+\tgather_results_from_workers(workers, num_workers);\n+\tfinish_workers(workers, num_workers);\n+\tret = handle_results(state);\n \n+done:\n \tfinish_parallel_checkout();\n \treturn ret;\n }\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 8eef59ffcd..f25f2874ae 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -1,10 +1,21 @@\n #ifndef PARALLEL_CHECKOUT_H\n #define PARALLEL_CHECKOUT_H\n \n-struct cache_entry;\n-struct checkout;\n-struct conv_attrs;\n+#include \"entry.h\"\n+#include \"convert.h\"\n \n+/****************************************************************\n+ * Users of parallel checkout\n+ ****************************************************************/\n+\n+enum pc_status {\n+\tPC_UNINITIALIZED = 0,\n+\tPC_ACCEPTING_ENTRIES,\n+\tPC_RUNNING,\n+\tPC_HANDLING_RESULTS,\n+};\n+\n+enum pc_status parallel_checkout_status(void);\n void init_parallel_checkout(void);\n \n /*\n@@ -14,7 +25,62 @@ void init_parallel_checkout(void);\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n \n-/* Write all the queued entries, returning 0 on success.*/\n+/* Write all the queued entries, returning 0 on success. */\n int run_parallel_checkout(struct checkout *state);\n \n+/****************************************************************\n+ * Interface with checkout--helper\n+ ****************************************************************/\n+\n+enum ci_status {\n+\tCI_PENDING = 0,\n+\tCI_SUCCESS,\n+\tCI_RETRY,\n+\tCI_FAILED,\n+};\n+\n+struct checkout_item {\n+\t/*\n+\t * In main process ce points to a istate->cache[] entry. Thus, it's not\n+\t * owned by us. In workers they own the memory, which *must be* released.\n+\t */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tsize_t id; /* position in parallel_checkout->items[] of main process */\n+\n+\t/* Output fields, sent from workers. */\n+\tenum ci_status status;\n+\tstruct stat st;\n+};\n+\n+/*\n+ * The fixed-size portion of `struct checkout_item` that is sent to the workers.\n+ * Following this will be 2 strings: ca.working_tree_encoding and ce.name; These\n+ * are NOT null terminated, since we have the size in the fixed portion.\n+ */\n+struct ci_fixed_portion {\n+\tsize_t id;\n+\tstruct object_id oid;\n+\tunsigned int ce_mode;\n+\tenum crlf_action attr_action;\n+\tenum crlf_action crlf_action;\n+\tint ident;\n+\tsize_t working_tree_encoding_len;\n+\tsize_t name_len;\n+};\n+\n+/*\n+ * The `struct checkout_item` fields returned by the workers. The order is\n+ * important here, specially stat being the last one, as it is omitted on error.\n+ */\n+struct ci_result {\n+\tsize_t id;\n+\tenum ci_status status;\n+\tstruct stat st;\n+};\n+\n+#define ci_result_base_size() offsetof(struct ci_result, st)\n+\n+void write_checkout_item(struct checkout *state, struct checkout_item *ci);\n+\n #endif /* PARALLEL_CHECKOUT_H */\n-- \n2.27.0\n\n"},{"id":"403295","messageId":"1263342110bfc96480146df0f876a5c936b61ce1.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 12/21] parallel-checkout: add configuration options","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:20Z","receivedAt":"2020-08-10T21:35:21Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Add the checkout.workers and checkout.workerThreshold settings, which\nallow users to configure and/or disable the parallel checkout feature as\ndesired. The first setting defines the number of workers and the second\ndefines the minimum number of entries to attempt parallel checkout.\n\nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n\nI still have to evaluate what is the best default value for\ncheckout.workersThreshold. For now, I used 0 so that the test suite uses\nparallel-checkout by default, exercising the new code. I'm open to\nsuggestions on how we can improve testing for it, once \ncheckout.workersThreshold is no longer 0.\n\nNote: the default number of workers can probably be better calculated as\nwell, multiplying the number of cores by some factor. My machine, for\nexample, has 8 logical cores but 10 workers leads to the fastest\nexecution.\n\n Documentation/config/checkout.txt | 16 ++++++++++++++++\n parallel-checkout.c | 26 +++++++++++++++++++++-----\n parallel-checkout.h | 11 +++++++++--\n unpack-trees.c | 10 +++++++---\n 4 files changed, 53 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt\nindex 6b646813ab..9dabdf9231 100644\n--- a/Documentation/config/checkout.txt\n+++ b/Documentation/config/checkout.txt\n@@ -16,3 +16,19 @@ will checkout the '' branch on another remote,\n and by linkgit:git-worktree[1] when 'git worktree add' refers to a\n remote branch. This setting might be used for other checkout-like\n commands or functionality in the future.\n+\n+checkout.workers::\n+\tThe number of worker processes to use when updating the working tree.\n+\tIf unset (or set to a value less than one), Git will use as many\n+\tworkers as the number of logical cores available. One means sequential\n+\texecution. This and the checkout.workersThreshold settings affect all\n+\tcommands which perform checkout. E.g. checkout, switch, clone,\n+\tsparse-checkout, read-tree, etc.\n+\n+checkout.workersThreshold::\n+\tIf set to a positive number, parallel checkout will not be attempted\n+\twhen the number of files to be updated is less than the defined limit.\n+\tWhen set to a negative number or unset, defaults to 0. The reasoning\n+\tbehind this config is that, when modifying a small number of files, a\n+\tsequential execution might be faster, as it avoids the cost of spawning\n+\tsubprocesses and inter-process communication.\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex ec42342bc8..e0fca4d380 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -4,6 +4,8 @@\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n #include \"streaming.h\"\n+#include \"thread-utils.h\"\n+#include \"config.h\"\n \n struct parallel_checkout {\n \tstruct checkout_item *items;\n@@ -18,6 +20,19 @@ enum pc_status parallel_checkout_status(void)\n \treturn pc_status;\n }\n \n+#define DEFAULT_WORKERS_THRESHOLD 0\n+\n+void get_parallel_checkout_configs(int *num_workers, int *threshold)\n+{\n+\tif (git_config_get_int(\"checkout.workers\", num_workers) ||\n+\t *num_workers < 1)\n+\t\t*num_workers = online_cpus();\n+\n+\tif (git_config_get_int(\"checkout.workersThreshold\", threshold) ||\n+\t *threshold < 0)\n+\t\t*threshold = DEFAULT_WORKERS_THRESHOLD;\n+}\n+\n void init_parallel_checkout(void)\n {\n \tif (parallel_checkout)\n@@ -480,22 +495,23 @@ static int run_checkout_sequentially(struct checkout *state)\n \treturn handle_results(state);\n }\n \n-static const int workers_threshold = 0;\n-\n-int run_parallel_checkout(struct checkout *state)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n {\n-\tint num_workers = online_cpus();\n \tint ret = 0;\n \tstruct child_process *workers;\n \n \tif (!parallel_checkout)\n \t\tBUG(\"cannot run parallel checkout: not initialized yet\");\n \n+\tif (num_workers < 1)\n+\t\tBUG(\"invalid number of workers for run_parallel_checkout: %d\",\n+\t\t num_workers);\n+\n \tpc_status = PC_RUNNING;\n \n \tif (parallel_checkout->nr == 0) {\n \t\tgoto done;\n-\t} else if (parallel_checkout->nr < workers_threshold || num_workers == 1) {\n+\t} else if (parallel_checkout->nr < threshold || num_workers == 1) {\n \t\tret = run_checkout_sequentially(state);\n \t\tgoto done;\n \t}\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex f25f2874ae..b4d412c8b5 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -18,6 +18,9 @@ enum pc_status {\n enum pc_status parallel_checkout_status(void);\n void init_parallel_checkout(void);\n \n+/* Reads the checkout.workers and checkout.workersThreshold settings. */\n+void get_parallel_checkout_configs(int *num_workers, int *threshold);\n+\n /*\n * Return -1 if parallel checkout is currently not enabled or if the entry is\n * not eligible for parallel checkout. Otherwise, enqueue the entry for later\n@@ -25,8 +28,12 @@ void init_parallel_checkout(void);\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n \n-/* Write all the queued entries, returning 0 on success. */\n-int run_parallel_checkout(struct checkout *state);\n+/*\n+ * Write all the queued entries, returning 0 on success. If the number of\n+ * entries is below the specified threshold, the operation is performed\n+ * sequentially.\n+ */\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n \n /****************************************************************\n * Interface with checkout--helper\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1b1da7485a..117ed42370 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -399,7 +399,7 @@ static int check_updates(struct unpack_trees_options *o,\n \tint errs = 0;\n \tstruct progress *progress;\n \tstruct checkout state = CHECKOUT_INIT;\n-\tint i;\n+\tint i, pc_workers, pc_threshold;\n \n \ttrace_performance_enter();\n \tstate.force = 1;\n@@ -462,8 +462,11 @@ static int check_updates(struct unpack_trees_options *o,\n \t\toid_array_clear(&to_fetch);\n \t}\n \n+\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n \tenable_delayed_checkout(&state);\n-\tinit_parallel_checkout();\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -477,7 +480,8 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n-\terrs |= run_parallel_checkout(&state);\n+\tif (pc_workers > 1)\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.27.0\n\n"},{"id":"403296","messageId":"e271d73091de42b30123ae1fffaadb61ad23de44.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 13/21] parallel-checkout: support progress displaying","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:21Z","receivedAt":"2020-08-10T21:35:24Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Original-patch-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n parallel-checkout.c | 40 +++++++++++++++++++++++++++++++++++++---\n parallel-checkout.h | 4 +++-\n unpack-trees.c | 11 ++++++++---\n 3 files changed, 48 insertions(+), 7 deletions(-)\n\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex e0fca4d380..78bf2de5ea 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -2,6 +2,7 @@\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n+#include \"progress.h\"\n #include \"run-command.h\"\n #include \"streaming.h\"\n #include \"thread-utils.h\"\n@@ -10,6 +11,8 @@\n struct parallel_checkout {\n \tstruct checkout_item *items;\n \tsize_t nr, alloc;\n+\tstruct progress *progress;\n+\tunsigned int *progress_cnt;\n };\n \n static struct parallel_checkout *parallel_checkout = NULL;\n@@ -121,6 +124,22 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \treturn 0;\n }\n \n+size_t pc_queue_size(void)\n+{\n+\tif (!parallel_checkout)\n+\t\treturn 0;\n+\treturn parallel_checkout->nr;\n+}\n+\n+static void advance_progress_meter(void)\n+{\n+\tif (parallel_checkout && parallel_checkout->progress) {\n+\t\t(*parallel_checkout->progress_cnt)++;\n+\t\tdisplay_progress(parallel_checkout->progress,\n+\t\t\t\t *parallel_checkout->progress_cnt);\n+\t}\n+}\n+\n static int handle_results(struct checkout *state)\n {\n \tint ret = 0;\n@@ -132,6 +151,10 @@ static int handle_results(struct checkout *state)\n \t\tstruct checkout_item *ci = ¶llel_checkout->items[i];\n \t\tstruct stat *st = &ci->st;\n \n+\t\t/*\n+\t\t * Note: progress meter was already incremented for CI_SUCCESS\n+\t\t * and CI_FAILED.\n+\t\t */\n \t\tswitch(ci->status) {\n \t\tcase CI_SUCCESS:\n \t\t\tupdate_ce_after_write(state, ci->ce, st);\n@@ -145,6 +168,7 @@ static int handle_results(struct checkout *state)\n \t\t\t * leading dirs in the entry's path.\n \t\t\t */\n \t\t\tret |= checkout_entry_ca(ci->ce, &ci->ca, state, NULL, NULL);\n+\t\t\tadvance_progress_meter();\n \t\t\tbreak;\n \t\tcase CI_FAILED:\n \t\t\tret = -1;\n@@ -434,6 +458,9 @@ static void parse_and_save_result(const char *line, int len)\n \t */\n \tif (res->status == CI_SUCCESS)\n \t\tci->st = res->st;\n+\n+\tif (res->status != CI_RETRY)\n+\t\tadvance_progress_meter();\n }\n \n static void gather_results_from_workers(struct child_process *workers,\n@@ -490,12 +517,17 @@ static void gather_results_from_workers(struct child_process *workers,\n static int run_checkout_sequentially(struct checkout *state)\n {\n \tsize_t i;\n-\tfor (i = 0; i < parallel_checkout->nr; ++i)\n-\t\twrite_checkout_item(state, ¶llel_checkout->items[i]);\n+\tfor (i = 0; i < parallel_checkout->nr; ++i) {\n+\t\tstruct checkout_item *ci = ¶llel_checkout->items[i];\n+\t\twrite_checkout_item(state, ci);\n+\t\tif (ci->status != CI_RETRY)\n+\t\t\tadvance_progress_meter();\n+\t}\n \treturn handle_results(state);\n }\n \n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt)\n {\n \tint ret = 0;\n \tstruct child_process *workers;\n@@ -508,6 +540,8 @@ int run_parallel_checkout(struct checkout *state, int num_workers, int threshold\n \t\t num_workers);\n \n \tpc_status = PC_RUNNING;\n+\tparallel_checkout->progress = progress;\n+\tparallel_checkout->progress_cnt = progress_cnt;\n \n \tif (parallel_checkout->nr == 0) {\n \t\tgoto done;\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex b4d412c8b5..2b81a5db6c 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -27,13 +27,15 @@ void get_parallel_checkout_configs(int *num_workers, int *threshold);\n * write and return 0.\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+size_t pc_queue_size(void);\n \n /*\n * Write all the queued entries, returning 0 on success. If the number of\n * entries is below the specified threshold, the operation is performed\n * sequentially.\n */\n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt);\n \n /****************************************************************\n * Interface with checkout--helper\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 117ed42370..e05e6ceff2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -471,17 +471,22 @@ static int check_updates(struct unpack_trees_options *o,\n \t\tstruct cache_entry *ce = index->cache[i];\n \n \t\tif (ce->ce_flags & CE_UPDATE) {\n+\t\t\tsize_t last_pc_queue_size = pc_queue_size();\n+\n \t\t\tif (ce->ce_flags & CE_WT_REMOVE)\n \t\t\t\tBUG(\"both update and delete flags are set on %s\",\n \t\t\t\t ce->name);\n-\t\t\tdisplay_progress(progress, ++cnt);\n \t\t\tce->ce_flags &= ~CE_UPDATE;\n \t\t\terrs |= checkout_entry(ce, &state, NULL, NULL);\n+\n+\t\t\tif (last_pc_queue_size == pc_queue_size())\n+\t\t\t\tdisplay_progress(progress, ++cnt);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tif (pc_workers > 1)\n-\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t\t progress, &cnt);\n+\tstop_progress(&progress);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.27.0\n\n"},{"id":"403297","messageId":"9050b9da6748b7b64508255741f925d9d57983af.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 14/21] make_transient_cache_entry(): optionally alloc from mem_pool","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:22Z","receivedAt":"2020-08-10T21:35:29Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Allow make_transient_cache_entry() to optionally receive a mem_pool\nstruct in which it should allocate the entry. This will be used in the\nfollowing patch, to store some transient entries which should persist\nuntil parallel checkout finishes.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/checkout--helper.c | 2 +-\n builtin/checkout.c | 2 +-\n builtin/difftool.c | 2 +-\n cache.h | 10 +++++-----\n read-cache.c | 12 ++++++++----\n unpack-trees.c | 2 +-\n 6 files changed, 17 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/checkout--helper.c b/builtin/checkout--helper.c\nindex 269cf02feb..d2ab40cb4c 100644\n--- a/builtin/checkout--helper.c\n+++ b/builtin/checkout--helper.c\n@@ -30,7 +30,7 @@ static void packet_to_ci(char *line, int len, struct checkout_item *ci)\n \t}\n \n \tmemset(ci, 0, sizeof(*ci));\n-\tci->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n+\tci->ce = make_empty_transient_cache_entry(fixed_portion->name_len, NULL);\n \tci->ce->ce_namelen = fixed_portion->name_len;\n \tci->ce->ce_mode = fixed_portion->ce_mode;\n \tmemcpy(ci->ce->name, variant, ci->ce->ce_namelen);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 3e09b29cfe..8e4a3c1df0 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -291,7 +291,7 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_transient_cache_entry(mode, &oid, path, 2);\n+\tce = make_transient_cache_entry(mode, &oid, path, 2, NULL);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL, nr_checkouts);\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex dfa22b67eb..5e7a57c8c2 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -323,7 +323,7 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_transient_cache_entry(mode, oid, path, 0);\n+\tce = make_transient_cache_entry(mode, oid, path, 0, NULL);\n \tret = checkout_entry(ce, state, NULL, NULL);\n \n \tdiscard_cache_entry(ce);\ndiff --git a/cache.h b/cache.h\nindex e6963cf8fe..e2b41c5f8b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -355,16 +355,16 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate,\n \t\t\t\t\t size_t name_len);\n \n /*\n- * Create a cache_entry that is not intended to be added to an index.\n- * Caller is responsible for discarding the cache_entry\n- * with `discard_cache_entry`.\n+ * Create a cache_entry that is not intended to be added to an index. If mp is\n+ * not NULL, the entry is allocated within the given memory pool. Caller is\n+ * responsible for discarding the cache_entry with `discard_cache_entry`.\n */\n struct cache_entry *make_transient_cache_entry(unsigned int mode,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t const char *path,\n-\t\t\t\t\t int stage);\n+\t\t\t\t\t int stage, struct mem_pool *mp);\n \n-struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n+struct cache_entry *make_empty_transient_cache_entry(size_t len, struct mem_pool *mp);\n \n /*\n * Discard cache entry.\ndiff --git a/read-cache.c b/read-cache.c\nindex 8ed1c29b54..eeb122cca4 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -811,8 +811,10 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t le\n \treturn mem_pool__ce_calloc(find_mem_pool(istate), len);\n }\n \n-struct cache_entry *make_empty_transient_cache_entry(size_t len)\n+struct cache_entry *make_empty_transient_cache_entry(size_t len, struct mem_pool *mp)\n {\n+\tif (mp)\n+\t\treturn mem_pool__ce_calloc(mp, len);\n \treturn xcalloc(1, cache_entry_size(len));\n }\n \n@@ -846,8 +848,10 @@ struct cache_entry *make_cache_entry(struct index_state *istate,\n \treturn ret;\n }\n \n-struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid,\n-\t\t\t\t\t const char *path, int stage)\n+struct cache_entry *make_transient_cache_entry(unsigned int mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t const char *path, int stage,\n+\t\t\t\t\t struct mem_pool *mp)\n {\n \tstruct cache_entry *ce;\n \tint len;\n@@ -858,7 +862,7 @@ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct o\n \t}\n \n \tlen = strlen(path);\n-\tce = make_empty_transient_cache_entry(len);\n+\tce = make_empty_transient_cache_entry(len, mp);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex e05e6ceff2..dcb40dc8fa 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1031,7 +1031,7 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \tsize_t len = traverse_path_len(info, tree_entry_len(n));\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n-\t\tmake_empty_transient_cache_entry(len) :\n+\t\tmake_empty_transient_cache_entry(len, NULL) :\n \t\tmake_empty_cache_entry(istate, len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n-- \n2.27.0\n\n"},{"id":"403298","messageId":"258b855e49463ed39f0e6865327714c614c786d5.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 15/21] builtin/checkout.c: complete parallel checkout support","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:23Z","receivedAt":"2020-08-10T21:35:33Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"There is one code path in builtin/checkout.c which still doesn't benefit\nfrom parallel checkout because it calls checkout_entry() directly,\ninstead of unpack_trees(). Let's add parallel support for this missing\nspot as well. Note: the transient cache entries allocated in\ncheckout_merged() are now allocated in a mem_pool which is only\ndiscarded after parallel checkout finishes. This is done because the\nentries need to be valid when run_parallel_checkout() is called.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/checkout.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 8e4a3c1df0..b9230d5009 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -27,6 +27,7 @@\n #include \"wt-status.h\"\n #include \"xdiff-interface.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n static const char * const checkout_usage[] = {\n \tN_(\"git checkout [] \"),\n@@ -230,7 +231,8 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos,\n \t\treturn error(_(\"path '%s' does not have their version\"), ce->name);\n }\n \n-static int checkout_merged(int pos, const struct checkout *state, int *nr_checkouts)\n+static int checkout_merged(int pos, const struct checkout *state,\n+\t\t\t int *nr_checkouts, struct mem_pool *ce_mem_pool)\n {\n \tstruct cache_entry *ce = active_cache[pos];\n \tconst char *path = ce->name;\n@@ -291,11 +293,10 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_transient_cache_entry(mode, &oid, path, 2, NULL);\n+\tce = make_transient_cache_entry(mode, &oid, path, 2, ce_mem_pool);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL, nr_checkouts);\n-\tdiscard_cache_entry(ce);\n \treturn status;\n }\n \n@@ -359,16 +360,22 @@ static int checkout_worktree(const struct checkout_opts *opts,\n \tint nr_checkouts = 0, nr_unmerged = 0;\n \tint errs = 0;\n \tint pos;\n+\tint pc_workers, pc_threshold;\n+\tstruct mem_pool *ce_mem_pool = NULL;\n \n \tstate.force = 1;\n \tstate.refresh_cache = 1;\n \tstate.istate = &the_index;\n \n+\tmem_pool_init(&ce_mem_pool, 0);\n+\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n \tinit_checkout_metadata(&state.meta, info->refname,\n \t\t\t info->commit ? &info->commit->object.oid : &info->oid,\n \t\t\t NULL);\n \n \tenable_delayed_checkout(&state);\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n \t\tif (ce->ce_flags & CE_MATCHED) {\n@@ -384,10 +391,15 @@ static int checkout_worktree(const struct checkout_opts *opts,\n \t\t\t\t\t\t &nr_checkouts, opts->overlay_mode);\n \t\t\telse if (opts->merge)\n \t\t\t\terrs |= checkout_merged(pos, &state,\n-\t\t\t\t\t\t\t&nr_unmerged);\n+\t\t\t\t\t\t\t&nr_unmerged,\n+\t\t\t\t\t\t\tce_mem_pool);\n \t\t\tpos = skip_same_name(ce, pos) - 1;\n \t\t}\n \t}\n+\tif (pc_workers > 1)\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t\t NULL, NULL);\n+\tmem_pool_discard(ce_mem_pool, should_validate_cache_entries());\n \tremove_marked_cache_entries(&the_index, 1);\n \tremove_scheduled_dirs();\n \terrs |= finish_delayed_checkout(&state, &nr_checkouts);\n-- \n2.27.0\n\n"},{"id":"403299","messageId":"feed9d435589a308759a3e6b623db13e469c242a.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 16/21] checkout-index: add parallel checkout support","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:24Z","receivedAt":"2020-08-10T21:35:38Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Signed-off-by: Matheus Tavares \n---\n builtin/checkout-index.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 0f1ff73129..33fb933c30 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -12,6 +12,7 @@\n #include \"cache-tree.h\"\n #include \"parse-options.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n #define CHECKOUT_ALL 4\n static int nul_term_line;\n@@ -160,6 +161,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tint prefix_length;\n \tint force = 0, quiet = 0, not_new = 0;\n \tint index_opt = 0;\n+\tint pc_workers, pc_threshold;\n \tstruct option builtin_checkout_index_options[] = {\n \t\tOPT_BOOL('a', \"all\", &all,\n \t\t\tN_(\"check out all files in the index\")),\n@@ -214,6 +216,14 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \t\thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n \t}\n \n+\tif (!to_tempfile)\n+\t\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\telse\n+\t\tpc_workers = 1;\n+\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n+\n \t/* Check out named files first */\n \tfor (i = 0; i < argc; i++) {\n \t\tconst char *arg = argv[i];\n@@ -256,6 +266,12 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tif (all)\n \t\tcheckout_all(prefix, prefix_length);\n \n+\tif (pc_workers > 1) {\n+\t\t/* Errors were already reported */\n+\t\trun_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t NULL, NULL);\n+\t}\n+\n \tif (is_lock_file_locked(&lock_file) &&\n \t write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\tdie(\"Unable to write new index file\");\n-- \n2.27.0\n\n"},{"id":"403300","messageId":"79de52b6952441e77d7276243b4b2ebe7ca16a1f.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 17/21] parallel-checkout: avoid stat() calls in workers","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:25Z","receivedAt":"2020-08-10T21:35:43Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The current parallel checkout implementation requires the workers to\nstat() the path components of each entry before writing, to make sure\nthey are all real directories and not symlinks or something else. The\nstat() info is cached, so this procedure should not be so bad\nperformance-wise. But the exact same check is already done by the main\nprocess, before enqueueing the entries for parallel checkout, to remove\nfiles that were in the way and create the leading dirs. The reason we\nstill need the second check is that, in case of path collisions, a\nsymlink X could be created after an entry x/f was enqueued, leading the\nparallel worker to wrongly create the file at X/f. If we postpone the\nsymlinks' checkouts, though, we can avoid the need of these stat() calls\nin the workers. Other types of path collisions are still possible, such\nas a regular file X being written before the worker tries to write x/f.\nBut that's OK, since the parallel checkout machinery will check the\nreturn of open() to detect such collisions (which would not be possible\nfor the symlink case, as open() would succeed).\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 10 +++++++\n parallel-checkout.c | 71 ++++++++++++++++++++++++++++-----------------\n parallel-checkout.h | 8 +++++\n unpack-trees.c | 4 ++-\n 4 files changed, 65 insertions(+), 28 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex b6c808dffa..6208df23df 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -477,6 +477,16 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\treturn write_entry(ce, topath, ca, state, 1);\n \t}\n \n+\t/*\n+\t * If a regular file x/f is queued for parallel checkout and a symlink\n+\t * X is created now, the worker could wrongly create the file at X/f\n+\t * due to path collision. Thus, symlinks are only created after\n+\t * parallel-eligible entries.\n+\t */\n+\tif (parallel_checkout_status() == PC_ACCEPTING_ENTRIES &&\n+\t S_ISLNK(ce->ce_mode))\n+\t\tenqueue_symlink_checkout(ce, nr_checkouts);\n+\n \tstrbuf_reset(&path);\n \tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n \tstrbuf_add(&path, ce->name, ce_namelen(ce));\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 78bf2de5ea..fee93460c1 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -140,6 +140,44 @@ static void advance_progress_meter(void)\n \t}\n }\n \n+struct symlink_checkout_item {\n+\tstruct cache_entry *ce;\n+\tint *nr_checkouts;\n+};\n+\n+static struct symlink_checkout_item *symlink_queue = NULL;\n+static size_t symlink_queue_nr = 0, symlink_queue_alloc = 0;\n+\n+void enqueue_symlink_checkout(struct cache_entry *ce, int *nr_checkouts)\n+{\n+\tassert(S_ISLNK(ce->ce_mode));\n+\tALLOC_GROW(symlink_queue, symlink_queue_nr + 1, symlink_queue_alloc);\n+\tsymlink_queue[symlink_queue_nr].ce = ce;\n+\tsymlink_queue[symlink_queue_nr].nr_checkouts = nr_checkouts;\n+\tsymlink_queue_nr++;\n+}\n+\n+size_t symlink_queue_size(void)\n+{\n+\treturn symlink_queue_nr;\n+}\n+\n+static int checkout_symlink_queue(struct checkout *state)\n+{\n+\tsize_t i;\n+\tint ret = 0;\n+\n+\tfor (i = 0; i < symlink_queue_nr; ++i) {\n+\t\tstruct symlink_checkout_item *sci = &symlink_queue[i];\n+\t\tret |= checkout_entry(sci->ce, state, NULL, sci->nr_checkouts);\n+\t\tadvance_progress_meter();\n+\t}\n+\n+\tFREE_AND_NULL(symlink_queue);\n+\tsymlink_queue_nr = symlink_queue_alloc = 0;\n+\treturn ret;\n+}\n+\n static int handle_results(struct checkout *state)\n {\n \tint ret = 0;\n@@ -257,16 +295,6 @@ static int close_and_clear(int *fd)\n \treturn ret;\n }\n \n-static int check_leading_dirs(const char *path, int len, int prefix_len)\n-{\n-\tconst char *slash = path + len;\n-\n-\twhile (slash > path && *slash != '/')\n-\t\tslash--;\n-\n-\treturn has_dirs_only_path(path, slash - path, prefix_len);\n-}\n-\n void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n {\n \tunsigned int mode = (ci->ce->ce_mode & 0100) ? 0777 : 0666;\n@@ -276,27 +304,15 @@ void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n \tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n \tstrbuf_add(&path, ci->ce->name, ci->ce->ce_namelen);\n \n-\t/*\n-\t * At this point, leading dirs should have already been created. But if\n-\t * a symlink being checked out has collided with one of the dirs, due to\n-\t * file system folding rules, it's possible that the dirs are no longer\n-\t * present. So we have to check again, and report any path collisions.\n-\t */\n-\tif (!check_leading_dirs(path.buf, path.len, state->base_dir_len)) {\n-\t\tci->status = CI_RETRY;\n-\t\tgoto out;\n-\t}\n-\n \tfd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, mode);\n \n \tif (fd < 0) {\n-\t\tif (errno == EEXIST || errno == EISDIR) {\n+\t\tif (errno == EEXIST || errno == EISDIR || errno == ENOENT ||\n+\t\t errno == ENOTDIR) {\n \t\t\t/*\n \t\t\t * Errors which probably represent a path collision.\n \t\t\t * Suppress the error message and mark the ci to be\n-\t\t\t * retried later, sequentially. ENOTDIR and ENOENT are\n-\t\t\t * also interesting, but check_leading_dirs() should\n-\t\t\t * have already caught these cases.\n+\t\t\t * retried later, sequentially.\n \t\t\t */\n \t\t\tci->status = CI_RETRY;\n \t\t} else {\n@@ -523,7 +539,7 @@ static int run_checkout_sequentially(struct checkout *state)\n \t\tif (ci->status != CI_RETRY)\n \t\t\tadvance_progress_meter();\n \t}\n-\treturn handle_results(state);\n+\treturn handle_results(state) | checkout_symlink_queue(state);\n }\n \n int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n@@ -553,7 +569,8 @@ int run_parallel_checkout(struct checkout *state, int num_workers, int threshold\n \tworkers = setup_workers(state, num_workers);\n \tgather_results_from_workers(workers, num_workers);\n \tfinish_workers(workers, num_workers);\n-\tret = handle_results(state);\n+\tret |= handle_results(state);\n+\tret |= checkout_symlink_queue(state);\n \n done:\n \tfinish_parallel_checkout();\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 2b81a5db6c..a4f7e5b7bd 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -29,6 +29,14 @@ void get_parallel_checkout_configs(int *num_workers, int *threshold);\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n size_t pc_queue_size(void);\n \n+/*\n+ * Enqueues a symlink to be checked out *sequentially* after the parallel\n+ * checkout finishes. This is done to avoid path collisions with leading dirs,\n+ * which could make parallel workers write a file to the wrong place.\n+ */\n+void enqueue_symlink_checkout(struct cache_entry *ce, int *nr_checkouts);\n+size_t symlink_queue_size(void);\n+\n /*\n * Write all the queued entries, returning 0 on success. If the number of\n * entries is below the specified threshold, the operation is performed\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dcb40dc8fa..01928d3d65 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -472,6 +472,7 @@ static int check_updates(struct unpack_trees_options *o,\n \n \t\tif (ce->ce_flags & CE_UPDATE) {\n \t\t\tsize_t last_pc_queue_size = pc_queue_size();\n+\t\t\tsize_t last_symlink_queue_size = symlink_queue_size();\n \n \t\t\tif (ce->ce_flags & CE_WT_REMOVE)\n \t\t\t\tBUG(\"both update and delete flags are set on %s\",\n@@ -479,7 +480,8 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t\tce->ce_flags &= ~CE_UPDATE;\n \t\t\terrs |= checkout_entry(ce, &state, NULL, NULL);\n \n-\t\t\tif (last_pc_queue_size == pc_queue_size())\n+\t\t\tif (last_pc_queue_size == pc_queue_size() &&\n+\t\t\t last_symlink_queue_size == symlink_queue_size())\n \t\t\t\tdisplay_progress(progress, ++cnt);\n \t\t}\n \t}\n-- \n2.27.0\n\n"},{"id":"403301","messageId":"747e78a0a34c3044a3edf07ca038bd85e7c0afef.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 18/21] entry: use is_dir_sep() when checking leading dirs","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:26Z","receivedAt":"2020-08-10T21:35:43Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The test 'prevent git~1 squatting on Windows' in t7415, adds the file\n'd./a/x' and the submodule 'd\\a' to the index, with `git -c\ncore.protectNTFS=false update-index --add --cacheinfo`. Then it performs\na clone with `--recurse-submodules`. Since \"d./\" and \"d\\\" represent the\nsame entry on NTFS, the operation is expected to fail, because the\nsubmodule directory is not empty by the time \"d\\a\" is cloned. With\nparallel checkout, this condition is still valid: although we call\ncheckout_entry() for gitlinks before we write regular files (which are\ndelayed for later parallel write), the actual submodule cloning only\nhappens after unpack_trees() returns, in builtin/clone.c:checkout().\n\nNote, however, that we do create the submodule directory (and leading\ndirectories) in unpack_trees(). But the current code iterates through\npath components only considering \"/\", not \"\\\", which is also valid on\nWindows. The reason we don't fail to create the leading dir \"d\" for the\ngitlink \"d\\a\" is because, by the time we call mkdir(\"d\\a\"), \"d\" was\nalready created for the regular file 'd./a/x'. Again, this is still true\nfor parallel checkout, since we create leading dirs sequentially, even\nfor entries that are delayed for later writing. But in a following patch,\nwe will allow checkout workers to create the leading directories in\nparallel, for better performance. Therefore, when checkout_entry() is\ncalled for the gitlink \"d\\a\", \"d\" won't be present yet, and mkdir(\"d\\a\")\nwill fail with ENOENT. To solve this, in preparation for the said patch,\nlet's use is_dir_sep() when checking path components, so that\ncheckout_entry() can correctly create \"d\" for the gitlink \"d\\a\".\n\nSigned-off-by: Matheus Tavares \n---\n\nI'm not sure if this is the right way to make t7415 work with\nparallel-checkout; or if we should, perhaps, change the test to add the\nsubmodule at 'a/d'. I'd love if someone more familiar with Windows could\nreview this one.\n\n\n entry.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 6208df23df..19f2c1d132 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -19,7 +19,7 @@ static void create_directories(const char *path, int path_len,\n \t\tdo {\n \t\t\tbuf[len] = path[len];\n \t\t\tlen++;\n-\t\t} while (len < path_len && path[len] != '/');\n+\t\t} while (len < path_len && !is_dir_sep(path[len]));\n \t\tif (len >= path_len)\n \t\t\tbreak;\n \t\tbuf[len] = 0;\n@@ -404,7 +404,7 @@ static int check_path(const char *path, int len, struct stat *st, int skiplen)\n {\n \tconst char *slash = path + len;\n \n-\twhile (path < slash && *slash != '/')\n+\twhile (path < slash && !is_dir_sep(*slash))\n \t\tslash--;\n \tif (!has_dirs_only_path(path, slash - path, skiplen)) {\n \t\terrno = ENOENT;\n-- \n2.27.0\n\n"},{"id":"403302","messageId":"e4aed47fa62cb2d7ed3411c5ef1464025572c2ec.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 19/21] symlinks: make has_dirs_only_path() track FL_NOENT","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:27Z","receivedAt":"2020-08-10T21:35:48Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"In the next patch, the parallel-checkout workers will be able to create\nthe leading directories of the entries being written by themselves, to\nincrease performance. But to do so, the main process will first need to\nremove non-directory files that can potentially be in the way (the\nreasoning is discussed in the next patch). This can be done without much\ncost by calling has_dirs_only_path() for each path component until we\nfind the first one which is not a real directory, which should then be\nremoved. This operations is cheap because it doesn't have to call stat()\nagain for each component, as the information is already cached from the\nprevious call at entry.c:check_path().\n\nHowever, when has_dirs_only_path() returns false, we don't know if the\ncomponent doesn't exist or if it exists as another file type. The best\nwe could do in this case would be to stat() the component again. When\nthere are many files to be checked out inside the same directory (yet\nto be created by a worker), we would have to call stat() for the same\ndirectory once for each path, even though there is nothing to be\nunlinked there. We can skip this stat() calls by making\nhas_dirs_only_path() also ask for FL_NOENT caching, and converting its\nreturn to a tri-state.\n\nNote: since we are now caching FL_NOENT, we also need to manually\ninvalidate the cache when we create a directory in a path previously\ncached as non-existent.\n\nWhile we are here, also remove duplicated comments in\nhas_dirs_only_path() and check_leading_path().\n\nSigned-off-by: Matheus Tavares \n---\n cache.h | 1 +\n entry.c | 11 +++++++++--\n parallel-checkout.c | 3 +++\n symlinks.c | 42 ++++++++++++++++++------------------------\n 4 files changed, 31 insertions(+), 26 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex e2b41c5f8b..7a08cd6f0e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1711,6 +1711,7 @@ int has_symlink_leading_path(const char *name, int len);\n int threaded_has_symlink_leading_path(struct cache_def *, const char *, int);\n int check_leading_path(const char *name, int len);\n int has_dirs_only_path(const char *name, int len, int prefix_len);\n+void reset_default_lstat_cache(void);\n void schedule_dir_for_removal(const char *name, int len);\n void remove_scheduled_dirs(void);\n \ndiff --git a/entry.c b/entry.c\nindex 19f2c1d132..e876adff19 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -14,6 +14,7 @@ static void create_directories(const char *path, int path_len,\n {\n \tchar *buf = xmallocz(path_len);\n \tint len = 0;\n+\tint reset_cache = 0;\n \n \twhile (len < path_len) {\n \t\tdo {\n@@ -31,7 +32,7 @@ static void create_directories(const char *path, int path_len,\n \t\t * we test the path components of the prefix with the\n \t\t * stat() function instead of the lstat() function.\n \t\t */\n-\t\tif (has_dirs_only_path(buf, len, state->base_dir_len))\n+\t\tif (has_dirs_only_path(buf, len, state->base_dir_len) > 0)\n \t\t\tcontinue; /* ok, it is already a directory. */\n \n \t\t/*\n@@ -45,8 +46,14 @@ static void create_directories(const char *path, int path_len,\n \t\t\t !unlink_or_warn(buf) && !mkdir(buf, 0777))\n \t\t\t\tcontinue;\n \t\t\tdie_errno(\"cannot create directory at '%s'\", buf);\n+\t\t} else {\n+\t\t\t/* The cache had FL_NOENT, but we now created a dir */\n+\t\t\treset_cache = 1;\n \t\t}\n \t}\n+\n+\tif (reset_cache)\n+\t\treset_default_lstat_cache();\n \tfree(buf);\n }\n \n@@ -406,7 +413,7 @@ static int check_path(const char *path, int len, struct stat *st, int skiplen)\n \n \twhile (path < slash && !is_dir_sep(*slash))\n \t\tslash--;\n-\tif (!has_dirs_only_path(path, slash - path, skiplen)) {\n+\tif (has_dirs_only_path(path, slash - path, skiplen) <= 0) {\n \t\terrno = ENOENT;\n \t\treturn -1;\n \t}\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex fee93460c1..4d72540256 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -185,6 +185,9 @@ static int handle_results(struct checkout *state)\n \n \tpc_status = PC_HANDLING_RESULTS;\n \n+\t/* Workers might have created dirs, so the cache must be invalidated */\n+\treset_default_lstat_cache();\n+\n \tfor (i = 0; i < parallel_checkout->nr; ++i) {\n \t\tstruct checkout_item *ci = ¶llel_checkout->items[i];\n \t\tstruct stat *st = &ci->st;\ndiff --git a/symlinks.c b/symlinks.c\nindex 69d458a24d..3adf6ef8a1 100644\n--- a/symlinks.c\n+++ b/symlinks.c\n@@ -47,6 +47,11 @@ static inline void reset_lstat_cache(struct cache_def *cache)\n \t */\n }\n \n+void reset_default_lstat_cache(void)\n+{\n+\treset_lstat_cache(&default_cache);\n+}\n+\n #define FL_DIR (1 << 0)\n #define FL_NOENT (1 << 1)\n #define FL_SYMLINK (1 << 2)\n@@ -210,15 +215,6 @@ int has_symlink_leading_path(const char *name, int len)\n \treturn threaded_has_symlink_leading_path(&default_cache, name, len);\n }\n \n-/*\n- * Return zero if path 'name' has a leading symlink component or\n- * if some leading path component does not exists.\n- *\n- * Return -1 if leading path exists and is a directory.\n- *\n- * Return path length if leading path exists and is neither a\n- * directory nor a symlink.\n- */\n int check_leading_path(const char *name, int len)\n {\n \treturn threaded_check_leading_path(&default_cache, name, len);\n@@ -246,30 +242,28 @@ static int threaded_check_leading_path(struct cache_def *cache, const char *name\n \t\treturn match_len;\n }\n \n-/*\n- * Return non-zero if all path components of 'name' exists as a\n- * directory. If prefix_len > 0, we will test with the stat()\n- * function instead of the lstat() function for a prefix length of\n- * 'prefix_len', thus we then allow for symlinks in the prefix part as\n- * long as those points to real existing directories.\n- */\n int has_dirs_only_path(const char *name, int len, int prefix_len)\n {\n \treturn threaded_has_dirs_only_path(&default_cache, name, len, prefix_len);\n }\n \n /*\n- * Return non-zero if all path components of 'name' exists as a\n- * directory. If prefix_len > 0, we will test with the stat()\n- * function instead of the lstat() function for a prefix length of\n- * 'prefix_len', thus we then allow for symlinks in the prefix part as\n- * long as those points to real existing directories.\n+ * Return a positive number if all path components of 'name' exist as\n+ * directories, a negative number if a component does not exist, and 0 otherwise\n+ * (e.g. a component exists but as another file type). If prefix_len > 0, we\n+ * will test with the stat() function instead of the lstat() function for a\n+ * prefix length of 'prefix_len', thus we return +1 for symlinks in the prefix\n+ * part as long as those points to real existing directories.\n */\n static int threaded_has_dirs_only_path(struct cache_def *cache, const char *name, int len, int prefix_len)\n {\n-\treturn lstat_cache(cache, name, len,\n-\t\t\t FL_DIR|FL_FULLPATH, prefix_len) &\n-\t\tFL_DIR;\n+\tint flags = lstat_cache(cache, name, len,\n+\t\t\t\tFL_NOENT|FL_DIR|FL_FULLPATH, prefix_len);\n+\tif (flags & FL_DIR)\n+\t\treturn 1;\n+\tif (flags & FL_NOENT)\n+\t\treturn -1;\n+\treturn 0;\n }\n \n static struct strbuf removal = STRBUF_INIT;\n-- \n2.27.0\n\n"},{"id":"403303","messageId":"d9e8bad5d4e6cf3c44c6e48d28c40536afcba115.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 20/21] parallel-checkout: create leading dirs in workers","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:28Z","receivedAt":"2020-08-10T21:35:53Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Allow the parallel workers to create the leading directories of the\nentries being checked out, instead of pre-creating them in the main\nprocess. This optimization should be more effective on file systems with\nhigher I/O latency.\n\nPart of the process of creating leading dirs is the removal of any\nnon-directory file that could be in the way. This is currently done\ninside entry.c:create_directories(). However, if we were to move this to\nthe workers as well, we would risk removing a file just written by\nanother worker, which collided with the one currently being written. In\na worse scenario, we could remove the file right after a worker have\nclosed it but before it called stat(). To avoid these problems, let's\nremove the non-directory files in the main process. And to avoid the\ncost of extra lstat() calls in this process, we use\nhas_dirs_only_path(), which will have the necessary information already\ncached from check_path().\n\nFinally, to create the leading dirs in the workers, we could re-use\ncreate_directories(). But, unlike the main process, we wouldn't have the\nstat() information cached. Thus, let's use raceproof_create_file(),\nwhich will only stat() the path components after a open() failure,\nsaving us time when creating subsequent files in the same directory.\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 45 ++++++++++++++++++++++++++++++++++++++++++---\n parallel-checkout.c | 42 ++++++++++++++++++++++++++++++++++++------\n 2 files changed, 78 insertions(+), 9 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex e876adff19..5dfd4d150d 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -57,6 +57,43 @@ static void create_directories(const char *path, int path_len,\n \tfree(buf);\n }\n \n+static void remove_non_dirs(const char *path, int path_len,\n+\t\t\t const struct checkout *state)\n+{\n+\tchar *buf = xmallocz(path_len);\n+\tint len = 0;\n+\n+\twhile (len < path_len) {\n+\t\tint ret;\n+\n+\t\tdo {\n+\t\t\tbuf[len] = path[len];\n+\t\t\tlen++;\n+\t\t} while (len < path_len && !is_dir_sep(path[len]));\n+\t\tif (len >= path_len)\n+\t\t\tbreak;\n+\t\tbuf[len] = 0;\n+\n+\t\tret = has_dirs_only_path(buf, len, state->base_dir_len);\n+\n+\t\tif (ret > 0)\n+\t\t\tcontinue; /* Is directory. */\n+\t\tif (ret < 0)\n+\t\t\tbreak; /* No entry */\n+\n+\t\t/* ret == 0: not a directory, let's unlink it. */\n+\n+\t\tif (!state->force)\n+\t\t\tdie(\"'%s' already exists, and it's not a directory\", buf);\n+\n+\t\tif (unlink(buf))\n+\t\t\tdie_errno(\"cannot unlink '%s'\", buf);\n+\t\telse\n+\t\t\tbreak;\n+\t}\n+\tfree(buf);\n+}\n+\n static void remove_subtree(struct strbuf *path)\n {\n \tDIR *dir = opendir(path->buf);\n@@ -555,8 +592,6 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t} else if (state->not_new)\n \t\treturn 0;\n \n-\tcreate_directories(path.buf, path.len, state);\n-\n \tif (nr_checkouts)\n \t\t(*nr_checkouts)++;\n \n@@ -565,9 +600,13 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\tca = &ca_buf;\n \t}\n \n-\tif (!enqueue_checkout(ce, ca))\n+\tif (!enqueue_checkout(ce, ca)) {\n+\t\t/* \"clean\" path so that workers can create leading dirs */\n+\t\tremove_non_dirs(path.buf, path.len, state);\n \t\treturn 0;\n+\t}\n \n+\tcreate_directories(path.buf, path.len, state);\n \treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 4d72540256..5b73d8fa4b 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -298,20 +298,48 @@ static int close_and_clear(int *fd)\n \treturn ret;\n }\n \n+struct ci_open_data {\n+\tint fd;\n+\tunsigned int mode;\n+};\n+\n+static int ci_open(const char *path, void *cb)\n+{\n+\tstruct ci_open_data *data = cb;\n+\tdata->fd = open(path, O_WRONLY | O_CREAT | O_EXCL, data->mode);\n+\n+\tif (data->fd < 0) {\n+\t\t/*\n+\t\t * EISDIR can only indicate path collisions among the entries\n+\t\t * being checked out. We don't need raceproof_create_file() to\n+\t\t * try removing empty dirs. Instead, just let the caller known\n+\t\t * that the path already exists, so that the collision can be\n+\t\t * properly handled later.\n+\t\t */\n+\t\tif (errno == EISDIR)\n+\t\t\terrno = EEXIST;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n {\n-\tunsigned int mode = (ci->ce->ce_mode & 0100) ? 0777 : 0666;\n+\tstruct ci_open_data open_data;\n \tint fd = -1, fstat_done = 0;\n \tstruct strbuf path = STRBUF_INIT;\n \n+\topen_data.mode = (ci->ce->ce_mode & 0100) ? 0777 : 0666;\n \tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n \tstrbuf_add(&path, ci->ce->name, ci->ce->ce_namelen);\n \n-\tfd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, mode);\n-\n-\tif (fd < 0) {\n-\t\tif (errno == EEXIST || errno == EISDIR || errno == ENOENT ||\n-\t\t errno == ENOTDIR) {\n+\t/*\n+\t * The main process already removed any non-directory file that was in\n+\t * the way. So if we find one, it's a path collision.\n+\t */\n+\tif (raceproof_create_file(path.buf, ci_open, &open_data)) {\n+\t\tif (errno == EEXIST || errno == ENOTDIR || errno == ENOENT) {\n \t\t\t/*\n \t\t\t * Errors which probably represent a path collision.\n \t\t\t * Suppress the error message and mark the ci to be\n@@ -325,6 +353,8 @@ void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n \t\tgoto out;\n \t}\n \n+\tfd = open_data.fd;\n+\n \tif (write_checkout_item_to_fd(fd, state, ci, path.buf)) {\n \t\t/* Error was already reported. */\n \t\tci->status = CI_FAILED;\n-- \n2.27.0\n\n"},{"id":"403304","messageId":"6356b499a4c12c86ec12ca6309c12e16662996f9.1597093021.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[RFC PATCH 21/21] parallel-checkout: skip checking the working tree on clone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-10T21:33:29Z","receivedAt":"2020-08-10T21:35:56Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"If the current checkout process is part of a clone, we can skip some\nsteps that check paths in the working tree, as we know it was previously\nempty. More specifically, we can enqueue the entry for parallel checkout\nbefore calling check_path() to see if the path was already present and\nup-to-date. We can also skip calling remove_non_dirs().\n\nNote: this optimization is only possible because the parallel checkout\nmachinery will detect path collisions, and call checkout_entry_ca()\nagain for them, going through the check_path() logic.\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 16 ++++++++++++++--\n 1 file changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 5dfd4d150d..8c03e23811 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -513,12 +513,24 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\treturn 0;\n \t}\n \n-\tif (topath) {\n+\tif (topath || state->clone) {\n \t\tif (S_ISREG(ce->ce_mode) && !ca) {\n \t\t\tconvert_attrs(state->istate, &ca_buf, ce->name);\n \t\t\tca = &ca_buf;\n \t\t}\n-\t\treturn write_entry(ce, topath, ca, state, 1);\n+\t\tif (topath)\n+\t\t\treturn write_entry(ce, topath, ca, state, 1);\n+\t\t/*\n+\t\t * Since we are cloning, there should be no previous files in\n+\t\t * the working tree. So we can skip calling remove_non_dirs()\n+\t\t * and check_path(). (parallel-checkout.c will take care of path\n+\t\t * collision.)\n+\t\t */\n+\t\tif (!enqueue_checkout(ce, ca)) {\n+\t\t\tif (nr_checkouts)\n+\t\t\t\t(*nr_checkouts)++;\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n \t/*\n-- \n2.27.0\n\n"},{"id":"403474","messageId":"a927c55a-3e5a-b5f9-04d2-3d5e1d298b7a@jeffhostetler.com","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH 00/21] [RFC] Parallel checkout","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-08-12T16:57:42Z","receivedAt":"2020-08-12T16:57:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 8/10/20 5:33 PM, Matheus Tavares wrote:\n> This series adds parallel workers to the checkout machinery. The cache\n> entries are distributed among helper processes which are responsible for\n> reading, filtering and writing the blobs to the working tree. This\n> should benefit all commands that call unpack_trees() or check_updates(),\n> such as: checkout, clone, sparse-checkout, checkout-index, etc.\n> \n> This proposal is based on two previous ones, by Duy [1] and Jeff [2]. It\n> uses some of the patches from these two series, with additional changes.\n> The final parallel version was benchmarked during three operations with\n> cold cache in the linux repo: cloning v5.8, checking out v5.8 from\n> v2.6.15 and checking out v5.8 from v5.7. The three tables below show the\n> mean run times and standard deviations for 5 runs in: a local file\n> system, a Linux NFS server and Amazon EFS. The number of workers was\n> chosen based on what produces the best result for each case.\n> \n > ...\n> \n> The first 4 patches come from [2]. I couldn't get in touch with Jeff yet\n> and ask for his approval on then, so I didn't include his Signed-off-by,\n> for the time being.\n\nThis looks like an interesting mixture of our efforts. Thanks for\npicking it up. I got re-tasked earlier this summer and had to put\nit on hold. I've given it a quick read and like the overall shape.\nI still need to give it an in-depth review and run some perf tests\non Windows and on the gigantic Windows and Office repos.\n\nPlease feel free to add my sign-off to those commits.\n\n > ...\n\nJeff\n"},{"id":"404069","messageId":"f3291797-3ef1-8a2b-d6d5-394f3c32c83c@jeffhostetler.com","threadId":"54015","inReplyTo":"7e7527ef3e8a9e71a012f1623e9642c47f7f741c.1597093021.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH 11/21] parallel-checkout: make it truly parallel","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-08-19T21:34:23Z","receivedAt":"2020-08-19T21:34:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 8/10/20 5:33 PM, Matheus Tavares wrote:\n> Use multiple worker processes to distribute the queued entries and call\n> write_checkout_item() in parallel for them. The items are distributed\n> uniformly in contiguous chunks. This minimizes the chances of two\n> workers writing to the same directory simultaneously, which could\n> affect performance due to lock contention in the kernel. Work stealing\n> (or any other format of re-distribution) is not implemented yet.\n> \n> For now, the number of workers is equal to the number of logical cores\n> available. But the next patch will add settings to configure this.\n> Distributed file systems, such as NFS and EFS, can benefit from using\n> more workers than the actual number of cores (see timings below).\n> \n> The parallel version was benchmarked during three operations in the\n> linux repo, with cold cache: cloning v5.8, checking out v5.8 from\n> v2.6.15 (checkout I) and checking out v5.8 from v5.7 (checkout II). The\n> three tables below show the mean run times and standard deviations for\n> 5 runs in a local file system, a Linux NFS server and Amazon EFS. The\n> numbers of workers were chosen based on what produces the best result\n> for each case.\n> \n> Local:\n> \n> Clone Checkout I Checkout II\n> Sequential 8.180 s ± 0.021 s 6.936 s ± 0.030 s 2.585 s ± 0.005 s\n> 10 workers 3.633 s ± 0.040 s 2.288 s ± 0.026 s 1.058 s ± 0.015 s\n> Speedup 2.25 ± 0.03 3.03 ± 0.04 2.44 ± 0.03\n> \n> Linux NFS server (v4.1, on EBS, single availability zone):\n> \n> Clone Checkout I Checkout II\n> Sequential 208.069 s ± 2.522 s 198.610 s ± 1.979 s 54.376 s ± 1.333 s\n> 32 workers 67.078 s ± 0.878 s 64.828 s ± 0.387 s 22.993 s ± 0.252 s\n> Speedup 3.10 ± 0.06 3.06 ± 0.04 2.36 ± 0.06\n> \n> EFS (v4.1, replicated over multiple availability zones):\n> \n> Clone Checkout I Checkout II\n> Sequential 1143.655 s ± 11.819 s 1277.891 s ± 10.481 s 396.891 s ± 7.505 s\n> 64 workers 173.242 s ± 1.484 s 282.421 s ± 1.521 s 165.424 s ± 9.564 s\n> Speedup 6.60 ± 0.09 4.52 ± 0.04 2.40 ± 0.15\n> \n> Local tests were executed in an i7-7700HQ (4 cores with hyper-threading)\n> running Manjaro Linux, with SSD. NFS and EFS tests were executed in an\n> Amazon EC2 c5n.large instance, with 2 vCPUs. The Linux NFS server was\n> running on a m6g.large instance with 1 TB, EBS GP2 volume. Before each\n> timing, the linux repository was removed (or checked out back), and\n> `sync && sysctl vm.drop_caches=3` was executed.\n> \n> Co-authored-by: Nguyễn Thái Ngọc Duy \n> Co-authored-by: Jeff Hostetler \n> Signed-off-by: Matheus Tavares \n> ---\n> .gitignore | 1 +\n> Makefile | 1 +\n> builtin.h | 1 +\n> builtin/checkout--helper.c | 135 +++++++++++++++++++++\n> entry.c | 13 +-\n> git.c | 2 +\n> parallel-checkout.c | 237 +++++++++++++++++++++++++++++++------\n> parallel-checkout.h | 74 +++++++++++-\n> 8 files changed, 425 insertions(+), 39 deletions(-)\n> create mode 100644 builtin/checkout--helper.c\n> \n> diff --git a/.gitignore b/.gitignore\n> index ee509a2ad2..6c01f0a58c 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -33,6 +33,7 @@\n> /git-check-mailmap\n> /git-check-ref-format\n> /git-checkout\n> +/git-checkout--helper\n> /git-checkout-index\n> /git-cherry\n> /git-cherry-pick\n> diff --git a/Makefile b/Makefile\n> index caab8e6401..926473d484 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1049,6 +1049,7 @@ BUILTIN_OBJS += builtin/check-attr.o\n> BUILTIN_OBJS += builtin/check-ignore.o\n> BUILTIN_OBJS += builtin/check-mailmap.o\n> BUILTIN_OBJS += builtin/check-ref-format.o\n> +BUILTIN_OBJS += builtin/checkout--helper.o\n> BUILTIN_OBJS += builtin/checkout-index.o\n> BUILTIN_OBJS += builtin/checkout.o\n> BUILTIN_OBJS += builtin/clean.o\n> diff --git a/builtin.h b/builtin.h\n> index a5ae15bfe5..5790c68750 100644\n> --- a/builtin.h\n> +++ b/builtin.h\n> @@ -122,6 +122,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix);\n> int cmd_bundle(int argc, const char **argv, const char *prefix);\n> int cmd_cat_file(int argc, const char **argv, const char *prefix);\n> int cmd_checkout(int argc, const char **argv, const char *prefix);\n> +int cmd_checkout__helper(int argc, const char **argv, const char *prefix);\n> int cmd_checkout_index(int argc, const char **argv, const char *prefix);\n> int cmd_check_attr(int argc, const char **argv, const char *prefix);\n> int cmd_check_ignore(int argc, const char **argv, const char *prefix);\n> diff --git a/builtin/checkout--helper.c b/builtin/checkout--helper.c\n> new file mode 100644\n> index 0000000000..269cf02feb\n> --- /dev/null\n> +++ b/builtin/checkout--helper.c\n> @@ -0,0 +1,135 @@\n> +#include \"builtin.h\"\n> +#include \"config.h\"\n> +#include \"entry.h\"\n> +#include \"parallel-checkout.h\"\n> +#include \"parse-options.h\"\n> +#include \"pkt-line.h\"\n> +\n> +static void packet_to_ci(char *line, int len, struct checkout_item *ci)\n> +{\n> +\tstruct ci_fixed_portion *fixed_portion;\n> +\tchar *encoding, *variant;\n> +\n> +\tif (len < sizeof(struct ci_fixed_portion))\n> +\t\tBUG(\"checkout worker received too short item (got %d, exp %d)\",\n> +\t\t len, (int)sizeof(struct ci_fixed_portion));\n> +\n> +\tfixed_portion = (struct ci_fixed_portion *)line;\n> +\n> +\tif (len - sizeof(struct ci_fixed_portion) !=\n> +\t\tfixed_portion->name_len + fixed_portion->working_tree_encoding_len)\n> +\t\tBUG(\"checkout worker received corrupted item\");\n> +\n> +\tvariant = line + sizeof(struct ci_fixed_portion);\n> +\tif (fixed_portion->working_tree_encoding_len) {\n> +\t\tencoding = xmemdupz(variant,\n> +\t\t\t\t fixed_portion->working_tree_encoding_len);\n> +\t\tvariant += fixed_portion->working_tree_encoding_len;\n> +\t} else {\n> +\t\tencoding = NULL;\n> +\t}\n> +\n> +\tmemset(ci, 0, sizeof(*ci));\n> +\tci->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n> +\tci->ce->ce_namelen = fixed_portion->name_len;\n> +\tci->ce->ce_mode = fixed_portion->ce_mode;\n> +\tmemcpy(ci->ce->name, variant, ci->ce->ce_namelen);\n> +\toidcpy(&ci->ce->oid, &fixed_portion->oid);\n> +\n> +\tci->id = fixed_portion->id;\n> +\tci->ca.attr_action = fixed_portion->attr_action;\n> +\tci->ca.crlf_action = fixed_portion->crlf_action;\n> +\tci->ca.ident = fixed_portion->ident;\n> +\tci->ca.working_tree_encoding = encoding;\n> +}\n> +\n> +static void report_result(struct checkout_item *ci)\n> +{\n> +\tstruct ci_result res = { 0 };\n> +\tsize_t size;\n> +\n> +\tres.id = ci->id;\n> +\tres.status = ci->status;\n> +\n> +\tif (ci->status == CI_SUCCESS) {\n> +\t\tres.st = ci->st;\n> +\t\tsize = sizeof(res);\n> +\t} else {\n> +\t\tsize = ci_result_base_size();\n> +\t}\n> +\n> +\tpacket_write(1, (const char *)&res, size);\n> +}\n> +\n> +/* Free the worker-side malloced data, but not the ci itself. */\n> +static void release_checkout_item_data(struct checkout_item *ci)\n> +{\n> +\tfree((char *)ci->ca.working_tree_encoding);\n> +\tdiscard_cache_entry(ci->ce);\n> +}\n> +\n> +static void worker_loop(struct checkout *state)\n> +{\n> +\tstruct checkout_item *items = NULL;\n> +\tsize_t i, nr = 0, alloc = 0;\n> +\n> +\twhile (1) {\n> +\t\tint len;\n> +\t\tchar *line = packet_read_line(0, &len);\n> +\n> +\t\tif (!line)\n> +\t\t\tbreak;\n> +\n> +\t\tALLOC_GROW(items, nr + 1, alloc);\n> +\t\tpacket_to_ci(line, len, &items[nr++]);\n> +\t}\n> +\n> +\tfor (i = 0; i < nr; ++i) {\n> +\t\tstruct checkout_item *ci = &items[i];\n> +\t\twrite_checkout_item(state, ci);\n> +\t\treport_result(ci);\n> +\t\trelease_checkout_item_data(ci);\n> +\t}\n> +\n> +\tpacket_flush(1);\n> +\n> +\tfree(items);\n> +}\n> +\n> +static const char * const checkout_helper_usage[] = {\n> +\tN_(\"git checkout--helper []\"),\n> +\tNULL\n> +};\n> +\n> +int cmd_checkout__helper(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstruct checkout state = CHECKOUT_INIT;\n> +\tstruct option checkout_helper_options[] = {\n> +\t\tOPT_STRING(0, \"prefix\", &state.base_dir, N_(\"string\"),\n> +\t\t\tN_(\"when creating files, prepend \")),\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n> +\t\tusage_with_options(checkout_helper_usage,\n> +\t\t\t\t checkout_helper_options);\n> +\n> +\tgit_config(git_default_config, NULL);\n> +\targc = parse_options(argc, argv, prefix, checkout_helper_options,\n> +\t\t\t checkout_helper_usage, 0);\n> +\tif (argc > 0)\n> +\t\tusage_with_options(checkout_helper_usage, checkout_helper_options);\n> +\n> +\tif (state.base_dir)\n> +\t\tstate.base_dir_len = strlen(state.base_dir);\n> +\n> +\t/*\n> +\t * Setting this on worker won't actually update the index. We just need\n> +\t * to pretend so to induce the checkout machinery to stat() the written\n> +\t * entries.\n> +\t */\n> +\tstate.refresh_cache = 1;\n> +\n> +\tworker_loop(&state);\n> +\treturn 0;\n> +}\n> diff --git a/entry.c b/entry.c\n> index 47c2c20d5a..b6c808dffa 100644\n> --- a/entry.c\n> +++ b/entry.c\n> @@ -427,8 +427,17 @@ static void mark_colliding_entries(const struct checkout *state,\n> \tfor (i = 0; i < state->istate->cache_nr; i++) {\n> \t\tstruct cache_entry *dup = state->istate->cache[i];\n> \n> -\t\tif (dup == ce)\n> -\t\t\tbreak;\n> +\t\tif (dup == ce) {\n> +\t\t\t/*\n> +\t\t\t * Parallel checkout creates the files in a racy order.\n> +\t\t\t * So the other side of the collision may appear after\n> +\t\t\t * the given cache_entry in the array.\n> +\t\t\t */\n> +\t\t\tif (parallel_checkout_status() == PC_HANDLING_RESULTS)\n> +\t\t\t\tcontinue;\n> +\t\t\telse\n> +\t\t\t\tbreak;\n> +\t\t}\n> \n> \t\tif (dup->ce_flags & (CE_MATCHED | CE_VALID | CE_SKIP_WORKTREE))\n> \t\t\tcontinue;\n> diff --git a/git.c b/git.c\n> index 8bd1d7551d..78c7bd412c 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -486,6 +486,8 @@ static struct cmd_struct commands[] = {\n> \t{ \"check-mailmap\", cmd_check_mailmap, RUN_SETUP },\n> \t{ \"check-ref-format\", cmd_check_ref_format, NO_PARSEOPT },\n> \t{ \"checkout\", cmd_checkout, RUN_SETUP | NEED_WORK_TREE },\n> +\t{ \"checkout--helper\", cmd_checkout__helper,\n> +\t\tRUN_SETUP | NEED_WORK_TREE | SUPPORT_SUPER_PREFIX },\n> \t{ \"checkout-index\", cmd_checkout_index,\n> \t\tRUN_SETUP | NEED_WORK_TREE},\n> \t{ \"cherry\", cmd_cherry, RUN_SETUP },\n> diff --git a/parallel-checkout.c b/parallel-checkout.c\n> index e3b44eeb34..ec42342bc8 100644\n> --- a/parallel-checkout.c\n> +++ b/parallel-checkout.c\n> @@ -1,39 +1,23 @@\n> #include \"cache.h\"\n> #include \"entry.h\"\n> #include \"parallel-checkout.h\"\n> +#include \"pkt-line.h\"\n> +#include \"run-command.h\"\n> #include \"streaming.h\"\n> \n> -enum ci_status {\n> -\tCI_PENDING = 0,\n> -\tCI_SUCCESS,\n> -\tCI_RETRY,\n> -\tCI_FAILED,\n> -};\n> -\n> -struct checkout_item {\n> -\t/* pointer to a istate->cache[] entry. Not owned by us. */\n> -\tstruct cache_entry *ce;\n> -\tstruct conv_attrs ca;\n> -\tstruct stat st;\n> -\tenum ci_status status;\n> -};\n> -\n> struct parallel_checkout {\n> \tstruct checkout_item *items;\n> \tsize_t nr, alloc;\n> };\n> \n> static struct parallel_checkout *parallel_checkout = NULL;\n> -\n> -enum pc_status {\n> -\tPC_UNINITIALIZED = 0,\n> -\tPC_ACCEPTING_ENTRIES,\n> -\tPC_RUNNING,\n> -\tPC_HANDLING_RESULTS,\n> -};\n> -\n> static enum pc_status pc_status = PC_UNINITIALIZED;\n> \n> +enum pc_status parallel_checkout_status(void)\n> +{\n> +\treturn pc_status;\n> +}\n> +\n> void init_parallel_checkout(void)\n> {\n> \tif (parallel_checkout)\n> @@ -113,9 +97,11 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n> \tALLOC_GROW(parallel_checkout->items, parallel_checkout->nr + 1,\n> \t\t parallel_checkout->alloc);\n> \n> -\tci = ¶llel_checkout->items[parallel_checkout->nr++];\n> +\tci = ¶llel_checkout->items[parallel_checkout->nr];\n> \tci->ce = ce;\n> \tmemcpy(&ci->ca, ca, sizeof(ci->ca));\n> +\tci->id = parallel_checkout->nr;\n> +\tparallel_checkout->nr++;\n> \n> \treturn 0;\n> }\n> @@ -200,7 +186,8 @@ static int write_checkout_item_to_fd(int fd, struct checkout *state,\n> \t/*\n> \t * checkout metadata is used to give context for external process\n> \t * filters. Files requiring such filters are not eligible for parallel\n> -\t * checkout, so pass NULL.\n> +\t * checkout, so pass NULL. Note: if that changes, the metadata must also\n> +\t * be passed from the main process to the workers.\n> \t */\n> \tret = convert_to_working_tree_ca(&ci->ca, ci->ce->name, new_blob, size,\n> \t\t\t\t\t &buf, NULL);\n> @@ -241,14 +228,14 @@ static int check_leading_dirs(const char *path, int len, int prefix_len)\n> \treturn has_dirs_only_path(path, slash - path, prefix_len);\n> }\n> \n> -static void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n> +void write_checkout_item(struct checkout *state, struct checkout_item *ci)\n> {\n> \tunsigned int mode = (ci->ce->ce_mode & 0100) ? 0777 : 0666;\n> \tint fd = -1, fstat_done = 0;\n> \tstruct strbuf path = STRBUF_INIT;\n> \n> \tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n> -\tstrbuf_add(&path, ci->ce->name, ce_namelen(ci->ce));\n> +\tstrbuf_add(&path, ci->ce->name, ci->ce->ce_namelen);\n> \n> \t/*\n> \t * At this point, leading dirs should have already been created. But if\n> @@ -311,30 +298,214 @@ static void write_checkout_item(struct checkout *state, struct checkout_item *ci\n> \tstrbuf_release(&path);\n> }\n> \n> -static int run_checkout_sequentially(struct checkout *state)\n> +static void send_one_item(int fd, struct checkout_item *ci)\n> +{\n> +\tsize_t len_data;\n> +\tchar *data, *variant;\n> +\tstruct ci_fixed_portion *fixed_portion;\n> +\tconst char *working_tree_encoding = ci->ca.working_tree_encoding;\n> +\tsize_t name_len = ci->ce->ce_namelen;\n> +\tsize_t working_tree_encoding_len = working_tree_encoding ?\n> +\t\t\t\t\t strlen(working_tree_encoding) : 0;\n> +\n> +\tlen_data = sizeof(struct ci_fixed_portion) + name_len +\n> +\t\t working_tree_encoding_len;\n> +\n> +\tdata = xcalloc(1, len_data);\n> +\n> +\tfixed_portion = (struct ci_fixed_portion *)data;\n> +\tfixed_portion->id = ci->id;\n> +\toidcpy(&fixed_portion->oid, &ci->ce->oid);\n> +\tfixed_portion->ce_mode = ci->ce->ce_mode;\n> +\tfixed_portion->attr_action = ci->ca.attr_action;\n> +\tfixed_portion->crlf_action = ci->ca.crlf_action;\n> +\tfixed_portion->ident = ci->ca.ident;\n> +\tfixed_portion->name_len = name_len;\n> +\tfixed_portion->working_tree_encoding_len = working_tree_encoding_len;\n> +\n> +\tvariant = data + sizeof(*fixed_portion);\n> +\tif (working_tree_encoding_len) {\n> +\t\tmemcpy(variant, working_tree_encoding, working_tree_encoding_len);\n> +\t\tvariant += working_tree_encoding_len;\n> +\t}\n> +\tmemcpy(variant, ci->ce->name, name_len);\n> +\n> +\tpacket_write(fd, data, len_data);\n> +\n> +\tfree(data);\n> +}\n> +\n> +static void send_batch(int fd, size_t start, size_t nr)\n> {\n> \tsize_t i;\n> +\tfor (i = 0; i < nr; ++i)\n> +\t\tsend_one_item(fd, ¶llel_checkout->items[start + i]);\n> +\tpacket_flush(fd);\n> +}\n> \n> -\tfor (i = 0; i < parallel_checkout->nr; ++i) {\n> -\t\tstruct checkout_item *ci = ¶llel_checkout->items[i];\n> -\t\twrite_checkout_item(state, ci);\n> +static struct child_process *setup_workers(struct checkout *state, int num_workers)\n> +{\n> +\tstruct child_process *workers;\n> +\tint i, workers_with_one_extra_item;\n> +\tsize_t base_batch_size, next_to_assign = 0;\n> +\n> +\tbase_batch_size = parallel_checkout->nr / num_workers;\n> +\tworkers_with_one_extra_item = parallel_checkout->nr % num_workers;\n> +\tALLOC_ARRAY(workers, num_workers);\n> +\n> +\tfor (i = 0; i < num_workers; ++i) {\n> +\t\tstruct child_process *cp = &workers[i];\n> +\t\tsize_t batch_size = base_batch_size;\n> +\n> +\t\tchild_process_init(cp);\n> +\t\tcp->git_cmd = 1;\n> +\t\tcp->in = -1;\n> +\t\tcp->out = -1;\n> +\t\tstrvec_push(&cp->args, \"checkout--helper\");\n> +\t\tif (state->base_dir_len)\n> +\t\t\tstrvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n> +\t\tif (start_command(cp))\n> +\t\t\tdie(_(\"failed to spawn checkout worker\"));\n\nWe should consider splitting this loop into one to start the helpers\nand another loop to later send them their assignments. This would\nbetter hide the process startup costs.\n\nWhen comparing this version with my pc-p4-core branch on Windows,\nI was seeing a delay of 0.8 seconds between each helper process\ngetting started. And on my version a delay of 0.2 between them.\n\nI was testing with a huge repo and the batch size was ~200k, so it\nforeground process was stuck in send_batch() for a while before it\ncould start the next helper process.\n\nIt still takes the same amount of time to send each batch, but\nthe 2nd thru nth helpers can be starting while we are sending the\nbatch to the 1st helper. (This might just be a Windows issue because\nof how slow process creation is on Windows....)\n\nWe could maybe also save a little time splitting the batches\nacross the helpers, but that's a refinement for later...\n\n> +\n> +\t\t/* distribute the extra work evenly */\n> +\t\tif (i < workers_with_one_extra_item)\n> +\t\t\tbatch_size++;\n> +\n> +\t\tsend_batch(cp->in, next_to_assign, batch_size);\n> +\t\tnext_to_assign += batch_size;\n> \t}\n> \n> +\treturn workers;\n> +}\n> +\n> +static void finish_workers(struct child_process *workers, int num_workers)\n> +{\n> +\tint i;\n> +\tfor (i = 0; i < num_workers; ++i) {\n> +\t\tstruct child_process *w = &workers[i];\n> +\t\tif (w->in >= 0)\n> +\t\t\tclose(w->in);\n> +\t\tif (w->out >= 0)\n> +\t\t\tclose(w->out);\n\nYou might also consider splitting this loop too. The net-net here\nis that the foreground process closes the handle to the child and\nwaits for the child to exit -- which it will because it get EOF on\nits stdin.\n\nBut the foreground process is stuck in a wait() for it to do so.\n\nYou could make finish_workers() just call close() on all the child\nhandles and then have an atexit() handler to actually wait() and\nreap them. This would let the children exit asynchronously (while\nthe caller here in the foreground process is updating the index\non disk, for example).\n\n\n> +\t\tif (finish_command(w))\n> +\t\t\tdie(_(\"checkout worker finished with error\"));\n> +\t}\n> +\tfree(workers);\n> +}\n> +\n> +static void parse_and_save_result(const char *line, int len)\n> +{\n> +\tstruct ci_result *res;\n> +\tstruct checkout_item *ci;\n> +\n> +\t/*\n> +\t * Worker should send either the full result struct or just the base\n> +\t * (i.e. no stat data).\n> +\t */\n> +\tif (len != ci_result_base_size() && len != sizeof(struct ci_result))\n> +\t\tBUG(\"received corrupted item from checkout worker\");\n> +\n> +\tres = (struct ci_result *)line;\n> +\n> +\tif (res->id > parallel_checkout->nr)\n> +\t\tBUG(\"checkout worker sent unknown item id\");\n> +\n> +\tci = ¶llel_checkout->items[res->id];\n> +\tci->status = res->status;\n> +\n> +\t/*\n> +\t * Worker only sends stat data on success. Otherwise, we *cannot* access\n> +\t * res->st as that will be an invalid address.\n> +\t */\n> +\tif (res->status == CI_SUCCESS)\n> +\t\tci->st = res->st;\n> +}\n> +\n> +static void gather_results_from_workers(struct child_process *workers,\n> +\t\t\t\t\tint num_workers)\n> +{\n> +\tint i, active_workers = num_workers;\n> +\tstruct pollfd *pfds;\n> +\n> +\tCALLOC_ARRAY(pfds, num_workers);\n> +\tfor (i = 0; i < num_workers; ++i) {\n> +\t\tpfds[i].fd = workers[i].out;\n> +\t\tpfds[i].events = POLLIN;\n> +\t}\n> +\n> +\twhile (active_workers) {\n> +\t\tint nr = poll(pfds, num_workers, -1);\n> +\n> +\t\tif (nr < 0) {\n> +\t\t\tif (errno == EINTR)\n> +\t\t\t\tcontinue;\n> +\t\t\tdie_errno(\"failed to poll checkout workers\");\n> +\t\t}\n> +\n> +\t\tfor (i = 0; i < num_workers && nr > 0; ++i) {\n> +\t\t\tstruct pollfd *pfd = &pfds[i];\n> +\n> +\t\t\tif (!pfd->revents)\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\tif (pfd->revents & POLLIN) {\n> +\t\t\t\tint len;\n> +\t\t\t\tconst char *line = packet_read_line(pfd->fd, &len);\n> +\n> +\t\t\t\tif (!line) {\n> +\t\t\t\t\tpfd->fd = -1;\n> +\t\t\t\t\tactive_workers--;\n> +\t\t\t\t} else {\n> +\t\t\t\t\tparse_and_save_result(line, len);\n> +\t\t\t\t}\n> +\t\t\t} else if (pfd->revents & POLLHUP) {\n> +\t\t\t\tpfd->fd = -1;\n> +\t\t\t\tactive_workers--;\n> +\t\t\t} else if (pfd->revents & (POLLNVAL | POLLERR)) {\n> +\t\t\t\tdie(_(\"error polling from checkout worker\"));\n> +\t\t\t}\n> +\n> +\t\t\tnr--;\n> +\t\t}\n> +\t}\n> +\n> +\tfree(pfds);\n> +}\n> +\n> +static int run_checkout_sequentially(struct checkout *state)\n> +{\n> +\tsize_t i;\n> +\tfor (i = 0; i < parallel_checkout->nr; ++i)\n> +\t\twrite_checkout_item(state, ¶llel_checkout->items[i]);\n> \treturn handle_results(state);\n> }\n> \n> +static const int workers_threshold = 0;\n> \n> int run_parallel_checkout(struct checkout *state)\n> {\n> -\tint ret;\n> +\tint num_workers = online_cpus();\n> +\tint ret = 0;\n> +\tstruct child_process *workers;\n> \n> \tif (!parallel_checkout)\n> \t\tBUG(\"cannot run parallel checkout: not initialized yet\");\n> \n> \tpc_status = PC_RUNNING;\n> \n> -\tret = run_checkout_sequentially(state);\n> +\tif (parallel_checkout->nr == 0) {\n> +\t\tgoto done;\n> +\t} else if (parallel_checkout->nr < workers_threshold || num_workers == 1) {\n> +\t\tret = run_checkout_sequentially(state);\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tworkers = setup_workers(state, num_workers);\n> +\tgather_results_from_workers(workers, num_workers);\n> +\tfinish_workers(workers, num_workers);\n> +\tret = handle_results(state);\n> \n> +done:\n> \tfinish_parallel_checkout();\n> \treturn ret;\n> }\n> diff --git a/parallel-checkout.h b/parallel-checkout.h\n> index 8eef59ffcd..f25f2874ae 100644\n> --- a/parallel-checkout.h\n> +++ b/parallel-checkout.h\n> @@ -1,10 +1,21 @@\n> #ifndef PARALLEL_CHECKOUT_H\n> #define PARALLEL_CHECKOUT_H\n> \n> -struct cache_entry;\n> -struct checkout;\n> -struct conv_attrs;\n> +#include \"entry.h\"\n> +#include \"convert.h\"\n> \n> +/****************************************************************\n> + * Users of parallel checkout\n> + ****************************************************************/\n> +\n> +enum pc_status {\n> +\tPC_UNINITIALIZED = 0,\n> +\tPC_ACCEPTING_ENTRIES,\n> +\tPC_RUNNING,\n> +\tPC_HANDLING_RESULTS,\n> +};\n> +\n> +enum pc_status parallel_checkout_status(void);\n> void init_parallel_checkout(void);\n> \n> /*\n> @@ -14,7 +25,62 @@ void init_parallel_checkout(void);\n> */\n> int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n> \n> -/* Write all the queued entries, returning 0 on success.*/\n> +/* Write all the queued entries, returning 0 on success. */\n> int run_parallel_checkout(struct checkout *state);\n> \n> +/****************************************************************\n> + * Interface with checkout--helper\n> + ****************************************************************/\n> +\n> +enum ci_status {\n> +\tCI_PENDING = 0,\n> +\tCI_SUCCESS,\n> +\tCI_RETRY,\n> +\tCI_FAILED,\n> +};\n> +\n> +struct checkout_item {\n> +\t/*\n> +\t * In main process ce points to a istate->cache[] entry. Thus, it's not\n> +\t * owned by us. In workers they own the memory, which *must be* released.\n> +\t */\n> +\tstruct cache_entry *ce;\n> +\tstruct conv_attrs ca;\n> +\tsize_t id; /* position in parallel_checkout->items[] of main process */\n> +\n> +\t/* Output fields, sent from workers. */\n> +\tenum ci_status status;\n> +\tstruct stat st;\n> +};\n> +\n> +/*\n> + * The fixed-size portion of `struct checkout_item` that is sent to the workers.\n> + * Following this will be 2 strings: ca.working_tree_encoding and ce.name; These\n> + * are NOT null terminated, since we have the size in the fixed portion.\n> + */\n> +struct ci_fixed_portion {\n> +\tsize_t id;\n> +\tstruct object_id oid;\n> +\tunsigned int ce_mode;\n> +\tenum crlf_action attr_action;\n> +\tenum crlf_action crlf_action;\n> +\tint ident;\n> +\tsize_t working_tree_encoding_len;\n> +\tsize_t name_len;\n> +};\n> +\n> +/*\n> + * The `struct checkout_item` fields returned by the workers. The order is\n> + * important here, specially stat being the last one, as it is omitted on error.\n> + */\n> +struct ci_result {\n> +\tsize_t id;\n> +\tenum ci_status status;\n> +\tstruct stat st;\n> +};\n> +\n> +#define ci_result_base_size() offsetof(struct ci_result, st)\n> +\n> +void write_checkout_item(struct checkout *state, struct checkout_item *ci);\n> +\n> #endif /* PARALLEL_CHECKOUT_H */\n> \n"},{"id":"404079","messageId":"CAHd-oW5QjLPmvRL0YojL=NiO-ciY7rNZkQuvCtnF-1vmR4ykKQ@mail.gmail.com","threadId":"54015","inReplyTo":"f3291797-3ef1-8a2b-d6d5-394f3c32c83c@jeffhostetler.com","subject":"Re: [RFC PATCH 11/21] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-08-20T01:33:13Z","receivedAt":"2020-08-20T01:33:29Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Aug 19, 2020 at 6:34 PM Jeff Hostetler wrote:\n>\n> On 8/10/20 5:33 PM, Matheus Tavares wrote:\n> >\n> > +static struct child_process *setup_workers(struct checkout *state, int num_workers)\n> > +{\n> > + struct child_process *workers;\n> > + int i, workers_with_one_extra_item;\n> > + size_t base_batch_size, next_to_assign = 0;\n> > +\n> > + base_batch_size = parallel_checkout->nr / num_workers;\n> > + workers_with_one_extra_item = parallel_checkout->nr % num_workers;\n> > + ALLOC_ARRAY(workers, num_workers);\n> > +\n> > + for (i = 0; i < num_workers; ++i) {\n> > + struct child_process *cp = &workers[i];\n> > + size_t batch_size = base_batch_size;\n> > +\n> > + child_process_init(cp);\n> > + cp->git_cmd = 1;\n> > + cp->in = -1;\n> > + cp->out = -1;\n> > + strvec_push(&cp->args, \"checkout--helper\");\n> > + if (state->base_dir_len)\n> > + strvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n> > + if (start_command(cp))\n> > + die(_(\"failed to spawn checkout worker\"));\n>\n> We should consider splitting this loop into one to start the helpers\n> and another loop to later send them their assignments. This would\n> better hide the process startup costs.\n>\n> When comparing this version with my pc-p4-core branch on Windows,\n> I was seeing a delay of 0.8 seconds between each helper process\n> getting started. And on my version a delay of 0.2 between them.\n>\n> I was testing with a huge repo and the batch size was ~200k, so it\n> foreground process was stuck in send_batch() for a while before it\n> could start the next helper process.\n>\n> It still takes the same amount of time to send each batch, but\n> the 2nd thru nth helpers can be starting while we are sending the\n> batch to the 1st helper. (This might just be a Windows issue because\n> of how slow process creation is on Windows....)\n\nThanks for the explanation. I will split the loop in v2.\n\n> We could maybe also save a little time splitting the batches\n> across the helpers, but that's a refinement for later...\n>\n> > +\n> > + /* distribute the extra work evenly */\n> > + if (i < workers_with_one_extra_item)\n> > + batch_size++;\n> > +\n> > + send_batch(cp->in, next_to_assign, batch_size);\n> > + next_to_assign += batch_size;\n> > }\n> >\n> > + return workers;\n> > +}\n> > +\n> > +static void finish_workers(struct child_process *workers, int num_workers)\n> > +{\n> > + int i;\n> > + for (i = 0; i < num_workers; ++i) {\n> > + struct child_process *w = &workers[i];\n> > + if (w->in >= 0)\n> > + close(w->in);\n> > + if (w->out >= 0)\n> > + close(w->out);\n>\n> You might also consider splitting this loop too. The net-net here\n> is that the foreground process closes the handle to the child and\n> waits for the child to exit -- which it will because it get EOF on\n> its stdin.\n>\n> But the foreground process is stuck in a wait() for it to do so.\n>\n> You could make finish_workers() just call close() on all the child\n> handles and then have an atexit() handler to actually wait() and\n> reap them. This would let the children exit asynchronously (while\n> the caller here in the foreground process is updating the index\n> on disk, for example).\n\nMakes sense, thanks. And I think we could achieve this by setting both\n`clean_on_exit` and `wait_after_clean` on the child_process struct,\nright? (BTW, I have just noticed that we probably want these flags set\nfor another reason too: we wouldn't want the workers to keep checking\nout files if the main process was killed.)\n\nMaybe the downside of using the atexit() handler, instead of calling\nfinish_command(), would be that we cannot free the `workers` array\nwhen cleaning up parallel-checkout, right? Also, we wouldn't be able\nto report an error if the worker ends with an error code (but at this\npoint it would have already sent all the results to the foreground\nprocess, anyway).\n\nDo you think that we can mitigate the wait() cost by just splitting\nthe loop in two, like we are going to do in setup_workers()?\n\n>\n> > + if (finish_command(w))\n> > + die(_(\"checkout worker finished with error\"));\n> > + }\n> > + free(workers);\n> > +}\n"},{"id":"404096","messageId":"5327b0e3-fdaa-11e1-f7da-832eb988555e@jeffhostetler.com","threadId":"54015","inReplyTo":"CAHd-oW5QjLPmvRL0YojL=NiO-ciY7rNZkQuvCtnF-1vmR4ykKQ@mail.gmail.com","subject":"Re: [RFC PATCH 11/21] parallel-checkout: make it truly parallel","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-08-20T14:39:49Z","receivedAt":"2020-08-20T14:40:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 8/19/20 9:33 PM, Matheus Tavares Bernardino wrote:\n> On Wed, Aug 19, 2020 at 6:34 PM Jeff Hostetler wrote:\n>>\n>> On 8/10/20 5:33 PM, Matheus Tavares wrote:\n>>>\n>>> +static struct child_process *setup_workers(struct checkout *state, int num_workers)\n>>> +{\n>>> + struct child_process *workers;\n>>> + int i, workers_with_one_extra_item;\n>>> + size_t base_batch_size, next_to_assign = 0;\n>>> +\n>>> + base_batch_size = parallel_checkout->nr / num_workers;\n>>> + workers_with_one_extra_item = parallel_checkout->nr % num_workers;\n>>> + ALLOC_ARRAY(workers, num_workers);\n>>> +\n>>> + for (i = 0; i < num_workers; ++i) {\n>>> + struct child_process *cp = &workers[i];\n>>> + size_t batch_size = base_batch_size;\n>>> +\n>>> + child_process_init(cp);\n>>> + cp->git_cmd = 1;\n>>> + cp->in = -1;\n>>> + cp->out = -1;\n>>> + strvec_push(&cp->args, \"checkout--helper\");\n>>> + if (state->base_dir_len)\n>>> + strvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n>>> + if (start_command(cp))\n>>> + die(_(\"failed to spawn checkout worker\"));\n>>\n>> We should consider splitting this loop into one to start the helpers\n>> and another loop to later send them their assignments. This would\n>> better hide the process startup costs.\n>>\n>> When comparing this version with my pc-p4-core branch on Windows,\n>> I was seeing a delay of 0.8 seconds between each helper process\n>> getting started. And on my version a delay of 0.2 between them.\n>>\n>> I was testing with a huge repo and the batch size was ~200k, so it\n>> foreground process was stuck in send_batch() for a while before it\n>> could start the next helper process.\n>>\n>> It still takes the same amount of time to send each batch, but\n>> the 2nd thru nth helpers can be starting while we are sending the\n>> batch to the 1st helper. (This might just be a Windows issue because\n>> of how slow process creation is on Windows....)\n> \n> Thanks for the explanation. I will split the loop in v2.\n> \n>> We could maybe also save a little time splitting the batches\n>> across the helpers, but that's a refinement for later...\n>>\n>>> +\n>>> + /* distribute the extra work evenly */\n>>> + if (i < workers_with_one_extra_item)\n>>> + batch_size++;\n>>> +\n>>> + send_batch(cp->in, next_to_assign, batch_size);\n>>> + next_to_assign += batch_size;\n>>> }\n>>>\n>>> + return workers;\n>>> +}\n>>> +\n>>> +static void finish_workers(struct child_process *workers, int num_workers)\n>>> +{\n>>> + int i;\n>>> + for (i = 0; i < num_workers; ++i) {\n>>> + struct child_process *w = &workers[i];\n>>> + if (w->in >= 0)\n>>> + close(w->in);\n>>> + if (w->out >= 0)\n>>> + close(w->out);\n>>\n>> You might also consider splitting this loop too. The net-net here\n>> is that the foreground process closes the handle to the child and\n>> waits for the child to exit -- which it will because it get EOF on\n>> its stdin.\n>>\n>> But the foreground process is stuck in a wait() for it to do so.\n>>\n>> You could make finish_workers() just call close() on all the child\n>> handles and then have an atexit() handler to actually wait() and\n>> reap them. This would let the children exit asynchronously (while\n>> the caller here in the foreground process is updating the index\n>> on disk, for example).\n> \n> Makes sense, thanks. And I think we could achieve this by setting both\n> `clean_on_exit` and `wait_after_clean` on the child_process struct,\n> right? (BTW, I have just noticed that we probably want these flags set\n> for another reason too: we wouldn't want the workers to keep checking\n> out files if the main process was killed.)\n> \n> Maybe the downside of using the atexit() handler, instead of calling\n> finish_command(), would be that we cannot free the `workers` array\n> when cleaning up parallel-checkout, right? Also, we wouldn't be able\n> to report an error if the worker ends with an error code (but at this\n> point it would have already sent all the results to the foreground\n> process, anyway).\n> \n> Do you think that we can mitigate the wait() cost by just splitting\n> the loop in two, like we are going to do in setup_workers()?\n\nYeah, I was thinking about that after I hit send, that it'd be simpler\nto do a close loop and then a wait loop and not bother with the atexit\ncomplexity and yet get us most of the gains.\n\n> \n>>\n>>> + if (finish_command(w))\n>>> + die(_(\"checkout worker finished with error\"));\n>>> + }\n>>> + free(workers);\n>>> +}\n"},{"id":"406142","messageId":"cover.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"[PATCH v2 00/19] Parallel Checkout (part I)","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:14Z","receivedAt":"2020-09-22T22:50:22Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This series adds helper workers to checkout, parallelizing the reading,\nfiltering and writing of multiple blobs to the working tree.\n\nSince v1, I got the chance to benchmark parallel checkout in more\nmachines. The results showed that the parallelization is most effective\nwith repositories located on SSDs or over distributed file systems. For\nlocal file systems on spinning disks, it does not always bring good\nperformances. In fact, it even brings a slowdown sometimes. But given\nthe results on the two first cases, I think it's worth having the\nparallel code as an optional (and non-default) setting.\n\nThe size of the repository being checked out and the compression level\non the packfiles also influence how much performance gain we can get\nfrom parallel checkout. For example, downloading the Linux repo from\nGitHub and from kernel.org I got packfiles with 2.9GB and 1.4GB,\nrespectively. The number of objects was the same, but GitHub's had a\nsmaller number of delta-chains with size >= 7 [A]. For this reason, the\nsequential checkout after GitHub's clone was considerably faster than\nthe sequential checkout after kernel.org's clone. And the speedup from\nparallel checkout was more modest (but it was faster in absolute values,\nnevertheless).\n\n[A]: https://docs.google.com/spreadsheets/d/1dDGLym77JAGCVYhKQHe44r3pqtrsvHrjS4NmD_Hqr6k/edit?usp=sharing\n\nV2 got bigger with tests and some additional optimizations, so I decided\nto divide the original series into two parts to facilitate reviewing.\nThis one is constituted of:\n\n- The first 9 patches are preparatory steps in convert.c and entry.c.\n- The middle 6 actually implement parallel checkout.\n- The last 4 add tests.\n\nPart II will contain some extra optimizations, like work stealing and\nthe creation of leading directories in parallel. With that, workers\nwon't need to stat() the path components again before opening the files\nfor writing. We will also skip some stat() calls during clone.\n\n\nMajor changes since v1:\n\nGeneral:\n- Added tests\n- Parallel checkout is no longer the default, since not all machines\n benefit from it.\n- Rebased on top of master to use the adjusted mem_pool API of\n en/mem-pool.\n\nPatch 10:\n- Converted BUG() to error(), in handle_results(), when we finish\n parallel checkout with pending entries. This is not really a BUG; it\n can happen when a worker dies before sending all of its results. Also,\n by emitting an error message instead of die()'ing, we can continue\n processing the next results and, thus, avoid wasting successful work.\n- Added missing initialization of ci->status on enqueue_entry().\n- Fixed bug on which collision report during clone would not be correct\n when the file that is first written appears after it's colliding pair\n in the cache array.\n- Reworded commit message and added comment in handle_results() to\n explain why we retry writing entries with path collisions.\n- Renamed CI_RETRY to CI_COLLISION, to make it easier to change the\n behavior on collided entries in the future, if necessary.\n- Some other minor changes like:\n * Removed unnecessary PC_HANDLING_RESULTS status.\n * Statically allocated the global parallel_checkout struct.\n * Renamed checkout_item to parallel_checkout_item.\n\nPatch 11:\n- Made parse_and_save_result() safer by checking that the received data\n has the expected size, instead of trusting ci->status and possibly\n accessing an invalid address on errors.\n- Limited the workers to the number of enqueued entries.\n- Added comment in packet_to_ci() mentioning why it's OK to encode\n NULL as a zero length string when sending the working_tree_encoding to\n workers.\n- Split subprocess' spawning and finalizing loops, to mitigate the\n spawn/wait cost.\n- Don't die() when a worker exits with an error code (only report the\n error), to avoid wasting good work by not updating the index with the \n stat information from the written entries.\n- Renamed checkout.workersThreshold to checkout.thresholdForParallelism.\n\n\nJeff Hostetler (4):\n convert: make convert_attrs() and convert structs public\n convert: add [async_]convert_to_working_tree_ca() variants\n convert: add get_stream_filter_ca() variant\n convert: add conv_attrs classification\n\nMatheus Tavares (15):\n entry: extract a header file for entry.c functions\n entry: make fstat_output() and read_blob_entry() public\n entry: extract cache_entry update from write_entry()\n entry: move conv_attrs lookup up to checkout_entry()\n entry: add checkout_entry_ca() which takes preloaded conv_attrs\n unpack-trees: add basic support for parallel checkout\n parallel-checkout: make it truly parallel\n parallel-checkout: support progress displaying\n make_transient_cache_entry(): optionally alloc from mem_pool\n builtin/checkout.c: complete parallel checkout support\n checkout-index: add parallel checkout support\n parallel-checkout: add tests for basic operations\n parallel-checkout: add tests related to clone collisions\n parallel-checkout: add tests related to .gitattributes\n ci: run test round with parallel-checkout enabled\n\n .gitignore | 1 +\n Documentation/config/checkout.txt | 21 +\n Makefile | 2 +\n apply.c | 1 +\n builtin.h | 1 +\n builtin/checkout--helper.c | 142 ++++++\n builtin/checkout-index.c | 17 +\n builtin/checkout.c | 21 +-\n builtin/difftool.c | 3 +-\n cache.h | 34 +-\n ci/run-build-and-tests.sh | 1 +\n convert.c | 121 +++--\n convert.h | 68 +++\n entry.c | 102 ++--\n entry.h | 54 ++\n git.c | 2 +\n parallel-checkout.c | 631 ++++++++++++++++++++++++\n parallel-checkout.h | 103 ++++\n read-cache.c | 12 +-\n t/README | 4 +\n t/lib-encoding.sh | 25 +\n t/lib-parallel-checkout.sh | 45 ++\n t/t0028-working-tree-encoding.sh | 25 +-\n t/t2080-parallel-checkout-basics.sh | 197 ++++++++\n t/t2081-parallel-checkout-collisions.sh | 116 +++++\n t/t2082-parallel-checkout-attributes.sh | 174 +++++++\n unpack-trees.c | 22 +-\n 27 files changed, 1793 insertions(+), 152 deletions(-)\n create mode 100644 builtin/checkout--helper.c\n create mode 100644 entry.h\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n create mode 100644 t/lib-encoding.sh\n create mode 100644 t/lib-parallel-checkout.sh\n create mode 100755 t/t2080-parallel-checkout-basics.sh\n create mode 100755 t/t2081-parallel-checkout-collisions.sh\n create mode 100755 t/t2082-parallel-checkout-attributes.sh\n\n-- \n2.28.0\n\n"},{"id":"406143","messageId":"b9d2a329d3e9c4ff631d1a43e2af46088dc994ab.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 01/19] convert: make convert_attrs() and convert structs public","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:15Z","receivedAt":"2020-09-22T22:50:34Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Jeff Hostetler \n\nMove convert_attrs() declaration from convert.c to convert.h, together\nwith the conv_attrs struct and the crlf_action enum. This function and\nthe data structures will be used outside convert.c in the upcoming\nparallel checkout implementation.\n\nSigned-off-by: Jeff Hostetler \n[matheus.bernardino: squash and reword msg]\nSigned-off-by: Matheus Tavares \n---\n convert.c | 23 ++---------------------\n convert.h | 24 ++++++++++++++++++++++++\n 2 files changed, 26 insertions(+), 21 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 8e6c292421..941a845692 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -24,17 +24,6 @@\n #define CONVERT_STAT_BITS_TXT_CRLF 0x2\n #define CONVERT_STAT_BITS_BIN 0x4\n \n-enum crlf_action {\n-\tCRLF_UNDEFINED,\n-\tCRLF_BINARY,\n-\tCRLF_TEXT,\n-\tCRLF_TEXT_INPUT,\n-\tCRLF_TEXT_CRLF,\n-\tCRLF_AUTO,\n-\tCRLF_AUTO_INPUT,\n-\tCRLF_AUTO_CRLF\n-};\n-\n struct text_stat {\n \t/* NUL, CR, LF and CRLF counts */\n \tunsigned nul, lonecr, lonelf, crlf;\n@@ -1297,18 +1286,10 @@ static int git_path_check_ident(struct attr_check_item *check)\n \treturn !!ATTR_TRUE(value);\n }\n \n-struct conv_attrs {\n-\tstruct convert_driver *drv;\n-\tenum crlf_action attr_action; /* What attr says */\n-\tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n-\tint ident;\n-\tconst char *working_tree_encoding; /* Supported encoding or default encoding if NULL */\n-};\n-\n static struct attr_check *check;\n \n-static void convert_attrs(const struct index_state *istate,\n-\t\t\t struct conv_attrs *ca, const char *path)\n+void convert_attrs(const struct index_state *istate,\n+\t\t struct conv_attrs *ca, const char *path)\n {\n \tstruct attr_check_item *ccheck = NULL;\n \ndiff --git a/convert.h b/convert.h\nindex e29d1026a6..aeb4a1be9a 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -37,6 +37,27 @@ enum eol {\n #endif\n };\n \n+enum crlf_action {\n+\tCRLF_UNDEFINED,\n+\tCRLF_BINARY,\n+\tCRLF_TEXT,\n+\tCRLF_TEXT_INPUT,\n+\tCRLF_TEXT_CRLF,\n+\tCRLF_AUTO,\n+\tCRLF_AUTO_INPUT,\n+\tCRLF_AUTO_CRLF\n+};\n+\n+struct convert_driver;\n+\n+struct conv_attrs {\n+\tstruct convert_driver *drv;\n+\tenum crlf_action attr_action; /* What attr says */\n+\tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n+\tint ident;\n+\tconst char *working_tree_encoding; /* Supported encoding or default encoding if NULL */\n+};\n+\n enum ce_delay_state {\n \tCE_NO_DELAY = 0,\n \tCE_CAN_DELAY = 1,\n@@ -102,6 +123,9 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n int would_convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t\t const char *path);\n \n+void convert_attrs(const struct index_state *istate,\n+\t\t struct conv_attrs *ca, const char *path);\n+\n /*\n * Initialize the checkout metadata with the given values. Any argument may be\n * NULL if it is not applicable. The treeish should be a commit if that is\n-- \n2.28.0\n\n"},{"id":"406144","messageId":"29bbdb78e98667393f17e167f3d72d1bfbb302bd.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 03/19] convert: add get_stream_filter_ca() variant","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:17Z","receivedAt":"2020-09-22T22:50:43Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Jeff Hostetler \n\nLike the previous patch, we will also need to call get_stream_filter()\nwith a precomputed `struct conv_attrs`, when we add support for parallel\ncheckout workers. So add the _ca() variant which takes the conversion\nattributes struct as a parameter.\n\nSigned-off-by: Jeff Hostetler \n[matheus.bernardino: move header comment to ca() variant and reword msg]\nSigned-off-by: Matheus Tavares \n---\n convert.c | 28 +++++++++++++++++-----------\n convert.h | 2 ++\n 2 files changed, 19 insertions(+), 11 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 55bcce891c..c112ea23cb 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -1960,34 +1960,31 @@ static struct stream_filter *ident_filter(const struct object_id *oid)\n }\n \n /*\n- * Return an appropriately constructed filter for the path, or NULL if\n+ * Return an appropriately constructed filter for the given ca, or NULL if\n * the contents cannot be filtered without reading the whole thing\n * in-core.\n *\n * Note that you would be crazy to set CRLF, smudge/clean or ident to a\n * large binary blob you would want us not to slurp into the memory!\n */\n-struct stream_filter *get_stream_filter(const struct index_state *istate,\n-\t\t\t\t\tconst char *path,\n-\t\t\t\t\tconst struct object_id *oid)\n+struct stream_filter *get_stream_filter_ca(const struct conv_attrs *ca,\n+\t\t\t\t\t const struct object_id *oid)\n {\n-\tstruct conv_attrs ca;\n \tstruct stream_filter *filter = NULL;\n \n-\tconvert_attrs(istate, &ca, path);\n-\tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n+\tif (ca->drv && (ca->drv->process || ca->drv->smudge || ca->drv->clean))\n \t\treturn NULL;\n \n-\tif (ca.working_tree_encoding)\n+\tif (ca->working_tree_encoding)\n \t\treturn NULL;\n \n-\tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\n+\tif (ca->crlf_action == CRLF_AUTO || ca->crlf_action == CRLF_AUTO_CRLF)\n \t\treturn NULL;\n \n-\tif (ca.ident)\n+\tif (ca->ident)\n \t\tfilter = ident_filter(oid);\n \n-\tif (output_eol(ca.crlf_action) == EOL_CRLF)\n+\tif (output_eol(ca->crlf_action) == EOL_CRLF)\n \t\tfilter = cascade_filter(filter, lf_to_crlf_filter());\n \telse\n \t\tfilter = cascade_filter(filter, &null_filter_singleton);\n@@ -1995,6 +1992,15 @@ struct stream_filter *get_stream_filter(const struct index_state *istate,\n \treturn filter;\n }\n \n+struct stream_filter *get_stream_filter(const struct index_state *istate,\n+\t\t\t\t\tconst char *path,\n+\t\t\t\t\tconst struct object_id *oid)\n+{\n+\tstruct conv_attrs ca;\n+\tconvert_attrs(istate, &ca, path);\n+\treturn get_stream_filter_ca(&ca, oid);\n+}\n+\n void free_stream_filter(struct stream_filter *filter)\n {\n \tfilter->vtbl->free(filter);\ndiff --git a/convert.h b/convert.h\nindex 46d537d1ae..262c1a1d46 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -169,6 +169,8 @@ struct stream_filter; /* opaque */\n struct stream_filter *get_stream_filter(const struct index_state *istate,\n \t\t\t\t\tconst char *path,\n \t\t\t\t\tconst struct object_id *);\n+struct stream_filter *get_stream_filter_ca(const struct conv_attrs *ca,\n+\t\t\t\t\t const struct object_id *oid);\n void free_stream_filter(struct stream_filter *);\n int is_null_stream_filter(struct stream_filter *);\n \n-- \n2.28.0\n\n"},{"id":"406145","messageId":"313c3bcbebb9460d62cc29692964111565043de0.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 02/19] convert: add [async_]convert_to_working_tree_ca() variants","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:16Z","receivedAt":"2020-09-22T22:50:44Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Jeff Hostetler \n\nSeparate the attribute gathering from the actual conversion by adding\n_ca() variants of the conversion functions. These variants receive a\nprecomputed 'struct conv_attrs', not relying, thus, on a index state.\nThey will be used in a future patch adding parallel checkout support,\nfor two reasons:\n\n- We will already load the conversion attributes in checkout_entry(),\n before conversion, to decide whether a path is eligible for parallel\n checkout. Therefore, it would be wasteful to load them again later,\n for the actual conversion.\n\n- The parallel workers will be responsible for reading, converting and\n writing blobs to the working tree. They won't have access to the main\n process' index state, so they cannot load the attributes. Instead,\n they will receive the preloaded ones and call the _ca() variant of\n the conversion functions. Furthermore, the attributes machinery is\n optimized to handle paths in sequential order, so it's better to leave\n it for the main process, anyway.\n\nSigned-off-by: Jeff Hostetler \n[matheus.bernardino: squash, remove one function definition and reword]\nSigned-off-by: Matheus Tavares \n---\n convert.c | 50 ++++++++++++++++++++++++++++++++++++--------------\n convert.h | 9 +++++++++\n 2 files changed, 45 insertions(+), 14 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex 941a845692..55bcce891c 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -1447,7 +1447,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tident_to_git(dst->buf, dst->len, dst, ca.ident);\n }\n \n-static int convert_to_working_tree_internal(const struct index_state *istate,\n+static int convert_to_working_tree_internal(const struct conv_attrs *ca,\n \t\t\t\t\t const char *path, const char *src,\n \t\t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t\t int normalizing,\n@@ -1455,11 +1455,8 @@ static int convert_to_working_tree_internal(const struct index_state *istate,\n \t\t\t\t\t struct delayed_checkout *dco)\n {\n \tint ret = 0, ret_filter = 0;\n-\tstruct conv_attrs ca;\n-\n-\tconvert_attrs(istate, &ca, path);\n \n-\tret |= ident_to_worktree(src, len, dst, ca.ident);\n+\tret |= ident_to_worktree(src, len, dst, ca->ident);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n@@ -1469,24 +1466,24 @@ static int convert_to_working_tree_internal(const struct index_state *istate,\n \t * is a smudge or process filter (even if the process filter doesn't\n \t * support smudge). The filters might expect CRLFs.\n \t */\n-\tif ((ca.drv && (ca.drv->smudge || ca.drv->process)) || !normalizing) {\n-\t\tret |= crlf_to_worktree(src, len, dst, ca.crlf_action);\n+\tif ((ca->drv && (ca->drv->smudge || ca->drv->process)) || !normalizing) {\n+\t\tret |= crlf_to_worktree(src, len, dst, ca->crlf_action);\n \t\tif (ret) {\n \t\t\tsrc = dst->buf;\n \t\t\tlen = dst->len;\n \t\t}\n \t}\n \n-\tret |= encode_to_worktree(path, src, len, dst, ca.working_tree_encoding);\n+\tret |= encode_to_worktree(path, src, len, dst, ca->working_tree_encoding);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n \n \tret_filter = apply_filter(\n-\t\tpath, src, len, -1, dst, ca.drv, CAP_SMUDGE, meta, dco);\n-\tif (!ret_filter && ca.drv && ca.drv->required)\n-\t\tdie(_(\"%s: smudge filter %s failed\"), path, ca.drv->name);\n+\t\tpath, src, len, -1, dst, ca->drv, CAP_SMUDGE, meta, dco);\n+\tif (!ret_filter && ca->drv && ca->drv->required)\n+\t\tdie(_(\"%s: smudge filter %s failed\"), path, ca->drv->name);\n \n \treturn ret | ret_filter;\n }\n@@ -1497,7 +1494,9 @@ int async_convert_to_working_tree(const struct index_state *istate,\n \t\t\t\t const struct checkout_metadata *meta,\n \t\t\t\t void *dco)\n {\n-\treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, dco);\n+\tstruct conv_attrs ca;\n+\tconvert_attrs(istate, &ca, path);\n+\treturn convert_to_working_tree_internal(&ca, path, src, len, dst, 0, meta, dco);\n }\n \n int convert_to_working_tree(const struct index_state *istate,\n@@ -1505,13 +1504,36 @@ int convert_to_working_tree(const struct index_state *istate,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta)\n {\n-\treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, NULL);\n+\tstruct conv_attrs ca;\n+\tconvert_attrs(istate, &ca, path);\n+\treturn convert_to_working_tree_internal(&ca, path, src, len, dst, 0, meta, NULL);\n+}\n+\n+int async_convert_to_working_tree_ca(const struct conv_attrs *ca,\n+\t\t\t\t const char *path, const char *src,\n+\t\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t\t const struct checkout_metadata *meta,\n+\t\t\t\t void *dco)\n+{\n+\treturn convert_to_working_tree_internal(ca, path, src, len, dst, 0, meta, dco);\n+}\n+\n+int convert_to_working_tree_ca(const struct conv_attrs *ca,\n+\t\t\t const char *path, const char *src,\n+\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t const struct checkout_metadata *meta)\n+{\n+\treturn convert_to_working_tree_internal(ca, path, src, len, dst, 0, meta, NULL);\n }\n \n int renormalize_buffer(const struct index_state *istate, const char *path,\n \t\t const char *src, size_t len, struct strbuf *dst)\n {\n-\tint ret = convert_to_working_tree_internal(istate, path, src, len, dst, 1, NULL, NULL);\n+\tstruct conv_attrs ca;\n+\tint ret;\n+\n+\tconvert_attrs(istate, &ca, path);\n+\tret = convert_to_working_tree_internal(&ca, path, src, len, dst, 1, NULL, NULL);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\ndiff --git a/convert.h b/convert.h\nindex aeb4a1be9a..46d537d1ae 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -100,11 +100,20 @@ int convert_to_working_tree(const struct index_state *istate,\n \t\t\t const char *path, const char *src,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta);\n+int convert_to_working_tree_ca(const struct conv_attrs *ca,\n+\t\t\t const char *path, const char *src,\n+\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t const struct checkout_metadata *meta);\n int async_convert_to_working_tree(const struct index_state *istate,\n \t\t\t\t const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t const struct checkout_metadata *meta,\n \t\t\t\t void *dco);\n+int async_convert_to_working_tree_ca(const struct conv_attrs *ca,\n+\t\t\t\t const char *path, const char *src,\n+\t\t\t\t size_t len, struct strbuf *dst,\n+\t\t\t\t const struct checkout_metadata *meta,\n+\t\t\t\t void *dco);\n int async_query_available_blobs(const char *cmd,\n \t\t\t\tstruct string_list *available_paths);\n int renormalize_buffer(const struct index_state *istate,\n-- \n2.28.0\n\n"},{"id":"406146","messageId":"a1cf5df961dd493801ac374c3a3f3c38573a61d7.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 04/19] convert: add conv_attrs classification","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:18Z","receivedAt":"2020-09-22T22:50:47Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"From: Jeff Hostetler \n\nCreate `enum conv_attrs_classification` to express the different ways\nthat attributes are handled for a blob during checkout.\n\nThis will be used in a later commit when deciding whether to add a file\nto the parallel or delayed queue during checkout. For now, we can also\nuse it in get_stream_filter_ca() to simplify the function (as the\nclassifying logic is the same).\n\nSigned-off-by: Jeff Hostetler \n[matheus.bernardino: use classification in get_stream_filter_ca()]\nSigned-off-by: Matheus Tavares \n---\n convert.c | 26 +++++++++++++++++++-------\n convert.h | 33 +++++++++++++++++++++++++++++++++\n 2 files changed, 52 insertions(+), 7 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex c112ea23cb..633ad6976a 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -1972,13 +1972,7 @@ struct stream_filter *get_stream_filter_ca(const struct conv_attrs *ca,\n {\n \tstruct stream_filter *filter = NULL;\n \n-\tif (ca->drv && (ca->drv->process || ca->drv->smudge || ca->drv->clean))\n-\t\treturn NULL;\n-\n-\tif (ca->working_tree_encoding)\n-\t\treturn NULL;\n-\n-\tif (ca->crlf_action == CRLF_AUTO || ca->crlf_action == CRLF_AUTO_CRLF)\n+\tif (classify_conv_attrs(ca) != CA_CLASS_STREAMABLE)\n \t\treturn NULL;\n \n \tif (ca->ident)\n@@ -2034,3 +2028,21 @@ void clone_checkout_metadata(struct checkout_metadata *dst,\n \tif (blob)\n \t\toidcpy(&dst->blob, blob);\n }\n+\n+enum conv_attrs_classification classify_conv_attrs(const struct conv_attrs *ca)\n+{\n+\tif (ca->drv) {\n+\t\tif (ca->drv->process)\n+\t\t\treturn CA_CLASS_INCORE_PROCESS;\n+\t\tif (ca->drv->smudge || ca->drv->clean)\n+\t\t\treturn CA_CLASS_INCORE_FILTER;\n+\t}\n+\n+\tif (ca->working_tree_encoding)\n+\t\treturn CA_CLASS_INCORE;\n+\n+\tif (ca->crlf_action == CRLF_AUTO || ca->crlf_action == CRLF_AUTO_CRLF)\n+\t\treturn CA_CLASS_INCORE;\n+\n+\treturn CA_CLASS_STREAMABLE;\n+}\ndiff --git a/convert.h b/convert.h\nindex 262c1a1d46..523ba9b140 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -190,4 +190,37 @@ int stream_filter(struct stream_filter *,\n \t\t const char *input, size_t *isize_p,\n \t\t char *output, size_t *osize_p);\n \n+enum conv_attrs_classification {\n+\t/*\n+\t * The blob must be loaded into a buffer before it can be\n+\t * smudged. All smudging is done in-proc.\n+\t */\n+\tCA_CLASS_INCORE,\n+\n+\t/*\n+\t * The blob must be loaded into a buffer, but uses a\n+\t * single-file driver filter, such as rot13.\n+\t */\n+\tCA_CLASS_INCORE_FILTER,\n+\n+\t/*\n+\t * The blob must be loaded into a buffer, but uses a\n+\t * long-running driver process, such as LFS. This might or\n+\t * might not use delayed operations. (The important thing is\n+\t * that there is a single subordinate long-running process\n+\t * handling all associated blobs and in case of delayed\n+\t * operations, may hold per-blob state.)\n+\t */\n+\tCA_CLASS_INCORE_PROCESS,\n+\n+\t/*\n+\t * The blob can be streamed and smudged without needing to\n+\t * completely read it into a buffer.\n+\t */\n+\tCA_CLASS_STREAMABLE,\n+};\n+\n+enum conv_attrs_classification classify_conv_attrs(\n+\tconst struct conv_attrs *ca);\n+\n #endif /* CONVERT_H */\n-- \n2.28.0\n\n"},{"id":"406147","messageId":"25b311745aac7aaf43335a754ac8af3b79b65404.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 05/19] entry: extract a header file for entry.c functions","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:19Z","receivedAt":"2020-09-22T22:50:50Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The declarations of entry.c's public functions and structures currently\nreside in cache.h. Although not many, they contribute to the size of\ncache.h and, when changed, cause the unnecessary recompilation of\nmodules that don't really use these functions. So let's move them to a\nnew entry.h header.\n\nOriginal-patch-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n apply.c | 1 +\n builtin/checkout-index.c | 1 +\n builtin/checkout.c | 1 +\n builtin/difftool.c | 1 +\n cache.h | 24 -----------------------\n entry.c | 9 +--------\n entry.h | 41 ++++++++++++++++++++++++++++++++++++++++\n unpack-trees.c | 1 +\n 8 files changed, 47 insertions(+), 32 deletions(-)\n create mode 100644 entry.h\n\ndiff --git a/apply.c b/apply.c\nindex 76dba93c97..ddec80b4b0 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -21,6 +21,7 @@\n #include \"quote.h\"\n #include \"rerere.h\"\n #include \"apply.h\"\n+#include \"entry.h\"\n \n struct gitdiff_data {\n \tstruct strbuf *root;\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex a854fd16e7..0f1ff73129 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -11,6 +11,7 @@\n #include \"quote.h\"\n #include \"cache-tree.h\"\n #include \"parse-options.h\"\n+#include \"entry.h\"\n \n #define CHECKOUT_ALL 4\n static int nul_term_line;\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 0951f8fee5..b18b9d6f3c 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -26,6 +26,7 @@\n #include \"unpack-trees.h\"\n #include \"wt-status.h\"\n #include \"xdiff-interface.h\"\n+#include \"entry.h\"\n \n static const char * const checkout_usage[] = {\n \tN_(\"git checkout [] \"),\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex 7ac432b881..dfa22b67eb 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -23,6 +23,7 @@\n #include \"lockfile.h\"\n #include \"object-store.h\"\n #include \"dir.h\"\n+#include \"entry.h\"\n \n static int trust_exit_code;\n \ndiff --git a/cache.h b/cache.h\nindex cee8aa5dc3..17350cafa2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1706,30 +1706,6 @@ const char *show_ident_date(const struct ident_split *id,\n */\n int ident_cmp(const struct ident_split *, const struct ident_split *);\n \n-struct checkout {\n-\tstruct index_state *istate;\n-\tconst char *base_dir;\n-\tint base_dir_len;\n-\tstruct delayed_checkout *delayed_checkout;\n-\tstruct checkout_metadata meta;\n-\tunsigned force:1,\n-\t\t quiet:1,\n-\t\t not_new:1,\n-\t\t clone:1,\n-\t\t refresh_cache:1;\n-};\n-#define CHECKOUT_INIT { NULL, \"\" }\n-\n-#define TEMPORARY_FILENAME_LENGTH 25\n-int checkout_entry(struct cache_entry *ce, const struct checkout *state, char *topath, int *nr_checkouts);\n-void enable_delayed_checkout(struct checkout *state);\n-int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n-/*\n- * Unlink the last component and schedule the leading directories for\n- * removal, such that empty directories get removed.\n- */\n-void unlink_entry(const struct cache_entry *ce);\n-\n struct cache_def {\n \tstruct strbuf path;\n \tint flags;\ndiff --git a/entry.c b/entry.c\nindex a0532f1f00..b0b8099699 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -6,6 +6,7 @@\n #include \"submodule.h\"\n #include \"progress.h\"\n #include \"fsmonitor.h\"\n+#include \"entry.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -429,14 +430,6 @@ static void mark_colliding_entries(const struct checkout *state,\n \t}\n }\n \n-/*\n- * Write the contents from ce out to the working tree.\n- *\n- * When topath[] is not NULL, instead of writing to the working tree\n- * file named by ce, a temporary file is created by this function and\n- * its name is returned in topath[], which must be able to hold at\n- * least TEMPORARY_FILENAME_LENGTH bytes long.\n- */\n int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \t\t char *topath, int *nr_checkouts)\n {\ndiff --git a/entry.h b/entry.h\nnew file mode 100644\nindex 0000000000..2d69185448\n--- /dev/null\n+++ b/entry.h\n@@ -0,0 +1,41 @@\n+#ifndef ENTRY_H\n+#define ENTRY_H\n+\n+#include \"cache.h\"\n+#include \"convert.h\"\n+\n+struct checkout {\n+\tstruct index_state *istate;\n+\tconst char *base_dir;\n+\tint base_dir_len;\n+\tstruct delayed_checkout *delayed_checkout;\n+\tstruct checkout_metadata meta;\n+\tunsigned force:1,\n+\t\t quiet:1,\n+\t\t not_new:1,\n+\t\t clone:1,\n+\t\t refresh_cache:1;\n+};\n+#define CHECKOUT_INIT { NULL, \"\" }\n+\n+#define TEMPORARY_FILENAME_LENGTH 25\n+\n+/*\n+ * Write the contents from ce out to the working tree.\n+ *\n+ * When topath[] is not NULL, instead of writing to the working tree\n+ * file named by ce, a temporary file is created by this function and\n+ * its name is returned in topath[], which must be able to hold at\n+ * least TEMPORARY_FILENAME_LENGTH bytes long.\n+ */\n+int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n+\t\t char *topath, int *nr_checkouts);\n+void enable_delayed_checkout(struct checkout *state);\n+int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n+/*\n+ * Unlink the last component and schedule the leading directories for\n+ * removal, such that empty directories get removed.\n+ */\n+void unlink_entry(const struct cache_entry *ce);\n+\n+#endif /* ENTRY_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 323280dd48..a511fadd89 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -16,6 +16,7 @@\n #include \"fsmonitor.h\"\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n+#include \"entry.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n-- \n2.28.0\n\n"},{"id":"406148","messageId":"dbee09e9366ee802306eba3cd598c62c64ef7e5f.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 06/19] entry: make fstat_output() and read_blob_entry() public","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:20Z","receivedAt":"2020-09-22T22:50:53Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"These two functions will be used by the parallel checkout code, so let's\nmake them public. Note: fstat_output() is renamed to\nfstat_checkout_output(), now that it has become public, seeking to avoid\nfuture name collisions.\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 8 ++++----\n entry.h | 2 ++\n 2 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex b0b8099699..b36071a610 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -84,7 +84,7 @@ static int create_file(const char *path, unsigned int mode)\n \treturn open(path, O_WRONLY | O_CREAT | O_EXCL, mode);\n }\n \n-static void *read_blob_entry(const struct cache_entry *ce, unsigned long *size)\n+void *read_blob_entry(const struct cache_entry *ce, unsigned long *size)\n {\n \tenum object_type type;\n \tvoid *blob_data = read_object_file(&ce->oid, &type, size);\n@@ -109,7 +109,7 @@ static int open_output_fd(char *path, const struct cache_entry *ce, int to_tempf\n \t}\n }\n \n-static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n+int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st)\n {\n \t/* use fstat() only when path == ce->name */\n \tif (fstat_is_reliable() &&\n@@ -132,7 +132,7 @@ static int streaming_write_entry(const struct cache_entry *ce, char *path,\n \t\treturn -1;\n \n \tresult |= stream_blob_to_fd(fd, &ce->oid, filter, 1);\n-\t*fstat_done = fstat_output(fd, state, statbuf);\n+\t*fstat_done = fstat_checkout_output(fd, state, statbuf);\n \tresult |= close(fd);\n \n \tif (result)\n@@ -346,7 +346,7 @@ static int write_entry(struct cache_entry *ce,\n \n \t\twrote = write_in_full(fd, new_blob, size);\n \t\tif (!to_tempfile)\n-\t\t\tfstat_done = fstat_output(fd, state, &st);\n+\t\t\tfstat_done = fstat_checkout_output(fd, state, &st);\n \t\tclose(fd);\n \t\tfree(new_blob);\n \t\tif (wrote < 0)\ndiff --git a/entry.h b/entry.h\nindex 2d69185448..f860e60846 100644\n--- a/entry.h\n+++ b/entry.h\n@@ -37,5 +37,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n * removal, such that empty directories get removed.\n */\n void unlink_entry(const struct cache_entry *ce);\n+void *read_blob_entry(const struct cache_entry *ce, unsigned long *size);\n+int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st);\n \n #endif /* ENTRY_H */\n-- \n2.28.0\n\n"},{"id":"406149","messageId":"b61b5c44f03819ea797727021261d54822b5f9ad.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 07/19] entry: extract cache_entry update from write_entry()","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:21Z","receivedAt":"2020-09-22T22:50:56Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This code will be used by the parallel checkout functions, outside\nentry.c, so extract it to a public function.\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 25 ++++++++++++++++---------\n entry.h | 2 ++\n 2 files changed, 18 insertions(+), 9 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex b36071a610..1d2df188e5 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -251,6 +251,18 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \treturn errs;\n }\n \n+void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,\n+\t\t\t struct stat *st)\n+{\n+\tif (state->refresh_cache) {\n+\t\tassert(state->istate);\n+\t\tfill_stat_cache_info(state->istate, ce, st);\n+\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\tmark_fsmonitor_invalid(state->istate, ce);\n+\t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n+\t}\n+}\n+\n static int write_entry(struct cache_entry *ce,\n \t\t char *path, const struct checkout *state, int to_tempfile)\n {\n@@ -371,15 +383,10 @@ static int write_entry(struct cache_entry *ce,\n \n finish:\n \tif (state->refresh_cache) {\n-\t\tassert(state->istate);\n-\t\tif (!fstat_done)\n-\t\t\tif (lstat(ce->name, &st) < 0)\n-\t\t\t\treturn error_errno(\"unable to stat just-written file %s\",\n-\t\t\t\t\t\t ce->name);\n-\t\tfill_stat_cache_info(state->istate, ce, &st);\n-\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\tmark_fsmonitor_invalid(state->istate, ce);\n-\t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n+\t\tif (!fstat_done && lstat(ce->name, &st) < 0)\n+\t\t\treturn error_errno(\"unable to stat just-written file %s\",\n+\t\t\t\t\t ce->name);\n+\t\tupdate_ce_after_write(state, ce , &st);\n \t}\n delayed:\n \treturn 0;\ndiff --git a/entry.h b/entry.h\nindex f860e60846..664aed1576 100644\n--- a/entry.h\n+++ b/entry.h\n@@ -39,5 +39,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n void unlink_entry(const struct cache_entry *ce);\n void *read_blob_entry(const struct cache_entry *ce, unsigned long *size);\n int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st);\n+void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,\n+\t\t\t struct stat *st);\n \n #endif /* ENTRY_H */\n-- \n2.28.0\n\n"},{"id":"406150","messageId":"667ad0dea70cb7f0bbf8f52467f15129b3ae1325.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 08/19] entry: move conv_attrs lookup up to checkout_entry()","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:22Z","receivedAt":"2020-09-22T22:50:59Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"In a following patch, checkout_entry() will use conv_attrs to decide\nwhether an entry should be enqueued for parallel checkout or not. But\nthe attributes lookup only happens lower in this call stack. To avoid\nthe unnecessary work of loading the attributes twice, let's move it up\nto checkout_entry(), and pass the loaded struct down to write_entry().\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 38 +++++++++++++++++++++++++++-----------\n 1 file changed, 27 insertions(+), 11 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 1d2df188e5..8237859b12 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -263,8 +263,9 @@ void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,\n \t}\n }\n \n-static int write_entry(struct cache_entry *ce,\n-\t\t char *path, const struct checkout *state, int to_tempfile)\n+/* Note: ca is used (and required) iff the entry refers to a regular file. */\n+static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca,\n+\t\t const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n \tstruct delayed_checkout *dco = state->delayed_checkout;\n@@ -281,8 +282,7 @@ static int write_entry(struct cache_entry *ce,\n \tclone_checkout_metadata(&meta, &state->meta, &ce->oid);\n \n \tif (ce_mode_s_ifmt == S_IFREG) {\n-\t\tstruct stream_filter *filter = get_stream_filter(state->istate, ce->name,\n-\t\t\t\t\t\t\t\t &ce->oid);\n+\t\tstruct stream_filter *filter = get_stream_filter_ca(ca, &ce->oid);\n \t\tif (filter &&\n \t\t !streaming_write_entry(ce, path, filter,\n \t\t\t\t\t state, to_tempfile,\n@@ -329,14 +329,17 @@ static int write_entry(struct cache_entry *ce,\n \t\t * Convert from git internal format to working tree format\n \t\t */\n \t\tif (dco && dco->state != CE_NO_DELAY) {\n-\t\t\tret = async_convert_to_working_tree(state->istate, ce->name, new_blob,\n-\t\t\t\t\t\t\t size, &buf, &meta, dco);\n+\t\t\tret = async_convert_to_working_tree_ca(ca, ce->name,\n+\t\t\t\t\t\t\t new_blob, size,\n+\t\t\t\t\t\t\t &buf, &meta, dco);\n \t\t\tif (ret && string_list_has_string(&dco->paths, ce->name)) {\n \t\t\t\tfree(new_blob);\n \t\t\t\tgoto delayed;\n \t\t\t}\n-\t\t} else\n-\t\t\tret = convert_to_working_tree(state->istate, ce->name, new_blob, size, &buf, &meta);\n+\t\t} else {\n+\t\t\tret = convert_to_working_tree_ca(ca, ce->name, new_blob,\n+\t\t\t\t\t\t\t size, &buf, &meta);\n+\t\t}\n \n \t\tif (ret) {\n \t\t\tfree(new_blob);\n@@ -442,6 +445,7 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n {\n \tstatic struct strbuf path = STRBUF_INIT;\n \tstruct stat st;\n+\tstruct conv_attrs ca;\n \n \tif (ce->ce_flags & CE_WT_REMOVE) {\n \t\tif (topath)\n@@ -454,8 +458,13 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \t\treturn 0;\n \t}\n \n-\tif (topath)\n-\t\treturn write_entry(ce, topath, state, 1);\n+\tif (topath) {\n+\t\tif (S_ISREG(ce->ce_mode)) {\n+\t\t\tconvert_attrs(state->istate, &ca, ce->name);\n+\t\t\treturn write_entry(ce, topath, &ca, state, 1);\n+\t\t}\n+\t\treturn write_entry(ce, topath, NULL, state, 1);\n+\t}\n \n \tstrbuf_reset(&path);\n \tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n@@ -517,9 +526,16 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \t\treturn 0;\n \n \tcreate_directories(path.buf, path.len, state);\n+\n \tif (nr_checkouts)\n \t\t(*nr_checkouts)++;\n-\treturn write_entry(ce, path.buf, state, 0);\n+\n+\tif (S_ISREG(ce->ce_mode)) {\n+\t\tconvert_attrs(state->istate, &ca, ce->name);\n+\t\treturn write_entry(ce, path.buf, &ca, state, 0);\n+\t}\n+\n+\treturn write_entry(ce, path.buf, NULL, state, 0);\n }\n \n void unlink_entry(const struct cache_entry *ce)\n-- \n2.28.0\n\n"},{"id":"406151","messageId":"4ddb34209e9340f4e709234262a4a9ce81ad9b51.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 09/19] entry: add checkout_entry_ca() which takes preloaded conv_attrs","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:23Z","receivedAt":"2020-09-22T22:51:02Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"The parallel checkout machinery will call checkout_entry() for entries\nthat could not be written in parallel due to path collisions. At this\npoint, we will already be holding the conversion attributes for each\nentry, and it would be wasteful to let checkout_entry() load these\nagain. Instead, let's add the checkout_entry_ca() variant, which\noptionally takes a preloaded conv_attrs struct.\n\nSigned-off-by: Matheus Tavares \n---\n entry.c | 23 ++++++++++++-----------\n entry.h | 13 +++++++++++--\n 2 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 8237859b12..9d79a5671f 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -440,12 +440,13 @@ static void mark_colliding_entries(const struct checkout *state,\n \t}\n }\n \n-int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n-\t\t char *topath, int *nr_checkouts)\n+int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n+\t\t const struct checkout *state, char *topath,\n+\t\t int *nr_checkouts)\n {\n \tstatic struct strbuf path = STRBUF_INIT;\n \tstruct stat st;\n-\tstruct conv_attrs ca;\n+\tstruct conv_attrs ca_buf;\n \n \tif (ce->ce_flags & CE_WT_REMOVE) {\n \t\tif (topath)\n@@ -459,11 +460,11 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \t}\n \n \tif (topath) {\n-\t\tif (S_ISREG(ce->ce_mode)) {\n-\t\t\tconvert_attrs(state->istate, &ca, ce->name);\n-\t\t\treturn write_entry(ce, topath, &ca, state, 1);\n+\t\tif (S_ISREG(ce->ce_mode) && !ca) {\n+\t\t\tconvert_attrs(state->istate, &ca_buf, ce->name);\n+\t\t\tca = &ca_buf;\n \t\t}\n-\t\treturn write_entry(ce, topath, NULL, state, 1);\n+\t\treturn write_entry(ce, topath, ca, state, 1);\n \t}\n \n \tstrbuf_reset(&path);\n@@ -530,12 +531,12 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n \tif (nr_checkouts)\n \t\t(*nr_checkouts)++;\n \n-\tif (S_ISREG(ce->ce_mode)) {\n-\t\tconvert_attrs(state->istate, &ca, ce->name);\n-\t\treturn write_entry(ce, path.buf, &ca, state, 0);\n+\tif (S_ISREG(ce->ce_mode) && !ca) {\n+\t\tconvert_attrs(state->istate, &ca_buf, ce->name);\n+\t\tca = &ca_buf;\n \t}\n \n-\treturn write_entry(ce, path.buf, NULL, state, 0);\n+\treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \n void unlink_entry(const struct cache_entry *ce)\ndiff --git a/entry.h b/entry.h\nindex 664aed1576..2081fbbbab 100644\n--- a/entry.h\n+++ b/entry.h\n@@ -27,9 +27,18 @@ struct checkout {\n * file named by ce, a temporary file is created by this function and\n * its name is returned in topath[], which must be able to hold at\n * least TEMPORARY_FILENAME_LENGTH bytes long.\n+ *\n+ * With checkout_entry_ca(), callers can optionally pass a preloaded\n+ * conv_attrs struct (to avoid reloading it), when ce refers to a\n+ * regular file. If ca is NULL, the attributes will be loaded\n+ * internally when (and if) needed.\n */\n-int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n-\t\t char *topath, int *nr_checkouts);\n+#define checkout_entry(ce, state, topath, nr_checkouts) \\\n+\t\tcheckout_entry_ca(ce, NULL, state, topath, nr_checkouts)\n+int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n+\t\t const struct checkout *state, char *topath,\n+\t\t int *nr_checkouts);\n+\n void enable_delayed_checkout(struct checkout *state);\n int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);\n /*\n-- \n2.28.0\n\n"},{"id":"406152","messageId":"af0d790973b1e52da0fc9c4bed95e93d995d68da.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 10/19] unpack-trees: add basic support for parallel checkout","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:24Z","receivedAt":"2020-09-22T22:51:06Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This new interface allows us to enqueue some of the entries being\nchecked out to later call write_entry() for them in parallel. For now,\nthe parallel checkout machinery is enabled by default and there is no\nuser configuration, but run_parallel_checkout() just writes the queued\nentries in sequence (without spawning additional workers). The next\npatch will actually implement the parallelism and, later, we will make\nit configurable.\n\nWhen there are path collisions among the entries being written (which\ncan happen e.g. with case-sensitive files in case-insensitive file\nsystems), the parallel checkout code detects the problem and marks the\nitem with PC_ITEM_COLLIDED. Later, these items are sequentially fed to\ncheckout_entry() again. This is similar to the way the sequential code\ndeals with collisions, overwriting the previously checked out entries\nwith the subsequent ones. The only difference is that, when we start\nwriting the entries in parallel, we won't be able to determine which of\nthe colliding entries will survive on disk (for the sequential\nalgorithm, it is always the last one).\n\nI also experimented with the idea of not overwriting colliding entries,\nand it seemed to work well in my simple tests. However, because just one\nentry of each colliding group would be actually written, the others\nwould have null lstat() fields on the index. This might not be a problem\nby itself, but it could cause performance penalties for subsequent\ncommands that need to refresh the index: when the st_size value cached\nis 0, read-cache.c:ie_modified() will go to the filesystem to see if the\ncontents match. As mentioned in the function:\n\n * Immediately after read-tree or update-index --cacheinfo,\n * the length field is zero, as we have never even read the\n * lstat(2) information once, and we cannot trust DATA_CHANGED\n * returned by ie_match_stat() which in turn was returned by\n * ce_match_stat_basic() to signal that the filesize of the\n * blob changed. We have to actually go to the filesystem to\n * see if the contents match, and if so, should answer \"unchanged\".\n\nSo, if we have N entries in a colliding group and we decide to write and\nlstat() only one of them, every subsequent git-status will have to read,\nconvert, and hash the written file N - 1 times, to check that the N - 1\nunwritten entries are dirty. By checking out all colliding entries (like\nthe sequential code does), we only pay the overhead once.\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n\nNote: currently, we have to check leading directories again before\nwriting each parallel-eligible entry, as explained in the respective\ncode comment. But I plan to remove this extra work on part II,\npostponing the checkout of symlinks to *after* the parallel-eligible\nentries.\n\n Makefile | 1 +\n entry.c | 17 +-\n parallel-checkout.c | 368 ++++++++++++++++++++++++++++++++++++++++++++\n parallel-checkout.h | 27 ++++\n unpack-trees.c | 6 +-\n 5 files changed, 416 insertions(+), 3 deletions(-)\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\ndiff --git a/Makefile b/Makefile\nindex f1b1bc8aa0..3edcdc534c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -932,6 +932,7 @@ LIB_OBJS += pack-revindex.o\n LIB_OBJS += pack-write.o\n LIB_OBJS += packfile.o\n LIB_OBJS += pager.o\n+LIB_OBJS += parallel-checkout.o\n LIB_OBJS += parse-options-cb.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\ndiff --git a/entry.c b/entry.c\nindex 9d79a5671f..6676954431 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -7,6 +7,7 @@\n #include \"progress.h\"\n #include \"fsmonitor.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -426,8 +427,17 @@ static void mark_colliding_entries(const struct checkout *state,\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-\t\tif (dup == ce)\n-\t\t\tbreak;\n+\t\tif (dup == ce) {\n+\t\t\t/*\n+\t\t\t * Parallel checkout creates the files in no particular\n+\t\t\t * order. So the other side of the collision may appear\n+\t\t\t * after the given cache_entry in the array.\n+\t\t\t */\n+\t\t\tif (parallel_checkout_status() == PC_RUNNING)\n+\t\t\t\tcontinue;\n+\t\t\telse\n+\t\t\t\tbreak;\n+\t\t}\n \n \t\tif (dup->ce_flags & (CE_MATCHED | CE_VALID | CE_SKIP_WORKTREE))\n \t\t\tcontinue;\n@@ -536,6 +546,9 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\tca = &ca_buf;\n \t}\n \n+\tif (!enqueue_checkout(ce, ca))\n+\t\treturn 0;\n+\n \treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nnew file mode 100644\nindex 0000000000..7dc8ab2a67\n--- /dev/null\n+++ b/parallel-checkout.c\n@@ -0,0 +1,368 @@\n+#include \"cache.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"streaming.h\"\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/* pointer to a istate->cache[] entry. Not owned by us. */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tstruct stat st;\n+\tenum pc_item_status status;\n+};\n+\n+struct parallel_checkout {\n+\tenum pc_status status;\n+\tstruct parallel_checkout_item *items;\n+\tsize_t nr, alloc;\n+};\n+\n+static struct parallel_checkout parallel_checkout = { 0 };\n+\n+enum pc_status parallel_checkout_status(void)\n+{\n+\treturn parallel_checkout.status;\n+}\n+\n+void init_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status != PC_UNINITIALIZED)\n+\t\tBUG(\"parallel checkout already initialized\");\n+\n+\tparallel_checkout.status = PC_ACCEPTING_ENTRIES;\n+}\n+\n+static void finish_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status == PC_UNINITIALIZED)\n+\t\tBUG(\"cannot finish parallel checkout: not initialized yet\");\n+\n+\tfree(parallel_checkout.items);\n+\tmemset(¶llel_checkout, 0, sizeof(parallel_checkout));\n+}\n+\n+static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n+\t\t\t\t\t const struct conv_attrs *ca)\n+{\n+\tenum conv_attrs_classification c;\n+\n+\tif (!S_ISREG(ce->ce_mode))\n+\t\treturn 0;\n+\n+\tc = classify_conv_attrs(ca);\n+\tswitch (c) {\n+\tcase CA_CLASS_INCORE:\n+\t\treturn 1;\n+\n+\tcase CA_CLASS_INCORE_FILTER:\n+\t\t/*\n+\t\t * It would be safe to allow concurrent instances of\n+\t\t * single-file smudge filters, like rot13, but we should not\n+\t\t * assume that all filters are parallel-process safe. So we\n+\t\t * don't allow this.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_INCORE_PROCESS:\n+\t\t/*\n+\t\t * The parallel queue and the delayed queue are not compatible,\n+\t\t * so they must be kept completely separated. And we can't tell\n+\t\t * if a long-running process will delay its response without\n+\t\t * actually asking it to perform the filtering. Therefore, this\n+\t\t * type of filter is not allowed in parallel checkout.\n+\t\t *\n+\t\t * Furthermore, there should only be one instance of the\n+\t\t * long-running process filter as we don't know how it is\n+\t\t * managing its own concurrency. So, spreading the entries that\n+\t\t * requisite such a filter among the parallel workers would\n+\t\t * require a lot more inter-process communication. We would\n+\t\t * probably have to designate a single process to interact with\n+\t\t * the filter and send all the necessary data to it, for each\n+\t\t * entry.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_STREAMABLE:\n+\t\treturn 1;\n+\n+\tdefault:\n+\t\tBUG(\"unsupported conv_attrs classification '%d'\", c);\n+\t}\n+}\n+\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n+{\n+\tstruct parallel_checkout_item *pc_item;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES ||\n+\t !is_eligible_for_parallel_checkout(ce, ca))\n+\t\treturn -1;\n+\n+\tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n+\t\t parallel_checkout.alloc);\n+\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item->ce = ce;\n+\tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n+\tpc_item->status = PC_ITEM_PENDING;\n+\n+\treturn 0;\n+}\n+\n+static int handle_results(struct checkout *state)\n+{\n+\tint ret = 0;\n+\tsize_t i;\n+\tint have_pending = 0;\n+\n+\t/*\n+\t * We first update the successfully written entries with the collected\n+\t * stat() data, so that they can be found by mark_colliding_entries(),\n+\t * in the next loop, when necessary.\n+\t */\n+\tfor (i = 0; i < parallel_checkout.nr; ++i) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\tif (pc_item->status == PC_ITEM_WRITTEN)\n+\t\t\tupdate_ce_after_write(state, pc_item->ce, &pc_item->st);\n+\t}\n+\n+\tfor (i = 0; i < parallel_checkout.nr; ++i) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\n+\t\tswitch(pc_item->status) {\n+\t\tcase PC_ITEM_WRITTEN:\n+\t\t\t/* Already handled */\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_COLLIDED:\n+\t\t\t/*\n+\t\t\t * The entry could not be checked out due to a path\n+\t\t\t * collision with another entry. Since there can only\n+\t\t\t * be one entry of each colliding group on the disk, we\n+\t\t\t * could skip trying to check out this one and move on.\n+\t\t\t * However, this would leave the unwritten entries with\n+\t\t\t * null stat() fields on the index, which could\n+\t\t\t * potentially slow down subsequent operations that\n+\t\t\t * require refreshing it: git would not be able to\n+\t\t\t * trust st_size and would have to go to the filesystem\n+\t\t\t * to see if the contents match (see ie_modified()).\n+\t\t\t *\n+\t\t\t * Instead, let's pay the overhead only once, now, and\n+\t\t\t * call checkout_entry_ca() again for this file, to\n+\t\t\t * have it's stat() data stored in the index. This also\n+\t\t\t * has the benefit of adding this entry and its\n+\t\t\t * colliding pair to the collision report message.\n+\t\t\t * Additionally, this overwriting behavior is consistent\n+\t\t\t * with what the sequential checkout does, so it doesn't\n+\t\t\t * add any extra overhead.\n+\t\t\t */\n+\t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n+\t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_PENDING:\n+\t\t\thave_pending = 1;\n+\t\t\t/* fall through */\n+\t\tcase PC_ITEM_FAILED:\n+\t\t\tret = -1;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unknown checkout item status in parallel checkout\");\n+\t\t}\n+\t}\n+\n+\tif (have_pending)\n+\t\terror(_(\"parallel checkout finished with pending entries\"));\n+\n+\treturn ret;\n+}\n+\n+static int reset_fd(int fd, const char *path)\n+{\n+\tif (lseek(fd, 0, SEEK_SET) != 0)\n+\t\treturn error_errno(\"failed to rewind descriptor of %s\", path);\n+\tif (ftruncate(fd, 0))\n+\t\treturn error_errno(\"failed to truncate file %s\", path);\n+\treturn 0;\n+}\n+\n+static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n+\t\t\t const char *path, struct checkout *state)\n+{\n+\tint ret;\n+\tstruct stream_filter *filter;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *new_blob;\n+\tunsigned long size;\n+\tsize_t newsize = 0;\n+\tssize_t wrote;\n+\n+\t/* Sanity check */\n+\tassert(is_eligible_for_parallel_checkout(pc_item->ce, &pc_item->ca));\n+\n+\tfilter = get_stream_filter_ca(&pc_item->ca, &pc_item->ce->oid);\n+\tif (filter) {\n+\t\tif (stream_blob_to_fd(fd, &pc_item->ce->oid, filter, 1)) {\n+\t\t\t/* On error, reset fd to try writing without streaming */\n+\t\t\tif (reset_fd(fd, path))\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tnew_blob = read_blob_entry(pc_item->ce, &size);\n+\tif (!new_blob)\n+\t\treturn error(\"unable to read sha1 file of %s (%s)\", path,\n+\t\t\t oid_to_hex(&pc_item->ce->oid));\n+\n+\t/*\n+\t * checkout metadata is used to give context for external process\n+\t * filters. Files requiring such filters are not eligible for parallel\n+\t * checkout, so pass NULL.\n+\t */\n+\tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n+\t\t\t\t\t new_blob, size, &buf, NULL);\n+\n+\tif (ret) {\n+\t\tfree(new_blob);\n+\t\tnew_blob = strbuf_detach(&buf, &newsize);\n+\t\tsize = newsize;\n+\t}\n+\n+\twrote = write_in_full(fd, new_blob, size);\n+\tfree(new_blob);\n+\tif (wrote < 0)\n+\t\treturn error(\"unable to write file %s\", path);\n+\n+\treturn 0;\n+}\n+\n+static int close_and_clear(int *fd)\n+{\n+\tint ret = 0;\n+\n+\tif (*fd >= 0) {\n+\t\tret = close(*fd);\n+\t\t*fd = -1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int check_leading_dirs(const char *path, int len, int prefix_len)\n+{\n+\tconst char *slash = path + len;\n+\n+\twhile (slash > path && *slash != '/')\n+\t\tslash--;\n+\n+\treturn has_dirs_only_path(path, slash - path, prefix_len);\n+}\n+\n+static void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t\t struct checkout *state)\n+{\n+\tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n+\tint fd = -1, fstat_done = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n+\tstrbuf_add(&path, pc_item->ce->name, pc_item->ce->ce_namelen);\n+\n+\t/*\n+\t * At this point, leading dirs should have already been created. But if\n+\t * a symlink being checked out has collided with one of the dirs, due to\n+\t * file system folding rules, it's possible that the dirs are no longer\n+\t * present. So we have to check again, and report any path collisions.\n+\t */\n+\tif (!check_leading_dirs(path.buf, path.len, state->base_dir_len)) {\n+\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\tgoto out;\n+\t}\n+\n+\tfd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, mode);\n+\n+\tif (fd < 0) {\n+\t\tif (errno == EEXIST || errno == EISDIR) {\n+\t\t\t/*\n+\t\t\t * Errors which probably represent a path collision.\n+\t\t\t * Suppress the error message and mark the item to be\n+\t\t\t * retried later, sequentially. ENOTDIR and ENOENT are\n+\t\t\t * also interesting, but check_leading_dirs() should\n+\t\t\t * have already caught these cases.\n+\t\t\t */\n+\t\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\t} else {\n+\t\t\terror_errno(\"failed to open file %s\", path.buf);\n+\t\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\t}\n+\t\tgoto out;\n+\t}\n+\n+\tif (write_pc_item_to_fd(pc_item, fd, path.buf, state)) {\n+\t\t/* Error was already reported. */\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tfstat_done = fstat_checkout_output(fd, state, &pc_item->st);\n+\n+\tif (close_and_clear(&fd)) {\n+\t\terror_errno(\"unable to close file %s\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tif (state->refresh_cache && !fstat_done && lstat(path.buf, &pc_item->st) < 0) {\n+\t\terror_errno(\"unable to stat just-written file %s\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tpc_item->status = PC_ITEM_WRITTEN;\n+\n+out:\n+\t/*\n+\t * No need to check close() return. At this point, either fd is already\n+\t * closed, or we are on an error path, that has already been reported.\n+\t */\n+\tclose_and_clear(&fd);\n+\tstrbuf_release(&path);\n+}\n+\n+static void write_items_sequentially(struct checkout *state)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < parallel_checkout.nr; ++i)\n+\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+}\n+\n+int run_parallel_checkout(struct checkout *state)\n+{\n+\tint ret;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n+\t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n+\n+\tparallel_checkout.status = PC_RUNNING;\n+\n+\twrite_items_sequentially(state);\n+\tret = handle_results(state);\n+\n+\tfinish_parallel_checkout();\n+\treturn ret;\n+}\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nnew file mode 100644\nindex 0000000000..e6d6fc01ea\n--- /dev/null\n+++ b/parallel-checkout.h\n@@ -0,0 +1,27 @@\n+#ifndef PARALLEL_CHECKOUT_H\n+#define PARALLEL_CHECKOUT_H\n+\n+struct cache_entry;\n+struct checkout;\n+struct conv_attrs;\n+\n+enum pc_status {\n+\tPC_UNINITIALIZED = 0,\n+\tPC_ACCEPTING_ENTRIES,\n+\tPC_RUNNING,\n+};\n+\n+enum pc_status parallel_checkout_status(void);\n+void init_parallel_checkout(void);\n+\n+/*\n+ * Return -1 if parallel checkout is currently not enabled or if the entry is\n+ * not eligible for parallel checkout. Otherwise, enqueue the entry for later\n+ * write and return 0.\n+ */\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+\n+/* Write all the queued entries, returning 0 on success.*/\n+int run_parallel_checkout(struct checkout *state);\n+\n+#endif /* PARALLEL_CHECKOUT_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a511fadd89..1b1da7485a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -17,6 +17,7 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -438,7 +439,6 @@ static int check_updates(struct unpack_trees_options *o,\n \tif (should_update_submodules())\n \t\tload_gitmodules_file(index, &state);\n \n-\tenable_delayed_checkout(&state);\n \tif (has_promisor_remote()) {\n \t\t/*\n \t\t * Prefetch the objects that are to be checked out in the loop\n@@ -461,6 +461,9 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t\t\t\t to_fetch.oid, to_fetch.nr);\n \t\toid_array_clear(&to_fetch);\n \t}\n+\n+\tenable_delayed_checkout(&state);\n+\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -474,6 +477,7 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n+\terrs |= run_parallel_checkout(&state);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.28.0\n\n"},{"id":"406153","messageId":"991169488b17e492c6d2c2f212267a66693aa7ec.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 11/19] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:25Z","receivedAt":"2020-09-22T22:51:09Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Use multiple worker processes to distribute the queued entries and call\nwrite_checkout_item() in parallel for them. The items are distributed\nuniformly in contiguous chunks. This minimizes the chances of two\nworkers writing to the same directory simultaneously, which could\naffect performance due to lock contention in the kernel. Work stealing\n(or any other format of re-distribution) is not implemented yet.\n\nThe parallel version was benchmarked during three operations in the\nlinux repo, with cold cache: cloning v5.8, checking out v5.8 from\nv2.6.15 (checkout I) and checking out v5.8 from v5.7 (checkout II). The\nfour tables below show the mean run times and standard deviations for\n5 runs in: a local file system with SSD, a local file system with HDD, a\nLinux NFS server, and Amazon EFS. The numbers of workers were chosen\nbased on what produces the best result for each case.\n\nLocal SSD:\n\n Clone Checkout I Checkout II\nSequential 8.171 s ± 0.206 s 8.735 s ± 0.230 s 4.166 s ± 0.246 s\n10 workers 3.277 s ± 0.138 s 3.774 s ± 0.188 s 2.561 s ± 0.120 s\nSpeedup 2.49 ± 0.12 2.31 ± 0.13 1.63 ± 0.12\n\nLocal HDD:\n\n Clone Checkout I Checkout II\nSequential 35.157 s ± 0.205 s 48.835 s ± 0.407 s 47.302 s ± 1.435 s\n8 workers 35.538 s ± 0.325 s 49.353 s ± 0.826 s 48.919 s ± 0.416 s\nSpeedup 0.99 ± 0.01 0.99 ± 0.02 0.97 ± 0.03\n\nLinux NFS server (v4.1, on EBS, single availability zone):\n\n Clone Checkout I Checkout II\nSequential 216.070 s ± 3.611 s 211.169 s ± 3.147 s 57.446 s ± 1.301 s\n32 workers 67.997 s ± 0.740 s 66.563 s ± 0.457 s 23.708 s ± 0.622 s\nSpeedup 3.18 ± 0.06 3.17 ± 0.05 2.42 ± 0.08\n\nEFS (v4.1, replicated over multiple availability zones):\n\n Clone Checkout I Checkout II\nSequential 1249.329 s ± 13.857 s 1438.979 s ± 78.792 s 543.919 s ± 18.745 s\n64 workers 225.864 s ± 12.433 s 316.345 s ± 1.887 s 183.648 s ± 10.095 s\nSpeedup 5.53 ± 0.31 4.55 ± 0.25 2.96 ± 0.19\n\nThe above benchmarks show that parallel checkout is most effective on\nrepositories located on an SSD or over a distributed file system. For\nlocal file systems on spinning disks, and/or older machines, the\nparallelism does not always bring a good performance. In fact, it can\neven increase the run time. For this reason, the sequential code is\nstill the default. Two settings are added to optionally enable and\nconfigure the new parallel version as desired.\n\nLocal SSD tests were executed in an i7-7700HQ (4 cores with\nhyper-threading) running Manjaro Linux. Local HDD tests were executed in\nan i7-2600 (also 4 cores with hyper-threading), HDD Seagate Barracuda\n7200 rpm SATA 3.0, running Debian 9.13. NFS and EFS tests were\nexecuted in an Amazon EC2 c5n.large instance, with 2 vCPUs. The Linux\nNFS server was running on a m6g.large instance with 1 TB, EBS GP2\nvolume. Before each timing, the linux repository was removed (or checked\nout back), and `sync && sysctl vm.drop_caches=3` was executed.\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n .gitignore | 1 +\n Documentation/config/checkout.txt | 21 +++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/checkout--helper.c | 142 ++++++++++++++++\n git.c | 2 +\n parallel-checkout.c | 273 +++++++++++++++++++++++++++---\n parallel-checkout.h | 84 ++++++++-\n unpack-trees.c | 10 +-\n 9 files changed, 501 insertions(+), 34 deletions(-)\n create mode 100644 builtin/checkout--helper.c\n\ndiff --git a/.gitignore b/.gitignore\nindex d0f692a355..6427739814 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -33,6 +33,7 @@\n /git-check-mailmap\n /git-check-ref-format\n /git-checkout\n+/git-checkout--helper\n /git-checkout-index\n /git-cherry\n /git-cherry-pick\ndiff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt\nindex 6b646813ab..44eb58bcd3 100644\n--- a/Documentation/config/checkout.txt\n+++ b/Documentation/config/checkout.txt\n@@ -16,3 +16,24 @@ will checkout the '' branch on another remote,\n and by linkgit:git-worktree[1] when 'git worktree add' refers to a\n remote branch. This setting might be used for other checkout-like\n commands or functionality in the future.\n+\n+checkout.workers::\n+\tThe number of parallel workers to use when updating the working tree.\n+\tThe default is one, i.e. sequential execution. If set to a value less\n+\tthan one, Git will use as many workers as the number of logical cores\n+\tavailable. This setting and checkout.thresholdForParallelism affect all\n+\tcommands that perform checkout. E.g. checkout, switch, clone, reset,\n+\tsparse-checkout, read-tree, etc.\n++\n+Note: parallel checkout usually delivers better performance for repositories\n+located on SSDs or over NFS. For repositories on spinning disks and/or machines\n+with a small number of cores, the default sequential checkout often performs\n+better. The size and compression level of a repository might also influence how\n+well the parallel version performs.\n+\n+checkout.thresholdForParallelism::\n+\tWhen running parallel checkout with a small number of files, the cost\n+\tof subprocess spawning and inter-process communication might outweigh\n+\tthe parallelization gains. This setting allows to define the minimum\n+\tnumber of files for which parallel checkout should be attempted. The\n+\tdefault is 100.\ndiff --git a/Makefile b/Makefile\nindex 3edcdc534c..e9c6616180 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1049,6 +1049,7 @@ BUILTIN_OBJS += builtin/check-attr.o\n BUILTIN_OBJS += builtin/check-ignore.o\n BUILTIN_OBJS += builtin/check-mailmap.o\n BUILTIN_OBJS += builtin/check-ref-format.o\n+BUILTIN_OBJS += builtin/checkout--helper.o\n BUILTIN_OBJS += builtin/checkout-index.o\n BUILTIN_OBJS += builtin/checkout.o\n BUILTIN_OBJS += builtin/clean.o\ndiff --git a/builtin.h b/builtin.h\nindex ba954e180c..b52243848d 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -123,6 +123,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix);\n int cmd_bundle(int argc, const char **argv, const char *prefix);\n int cmd_cat_file(int argc, const char **argv, const char *prefix);\n int cmd_checkout(int argc, const char **argv, const char *prefix);\n+int cmd_checkout__helper(int argc, const char **argv, const char *prefix);\n int cmd_checkout_index(int argc, const char **argv, const char *prefix);\n int cmd_check_attr(int argc, const char **argv, const char *prefix);\n int cmd_check_ignore(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/checkout--helper.c b/builtin/checkout--helper.c\nnew file mode 100644\nindex 0000000000..67fe37cf11\n--- /dev/null\n+++ b/builtin/checkout--helper.c\n@@ -0,0 +1,142 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"parse-options.h\"\n+#include \"pkt-line.h\"\n+\n+static void packet_to_pc_item(char *line, int len,\n+\t\t\t struct parallel_checkout_item *pc_item)\n+{\n+\tstruct pc_item_fixed_portion *fixed_portion;\n+\tchar *encoding, *variant;\n+\n+\tif (len < sizeof(struct pc_item_fixed_portion))\n+\t\tBUG(\"checkout worker received too short item (got %dB, exp %dB)\",\n+\t\t len, (int)sizeof(struct pc_item_fixed_portion));\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)line;\n+\n+\tif (len - sizeof(struct pc_item_fixed_portion) !=\n+\t\tfixed_portion->name_len + fixed_portion->working_tree_encoding_len)\n+\t\tBUG(\"checkout worker received corrupted item\");\n+\n+\tvariant = line + sizeof(struct pc_item_fixed_portion);\n+\n+\t/*\n+\t * Note: the main process uses zero length to communicate that the\n+\t * encoding is NULL. There is no use case in actually sending an empty\n+\t * string since it's considered as NULL when ca.working_tree_encoding\n+\t * is set at git_path_check_encoding().\n+\t */\n+\tif (fixed_portion->working_tree_encoding_len) {\n+\t\tencoding = xmemdupz(variant,\n+\t\t\t\t fixed_portion->working_tree_encoding_len);\n+\t\tvariant += fixed_portion->working_tree_encoding_len;\n+\t} else {\n+\t\tencoding = NULL;\n+\t}\n+\n+\tmemset(pc_item, 0, sizeof(*pc_item));\n+\tpc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n+\tpc_item->ce->ce_namelen = fixed_portion->name_len;\n+\tpc_item->ce->ce_mode = fixed_portion->ce_mode;\n+\tmemcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen);\n+\toidcpy(&pc_item->ce->oid, &fixed_portion->oid);\n+\n+\tpc_item->id = fixed_portion->id;\n+\tpc_item->ca.crlf_action = fixed_portion->crlf_action;\n+\tpc_item->ca.ident = fixed_portion->ident;\n+\tpc_item->ca.working_tree_encoding = encoding;\n+}\n+\n+static void report_result(struct parallel_checkout_item *pc_item)\n+{\n+\tstruct pc_item_result res = { 0 };\n+\tsize_t size;\n+\n+\tres.id = pc_item->id;\n+\tres.status = pc_item->status;\n+\n+\tif (pc_item->status == PC_ITEM_WRITTEN) {\n+\t\tres.st = pc_item->st;\n+\t\tsize = sizeof(res);\n+\t} else {\n+\t\tsize = PC_ITEM_RESULT_BASE_SIZE;\n+\t}\n+\n+\tpacket_write(1, (const char *)&res, size);\n+}\n+\n+/* Free the worker-side malloced data, but not pc_item itself. */\n+static void release_pc_item_data(struct parallel_checkout_item *pc_item)\n+{\n+\tfree((char *)pc_item->ca.working_tree_encoding);\n+\tdiscard_cache_entry(pc_item->ce);\n+}\n+\n+static void worker_loop(struct checkout *state)\n+{\n+\tstruct parallel_checkout_item *items = NULL;\n+\tsize_t i, nr = 0, alloc = 0;\n+\n+\twhile (1) {\n+\t\tint len;\n+\t\tchar *line = packet_read_line(0, &len);\n+\n+\t\tif (!line)\n+\t\t\tbreak;\n+\n+\t\tALLOC_GROW(items, nr + 1, alloc);\n+\t\tpacket_to_pc_item(line, len, &items[nr++]);\n+\t}\n+\n+\tfor (i = 0; i < nr; ++i) {\n+\t\tstruct parallel_checkout_item *pc_item = &items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\treport_result(pc_item);\n+\t\trelease_pc_item_data(pc_item);\n+\t}\n+\n+\tpacket_flush(1);\n+\n+\tfree(items);\n+}\n+\n+static const char * const checkout_helper_usage[] = {\n+\tN_(\"git checkout--helper []\"),\n+\tNULL\n+};\n+\n+int cmd_checkout__helper(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct checkout state = CHECKOUT_INIT;\n+\tstruct option checkout_helper_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &state.base_dir, N_(\"string\"),\n+\t\t\tN_(\"when creating files, prepend \")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(checkout_helper_usage,\n+\t\t\t\t checkout_helper_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix, checkout_helper_options,\n+\t\t\t checkout_helper_usage, 0);\n+\tif (argc > 0)\n+\t\tusage_with_options(checkout_helper_usage, checkout_helper_options);\n+\n+\tif (state.base_dir)\n+\t\tstate.base_dir_len = strlen(state.base_dir);\n+\n+\t/*\n+\t * Setting this on worker won't actually update the index. We just need\n+\t * to pretend so to induce the checkout machinery to stat() the written\n+\t * entries.\n+\t */\n+\tstate.refresh_cache = 1;\n+\n+\tworker_loop(&state);\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 01c456edce..a09357fc56 100644\n--- a/git.c\n+++ b/git.c\n@@ -487,6 +487,8 @@ static struct cmd_struct commands[] = {\n \t{ \"check-mailmap\", cmd_check_mailmap, RUN_SETUP },\n \t{ \"check-ref-format\", cmd_check_ref_format, NO_PARSEOPT },\n \t{ \"checkout\", cmd_checkout, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"checkout--helper\", cmd_checkout__helper,\n+\t\tRUN_SETUP | NEED_WORK_TREE | SUPPORT_SUPER_PREFIX },\n \t{ \"checkout-index\", cmd_checkout_index,\n \t\tRUN_SETUP | NEED_WORK_TREE},\n \t{ \"cherry\", cmd_cherry, RUN_SETUP },\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 7dc8ab2a67..7ea0faa526 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,28 +1,15 @@\n #include \"cache.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n+#include \"pkt-line.h\"\n+#include \"run-command.h\"\n #include \"streaming.h\"\n+#include \"thread-utils.h\"\n+#include \"config.h\"\n \n-enum pc_item_status {\n-\tPC_ITEM_PENDING = 0,\n-\tPC_ITEM_WRITTEN,\n-\t/*\n-\t * The entry could not be written because there was another file\n-\t * already present in its path or leading directories. Since\n-\t * checkout_entry_ca() removes such files from the working tree before\n-\t * enqueueing the entry for parallel checkout, it means that there was\n-\t * a path collision among the entries being written.\n-\t */\n-\tPC_ITEM_COLLIDED,\n-\tPC_ITEM_FAILED,\n-};\n-\n-struct parallel_checkout_item {\n-\t/* pointer to a istate->cache[] entry. Not owned by us. */\n-\tstruct cache_entry *ce;\n-\tstruct conv_attrs ca;\n-\tstruct stat st;\n-\tenum pc_item_status status;\n+struct pc_worker {\n+\tstruct child_process cp;\n+\tsize_t next_to_complete, nr_to_complete;\n };\n \n struct parallel_checkout {\n@@ -38,6 +25,19 @@ enum pc_status parallel_checkout_status(void)\n \treturn parallel_checkout.status;\n }\n \n+#define DEFAULT_THRESHOLD_FOR_PARALLELISM 100\n+\n+void get_parallel_checkout_configs(int *num_workers, int *threshold)\n+{\n+\tif (git_config_get_int(\"checkout.workers\", num_workers))\n+\t\t*num_workers = 1;\n+\telse if (*num_workers < 1)\n+\t\t*num_workers = online_cpus();\n+\n+\tif (git_config_get_int(\"checkout.thresholdForParallelism\", threshold))\n+\t\t*threshold = DEFAULT_THRESHOLD_FOR_PARALLELISM;\n+}\n+\n void init_parallel_checkout(void)\n {\n \tif (parallel_checkout.status != PC_UNINITIALIZED)\n@@ -115,10 +115,12 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n \t\t parallel_checkout.alloc);\n \n-\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr];\n \tpc_item->ce = ce;\n \tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n \tpc_item->status = PC_ITEM_PENDING;\n+\tpc_item->id = parallel_checkout.nr;\n+\tparallel_checkout.nr++;\n \n \treturn 0;\n }\n@@ -231,7 +233,8 @@ static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n \t/*\n \t * checkout metadata is used to give context for external process\n \t * filters. Files requiring such filters are not eligible for parallel\n-\t * checkout, so pass NULL.\n+\t * checkout, so pass NULL. Note: if that changes, the metadata must also\n+\t * be passed from the main process to the workers.\n \t */\n \tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n \t\t\t\t\t new_blob, size, &buf, NULL);\n@@ -272,8 +275,8 @@ static int check_leading_dirs(const char *path, int len, int prefix_len)\n \treturn has_dirs_only_path(path, slash - path, prefix_len);\n }\n \n-static void write_pc_item(struct parallel_checkout_item *pc_item,\n-\t\t\t struct checkout *state)\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state)\n {\n \tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n \tint fd = -1, fstat_done = 0;\n@@ -343,6 +346,214 @@ static void write_pc_item(struct parallel_checkout_item *pc_item,\n \tstrbuf_release(&path);\n }\n \n+static void send_one_item(int fd, struct parallel_checkout_item *pc_item)\n+{\n+\tsize_t len_data;\n+\tchar *data, *variant;\n+\tstruct pc_item_fixed_portion *fixed_portion;\n+\tconst char *working_tree_encoding = pc_item->ca.working_tree_encoding;\n+\tsize_t name_len = pc_item->ce->ce_namelen;\n+\tsize_t working_tree_encoding_len = working_tree_encoding ?\n+\t\t\t\t\t strlen(working_tree_encoding) : 0;\n+\n+\tlen_data = sizeof(struct pc_item_fixed_portion) + name_len +\n+\t\t working_tree_encoding_len;\n+\n+\tdata = xcalloc(1, len_data);\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)data;\n+\tfixed_portion->id = pc_item->id;\n+\toidcpy(&fixed_portion->oid, &pc_item->ce->oid);\n+\tfixed_portion->ce_mode = pc_item->ce->ce_mode;\n+\tfixed_portion->crlf_action = pc_item->ca.crlf_action;\n+\tfixed_portion->ident = pc_item->ca.ident;\n+\tfixed_portion->name_len = name_len;\n+\tfixed_portion->working_tree_encoding_len = working_tree_encoding_len;\n+\n+\tvariant = data + sizeof(*fixed_portion);\n+\tif (working_tree_encoding_len) {\n+\t\tmemcpy(variant, working_tree_encoding, working_tree_encoding_len);\n+\t\tvariant += working_tree_encoding_len;\n+\t}\n+\tmemcpy(variant, pc_item->ce->name, name_len);\n+\n+\tpacket_write(fd, data, len_data);\n+\n+\tfree(data);\n+}\n+\n+static void send_batch(int fd, size_t start, size_t nr)\n+{\n+\tsize_t i;\n+\tfor (i = 0; i < nr; ++i)\n+\t\tsend_one_item(fd, ¶llel_checkout.items[start + i]);\n+\tpacket_flush(fd);\n+}\n+\n+static struct pc_worker *setup_workers(struct checkout *state, int num_workers)\n+{\n+\tstruct pc_worker *workers;\n+\tint i, workers_with_one_extra_item;\n+\tsize_t base_batch_size, next_to_assign = 0;\n+\n+\tALLOC_ARRAY(workers, num_workers);\n+\n+\tfor (i = 0; i < num_workers; ++i) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->git_cmd = 1;\n+\t\tcp->in = -1;\n+\t\tcp->out = -1;\n+\t\tcp->clean_on_exit = 1;\n+\t\tstrvec_push(&cp->args, \"checkout--helper\");\n+\t\tif (state->base_dir_len)\n+\t\t\tstrvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n+\t\tif (start_command(cp))\n+\t\t\tdie(_(\"failed to spawn checkout worker\"));\n+\t}\n+\n+\tbase_batch_size = parallel_checkout.nr / num_workers;\n+\tworkers_with_one_extra_item = parallel_checkout.nr % num_workers;\n+\n+\tfor (i = 0; i < num_workers; ++i) {\n+\t\tstruct pc_worker *worker = &workers[i];\n+\t\tsize_t batch_size = base_batch_size;\n+\n+\t\t/* distribute the extra work evenly */\n+\t\tif (i < workers_with_one_extra_item)\n+\t\t\tbatch_size++;\n+\n+\t\tsend_batch(worker->cp.in, next_to_assign, batch_size);\n+\t\tworker->next_to_complete = next_to_assign;\n+\t\tworker->nr_to_complete = batch_size;\n+\n+\t\tnext_to_assign += batch_size;\n+\t}\n+\n+\treturn workers;\n+}\n+\n+static void finish_workers(struct pc_worker *workers, int num_workers)\n+{\n+\tint i;\n+\n+\t/*\n+\t * Close pipes before calling finish_command() to let the workers\n+\t * exit asynchronously and avoid spending extra time on wait().\n+\t */\n+\tfor (i = 0; i < num_workers; ++i) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\t\tif (cp->in >= 0)\n+\t\t\tclose(cp->in);\n+\t\tif (cp->out >= 0)\n+\t\t\tclose(cp->out);\n+\t}\n+\n+\tfor (i = 0; i < num_workers; ++i) {\n+\t\tif (finish_command(&workers[i].cp))\n+\t\t\terror(_(\"checkout worker %d finished with error\"), i);\n+\t}\n+\n+\tfree(workers);\n+}\n+\n+#define ASSERT_PC_ITEM_RESULT_SIZE(got, exp) \\\n+{ \\\n+\tif (got != exp) \\\n+\t\tBUG(\"corrupted result from checkout worker (got %dB, exp %dB)\", \\\n+\t\t got, exp); \\\n+} while(0)\n+\n+static void parse_and_save_result(const char *line, int len,\n+\t\t\t\t struct pc_worker *worker)\n+{\n+\tstruct pc_item_result *res;\n+\tstruct parallel_checkout_item *pc_item;\n+\tstruct stat *st = NULL;\n+\n+\tif (len < PC_ITEM_RESULT_BASE_SIZE)\n+\t\tBUG(\"too short result from checkout worker (got %dB, exp %dB)\",\n+\t\t len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\n+\tres = (struct pc_item_result *)line;\n+\n+\t/*\n+\t * Worker should send either the full result struct on success, or\n+\t * just the base (i.e. no stat data), otherwise.\n+\t */\n+\tif (res->status == PC_ITEM_WRITTEN) {\n+\t\tASSERT_PC_ITEM_RESULT_SIZE(len, (int)sizeof(struct pc_item_result));\n+\t\tst = &res->st;\n+\t} else {\n+\t\tASSERT_PC_ITEM_RESULT_SIZE(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\t}\n+\n+\tif (!worker->nr_to_complete || res->id != worker->next_to_complete)\n+\t\tBUG(\"checkout worker sent unexpected item id\");\n+\n+\tworker->next_to_complete++;\n+\tworker->nr_to_complete--;\n+\n+\tpc_item = ¶llel_checkout.items[res->id];\n+\tpc_item->status = res->status;\n+\tif (st)\n+\t\tpc_item->st = *st;\n+}\n+\n+\n+static void gather_results_from_workers(struct pc_worker *workers,\n+\t\t\t\t\tint num_workers)\n+{\n+\tint i, active_workers = num_workers;\n+\tstruct pollfd *pfds;\n+\n+\tCALLOC_ARRAY(pfds, num_workers);\n+\tfor (i = 0; i < num_workers; ++i) {\n+\t\tpfds[i].fd = workers[i].cp.out;\n+\t\tpfds[i].events = POLLIN;\n+\t}\n+\n+\twhile (active_workers) {\n+\t\tint nr = poll(pfds, num_workers, -1);\n+\n+\t\tif (nr < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tdie_errno(\"failed to poll checkout workers\");\n+\t\t}\n+\n+\t\tfor (i = 0; i < num_workers && nr > 0; ++i) {\n+\t\t\tstruct pc_worker *worker = &workers[i];\n+\t\t\tstruct pollfd *pfd = &pfds[i];\n+\n+\t\t\tif (!pfd->revents)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (pfd->revents & POLLIN) {\n+\t\t\t\tint len;\n+\t\t\t\tconst char *line = packet_read_line(pfd->fd, &len);\n+\n+\t\t\t\tif (!line) {\n+\t\t\t\t\tpfd->fd = -1;\n+\t\t\t\t\tactive_workers--;\n+\t\t\t\t} else {\n+\t\t\t\t\tparse_and_save_result(line, len, worker);\n+\t\t\t\t}\n+\t\t\t} else if (pfd->revents & POLLHUP) {\n+\t\t\t\tpfd->fd = -1;\n+\t\t\t\tactive_workers--;\n+\t\t\t} else if (pfd->revents & (POLLNVAL | POLLERR)) {\n+\t\t\t\tdie(_(\"error polling from checkout worker\"));\n+\t\t\t}\n+\n+\t\t\tnr--;\n+\t\t}\n+\t}\n+\n+\tfree(pfds);\n+}\n+\n static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n@@ -351,7 +562,7 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n-int run_parallel_checkout(struct checkout *state)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n {\n \tint ret;\n \n@@ -360,7 +571,17 @@ int run_parallel_checkout(struct checkout *state)\n \n \tparallel_checkout.status = PC_RUNNING;\n \n-\twrite_items_sequentially(state);\n+\tif (parallel_checkout.nr < num_workers)\n+\t\tnum_workers = parallel_checkout.nr;\n+\n+\tif (num_workers <= 1 || parallel_checkout.nr < threshold) {\n+\t\twrite_items_sequentially(state);\n+\t} else {\n+\t\tstruct pc_worker *workers = setup_workers(state, num_workers);\n+\t\tgather_results_from_workers(workers, num_workers);\n+\t\tfinish_workers(workers, num_workers);\n+\t}\n+\n \tret = handle_results(state);\n \n \tfinish_parallel_checkout();\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex e6d6fc01ea..0c9984584e 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -1,9 +1,12 @@\n #ifndef PARALLEL_CHECKOUT_H\n #define PARALLEL_CHECKOUT_H\n \n-struct cache_entry;\n-struct checkout;\n-struct conv_attrs;\n+#include \"entry.h\"\n+#include \"convert.h\"\n+\n+/****************************************************************\n+ * Users of parallel checkout\n+ ****************************************************************/\n \n enum pc_status {\n \tPC_UNINITIALIZED = 0,\n@@ -12,6 +15,7 @@ enum pc_status {\n };\n \n enum pc_status parallel_checkout_status(void);\n+void get_parallel_checkout_configs(int *num_workers, int *threshold);\n void init_parallel_checkout(void);\n \n /*\n@@ -21,7 +25,77 @@ void init_parallel_checkout(void);\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n \n-/* Write all the queued entries, returning 0 on success.*/\n-int run_parallel_checkout(struct checkout *state);\n+/*\n+ * Write all the queued entries, returning 0 on success. If the number of\n+ * entries is smaller than the specified threshold, the operation is performed\n+ * sequentially.\n+ */\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n+\n+/****************************************************************\n+ * Interface with checkout--helper\n+ ****************************************************************/\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/*\n+\t * In main process ce points to a istate->cache[] entry. Thus, it's not\n+\t * owned by us. In workers they own the memory, which *must be* released.\n+\t */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tsize_t id; /* position in parallel_checkout.items[] of main process */\n+\n+\t/* Output fields, sent from workers. */\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+/*\n+ * The fixed-size portion of `struct parallel_checkout_item` that is sent to the\n+ * workers. Following this will be 2 strings: ca.working_tree_encoding and\n+ * ce.name; These are NOT null terminated, since we have the size in the fixed\n+ * portion.\n+ *\n+ * Note that not all fields of conv_attrs and cache_entry are passed, only the\n+ * ones that will be required by the workers to smudge and write the entry.\n+ */\n+struct pc_item_fixed_portion {\n+\tsize_t id;\n+\tstruct object_id oid;\n+\tunsigned int ce_mode;\n+\tenum crlf_action crlf_action;\n+\tint ident;\n+\tsize_t working_tree_encoding_len;\n+\tsize_t name_len;\n+};\n+\n+/*\n+ * The fields of `struct parallel_checkout_item` that are returned by the\n+ * workers. Note: `st` must be the last one, as it is omitted on error.\n+ */\n+struct pc_item_result {\n+\tsize_t id;\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+#define PC_ITEM_RESULT_BASE_SIZE offsetof(struct pc_item_result, st)\n+\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state);\n \n #endif /* PARALLEL_CHECKOUT_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1b1da7485a..117ed42370 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -399,7 +399,7 @@ static int check_updates(struct unpack_trees_options *o,\n \tint errs = 0;\n \tstruct progress *progress;\n \tstruct checkout state = CHECKOUT_INIT;\n-\tint i;\n+\tint i, pc_workers, pc_threshold;\n \n \ttrace_performance_enter();\n \tstate.force = 1;\n@@ -462,8 +462,11 @@ static int check_updates(struct unpack_trees_options *o,\n \t\toid_array_clear(&to_fetch);\n \t}\n \n+\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n \tenable_delayed_checkout(&state);\n-\tinit_parallel_checkout();\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -477,7 +480,8 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n-\terrs |= run_parallel_checkout(&state);\n+\tif (pc_workers > 1)\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.28.0\n\n"},{"id":"406154","messageId":"7ceadf2427b5c9b04c0943c0b257b8ebcaa13f19.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 12/19] parallel-checkout: support progress displaying","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:26Z","receivedAt":"2020-09-22T22:51:11Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Original-patch-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n parallel-checkout.c | 34 +++++++++++++++++++++++++++++++---\n parallel-checkout.h | 4 +++-\n unpack-trees.c | 11 ++++++++---\n 3 files changed, 42 insertions(+), 7 deletions(-)\n\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 7ea0faa526..5156b14c53 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -2,6 +2,7 @@\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n+#include \"progress.h\"\n #include \"run-command.h\"\n #include \"streaming.h\"\n #include \"thread-utils.h\"\n@@ -16,6 +17,8 @@ struct parallel_checkout {\n \tenum pc_status status;\n \tstruct parallel_checkout_item *items;\n \tsize_t nr, alloc;\n+\tstruct progress *progress;\n+\tunsigned int *progress_cnt;\n };\n \n static struct parallel_checkout parallel_checkout = { 0 };\n@@ -125,6 +128,20 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \treturn 0;\n }\n \n+size_t pc_queue_size(void)\n+{\n+\treturn parallel_checkout.nr;\n+}\n+\n+static void advance_progress_meter(void)\n+{\n+\tif (parallel_checkout.progress) {\n+\t\t(*parallel_checkout.progress_cnt)++;\n+\t\tdisplay_progress(parallel_checkout.progress,\n+\t\t\t\t *parallel_checkout.progress_cnt);\n+\t}\n+}\n+\n static int handle_results(struct checkout *state)\n {\n \tint ret = 0;\n@@ -173,6 +190,7 @@ static int handle_results(struct checkout *state)\n \t\t\t */\n \t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n \t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tadvance_progress_meter();\n \t\t\tbreak;\n \t\tcase PC_ITEM_PENDING:\n \t\t\thave_pending = 1;\n@@ -499,6 +517,9 @@ static void parse_and_save_result(const char *line, int len,\n \tpc_item->status = res->status;\n \tif (st)\n \t\tpc_item->st = *st;\n+\n+\tif (res->status != PC_ITEM_COLLIDED)\n+\t\tadvance_progress_meter();\n }\n \n \n@@ -558,11 +579,16 @@ static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n \n-\tfor (i = 0; i < parallel_checkout.nr; ++i)\n-\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+\tfor (i = 0; i < parallel_checkout.nr; ++i) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\tif (pc_item->status != PC_ITEM_COLLIDED)\n+\t\t\tadvance_progress_meter();\n+\t}\n }\n \n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt)\n {\n \tint ret;\n \n@@ -570,6 +596,8 @@ int run_parallel_checkout(struct checkout *state, int num_workers, int threshold\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n+\tparallel_checkout.progress = progress;\n+\tparallel_checkout.progress_cnt = progress_cnt;\n \n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 0c9984584e..6c3a016c0b 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -24,13 +24,15 @@ void init_parallel_checkout(void);\n * write and return 0.\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+size_t pc_queue_size(void);\n \n /*\n * Write all the queued entries, returning 0 on success. If the number of\n * entries is smaller than the specified threshold, the operation is performed\n * sequentially.\n */\n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt);\n \n /****************************************************************\n * Interface with checkout--helper\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 117ed42370..e05e6ceff2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -471,17 +471,22 @@ static int check_updates(struct unpack_trees_options *o,\n \t\tstruct cache_entry *ce = index->cache[i];\n \n \t\tif (ce->ce_flags & CE_UPDATE) {\n+\t\t\tsize_t last_pc_queue_size = pc_queue_size();\n+\n \t\t\tif (ce->ce_flags & CE_WT_REMOVE)\n \t\t\t\tBUG(\"both update and delete flags are set on %s\",\n \t\t\t\t ce->name);\n-\t\t\tdisplay_progress(progress, ++cnt);\n \t\t\tce->ce_flags &= ~CE_UPDATE;\n \t\t\terrs |= checkout_entry(ce, &state, NULL, NULL);\n+\n+\t\t\tif (last_pc_queue_size == pc_queue_size())\n+\t\t\t\tdisplay_progress(progress, ++cnt);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tif (pc_workers > 1)\n-\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t\t progress, &cnt);\n+\tstop_progress(&progress);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.28.0\n\n"},{"id":"406155","messageId":"f13b4c17f4829911b7b8274fd0161a6b7784e9fb.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 13/19] make_transient_cache_entry(): optionally alloc from mem_pool","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:27Z","receivedAt":"2020-09-22T22:51:14Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Allow make_transient_cache_entry() to optionally receive a mem_pool\nstruct in which it should allocate the entry. This will be used in the\nfollowing patch, to store some transient entries which should persist\nuntil parallel checkout finishes.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/checkout--helper.c | 2 +-\n builtin/checkout.c | 2 +-\n builtin/difftool.c | 2 +-\n cache.h | 10 +++++-----\n read-cache.c | 12 ++++++++----\n unpack-trees.c | 2 +-\n 6 files changed, 17 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/checkout--helper.c b/builtin/checkout--helper.c\nindex 67fe37cf11..9646ed9eeb 100644\n--- a/builtin/checkout--helper.c\n+++ b/builtin/checkout--helper.c\n@@ -38,7 +38,7 @@ static void packet_to_pc_item(char *line, int len,\n \t}\n \n \tmemset(pc_item, 0, sizeof(*pc_item));\n-\tpc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n+\tpc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len, NULL);\n \tpc_item->ce->ce_namelen = fixed_portion->name_len;\n \tpc_item->ce->ce_mode = fixed_portion->ce_mode;\n \tmemcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex b18b9d6f3c..c0bf5e6711 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -291,7 +291,7 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_transient_cache_entry(mode, &oid, path, 2);\n+\tce = make_transient_cache_entry(mode, &oid, path, 2, NULL);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL, nr_checkouts);\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex dfa22b67eb..5e7a57c8c2 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -323,7 +323,7 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_transient_cache_entry(mode, oid, path, 0);\n+\tce = make_transient_cache_entry(mode, oid, path, 0, NULL);\n \tret = checkout_entry(ce, state, NULL, NULL);\n \n \tdiscard_cache_entry(ce);\ndiff --git a/cache.h b/cache.h\nindex 17350cafa2..a394263f0e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -355,16 +355,16 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate,\n \t\t\t\t\t size_t name_len);\n \n /*\n- * Create a cache_entry that is not intended to be added to an index.\n- * Caller is responsible for discarding the cache_entry\n- * with `discard_cache_entry`.\n+ * Create a cache_entry that is not intended to be added to an index. If mp is\n+ * not NULL, the entry is allocated within the given memory pool. Caller is\n+ * responsible for discarding the cache_entry with `discard_cache_entry`.\n */\n struct cache_entry *make_transient_cache_entry(unsigned int mode,\n \t\t\t\t\t const struct object_id *oid,\n \t\t\t\t\t const char *path,\n-\t\t\t\t\t int stage);\n+\t\t\t\t\t int stage, struct mem_pool *mp);\n \n-struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n+struct cache_entry *make_empty_transient_cache_entry(size_t len, struct mem_pool *mp);\n \n /*\n * Discard cache entry.\ndiff --git a/read-cache.c b/read-cache.c\nindex ecf6f68994..f9bac760af 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -813,8 +813,10 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t le\n \treturn mem_pool__ce_calloc(find_mem_pool(istate), len);\n }\n \n-struct cache_entry *make_empty_transient_cache_entry(size_t len)\n+struct cache_entry *make_empty_transient_cache_entry(size_t len, struct mem_pool *mp)\n {\n+\tif (mp)\n+\t\treturn mem_pool__ce_calloc(mp, len);\n \treturn xcalloc(1, cache_entry_size(len));\n }\n \n@@ -848,8 +850,10 @@ struct cache_entry *make_cache_entry(struct index_state *istate,\n \treturn ret;\n }\n \n-struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid,\n-\t\t\t\t\t const char *path, int stage)\n+struct cache_entry *make_transient_cache_entry(unsigned int mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t const char *path, int stage,\n+\t\t\t\t\t struct mem_pool *mp)\n {\n \tstruct cache_entry *ce;\n \tint len;\n@@ -860,7 +864,7 @@ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct o\n \t}\n \n \tlen = strlen(path);\n-\tce = make_empty_transient_cache_entry(len);\n+\tce = make_empty_transient_cache_entry(len, mp);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex e05e6ceff2..dcb40dc8fa 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1031,7 +1031,7 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \tsize_t len = traverse_path_len(info, tree_entry_len(n));\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n-\t\tmake_empty_transient_cache_entry(len) :\n+\t\tmake_empty_transient_cache_entry(len, NULL) :\n \t\tmake_empty_cache_entry(istate, len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n-- \n2.28.0\n\n"},{"id":"406156","messageId":"d7885a1130b6781a349d2c2647eaabe8d4ca115f.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 14/19] builtin/checkout.c: complete parallel checkout support","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:28Z","receivedAt":"2020-09-22T22:51:18Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"There is one code path in builtin/checkout.c which still doesn't benefit\nfrom parallel checkout because it calls checkout_entry() directly,\ninstead of unpack_trees(). Let's add parallel support for this missing\nspot as well. Note: the transient cache entries allocated in\ncheckout_merged() are now allocated in a mem_pool which is only\ndiscarded after parallel checkout finishes. This is done because the\nentries need to be valid when run_parallel_checkout() is called.\n\nSigned-off-by: Matheus Tavares \n---\n builtin/checkout.c | 20 ++++++++++++++++----\n 1 file changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex c0bf5e6711..ddc4079b85 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -27,6 +27,7 @@\n #include \"wt-status.h\"\n #include \"xdiff-interface.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n static const char * const checkout_usage[] = {\n \tN_(\"git checkout [] \"),\n@@ -230,7 +231,8 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos,\n \t\treturn error(_(\"path '%s' does not have their version\"), ce->name);\n }\n \n-static int checkout_merged(int pos, const struct checkout *state, int *nr_checkouts)\n+static int checkout_merged(int pos, const struct checkout *state,\n+\t\t\t int *nr_checkouts, struct mem_pool *ce_mem_pool)\n {\n \tstruct cache_entry *ce = active_cache[pos];\n \tconst char *path = ce->name;\n@@ -291,11 +293,10 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_transient_cache_entry(mode, &oid, path, 2, NULL);\n+\tce = make_transient_cache_entry(mode, &oid, path, 2, ce_mem_pool);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL, nr_checkouts);\n-\tdiscard_cache_entry(ce);\n \treturn status;\n }\n \n@@ -359,16 +360,22 @@ static int checkout_worktree(const struct checkout_opts *opts,\n \tint nr_checkouts = 0, nr_unmerged = 0;\n \tint errs = 0;\n \tint pos;\n+\tint pc_workers, pc_threshold;\n+\tstruct mem_pool ce_mem_pool;\n \n \tstate.force = 1;\n \tstate.refresh_cache = 1;\n \tstate.istate = &the_index;\n \n+\tmem_pool_init(&ce_mem_pool, 0);\n+\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n \tinit_checkout_metadata(&state.meta, info->refname,\n \t\t\t info->commit ? &info->commit->object.oid : &info->oid,\n \t\t\t NULL);\n \n \tenable_delayed_checkout(&state);\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n \t\tif (ce->ce_flags & CE_MATCHED) {\n@@ -384,10 +391,15 @@ static int checkout_worktree(const struct checkout_opts *opts,\n \t\t\t\t\t\t &nr_checkouts, opts->overlay_mode);\n \t\t\telse if (opts->merge)\n \t\t\t\terrs |= checkout_merged(pos, &state,\n-\t\t\t\t\t\t\t&nr_unmerged);\n+\t\t\t\t\t\t\t&nr_unmerged,\n+\t\t\t\t\t\t\t&ce_mem_pool);\n \t\t\tpos = skip_same_name(ce, pos) - 1;\n \t\t}\n \t}\n+\tif (pc_workers > 1)\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t\t NULL, NULL);\n+\tmem_pool_discard(&ce_mem_pool, should_validate_cache_entries());\n \tremove_marked_cache_entries(&the_index, 1);\n \tremove_scheduled_dirs();\n \terrs |= finish_delayed_checkout(&state, &nr_checkouts);\n-- \n2.28.0\n\n"},{"id":"406157","messageId":"1cf9b807f780467a3837b5e13939ead7c67eaef5.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 15/19] checkout-index: add parallel checkout support","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:29Z","receivedAt":"2020-09-22T22:51:22Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Signed-off-by: Matheus Tavares \n---\n builtin/checkout-index.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 0f1ff73129..33fb933c30 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -12,6 +12,7 @@\n #include \"cache-tree.h\"\n #include \"parse-options.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n #define CHECKOUT_ALL 4\n static int nul_term_line;\n@@ -160,6 +161,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tint prefix_length;\n \tint force = 0, quiet = 0, not_new = 0;\n \tint index_opt = 0;\n+\tint pc_workers, pc_threshold;\n \tstruct option builtin_checkout_index_options[] = {\n \t\tOPT_BOOL('a', \"all\", &all,\n \t\t\tN_(\"check out all files in the index\")),\n@@ -214,6 +216,14 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \t\thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n \t}\n \n+\tif (!to_tempfile)\n+\t\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\telse\n+\t\tpc_workers = 1;\n+\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n+\n \t/* Check out named files first */\n \tfor (i = 0; i < argc; i++) {\n \t\tconst char *arg = argv[i];\n@@ -256,6 +266,12 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tif (all)\n \t\tcheckout_all(prefix, prefix_length);\n \n+\tif (pc_workers > 1) {\n+\t\t/* Errors were already reported */\n+\t\trun_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t NULL, NULL);\n+\t}\n+\n \tif (is_lock_file_locked(&lock_file) &&\n \t write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\tdie(\"Unable to write new index file\");\n-- \n2.28.0\n\n"},{"id":"406158","messageId":"64b41d537e68a45f2bb0a0c3078f2cd314b5a57d.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 16/19] parallel-checkout: add tests for basic operations","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:30Z","receivedAt":"2020-09-22T22:51:24Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Add tests to populate the working tree during clone and checkout using\nthe sequential and parallel modes, to confirm that they produce\nidentical results. Also test basic checkout mechanics, such as checking\nfor symlinks in the leading directories and the abidance to --force.\n\nNote: some helper functions are added to a common lib file which is only\nincluded by t2080 for now. But it will also be used by another\nparallel-checkout test in a following patch.\n\nOriginal-patch-by: Jeff Hostetler \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n t/lib-parallel-checkout.sh | 39 ++++++\n t/t2080-parallel-checkout-basics.sh | 197 ++++++++++++++++++++++++++++\n 2 files changed, 236 insertions(+)\n create mode 100644 t/lib-parallel-checkout.sh\n create mode 100755 t/t2080-parallel-checkout-basics.sh\n\ndiff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh\nnew file mode 100644\nindex 0000000000..c95ca27711\n--- /dev/null\n+++ b/t/lib-parallel-checkout.sh\n@@ -0,0 +1,39 @@\n+# Helpers for t208* tests\n+\n+# Runs `git -c checkout.workers=$1 -c checkout.thesholdForParallelism=$2 ${@:4}`\n+# and checks that the number of workers spawned is equal to $3.\n+git_pc()\n+{\n+\tif test $# -lt 4\n+\tthen\n+\t\tBUG \"too few arguments to git_pc()\"\n+\tfi\n+\n+\tworkers=$1 threshold=$2 expected_workers=$3 &&\n+\tshift && shift && shift &&\n+\n+\trm -f trace &&\n+\tGIT_TRACE2=\"$(pwd)/trace\" git \\\n+\t\t-c checkout.workers=$workers \\\n+\t\t-c checkout.thresholdForParallelism=$threshold \\\n+\t\t-c advice.detachedHead=0 \\\n+\t\t$@ &&\n+\n+\t# Check that the expected number of workers has been used. Note that it\n+\t# can be different than the requested number in two cases: when the\n+\t# quantity of entries to be checked out is less than the number of\n+\t# workers; and when the threshold has not been reached.\n+\t#\n+\tlocal workers_in_trace=$(grep \"child_start\\[.\\+\\] git checkout--helper\" trace | wc -l) &&\n+\ttest $workers_in_trace -eq $expected_workers &&\n+\trm -f trace\n+}\n+\n+# Verify that both the working tree and the index were created correctly\n+verify_checkout()\n+{\n+\tgit -C $1 diff-index --quiet HEAD -- &&\n+\tgit -C $1 diff-index --quiet --cached HEAD -- &&\n+\tgit -C $1 status --porcelain >$1.status &&\n+\ttest_must_be_empty $1.status\n+}\ndiff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh\nnew file mode 100755\nindex 0000000000..c088a06ecc\n--- /dev/null\n+++ b/t/t2080-parallel-checkout-basics.sh\n@@ -0,0 +1,197 @@\n+#!/bin/sh\n+\n+test_description='parallel-checkout basics\n+\n+Ensure that parallel-checkout basically works on clone and checkout, spawning\n+the required number of workers and correctly populating both the index and\n+working tree.\n+'\n+\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY/lib-parallel-checkout.sh\"\n+\n+# NEEDSWORK: cloning a SHA1 repo with GIT_TEST_DEFAULT_HASH set to \"sha256\"\n+# currently produces a wrong result (See\n+# https://lore.kernel.org/git/20200911151717.43475-1-matheus.bernardino@usp.br/).\n+# So we skip the \"parallel-checkout during clone\" tests when this test flag is\n+# set to \"sha256\". Remove this when the bug is fixed.\n+#\n+if test \"$GIT_TEST_DEFAULT_HASH\" = \"sha256\"\n+then\n+\tskip_all=\"t2080 currently don't work with GIT_TEST_DEFAULT_HASH=sha256\"\n+\ttest_done\n+fi\n+\n+R_BASE=$GIT_BUILD_DIR\n+\n+test_expect_success 'sequential clone' '\n+\tgit_pc 1 0 0 clone --quiet -- $R_BASE r_sequential &&\n+\tverify_checkout r_sequential\n+'\n+\n+test_expect_success 'parallel clone' '\n+\tgit_pc 2 0 2 clone --quiet -- $R_BASE r_parallel &&\n+\tverify_checkout r_parallel\n+'\n+\n+test_expect_success 'fallback to sequential clone (threshold)' '\n+\tgit -C $R_BASE ls-files >files &&\n+\tnr_files=$(wc -l a &&\n+\t\tmkdir e &&\n+\t\techo e/x >e/x &&\n+\t\tln -s e b &&\n+\t\tgit add -A &&\n+\t\tgit commit -m B1 &&\n+\n+\t\tgit checkout -b B2 &&\n+\t\techo modified >a &&\n+\t\trm -rf e &&\n+\t\trm b &&\n+\t\tmkdir b &&\n+\t\techo b/f >b/f &&\n+\t\tln -s b e &&\n+\t\tgit init d &&\n+\t\ttest_commit -C d f &&\n+\t\tgit submodule add ./d &&\n+\t\tgit add -A &&\n+\t\tgit commit -m B2 &&\n+\n+\t\tgit checkout --recurse-submodules B1\n+\t)\n+'\n+\n+test_expect_success SYMLINKS 'sequential checkout' '\n+\tcp -R various various_sequential &&\n+\tgit_pc 1 0 0 -C various_sequential checkout --recurse-submodules B2 &&\n+\tverify_checkout various_sequential\n+'\n+\n+test_expect_success SYMLINKS 'parallel checkout' '\n+\tcp -R various various_parallel &&\n+\tgit_pc 2 0 2 -C various_parallel checkout --recurse-submodules B2 &&\n+\tverify_checkout various_parallel\n+'\n+\n+test_expect_success SYMLINKS 'fallback to sequential checkout (threshold)' '\n+\tcp -R various various_sequential_fallback &&\n+\tgit_pc 2 100 0 -C various_sequential_fallback checkout --recurse-submodules B2 &&\n+\tverify_checkout various_sequential_fallback\n+'\n+\n+test_expect_success SYMLINKS 'compare working trees from checkouts' '\n+\trm -rf various_sequential/.git &&\n+\trm -rf various_parallel/.git &&\n+\trm -rf various_sequential_fallback/.git &&\n+\tdiff -qr various_sequential various_parallel &&\n+\tdiff -qr various_sequential various_sequential_fallback\n+'\n+\n+test_cmp_str()\n+{\n+\techo \"$1\" >tmp &&\n+\ttest_cmp tmp \"$2\"\n+}\n+\n+test_expect_success 'parallel checkout respects --[no]-force' '\n+\tgit init dirty &&\n+\t(\n+\t\tcd dirty &&\n+\t\tmkdir D &&\n+\t\ttest_commit D/F &&\n+\t\ttest_commit F &&\n+\n+\t\techo changed >F.t &&\n+\t\trm -rf D &&\n+\t\techo changed >D &&\n+\n+\t\t# We expect 0 workers because there is nothing to be updated\n+\t\tgit_pc 2 0 0 checkout HEAD &&\n+\t\ttest_path_is_file D &&\n+\t\ttest_cmp_str changed D &&\n+\t\ttest_cmp_str changed F.t &&\n+\n+\t\tgit_pc 2 0 2 checkout --force HEAD &&\n+\t\ttest_path_is_dir D &&\n+\t\ttest_cmp_str D/F D/F.t &&\n+\t\ttest_cmp_str F F.t\n+\t)\n+'\n+\n+test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading dirs' '\n+\tgit init symlinks &&\n+\t(\n+\t\tcd symlinks &&\n+\t\tmkdir D E &&\n+\n+\t\t# Create two entries in D to have enough work for 2 parallel\n+\t\t# workers\n+\t\ttest_commit D/A &&\n+\t\ttest_commit D/B &&\n+\t\ttest_commit E/C &&\n+\t\trm -rf D &&\n+\t\tln -s E D &&\n+\n+\t\tgit_pc 2 0 2 checkout --force HEAD &&\n+\t\t! test -L D &&\n+\t\ttest_cmp_str D/A D/A.t &&\n+\t\ttest_cmp_str D/B D/B.t\n+\t)\n+'\n+\n+test_expect_success SYMLINKS,CASE_INSENSITIVE_FS 'symlink colliding with leading dir' '\n+\tgit init colliding-symlink &&\n+\t(\n+\t\tcd colliding-symlink &&\n+\t\tfile_hex=$(git hash-object -w --stdin tree &&\n+\t\tprintf \"100644 E/B\\0${file_oct}\" >>tree &&\n+\t\tprintf \"120000 e\\0${sym_oct}\" >>tree &&\n+\n+\t\ttree_hex=$(git hash-object -w -t tree --stdin \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n t/t2081-parallel-checkout-collisions.sh | 115 ++++++++++++++++++++++++\n 1 file changed, 115 insertions(+)\n create mode 100755 t/t2081-parallel-checkout-collisions.sh\n\ndiff --git a/t/t2081-parallel-checkout-collisions.sh b/t/t2081-parallel-checkout-collisions.sh\nnew file mode 100755\nindex 0000000000..3ce195b892\n--- /dev/null\n+++ b/t/t2081-parallel-checkout-collisions.sh\n@@ -0,0 +1,115 @@\n+#!/bin/sh\n+\n+test_description='parallel-checkout collisions'\n+\n+. ./test-lib.sh\n+\n+# When there are pathname collisions during a clone, Git should report a warning\n+# listing all of the colliding entries. The sequential code detects a collision\n+# by calling lstat() before trying to open(O_CREAT) the file. Then, to find the\n+# colliding pair of an item k, it searches cache_entry[0, k-1].\n+#\n+# This is not sufficient in parallel-checkout mode since colliding files may be\n+# created in a racy order. The tests in this file make sure the collision\n+# detection code is extended for parallel-checkout. This is done in two parts:\n+#\n+# - First, two parallel workers create four colliding files racily.\n+# - Then this exercise is repeated but forcing the colliding pair to appear in\n+# the second half of the cache_entry's array.\n+#\n+# The second item uses the fact that files with clean/smudge filters are not\n+# parallel-eligible; and that they are processed sequentially *before* any\n+# worker is spawned. We set a filter attribute to the last entry in the\n+# cache_entry[] array, making it non-eligible, so that it is populated first.\n+# This way, we can test if the collision detection code is correctly looking\n+# for collision pairs in the second half of the array.\n+\n+test_expect_success CASE_INSENSITIVE_FS 'setup' '\n+\tfile_hex=$(git hash-object -w --stdin tree &&\n+\tprintf \"100644 FILE_x\\0${file_oct}\" >>tree &&\n+\tprintf \"100644 file_X\\0${file_oct}\" >>tree &&\n+\tprintf \"100644 file_x\\0${file_oct}\" >>tree &&\n+\tprintf \"100644 .gitattributes\\0${attr_oct}\" >>tree &&\n+\n+\ttree_hex=$(git hash-object -w -t tree --stdin >filter.log\n+\tEOF\n+'\n+\n+clone_and_check_collision()\n+{\n+\tid=$1 workers=$2 threshold=$3 expected_workers=$4 filter=$5 &&\n+\n+\tfilter_opts=\n+\tif test \"$filter\" -eq \"use_filter\"\n+\tthen\n+\t\t# We use `core.ignoreCase=0` so that only `file_x`\n+\t\t# matches the pattern in .gitattributes.\n+\t\t#\n+\t\tfilter_opts='-c filter.logger.smudge=\"../logger_script %f\" -c core.ignoreCase=0'\n+\tfi &&\n+\n+\ttest_path_is_missing $id.trace &&\n+\tGIT_TRACE2=\"$(pwd)/$id.trace\" git \\\n+\t\t-c checkout.workers=$workers \\\n+\t\t-c checkout.thresholdForParallelism=$threshold \\\n+\t\t$filter_opts clone --branch=collisions -- . r_$id 2>$id.warning &&\n+\n+\t# Check that checkout spawned the right number of workers\n+\tworkers_in_trace=$(grep \"child_start\\[.\\] git checkout--helper\" $id.trace | wc -l) &&\n+\ttest $workers_in_trace -eq $expected_workers &&\n+\n+\tif test $filter -eq \"use_filter\"\n+\tthen\n+\t\t# Make sure only 'file_x' was filtered\n+\t\ttest_path_is_file r_$id/filter.log &&\n+\t\techo file_x >expected.filter.log &&\n+\t\ttest_cmp r_$id/filter.log expected.filter.log\n+\telse\n+\t\ttest_path_is_missing r_$id/filter.log\n+\tfi &&\n+\n+\tgrep FILE_X $id.warning &&\n+\tgrep FILE_x $id.warning &&\n+\tgrep file_X $id.warning &&\n+\tgrep file_x $id.warning &&\n+\ttest_i18ngrep \"the following paths have collided\" $id.warning\n+}\n+\n+test_expect_success CASE_INSENSITIVE_FS 'collision detection on parallel clone' '\n+\tclone_and_check_collision parallel 2 0 2\n+'\n+\n+test_expect_success CASE_INSENSITIVE_FS 'collision detection on fallback to sequential clone' '\n+\tgit ls-tree --name-only -r collisions >files &&\n+\tnr_files=$(wc -l files &&\n+\tnr_files=$(wc -l \nSigned-off-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n t/lib-encoding.sh | 25 ++++\n t/t0028-working-tree-encoding.sh | 25 +---\n t/t2082-parallel-checkout-attributes.sh | 174 ++++++++++++++++++++++++\n 3 files changed, 200 insertions(+), 24 deletions(-)\n create mode 100644 t/lib-encoding.sh\n create mode 100755 t/t2082-parallel-checkout-attributes.sh\n\ndiff --git a/t/lib-encoding.sh b/t/lib-encoding.sh\nnew file mode 100644\nindex 0000000000..c52ffbbed5\n--- /dev/null\n+++ b/t/lib-encoding.sh\n@@ -0,0 +1,25 @@\n+# Encoding helpers used by t0028 and t2082\n+\n+test_lazy_prereq NO_UTF16_BOM '\n+\ttest $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6\n+'\n+\n+test_lazy_prereq NO_UTF32_BOM '\n+\ttest $(printf abc | iconv -f UTF-8 -t UTF-32 | wc -c) = 12\n+'\n+\n+write_utf16 () {\n+\tif test_have_prereq NO_UTF16_BOM\n+\tthen\n+\t\tprintf '\\376\\377'\n+\tfi &&\n+\ticonv -f UTF-8 -t UTF-16\n+}\n+\n+write_utf32 () {\n+\tif test_have_prereq NO_UTF32_BOM\n+\tthen\n+\t\tprintf '\\0\\0\\376\\377'\n+\tfi &&\n+\ticonv -f UTF-8 -t UTF-32\n+}\ndiff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh\nindex bfc4fb9af5..4fffc3a639 100755\n--- a/t/t0028-working-tree-encoding.sh\n+++ b/t/t0028-working-tree-encoding.sh\n@@ -3,33 +3,10 @@\n test_description='working-tree-encoding conversion via gitattributes'\n \n . ./test-lib.sh\n+. \"$TEST_DIRECTORY/lib-encoding.sh\"\n \n GIT_TRACE_WORKING_TREE_ENCODING=1 && export GIT_TRACE_WORKING_TREE_ENCODING\n \n-test_lazy_prereq NO_UTF16_BOM '\n-\ttest $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6\n-'\n-\n-test_lazy_prereq NO_UTF32_BOM '\n-\ttest $(printf abc | iconv -f UTF-8 -t UTF-32 | wc -c) = 12\n-'\n-\n-write_utf16 () {\n-\tif test_have_prereq NO_UTF16_BOM\n-\tthen\n-\t\tprintf '\\376\\377'\n-\tfi &&\n-\ticonv -f UTF-8 -t UTF-16\n-}\n-\n-write_utf32 () {\n-\tif test_have_prereq NO_UTF32_BOM\n-\tthen\n-\t\tprintf '\\0\\0\\376\\377'\n-\tfi &&\n-\ticonv -f UTF-8 -t UTF-32\n-}\n-\n test_expect_success 'setup test files' '\n \tgit config core.eol lf &&\n \ndiff --git a/t/t2082-parallel-checkout-attributes.sh b/t/t2082-parallel-checkout-attributes.sh\nnew file mode 100755\nindex 0000000000..6800574588\n--- /dev/null\n+++ b/t/t2082-parallel-checkout-attributes.sh\n@@ -0,0 +1,174 @@\n+#!/bin/sh\n+\n+test_description='parallel-checkout: attributes\n+\n+Verify that parallel-checkout correctly creates files that require\n+conversions, as specified in .gitattributes. The main point here is\n+to check that the conv_attr data is correctly sent to the workers\n+and that it contains sufficient information to smudge files\n+properly (without access to the index or attribute stack).\n+'\n+\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY/lib-parallel-checkout.sh\"\n+. \"$TEST_DIRECTORY/lib-encoding.sh\"\n+\n+test_expect_success 'parallel-checkout with ident' '\n+\tgit init ident &&\n+\t(\n+\t\tcd ident &&\n+\t\techo \"A ident\" >.gitattributes &&\n+\t\techo \"\\$Id\\$\" >A &&\n+\t\techo \"\\$Id\\$\" >B &&\n+\t\tgit add -A &&\n+\t\tgit commit -m id &&\n+\n+\t\trm A B &&\n+\t\tgit_pc 2 0 2 reset --hard &&\n+\t\thexsz=$(test_oid hexsz) &&\n+\t\tgrep -E \"\\\\\\$Id: [0-9a-f]{$hexsz} \\\\\\$\" A &&\n+\t\tgrep \"\\\\\\$Id\\\\\\$\" B\n+\t)\n+'\n+\n+test_expect_success 'parallel-checkout with re-encoding' '\n+\tgit init encoding &&\n+\t(\n+\t\tcd encoding &&\n+\t\techo text >utf8-text &&\n+\t\tcat utf8-text | write_utf16 >utf16-text &&\n+\n+\t\techo \"A working-tree-encoding=UTF-16\" >.gitattributes &&\n+\t\tcp utf16-text A &&\n+\t\tcp utf16-text B &&\n+\t\tgit add A B .gitattributes &&\n+\t\tgit commit -m encoding &&\n+\n+\t\t# Check that A (and only A) is stored in UTF-8\n+\t\tgit cat-file -p :A >A.internal &&\n+\t\ttest_cmp_bin utf8-text A.internal &&\n+\t\tgit cat-file -p :B >B.internal &&\n+\t\ttest_cmp_bin utf16-text B.internal &&\n+\n+\t\t# Check that A is re-encoded during checkout\n+\t\trm A B &&\n+\t\tgit_pc 2 0 2 checkout A B &&\n+\t\ttest_cmp_bin utf16-text A\n+\t)\n+'\n+\n+test_expect_success 'parallel-checkout with eol conversions' '\n+\tgit init eol &&\n+\t(\n+\t\tcd eol &&\n+\t\tgit config core.autocrlf false &&\n+\t\tprintf \"multi\\r\\nline\\r\\ntext\" >crlf-text &&\n+\t\tprintf \"multi\\nline\\ntext\" >lf-text &&\n+\n+\t\techo \"A text eol=crlf\" >.gitattributes &&\n+\t\techo \"B -text\" >>.gitattributes &&\n+\t\tcp crlf-text A &&\n+\t\tcp crlf-text B &&\n+\t\tgit add A B .gitattributes &&\n+\t\tgit commit -m eol &&\n+\n+\t\t# Check that A (and only A) is stored with LF format\n+\t\tgit cat-file -p :A >A.internal &&\n+\t\ttest_cmp_bin lf-text A.internal &&\n+\t\tgit cat-file -p :B >B.internal &&\n+\t\ttest_cmp_bin crlf-text B.internal &&\n+\n+\t\t# Check that A is converted to CRLF during checkout\n+\t\trm A B &&\n+\t\tgit_pc 2 0 2 checkout A B &&\n+\t\ttest_cmp_bin crlf-text A\n+\t)\n+'\n+\n+test_cmp_str()\n+{\n+\techo \"$1\" >tmp &&\n+\ttest_cmp tmp \"$2\"\n+}\n+\n+# Entries that require an external filter are not eligible for parallel\n+# checkout. Check that both the parallel-eligible and non-eligible entries are\n+# properly writen in a single checkout process.\n+#\n+test_expect_success 'parallel-checkout and external filter' '\n+\tgit init filter &&\n+\t(\n+\t\tcd filter &&\n+\t\tgit config filter.x2y.clean \"tr x y\" &&\n+\t\tgit config filter.x2y.smudge \"tr y x\" &&\n+\t\tgit config filter.x2y.required true &&\n+\n+\t\techo \"A filter=x2y\" >.gitattributes &&\n+\t\techo x >A &&\n+\t\techo x >B &&\n+\t\techo x >C &&\n+\t\tgit add -A &&\n+\t\tgit commit -m filter &&\n+\n+\t\t# Check that A (and only A) was cleaned\n+\t\tgit cat-file -p :A >A.internal &&\n+\t\ttest_cmp_str y A.internal &&\n+\t\tgit cat-file -p :B >B.internal &&\n+\t\ttest_cmp_str x B.internal &&\n+\t\tgit cat-file -p :C >C.internal &&\n+\t\ttest_cmp_str x C.internal &&\n+\n+\t\trm A B C *.internal &&\n+\t\tgit_pc 2 0 2 checkout A B C &&\n+\t\ttest_cmp_str x A &&\n+\t\ttest_cmp_str x B &&\n+\t\ttest_cmp_str x C\n+\t)\n+'\n+\n+# The delayed queue is independent from the parallel queue, and they should be\n+# able to work together in the same checkout process.\n+#\n+test_expect_success PERL 'parallel-checkout and delayed checkout' '\n+\twrite_script rot13-filter.pl \"$PERL_PATH\" \\\n+\t\t<\"$TEST_DIRECTORY\"/t0021/rot13-filter.pl &&\n+\ttest_config_global filter.delay.process \\\n+\t\t\"\\\"$(pwd)/rot13-filter.pl\\\" \\\"$(pwd)/delayed.log\\\" clean smudge delay\" &&\n+\ttest_config_global filter.delay.required true &&\n+\n+\techo \"a b c\" >delay-content &&\n+\techo \"n o p\" >delay-rot13-content &&\n+\n+\tgit init delayed &&\n+\t(\n+\t\tcd delayed &&\n+\t\techo \"*.a filter=delay\" >.gitattributes &&\n+\t\tcp ../delay-content test-delay10.a &&\n+\t\tcp ../delay-content test-delay11.a &&\n+\t\techo parallel >parallel1.b &&\n+\t\techo parallel >parallel2.b &&\n+\t\tgit add -A &&\n+\t\tgit commit -m delayed &&\n+\n+\t\t# Check that the stored data was cleaned\n+\t\tgit cat-file -p :test-delay10.a > delay10.internal &&\n+\t\ttest_cmp delay10.internal ../delay-rot13-content &&\n+\t\tgit cat-file -p :test-delay11.a > delay11.internal &&\n+\t\ttest_cmp delay11.internal ../delay-rot13-content &&\n+\t\trm *.internal &&\n+\n+\t\trm *.a *.b\n+\t) &&\n+\n+\tgit_pc 2 0 2 -C delayed checkout -f &&\n+\tverify_checkout delayed &&\n+\n+\t# Check that the *.a files got to the delay queue and were filtered\n+\tgrep \"smudge test-delay10.a .* \\[DELAYED\\]\" delayed.log &&\n+\tgrep \"smudge test-delay11.a .* \\[DELAYED\\]\" delayed.log &&\n+\ttest_cmp delayed/test-delay10.a delay-content &&\n+\ttest_cmp delayed/test-delay11.a delay-content\n+'\n+\n+test_done\n-- \n2.28.0\n\n"},{"id":"406161","messageId":"b4cb5905d2240b5605ea6f9190ef02f49bf7042e.1600814153.git.matheus.bernardino@usp.br","threadId":"54015","inReplyTo":"cover.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH v2 19/19] ci: run test round with parallel-checkout enabled","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-22T22:49:33Z","receivedAt":"2020-09-22T22:51:34Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"We already have tests for the basic parallel-checkout operations. But\nthis code can also run in other commands, such as git-read-tree and\ngit-sparse-checkout, which are currently not tested with multiple\nworkers. To promote a wider test coverage without duplicating tests:\n\n1. Add the GIT_TEST_CHECKOUT_WORKERS environment variable, to optionally\n force parallel-checkout execution during the whole test suite.\n\n2. Include this variable in the second test round of the linux-gcc job\n of our ci scripts. This round runs `make test` again with some\n optional GIT_TEST_* variables enabled, so there is no additional\n overhead in exercising the parallel-checkout code here.\n\nNote: the specific parallel-checkout tests t208* cannot be used in\ncombination with GIT_TEST_CHECKOUT_WORKERS as they need to set and check\nthe number of workers by themselves. So skip those tests when this flag\nis set.\n\nSigned-off-by: Matheus Tavares \n---\n ci/run-build-and-tests.sh | 1 +\n parallel-checkout.c | 14 ++++++++++++++\n t/README | 4 ++++\n t/lib-parallel-checkout.sh | 6 ++++++\n t/t2081-parallel-checkout-collisions.sh | 1 +\n 5 files changed, 26 insertions(+)\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 6c27b886b8..aa32ddc361 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -22,6 +22,7 @@ linux-gcc)\n \texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n \texport GIT_TEST_ADD_I_USE_BUILTIN=1\n+\texport GIT_TEST_CHECKOUT_WORKERS=2\n \tmake test\n \t;;\n linux-clang)\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 5156b14c53..94b44d2a48 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -32,6 +32,20 @@ enum pc_status parallel_checkout_status(void)\n \n void get_parallel_checkout_configs(int *num_workers, int *threshold)\n {\n+\tchar *env_workers = getenv(\"GIT_TEST_CHECKOUT_WORKERS\");\n+\n+\tif (env_workers && *env_workers) {\n+\t\tif (strtol_i(env_workers, 10, num_workers)) {\n+\t\t\tdie(\"invalid value for GIT_TEST_CHECKOUT_WORKERS: '%s'\",\n+\t\t\t env_workers);\n+\t\t}\n+\t\tif (*num_workers < 1)\n+\t\t\t*num_workers = online_cpus();\n+\n+\t\t*threshold = 0;\n+\t\treturn;\n+\t}\n+\n \tif (git_config_get_int(\"checkout.workers\", num_workers))\n \t\t*num_workers = 1;\n \telse if (*num_workers < 1)\ndiff --git a/t/README b/t/README\nindex 2adaf7c2d2..cd1b15c55a 100644\n--- a/t/README\n+++ b/t/README\n@@ -425,6 +425,10 @@ GIT_TEST_DEFAULT_HASH= specifies which hash algorithm to\n use in the test scripts. Recognized values for are \"sha1\"\n and \"sha256\".\n \n+GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting\n+to and 'checkout.thresholdForParallelism' to 0, forcing the\n+execution of the parallel-checkout code.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh\nindex c95ca27711..80bb0a0900 100644\n--- a/t/lib-parallel-checkout.sh\n+++ b/t/lib-parallel-checkout.sh\n@@ -1,5 +1,11 @@\n # Helpers for t208* tests\n \n+if ! test -z \"$GIT_TEST_CHECKOUT_WORKERS\"\n+then\n+\tskip_all=\"skipping test, GIT_TEST_CHECKOUT_WORKERS is set\"\n+\ttest_done\n+fi\n+\n # Runs `git -c checkout.workers=$1 -c checkout.thesholdForParallelism=$2 ${@:4}`\n # and checks that the number of workers spawned is equal to $3.\n git_pc()\ndiff --git a/t/t2081-parallel-checkout-collisions.sh b/t/t2081-parallel-checkout-collisions.sh\nindex 3ce195b892..5dbff54bfb 100755\n--- a/t/t2081-parallel-checkout-collisions.sh\n+++ b/t/t2081-parallel-checkout-collisions.sh\n@@ -3,6 +3,7 @@\n test_description='parallel-checkout collisions'\n \n . ./test-lib.sh\n+. \"$TEST_DIRECTORY/lib-parallel-checkout.sh\"\n \n # When there are pathname collisions during a clone, Git should report a warning\n # listing all of the colliding entries. The sequential code detects a collision\n-- \n2.28.0\n\n"},{"id":"406640","messageId":"CAN0heSoJvkZ+gvQwZMPW5Q0HqB0mOUnVqXiOqnPk48cDxMzAqw@mail.gmail.com","threadId":"54015","inReplyTo":"991169488b17e492c6d2c2f212267a66693aa7ec.1600814153.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v2 11/19] parallel-checkout: make it truly parallel","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-09-29T19:52:19Z","receivedAt":"2020-09-29T19:52:33Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Hi Matheus,\n\nOn Wed, 23 Sep 2020 at 00:53, Matheus Tavares wrote:\n> --- a/Documentation/config/checkout.txt\n> +++ b/Documentation/config/checkout.txt\n> @@ -16,3 +16,24 @@ will checkout the '' branch on another remote,\n> and by linkgit:git-worktree[1] when 'git worktree add' refers to a\n> remote branch. This setting might be used for other checkout-like\n> commands or functionality in the future.\n> +\n> +checkout.workers::\n> + The number of parallel workers to use when updating the working tree.\n> + The default is one, i.e. sequential execution. If set to a value less\n> + than one, Git will use as many workers as the number of logical cores\n> + available. This setting and checkout.thresholdForParallelism affect all\n\nIf you end up rerolling this patch series for other reasons, you might\nwant to consider using `backticks` around\n`checkout.thresholdForParallelism` so that it gets typeset as monospace.\n\n> + commands that perform checkout. E.g. checkout, switch, clone, reset,\n> + sparse-checkout, read-tree, etc.\n\nSimilarly here. Or perhaps go for \"linkgit:git-checkout[1],\nlinkgit:git-switch[1]\" etc.\n\nBTW, as far as \".e.g.,\" goes, this list looks quite long. :) I almost\nget the feeling you've made it fairly exhaustive and added the \"e.g.,\"\nmore as future proofing than anything else. I don't think anyone would\ncomplain if you left out, say, the plumbing `git read-tree` from the list.\n\n> +Note: parallel checkout usually delivers better performance for repositories\n> +located on SSDs or over NFS. For repositories on spinning disks and/or machines\n> +with a small number of cores, the default sequential checkout often performs\n> +better. The size and compression level of a repository might also influence how\n> +well the parallel version performs.\n> +\n> +checkout.thresholdForParallelism::\n\nSorry if this has already been discussed, but this \"For\" looks somewhat\nodd. Basically every config knob is \"somethingForSomething\". ;-) Could\nthis be `checkout.parallelismThreshold`?\n\n\n> + When running parallel checkout with a small number of files, the cost\n> + of subprocess spawning and inter-process communication might outweigh\n> + the parallelization gains. This setting allows to define the minimum\n> + number of files for which parallel checkout should be attempted. The\n> + default is 100.\n\nMartin\n"},{"id":"406681","messageId":"CAHd-oW7ErMKqeeoF40Mq418SusbVhkL2R7W73=4mbXUiamY1mA@mail.gmail.com","threadId":"54015","inReplyTo":"CAN0heSoJvkZ+gvQwZMPW5Q0HqB0mOUnVqXiOqnPk48cDxMzAqw@mail.gmail.com","subject":"Re: [PATCH v2 11/19] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-09-30T14:02:02Z","receivedAt":"2020-09-30T14:02:20Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Martin\n\nOn Tue, Sep 29, 2020 at 4:52 PM Martin Ågren wrote:\n>\n> Hi Matheus,\n>\n> On Wed, 23 Sep 2020 at 00:53, Matheus Tavares wrote:\n> > --- a/Documentation/config/checkout.txt\n> > +++ b/Documentation/config/checkout.txt\n> > @@ -16,3 +16,24 @@ will checkout the '' branch on another remote,\n> > and by linkgit:git-worktree[1] when 'git worktree add' refers to a\n> > remote branch. This setting might be used for other checkout-like\n> > commands or functionality in the future.\n> > +\n> > +checkout.workers::\n> > + The number of parallel workers to use when updating the working tree.\n> > + The default is one, i.e. sequential execution. If set to a value less\n> > + than one, Git will use as many workers as the number of logical cores\n> > + available. This setting and checkout.thresholdForParallelism affect all\n>\n> If you end up rerolling this patch series for other reasons, you might\n> want to consider using `backticks` around\n> `checkout.thresholdForParallelism` so that it gets typeset as monospace.\n\nSure, thanks!\n\n> > + commands that perform checkout. E.g. checkout, switch, clone, reset,\n> > + sparse-checkout, read-tree, etc.\n>\n> Similarly here. Or perhaps go for \"linkgit:git-checkout[1],\n> linkgit:git-switch[1]\" etc.\n>\n> BTW, as far as \".e.g.,\" goes, this list looks quite long. :) I almost\n> get the feeling you've made it fairly exhaustive and added the \"e.g.,\"\n> more as future proofing than anything else. I don't think anyone would\n> complain if you left out, say, the plumbing `git read-tree` from the list.\n\nYeah, I might have gotten a little too carried away there hehe :)\n\n> > +Note: parallel checkout usually delivers better performance for repositories\n> > +located on SSDs or over NFS. For repositories on spinning disks and/or machines\n> > +with a small number of cores, the default sequential checkout often performs\n> > +better. The size and compression level of a repository might also influence how\n> > +well the parallel version performs.\n> > +\n> > +checkout.thresholdForParallelism::\n>\n> Sorry if this has already been discussed, but this \"For\" looks somewhat\n> odd. Basically every config knob is \"somethingForSomething\". ;-) Could\n> this be `checkout.parallelismThreshold`?\n\nThanks for the suggestion. TBH, I spent more time than I'd like to\nadmit trying to come up with a name for this setting... But\n`checkout.parallelismThreshold` does sound better.\n"},{"id":"406766","messageId":"3eec6883-d9b4-33d9-5a0d-45e82fba29d2@jeffhostetler.com","threadId":"54015","inReplyTo":"0b7d7704-159f-c186-9551-1989af8f572d@jeffhostetler.com","subject":"Re: [PATCH v2 08/19] entry: move conv_attrs lookup up to checkout_entry()","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-10-01T15:59:34Z","receivedAt":"2020-10-01T15:59:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/1/20 11:53 AM, Jeff Hostetler wrote:\n> \n> \n> On 9/22/20 6:49 PM, Matheus Tavares wrote:\n>> In a following patch, checkout_entry() will use conv_attrs to decide\n>> whether an entry should be enqueued for parallel checkout or not. But\n>> the attributes lookup only happens lower in this call stack. To avoid\n>> the unnecessary work of loading the attributes twice, let's move it up\n>> to checkout_entry(), and pass the loaded struct down to write_entry().\n>>\n>> Signed-off-by: Matheus Tavares \n>> ---\n>>   entry.c | 38 +++++++++++++++++++++++++++-----------\n>>   1 file changed, 27 insertions(+), 11 deletions(-)\n>>\n>> diff --git a/entry.c b/entry.c\n>> index 1d2df188e5..8237859b12 100644\n>> --- a/entry.c\n>> +++ b/entry.c\n>> @@ -263,8 +263,9 @@ void update_ce_after_write(const struct checkout \n>> *state, struct cache_entry *ce,\n>>       }\n>>   }\n>> -static int write_entry(struct cache_entry *ce,\n>> -               char *path, const struct checkout *state, int \n>> to_tempfile)\n>> +/* Note: ca is used (and required) iff the entry refers to a regular \n>> file. */\n>> +static int write_entry(struct cache_entry *ce, char *path, struct \n>> conv_attrs *ca,\n>> +               const struct checkout *state, int to_tempfile)\n>>   {\n>>       unsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n>>       struct delayed_checkout *dco = state->delayed_checkout;\n>> @@ -281,8 +282,7 @@ static int write_entry(struct cache_entry *ce,\n>>       clone_checkout_metadata(&meta, &state->meta, &ce->oid);\n>>       if (ce_mode_s_ifmt == S_IFREG) {\n>> -        struct stream_filter *filter = \n>> get_stream_filter(state->istate, ce->name,\n>> -                                 &ce->oid);\n>> +        struct stream_filter *filter = get_stream_filter_ca(ca, \n>> &ce->oid);\n>>           if (filter &&\n>>               !streaming_write_entry(ce, path, filter,\n>>                          state, to_tempfile,\n>> @@ -329,14 +329,17 @@ static int write_entry(struct cache_entry *ce,\n>>            * Convert from git internal format to working tree format\n>>            */\n>>           if (dco && dco->state != CE_NO_DELAY) {\n>> -            ret = async_convert_to_working_tree(state->istate, \n>> ce->name, new_blob,\n>> -                                size, &buf, &meta, dco);\n>> +            ret = async_convert_to_working_tree_ca(ca, ce->name,\n>> +                                   new_blob, size,\n>> +                                   &buf, &meta, dco);\n>>               if (ret && string_list_has_string(&dco->paths, ce->name)) {\n>>                   free(new_blob);\n>>                   goto delayed;\n>>               }\n>> -        } else\n>> -            ret = convert_to_working_tree(state->istate, ce->name, \n>> new_blob, size, &buf, &meta);\n>> +        } else {\n>> +            ret = convert_to_working_tree_ca(ca, ce->name, new_blob,\n>> +                             size, &buf, &meta);\n>> +        }\n>>           if (ret) {\n>>               free(new_blob);\n>> @@ -442,6 +445,7 @@ int checkout_entry(struct cache_entry *ce, const \n>> struct checkout *state,\n>>   {\n>>       static struct strbuf path = STRBUF_INIT;\n>>       struct stat st;\n>> +    struct conv_attrs ca;\n> \n> I have to wonder if it would be clearer to move this declaration of `ca`\n> into the two `if { ... }` blocks where it is used -- to indicate that it\n> is only defined in two cases where we call `convert_attrs()`.\n> \n> There are several other calls to `write_entry()` that pass NULL and it\n> could cause confusion.\n\n\nNevermind, I see what you did in step 9 and this makes sense.\n\n> \n> \n>>       if (ce->ce_flags & CE_WT_REMOVE) {\n>>           if (topath)\n>> @@ -454,8 +458,13 @@ int checkout_entry(struct cache_entry *ce, const \n>> struct checkout *state,\n>>           return 0;\n>>       }\n>> -    if (topath)\n>> -        return write_entry(ce, topath, state, 1);\n>> +    if (topath) {\n>> +        if (S_ISREG(ce->ce_mode)) {\n>> +            convert_attrs(state->istate, &ca, ce->name);\n>> +            return write_entry(ce, topath, &ca, state, 1);\n>> +        }\n>> +        return write_entry(ce, topath, NULL, state, 1);\n>> +    }\n>>       strbuf_reset(&path);\n>>       strbuf_add(&path, state->base_dir, state->base_dir_len);\n>> @@ -517,9 +526,16 @@ int checkout_entry(struct cache_entry *ce, const \n>> struct checkout *state,\n>>           return 0;\n>>       create_directories(path.buf, path.len, state);\n>> +\n>>       if (nr_checkouts)\n>>           (*nr_checkouts)++;\n>> -    return write_entry(ce, path.buf, state, 0);\n>> +\n>> +    if (S_ISREG(ce->ce_mode)) {\n>> +        convert_attrs(state->istate, &ca, ce->name);\n>> +        return write_entry(ce, path.buf, &ca, state, 0);\n>> +    }\n>> +\n>> +    return write_entry(ce, path.buf, NULL, state, 0);\n>>   }\n>>   void unlink_entry(const struct cache_entry *ce)\n>>\n"},{"id":"406767","messageId":"0b7d7704-159f-c186-9551-1989af8f572d@jeffhostetler.com","threadId":"54015","inReplyTo":"667ad0dea70cb7f0bbf8f52467f15129b3ae1325.1600814153.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v2 08/19] entry: move conv_attrs lookup up to checkout_entry()","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-10-01T15:53:28Z","receivedAt":"2020-10-01T16:00:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 9/22/20 6:49 PM, Matheus Tavares wrote:\n> In a following patch, checkout_entry() will use conv_attrs to decide\n> whether an entry should be enqueued for parallel checkout or not. But\n> the attributes lookup only happens lower in this call stack. To avoid\n> the unnecessary work of loading the attributes twice, let's move it up\n> to checkout_entry(), and pass the loaded struct down to write_entry().\n> \n> Signed-off-by: Matheus Tavares \n> ---\n> entry.c | 38 +++++++++++++++++++++++++++-----------\n> 1 file changed, 27 insertions(+), 11 deletions(-)\n> \n> diff --git a/entry.c b/entry.c\n> index 1d2df188e5..8237859b12 100644\n> --- a/entry.c\n> +++ b/entry.c\n> @@ -263,8 +263,9 @@ void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,\n> \t}\n> }\n> \n> -static int write_entry(struct cache_entry *ce,\n> -\t\t char *path, const struct checkout *state, int to_tempfile)\n> +/* Note: ca is used (and required) iff the entry refers to a regular file. */\n> +static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca,\n> +\t\t const struct checkout *state, int to_tempfile)\n> {\n> \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n> \tstruct delayed_checkout *dco = state->delayed_checkout;\n> @@ -281,8 +282,7 @@ static int write_entry(struct cache_entry *ce,\n> \tclone_checkout_metadata(&meta, &state->meta, &ce->oid);\n> \n> \tif (ce_mode_s_ifmt == S_IFREG) {\n> -\t\tstruct stream_filter *filter = get_stream_filter(state->istate, ce->name,\n> -\t\t\t\t\t\t\t\t &ce->oid);\n> +\t\tstruct stream_filter *filter = get_stream_filter_ca(ca, &ce->oid);\n> \t\tif (filter &&\n> \t\t !streaming_write_entry(ce, path, filter,\n> \t\t\t\t\t state, to_tempfile,\n> @@ -329,14 +329,17 @@ static int write_entry(struct cache_entry *ce,\n> \t\t * Convert from git internal format to working tree format\n> \t\t */\n> \t\tif (dco && dco->state != CE_NO_DELAY) {\n> -\t\t\tret = async_convert_to_working_tree(state->istate, ce->name, new_blob,\n> -\t\t\t\t\t\t\t size, &buf, &meta, dco);\n> +\t\t\tret = async_convert_to_working_tree_ca(ca, ce->name,\n> +\t\t\t\t\t\t\t new_blob, size,\n> +\t\t\t\t\t\t\t &buf, &meta, dco);\n> \t\t\tif (ret && string_list_has_string(&dco->paths, ce->name)) {\n> \t\t\t\tfree(new_blob);\n> \t\t\t\tgoto delayed;\n> \t\t\t}\n> -\t\t} else\n> -\t\t\tret = convert_to_working_tree(state->istate, ce->name, new_blob, size, &buf, &meta);\n> +\t\t} else {\n> +\t\t\tret = convert_to_working_tree_ca(ca, ce->name, new_blob,\n> +\t\t\t\t\t\t\t size, &buf, &meta);\n> +\t\t}\n> \n> \t\tif (ret) {\n> \t\t\tfree(new_blob);\n> @@ -442,6 +445,7 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n> {\n> \tstatic struct strbuf path = STRBUF_INIT;\n> \tstruct stat st;\n> +\tstruct conv_attrs ca;\n\nI have to wonder if it would be clearer to move this declaration of `ca`\ninto the two `if { ... }` blocks where it is used -- to indicate that it\nis only defined in two cases where we call `convert_attrs()`.\n\nThere are several other calls to `write_entry()` that pass NULL and it\ncould cause confusion.\n\n\n> \n> \tif (ce->ce_flags & CE_WT_REMOVE) {\n> \t\tif (topath)\n> @@ -454,8 +458,13 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n> \t\treturn 0;\n> \t}\n> \n> -\tif (topath)\n> -\t\treturn write_entry(ce, topath, state, 1);\n> +\tif (topath) {\n> +\t\tif (S_ISREG(ce->ce_mode)) {\n> +\t\t\tconvert_attrs(state->istate, &ca, ce->name);\n> +\t\t\treturn write_entry(ce, topath, &ca, state, 1);\n> +\t\t}\n> +\t\treturn write_entry(ce, topath, NULL, state, 1);\n> +\t}\n> \n> \tstrbuf_reset(&path);\n> \tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n> @@ -517,9 +526,16 @@ int checkout_entry(struct cache_entry *ce, const struct checkout *state,\n> \t\treturn 0;\n> \n> \tcreate_directories(path.buf, path.len, state);\n> +\n> \tif (nr_checkouts)\n> \t\t(*nr_checkouts)++;\n> -\treturn write_entry(ce, path.buf, state, 0);\n> +\n> +\tif (S_ISREG(ce->ce_mode)) {\n> +\t\tconvert_attrs(state->istate, &ca, ce->name);\n> +\t\treturn write_entry(ce, path.buf, &ca, state, 0);\n> +\t}\n> +\n> +\treturn write_entry(ce, path.buf, NULL, state, 0);\n> }\n> \n> void unlink_entry(const struct cache_entry *ce)\n> \n"},{"id":"406782","messageId":"956e4570-a42d-d9c6-f904-de9d793852d5@jeffhostetler.com","threadId":"54015","inReplyTo":"cover.1597093021.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH 00/21] [RFC] Parallel checkout","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-10-01T16:42:07Z","receivedAt":"2020-10-01T16:42:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 8/10/20 5:33 PM, Matheus Tavares wrote:\n> This series adds parallel workers to the checkout machinery. The cache\n> entries are distributed among helper processes which are responsible for\n> reading, filtering and writing the blobs to the working tree. This\n> should benefit all commands that call unpack_trees() or check_updates(),\n> such as: checkout, clone, sparse-checkout, checkout-index, etc.\n\nThis series looks very good!\nThanks for your attention to detail.\n\nJeff\n\n"},{"id":"406893","messageId":"20201005061732.GA2288306@coredump.intra.peff.net","threadId":"54015","inReplyTo":"af0d790973b1e52da0fc9c4bed95e93d995d68da.1600814153.git.matheus.bernardino@usp.br","subject":"[PATCH] parallel-checkout: drop unused checkout state parameter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-05T06:17:32Z","receivedAt":"2020-10-05T06:17:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 22, 2020 at 07:49:24PM -0300, Matheus Tavares wrote:\n\n> +static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n> +\t\t\t const char *path, struct checkout *state)\n\nThe \"state\" parameter is unused here. Maybe this on top of\nmt/parallel-checkout-part-1?\n\n-- >8 --\nSubject: parallel-checkout: drop unused checkout state parameter\n\nThe write_pc_item_to_fd() function takes a \"struct checkout *state\"\nparameter, but never uses it. This was true in its introduction in\nfa33dd99f0 (unpack-trees: add basic support for parallel checkout,\n2020-09-22). Its caller, write_pc_item(), has already pulled the useful\nbits from the state struct into the \"path\" variable. Let's drop the\nuseless parameter.\n\nSigned-off-by: Jeff King \n---\n parallel-checkout.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 94b44d2a48..d077618719 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -233,7 +233,7 @@ static int reset_fd(int fd, const char *path)\n }\n \n static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n-\t\t\t const char *path, struct checkout *state)\n+\t\t\t const char *path)\n {\n \tint ret;\n \tstruct stream_filter *filter;\n@@ -347,7 +347,7 @@ void write_pc_item(struct parallel_checkout_item *pc_item,\n \t\tgoto out;\n \t}\n \n-\tif (write_pc_item_to_fd(pc_item, fd, path.buf, state)) {\n+\tif (write_pc_item_to_fd(pc_item, fd, path.buf)) {\n \t\t/* Error was already reported. */\n \t\tpc_item->status = PC_ITEM_FAILED;\n \t\tgoto out;\n-- \n2.28.0.1295.g4824feede7\n\n"},{"id":"406960","messageId":"CAHd-oW5=ipx9RY_W44GFUSO=O1eDwWirfrjJfpv-xy8yqe6L4w@mail.gmail.com","threadId":"54015","inReplyTo":"20201005061732.GA2288306@coredump.intra.peff.net","subject":"Re: [PATCH] parallel-checkout: drop unused checkout state parameter","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-10-05T13:13:21Z","receivedAt":"2020-10-05T13:13:37Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Mon, Oct 5, 2020 at 3:17 AM Jeff King wrote:\n>\n> On Tue, Sep 22, 2020 at 07:49:24PM -0300, Matheus Tavares wrote:\n>\n> > +static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n> > + const char *path, struct checkout *state)\n>\n> The \"state\" parameter is unused here. Maybe this on top of\n> mt/parallel-checkout-part-1?\n>\n> -- >8 --\n> Subject: parallel-checkout: drop unused checkout state parameter\n>\n> The write_pc_item_to_fd() function takes a \"struct checkout *state\"\n> parameter, but never uses it. This was true in its introduction in\n> fa33dd99f0 (unpack-trees: add basic support for parallel checkout,\n> 2020-09-22). Its caller, write_pc_item(), has already pulled the useful\n> bits from the state struct into the \"path\" variable. Let's drop the\n> useless parameter.\n>\n> Signed-off-by: Jeff King \n\nGood catch, thanks.\n\nI was going to suggest squashing this into fa33dd99f0, but I noticed\nthat mt/parallel-checkout-part-1 is already in next. We don't re-roll\nseries that are already in next, right?\n"},{"id":"406963","messageId":"20201005134504.GA2910591@coredump.intra.peff.net","threadId":"54015","inReplyTo":"CAHd-oW5=ipx9RY_W44GFUSO=O1eDwWirfrjJfpv-xy8yqe6L4w@mail.gmail.com","subject":"Re: [PATCH] parallel-checkout: drop unused checkout state parameter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-05T13:45:04Z","receivedAt":"2020-10-05T13:45:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 05, 2020 at 10:13:21AM -0300, Matheus Tavares Bernardino wrote:\n\n> I was going to suggest squashing this into fa33dd99f0, but I noticed\n> that mt/parallel-checkout-part-1 is already in next. We don't re-roll\n> series that are already in next, right?\n\nCorrect. That's also why I noticed it; I build my day-to-day Git by\nintegrating next with my personal topics, and one of my topics turns on\n-Wunused-parameter. :)\n\n-Peff\n"},{"id":"407944","messageId":"20201020013558.GA15198@google.com","threadId":"54015","inReplyTo":"64b41d537e68a45f2bb0a0c3078f2cd314b5a57d.1600814153.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-20T01:35:58Z","receivedAt":"2020-10-20T01:36:04Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nMatheus Tavares wrote:\n\n> Add tests to populate the working tree during clone and checkout using\n> the sequential and parallel modes, to confirm that they produce\n> identical results. Also test basic checkout mechanics, such as checking\n> for symlinks in the leading directories and the abidance to --force.\n\nThanks for implementing parallel checkout! I'm excited about the\nfeature. And thanks for including these tests.\n\n[...]\n> --- /dev/null\n> +++ b/t/lib-parallel-checkout.sh\n> @@ -0,0 +1,39 @@\n[...]\n> +# Runs `git -c checkout.workers=$1 -c checkout.thesholdForParallelism=$2 ${@:4}`\n> +# and checks that the number of workers spawned is equal to $3.\n> +git_pc()\n\nnit: what does git_pc mean? Can this spell it out more verbosely, or\ncould callers take on more of the burden? (Perhaps it would make sense\nto use a helper that uses test_config to set the relevant configuration,\nand then the caller can use plain \"git clone\"?)\n\n[...]\n> +\tGIT_TRACE2=\"$(pwd)/trace\" git \\\n> +\t\t-c checkout.workers=$workers \\\n> +\t\t-c checkout.thresholdForParallelism=$threshold \\\n> +\t\t-c advice.detachedHead=0 \\\n> +\t\t$@ &&\n\n$@ needs to be quoted, or else it will act like $* (and in particular it\nwon't handle parameters with embedded spaces).\n\n> +\n> +\t# Check that the expected number of workers has been used. Note that it\n> +\t# can be different than the requested number in two cases: when the\n> +\t# quantity of entries to be checked out is less than the number of\n> +\t# workers; and when the threshold has not been reached.\n> +\t#\n> +\tlocal workers_in_trace=$(grep \"child_start\\[.\\+\\] git checkout--helper\" trace | wc -l) &&\n\nDo we use grep's \\+ operator in other tests? I thought we preferred to\nuse the more portable *, but it may be that I'm out of date.\n\n[...]\n> +# Verify that both the working tree and the index were created correctly\n> +verify_checkout()\n> +{\n> +\tgit -C $1 diff-index --quiet HEAD -- &&\n> +\tgit -C $1 diff-index --quiet --cached HEAD -- &&\n> +\tgit -C $1 status --porcelain >$1.status &&\n> +\ttest_must_be_empty $1.status\n> +}\n\nLike git_pc, this is not easy to take in at a glance.\n\n\"$1\" needs to be quoted if we are to handle paths with spaces.\n\n[...]\n> --- /dev/null\n> +++ b/t/t2080-parallel-checkout-basics.sh\n> @@ -0,0 +1,197 @@\n> +#!/bin/sh\n> +\n> +test_description='parallel-checkout basics\n> +\n> +Ensure that parallel-checkout basically works on clone and checkout, spawning\n> +the required number of workers and correctly populating both the index and\n> +working tree.\n> +'\n> +\n> +TEST_NO_CREATE_REPO=1\n> +. ./test-lib.sh\n> +. \"$TEST_DIRECTORY/lib-parallel-checkout.sh\"\n> +\n> +# NEEDSWORK: cloning a SHA1 repo with GIT_TEST_DEFAULT_HASH set to \"sha256\"\n> +# currently produces a wrong result (See\n> +# https://lore.kernel.org/git/20200911151717.43475-1-matheus.bernardino@usp.br/).\n> +# So we skip the \"parallel-checkout during clone\" tests when this test flag is\n> +# set to \"sha256\". Remove this when the bug is fixed.\n> +#\n> +if test \"$GIT_TEST_DEFAULT_HASH\" = \"sha256\"\n> +then\n> +\tskip_all=\"t2080 currently don't work with GIT_TEST_DEFAULT_HASH=sha256\"\n> +\ttest_done\n> +fi\n> +\n> +R_BASE=$GIT_BUILD_DIR\n> +\n> +test_expect_success 'sequential clone' '\n> +\tgit_pc 1 0 0 clone --quiet -- $R_BASE r_sequential &&\n\nThis fails when I run it when building from a tarball, which is\npresenting me from releasing this patch series to Debian experimental.\n\nCan we use an artificial repo instead of git.git? Using git.git as\ntest data seems like a recipe for hard-to-reproduce test failures.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"407953","messageId":"20201020025525.GG54484@nand.local","threadId":"54015","inReplyTo":"20201020013558.GA15198@google.com","subject":"Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-20T02:55:25Z","receivedAt":"2020-10-20T02:55:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 19, 2020 at 06:35:58PM -0700, Jonathan Nieder wrote:\n> > +\n> > +\t# Check that the expected number of workers has been used. Note that it\n> > +\t# can be different than the requested number in two cases: when the\n> > +\t# quantity of entries to be checked out is less than the number of\n> > +\t# workers; and when the threshold has not been reached.\n> > +\t#\n> > +\tlocal workers_in_trace=$(grep \"child_start\\[.\\+\\] git checkout--helper\" trace | wc -l) &&\n>\n> Do we use grep's \\+ operator in other tests? I thought we preferred to\n> use the more portable *, but it may be that I'm out of date.\n\nYou're not out-of-date; I looked at this myself a couple of months ago:\n\n https://lore.kernel.org/git/20200812140352.GC74542@syl.lan/\n\nThanks,\nTaylor\n"},{"id":"407956","messageId":"CAHd-oW7=Bd+StX_t+6iuaas0SWzEdbQCj5aWgasoOYT2kfVw7g@mail.gmail.com","threadId":"54015","inReplyTo":"20201020013558.GA15198@google.com","subject":"Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-10-20T03:18:21Z","receivedAt":"2020-10-20T03:18:39Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Jonathan\n\nOn Mon, Oct 19, 2020 at 10:36 PM Jonathan Nieder wrote:\n>\n> Hi,\n>\n> Matheus Tavares wrote:\n>\n> > Add tests to populate the working tree during clone and checkout using\n> > the sequential and parallel modes, to confirm that they produce\n> > identical results. Also test basic checkout mechanics, such as checking\n> > for symlinks in the leading directories and the abidance to --force.\n>\n> Thanks for implementing parallel checkout! I'm excited about the\n> feature. And thanks for including these tests.\n\nThanks for the comments and feedback :)\n\n> [...]\n> > --- /dev/null\n> > +++ b/t/lib-parallel-checkout.sh\n> > @@ -0,0 +1,39 @@\n> [...]\n> > +# Runs `git -c checkout.workers=$1 -c checkout.thesholdForParallelism=$2 ${@:4}`\n> > +# and checks that the number of workers spawned is equal to $3.\n> > +git_pc()\n>\n> nit: what does git_pc mean?\n\nThe idea was \"git w/ parallel-checkout\". But I realize it may have\ngotten too abbreviated...\n\n> Can this spell it out more verbosely, or\n> could callers take on more of the burden? (Perhaps it would make sense\n> to use a helper that uses test_config to set the relevant configuration,\n> and then the caller can use plain \"git clone\"?)\n\nHmm, it's possible, but I think we might end up with quite a lot of\nrepetition (to always check that checkout spawned the right number of\nworkers).\n\n> [...]\n> > + GIT_TRACE2=\"$(pwd)/trace\" git \\\n> > + -c checkout.workers=$workers \\\n> > + -c checkout.thresholdForParallelism=$threshold \\\n> > + -c advice.detachedHead=0 \\\n> > + $@ &&\n>\n> $@ needs to be quoted, or else it will act like $* (and in particular it\n> won't handle parameters with embedded spaces).\n\nNice catch, thanks! I will send a patch for this tomorrow.\n\n> > +\n> > + # Check that the expected number of workers has been used. Note that it\n> > + # can be different than the requested number in two cases: when the\n> > + # quantity of entries to be checked out is less than the number of\n> > + # workers; and when the threshold has not been reached.\n> > + #\n> > + local workers_in_trace=$(grep \"child_start\\[.\\+\\] git checkout--helper\" trace | wc -l) &&\n>\n> Do we use grep's \\+ operator in other tests? I thought we preferred to\n> use the more portable *, but it may be that I'm out of date.\n\nOh, I didn't know about the portability issue with \\+. This is already\nin `next`, but I guess it's worth sending a follow-up patch to fix it,\nright? (I see we have a second \\+ occurrence in t7508, which could be\nchanged in the same patch.)\n\n> [...]\n> > +# Verify that both the working tree and the index were created correctly\n> > +verify_checkout()\n> > +{\n> > + git -C $1 diff-index --quiet HEAD -- &&\n> > + git -C $1 diff-index --quiet --cached HEAD -- &&\n> > + git -C $1 status --porcelain >$1.status &&\n> > + test_must_be_empty $1.status\n> > +}\n>\n> Like git_pc, this is not easy to take in at a glance.\n>\n> \"$1\" needs to be quoted if we are to handle paths with spaces.\n\nThanks, again :) Currently, this function doesn't get paths with\nspaces, but I agree that it's better to be cautious here.\n\n> [...]\n> > --- /dev/null\n> > +++ b/t/t2080-parallel-checkout-basics.sh\n> > @@ -0,0 +1,197 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='parallel-checkout basics\n> > +\n> > +Ensure that parallel-checkout basically works on clone and checkout, spawning\n> > +the required number of workers and correctly populating both the index and\n> > +working tree.\n> > +'\n> > +\n> > +TEST_NO_CREATE_REPO=1\n> > +. ./test-lib.sh\n> > +. \"$TEST_DIRECTORY/lib-parallel-checkout.sh\"\n> > +\n> > +# NEEDSWORK: cloning a SHA1 repo with GIT_TEST_DEFAULT_HASH set to \"sha256\"\n> > +# currently produces a wrong result (See\n> > +# https://lore.kernel.org/git/20200911151717.43475-1-matheus.bernardino@usp.br/).\n> > +# So we skip the \"parallel-checkout during clone\" tests when this test flag is\n> > +# set to \"sha256\". Remove this when the bug is fixed.\n> > +#\n> > +if test \"$GIT_TEST_DEFAULT_HASH\" = \"sha256\"\n> > +then\n> > + skip_all=\"t2080 currently don't work with GIT_TEST_DEFAULT_HASH=sha256\"\n> > + test_done\n> > +fi\n> > +\n> > +R_BASE=$GIT_BUILD_DIR\n> > +\n> > +test_expect_success 'sequential clone' '\n> > + git_pc 1 0 0 clone --quiet -- $R_BASE r_sequential &&\n>\n> This fails when I run it when building from a tarball, which is\n> presenting me from releasing this patch series to Debian experimental.\n\nSorry for the trouble :( It didn't occur to me, while writing the\ntest, that it could also be run from the tarball.\n\n> Can we use an artificial repo instead of git.git? Using git.git as\n> test data seems like a recipe for hard-to-reproduce test failures.\n\nI think we could maybe drop these tests. There are already some\nsimilar tests below these, which use an artificial repository. The\ngoal of using git.git in this section was to test parallel-checkout\nwith a real-world repo, and hopefully catch errors that we might not\nsee with small artificial ones. But you have a very valid concern, as\nwell. Hmm, I'm not sure what is the best solution to this case. What\ndo you think?\n"},{"id":"407957","messageId":"20201020041625.GB15198@google.com","threadId":"54015","inReplyTo":"CAHd-oW7=Bd+StX_t+6iuaas0SWzEdbQCj5aWgasoOYT2kfVw7g@mail.gmail.com","subject":"Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-10-20T04:16:25Z","receivedAt":"2020-10-20T04:16:31Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Matheus Tavares Bernardino wrote:\n> On Mon, Oct 19, 2020 at 10:36 PM Jonathan Nieder wrote:\n\n>> Can we use an artificial repo instead of git.git? Using git.git as\n>> test data seems like a recipe for hard-to-reproduce test failures.\n>\n> I think we could maybe drop these tests. There are already some\n> similar tests below these, which use an artificial repository. The\n> goal of using git.git in this section was to test parallel-checkout\n> with a real-world repo, and hopefully catch errors that we might not\n> see with small artificial ones. But you have a very valid concern, as\n> well. Hmm, I'm not sure what is the best solution to this case. What\n> do you think?\n\nI see. I suppose my preference would be to have a real-world example\nin t/perf/ (see t/perf/README for how it allows an arbitrary repo to\nbe passed in) instead of in the regression tests. In the regression\ntestsuite I'd focus more on particular behaviors I want to test (e.g.,\na file being replaced by a directory, that kind of thing).\n\nBehaviors exercised by git.git are in some sense the *least* important\nthing to test here, since developers in the Git project know to\nadvocate for those and exercise them day-to-day. Where the testsuite\nshines is in being able to advocate for use cases that are exercised\nby other populations --- a testsuite failure can be a reminder to not\nforget about the features other people need that are not part of our\nown daily lives.\n\nThanks,\nJonathan\n"},{"id":"407980","messageId":"CAHd-oW4A0S6Wb58HA7nRo37sU7YTEbnbsFmwLKN_Onc6ART=PA@mail.gmail.com","threadId":"54015","inReplyTo":"20201020025525.GG54484@nand.local","subject":"Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-10-20T13:18:59Z","receivedAt":"2020-10-20T13:19:16Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Mon, Oct 19, 2020 at 11:55 PM Taylor Blau wrote:\n>\n> On Mon, Oct 19, 2020 at 06:35:58PM -0700, Jonathan Nieder wrote:\n> > > +\n> > > + # Check that the expected number of workers has been used. Note that it\n> > > + # can be different than the requested number in two cases: when the\n> > > + # quantity of entries to be checked out is less than the number of\n> > > + # workers; and when the threshold has not been reached.\n> > > + #\n> > > + local workers_in_trace=$(grep \"child_start\\[.\\+\\] git checkout--helper\" trace | wc -l) &&\n> >\n> > Do we use grep's \\+ operator in other tests? I thought we preferred to\n> > use the more portable *, but it may be that I'm out of date.\n>\n> You're not out-of-date; I looked at this myself a couple of months ago:\n>\n> https://lore.kernel.org/git/20200812140352.GC74542@syl.lan/\n\nThanks for the pointer, I'll replace .\\+ by ..*, then.\n\nI noticed we also have some uses of + and ? in tests, with `grep -E`\n(or egrep). Are we OK with ERE or did these maybe just slip in by\naccident?\n"},{"id":"407996","messageId":"xmqq4kmod8ab.fsf@gitster.c.googlers.com","threadId":"54015","inReplyTo":"CAHd-oW4A0S6Wb58HA7nRo37sU7YTEbnbsFmwLKN_Onc6ART=PA@mail.gmail.com","subject":"Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-20T19:09:00Z","receivedAt":"2020-10-20T19:09:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares Bernardino writes:\n\n> I noticed we also have some uses of + and ? in tests, with `grep -E`\n> (or egrep). Are we OK with ERE or did these maybe just slip in by\n> accident?\n\nWe are OK with 'grep -E' and 'egrep' and write '+' and '?' as valid\nERE elements. What we are not OK with is to invoke ERE elements in\nan expression that is supposed to be a BRE by prefixing a backslash,\ne.g. '\\+'. Perhaps it is a GNU extension?\n\nWe need to remove '\\+' in t/perf/bisect_regression used with sed.\nWhat is sad is that this trick and \"sed -E\" are both GNUisms, and\nthere is no portable way to use ERE with sed X-<.\n\nBut we could resort to Perl in truly tricky cases ;-).\n\n\n\n"},{"id":"407998","messageId":"xmqqy2k0btfw.fsf@gitster.c.googlers.com","threadId":"54015","inReplyTo":"CAHd-oW7=Bd+StX_t+6iuaas0SWzEdbQCj5aWgasoOYT2kfVw7g@mail.gmail.com","subject":"Re: [PATCH v2 16/19] parallel-checkout: add tests for basic operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-20T19:14:59Z","receivedAt":"2020-10-20T19:15:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares Bernardino writes:\n\n> Oh, I didn't know about the portability issue with \\+. This is already\n> in `next`, but I guess it's worth sending a follow-up patch to fix it,\n> right? (I see we have a second \\+ occurrence in t7508, which could be\n> changed in the same patch.)\n\nNote that soon, typically a week, after a release, the tip of the\nnext branch is rewound and all the topics that did not graduate to\nmaster has a chance to get a clean start. This may be a good use\ncase for that chance.\n"}]}