{"thread":{"id":"55342","subject":"[PATCH 1/5] unpack-trees: add basic support for parallel checkout","startedAt":"2021-03-17T21:13:09Z","lastAt":"2021-04-19T19:53:56Z","messageCount":40,"participants":["Matheus Tavares","Junio C Hamano","Christian Couder","Matheus Tavares Bernardino"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"419573","messageId":"a3383042e2ee6c454e024db7a9154dc979a5af9e.1616015337.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1616015337.git.matheus.bernardino@usp.br","subject":"[PATCH 1/5] unpack-trees: add basic support for parallel checkout","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-03-17T21:12:19Z","receivedAt":"2021-03-17T21:13:09Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This new interface allows us to enqueue some of the entries being\nchecked out to later uncompress, smudge, and write them in parallel. For\nnow, the parallel checkout machinery is enabled by default and there is\nno user configuration, but run_parallel_checkout() just writes the\nqueued entries in sequence (without spawning additional workers). The\nnext patch will actually implement the parallelism and, later, we will\nmake it configurable.\n\nNote that, to avoid potential data races, not all entries are eligible\nfor parallel checkout. Also, paths that collide on disk (e.g.\ncase-sensitive paths in case-insensitive file systems), are detected by\nthe parallel checkout code and skipped, so that they can be safely\nsequentially handled later. The collision detection works like the\nfollowing:\n\n- If the collision was at basename (e.g. 'a/b' and 'a/B'), the framework\n detects it by looking for EEXIST and EISDIR errors after an\n open(O_CREAT | O_EXCL) failure.\n\n- If the collision was at dirname (e.g. 'a/b' and 'A'), it is detected\n at the has_dirs_only_path() check, which is done for the leading path\n of each item in the parallel checkout queue.\n\nBoth verifications rely on the fact that, before enqueueing an entry for\nparallel checkout, checkout_entry() makes sure that there is no file at\nthe entry's path and that its leading components are all real\ndirectories. So, any later change in these conditions indicates that\nthere was a collision (either between two parallel-eligible entries or\nbetween an eligible and an ineligible one).\n\nAfter all parallel-eligible entries have been processed, the collided\n(and thus, skipped) entries are sequentially fed to checkout_entry()\nagain. This is similar to the way the current code deals with\ncollisions, overwriting the previously checked out entries with the\nsubsequent ones. The only difference is that, since we no longer create\nthe files in the same order that they appear on index, we are not able\nto determine which of the colliding entries will survive on disk (for\nthe classic code, it is always the last entry).\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Makefile | 1 +\n entry.c | 17 +-\n parallel-checkout.c | 368 ++++++++++++++++++++++++++++++++++++++++++++\n parallel-checkout.h | 32 ++++\n unpack-trees.c | 6 +-\n 5 files changed, 421 insertions(+), 3 deletions(-)\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\ndiff --git a/Makefile b/Makefile\nindex dfb0f1000f..f7b9ab49f9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -938,6 +938,7 @@ LIB_OBJS += pack-revindex.o\n LIB_OBJS += pack-write.o\n LIB_OBJS += packfile.o\n LIB_OBJS += pager.o\n+LIB_OBJS += parallel-checkout.o\n LIB_OBJS += parse-options-cb.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\ndiff --git a/entry.c b/entry.c\nindex 2ce16414a7..6a22c45050 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -7,6 +7,7 @@\n #include \"progress.h\"\n #include \"fsmonitor.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -426,8 +427,17 @@ static void mark_colliding_entries(const struct checkout *state,\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-\t\tif (dup == ce)\n-\t\t\tbreak;\n+\t\tif (dup == ce) {\n+\t\t\t/*\n+\t\t\t * Parallel checkout doesn't create the files in index\n+\t\t\t * order. So the other side of the collision may appear\n+\t\t\t * after the given cache_entry in the array.\n+\t\t\t */\n+\t\t\tif (parallel_checkout_status() == PC_RUNNING)\n+\t\t\t\tcontinue;\n+\t\t\telse\n+\t\t\t\tbreak;\n+\t\t}\n \n \t\tif (dup->ce_flags & (CE_MATCHED | CE_VALID | CE_SKIP_WORKTREE))\n \t\t\tcontinue;\n@@ -536,6 +546,9 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\tca = &ca_buf;\n \t}\n \n+\tif (!enqueue_checkout(ce, ca))\n+\t\treturn 0;\n+\n \treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nnew file mode 100644\nindex 0000000000..80a60eb2d3\n--- /dev/null\n+++ b/parallel-checkout.c\n@@ -0,0 +1,368 @@\n+#include \"cache.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"streaming.h\"\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/* pointer to a istate->cache[] entry. Not owned by us. */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tstruct stat st;\n+\tenum pc_item_status status;\n+};\n+\n+struct parallel_checkout {\n+\tenum pc_status status;\n+\tstruct parallel_checkout_item *items;\n+\tsize_t nr, alloc;\n+};\n+\n+static struct parallel_checkout parallel_checkout;\n+\n+enum pc_status parallel_checkout_status(void)\n+{\n+\treturn parallel_checkout.status;\n+}\n+\n+void init_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status != PC_UNINITIALIZED)\n+\t\tBUG(\"parallel checkout already initialized\");\n+\n+\tparallel_checkout.status = PC_ACCEPTING_ENTRIES;\n+}\n+\n+static void finish_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status == PC_UNINITIALIZED)\n+\t\tBUG(\"cannot finish parallel checkout: not initialized yet\");\n+\n+\tfree(parallel_checkout.items);\n+\tmemset(¶llel_checkout, 0, sizeof(parallel_checkout));\n+}\n+\n+static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n+\t\t\t\t\t const struct conv_attrs *ca)\n+{\n+\tenum conv_attrs_classification c;\n+\n+\t/*\n+\t * Symlinks cannot be checked out in parallel as, in case of path\n+\t * collision, they could racily replace leading directories of other\n+\t * entries being checked out. Submodules are checked out in child\n+\t * processes, which have their own parallel checkout queues.\n+\t */\n+\tif (!S_ISREG(ce->ce_mode))\n+\t\treturn 0;\n+\n+\tc = classify_conv_attrs(ca);\n+\tswitch (c) {\n+\tcase CA_CLASS_INCORE:\n+\t\treturn 1;\n+\n+\tcase CA_CLASS_INCORE_FILTER:\n+\t\t/*\n+\t\t * It would be safe to allow concurrent instances of\n+\t\t * single-file smudge filters, like rot13, but we should not\n+\t\t * assume that all filters are parallel-process safe. So we\n+\t\t * don't allow this.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_INCORE_PROCESS:\n+\t\t/*\n+\t\t * The parallel queue and the delayed queue are not compatible,\n+\t\t * so they must be kept completely separated. And we can't tell\n+\t\t * if a long-running process will delay its response without\n+\t\t * actually asking it to perform the filtering. Therefore, this\n+\t\t * type of filter is not allowed in parallel checkout.\n+\t\t *\n+\t\t * Furthermore, there should only be one instance of the\n+\t\t * long-running process filter as we don't know how it is\n+\t\t * managing its own concurrency. So, spreading the entries that\n+\t\t * requisite such a filter among the parallel workers would\n+\t\t * require a lot more inter-process communication. We would\n+\t\t * probably have to designate a single process to interact with\n+\t\t * the filter and send all the necessary data to it, for each\n+\t\t * entry.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_STREAMABLE:\n+\t\treturn 1;\n+\n+\tdefault:\n+\t\tBUG(\"unsupported conv_attrs classification '%d'\", c);\n+\t}\n+}\n+\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n+{\n+\tstruct parallel_checkout_item *pc_item;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES ||\n+\t !is_eligible_for_parallel_checkout(ce, ca))\n+\t\treturn -1;\n+\n+\tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n+\t\t parallel_checkout.alloc);\n+\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item->ce = ce;\n+\tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n+\tpc_item->status = PC_ITEM_PENDING;\n+\n+\treturn 0;\n+}\n+\n+static int handle_results(struct checkout *state)\n+{\n+\tint ret = 0;\n+\tsize_t i;\n+\tint have_pending = 0;\n+\n+\t/*\n+\t * We first update the successfully written entries with the collected\n+\t * stat() data, so that they can be found by mark_colliding_entries(),\n+\t * in the next loop, when necessary.\n+\t */\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\tif (pc_item->status == PC_ITEM_WRITTEN)\n+\t\t\tupdate_ce_after_write(state, pc_item->ce, &pc_item->st);\n+\t}\n+\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\n+\t\tswitch(pc_item->status) {\n+\t\tcase PC_ITEM_WRITTEN:\n+\t\t\t/* Already handled */\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_COLLIDED:\n+\t\t\t/*\n+\t\t\t * The entry could not be checked out due to a path\n+\t\t\t * collision with another entry. Since there can only\n+\t\t\t * be one entry of each colliding group on the disk, we\n+\t\t\t * could skip trying to check out this one and move on.\n+\t\t\t * However, this would leave the unwritten entries with\n+\t\t\t * null stat() fields on the index, which could\n+\t\t\t * potentially slow down subsequent operations that\n+\t\t\t * require refreshing it: git would not be able to\n+\t\t\t * trust st_size and would have to go to the filesystem\n+\t\t\t * to see if the contents match (see ie_modified()).\n+\t\t\t *\n+\t\t\t * Instead, let's pay the overhead only once, now, and\n+\t\t\t * call checkout_entry_ca() again for this file, to\n+\t\t\t * have it's stat() data stored in the index. This also\n+\t\t\t * has the benefit of adding this entry and its\n+\t\t\t * colliding pair to the collision report message.\n+\t\t\t * Additionally, this overwriting behavior is consistent\n+\t\t\t * with what the sequential checkout does, so it doesn't\n+\t\t\t * add any extra overhead.\n+\t\t\t */\n+\t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n+\t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_PENDING:\n+\t\t\thave_pending = 1;\n+\t\t\t/* fall through */\n+\t\tcase PC_ITEM_FAILED:\n+\t\t\tret = -1;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unknown checkout item status in parallel checkout\");\n+\t\t}\n+\t}\n+\n+\tif (have_pending)\n+\t\terror(_(\"parallel checkout finished with pending entries\"));\n+\n+\treturn ret;\n+}\n+\n+static int reset_fd(int fd, const char *path)\n+{\n+\tif (lseek(fd, 0, SEEK_SET) != 0)\n+\t\treturn error_errno(\"failed to rewind descriptor of %s\", path);\n+\tif (ftruncate(fd, 0))\n+\t\treturn error_errno(\"failed to truncate file %s\", path);\n+\treturn 0;\n+}\n+\n+static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n+\t\t\t const char *path)\n+{\n+\tint ret;\n+\tstruct stream_filter *filter;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *new_blob;\n+\tunsigned long size;\n+\tsize_t newsize = 0;\n+\tssize_t wrote;\n+\n+\t/* Sanity check */\n+\tassert(is_eligible_for_parallel_checkout(pc_item->ce, &pc_item->ca));\n+\n+\tfilter = get_stream_filter_ca(&pc_item->ca, &pc_item->ce->oid);\n+\tif (filter) {\n+\t\tif (stream_blob_to_fd(fd, &pc_item->ce->oid, filter, 1)) {\n+\t\t\t/* On error, reset fd to try writing without streaming */\n+\t\t\tif (reset_fd(fd, path))\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tnew_blob = read_blob_entry(pc_item->ce, &size);\n+\tif (!new_blob)\n+\t\treturn error(\"unable to read sha1 file of %s (%s)\", path,\n+\t\t\t oid_to_hex(&pc_item->ce->oid));\n+\n+\t/*\n+\t * checkout metadata is used to give context for external process\n+\t * filters. Files requiring such filters are not eligible for parallel\n+\t * checkout, so pass NULL.\n+\t */\n+\tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n+\t\t\t\t\t new_blob, size, &buf, NULL);\n+\n+\tif (ret) {\n+\t\tfree(new_blob);\n+\t\tnew_blob = strbuf_detach(&buf, &newsize);\n+\t\tsize = newsize;\n+\t}\n+\n+\twrote = write_in_full(fd, new_blob, size);\n+\tfree(new_blob);\n+\tif (wrote < 0)\n+\t\treturn error(\"unable to write file %s\", path);\n+\n+\treturn 0;\n+}\n+\n+static int close_and_clear(int *fd)\n+{\n+\tint ret = 0;\n+\n+\tif (*fd >= 0) {\n+\t\tret = close(*fd);\n+\t\t*fd = -1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t\t struct checkout *state)\n+{\n+\tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n+\tint fd = -1, fstat_done = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tconst char *dir_sep;\n+\n+\tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n+\tstrbuf_add(&path, pc_item->ce->name, pc_item->ce->ce_namelen);\n+\n+\tdir_sep = find_last_dir_sep(path.buf);\n+\n+\t/*\n+\t * The leading dirs should have been already created by now. But, in\n+\t * case of path collisions, one of the dirs could have been replaced by\n+\t * a symlink (checked out after we enqueued this entry for parallel\n+\t * checkout). Thus, we must check the leading dirs again.\n+\t */\n+\tif (dir_sep && !has_dirs_only_path(path.buf, dir_sep - path.buf,\n+\t\t\t\t\t state->base_dir_len)) {\n+\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\tgoto out;\n+\t}\n+\n+\tfd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, mode);\n+\n+\tif (fd < 0) {\n+\t\tif (errno == EEXIST || errno == EISDIR) {\n+\t\t\t/*\n+\t\t\t * Errors which probably represent a path collision.\n+\t\t\t * Suppress the error message and mark the item to be\n+\t\t\t * retried later, sequentially. ENOTDIR and ENOENT are\n+\t\t\t * also interesting, but the above has_dirs_only_path()\n+\t\t\t * call should have already caught these cases.\n+\t\t\t */\n+\t\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\t} else {\n+\t\t\terror_errno(\"failed to open file %s\", path.buf);\n+\t\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\t}\n+\t\tgoto out;\n+\t}\n+\n+\tif (write_pc_item_to_fd(pc_item, fd, path.buf)) {\n+\t\t/* Error was already reported. */\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tfstat_done = fstat_checkout_output(fd, state, &pc_item->st);\n+\n+\tif (close_and_clear(&fd)) {\n+\t\terror_errno(\"unable to close file %s\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tif (state->refresh_cache && !fstat_done && lstat(path.buf, &pc_item->st) < 0) {\n+\t\terror_errno(\"unable to stat just-written file %s\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tpc_item->status = PC_ITEM_WRITTEN;\n+\n+out:\n+\t/*\n+\t * No need to check close() return at this point. Either fd is already\n+\t * closed, or we are on an error path.\n+\t */\n+\tclose_and_clear(&fd);\n+\tstrbuf_release(&path);\n+}\n+\n+static void write_items_sequentially(struct checkout *state)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < parallel_checkout.nr; i++)\n+\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+}\n+\n+int run_parallel_checkout(struct checkout *state)\n+{\n+\tint ret;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n+\t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n+\n+\tparallel_checkout.status = PC_RUNNING;\n+\n+\twrite_items_sequentially(state);\n+\tret = handle_results(state);\n+\n+\tfinish_parallel_checkout();\n+\treturn ret;\n+}\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nnew file mode 100644\nindex 0000000000..4ad2a519b3\n--- /dev/null\n+++ b/parallel-checkout.h\n@@ -0,0 +1,32 @@\n+#ifndef PARALLEL_CHECKOUT_H\n+#define PARALLEL_CHECKOUT_H\n+\n+struct cache_entry;\n+struct checkout;\n+struct conv_attrs;\n+\n+enum pc_status {\n+\tPC_UNINITIALIZED = 0,\n+\tPC_ACCEPTING_ENTRIES,\n+\tPC_RUNNING,\n+};\n+\n+enum pc_status parallel_checkout_status(void);\n+\n+/*\n+ * Put parallel checkout into the PC_ACCEPTING_ENTRIES state. Should be used\n+ * only when in the PC_UNINITIALIZED state.\n+ */\n+void init_parallel_checkout(void);\n+\n+/*\n+ * Return -1 if parallel checkout is currently not accepting entries or if the\n+ * entry is not eligible for parallel checkout. Otherwise, enqueue the entry\n+ * for later write and return 0.\n+ */\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+\n+/* Write all the queued entries, returning 0 on success.*/\n+int run_parallel_checkout(struct checkout *state);\n+\n+#endif /* PARALLEL_CHECKOUT_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 5b3dd38f8c..b9548de96a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -17,6 +17,7 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -441,7 +442,6 @@ static int check_updates(struct unpack_trees_options *o,\n \tif (should_update_submodules())\n \t\tload_gitmodules_file(index, &state);\n \n-\tenable_delayed_checkout(&state);\n \tif (has_promisor_remote()) {\n \t\t/*\n \t\t * Prefetch the objects that are to be checked out in the loop\n@@ -464,6 +464,9 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t\t\t\t to_fetch.oid, to_fetch.nr);\n \t\toid_array_clear(&to_fetch);\n \t}\n+\n+\tenable_delayed_checkout(&state);\n+\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -477,6 +480,7 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n+\terrs |= run_parallel_checkout(&state);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"419574","messageId":"cover.1616015337.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":null,"subject":"[PATCH 0/5] Parallel Checkout (part 2)","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-03-17T21:12:18Z","receivedAt":"2021-03-17T21:13:09Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This is the next step in the parallel checkout implementation. An\noverview of the complete series can be seen at [1]. \n\nThe last patch in this series adds a design doc, so it may help to\nreview it first. Also, there is no need to have any familiarity with\npart-1, as this part doesn't have any semantic dependency with that.\n\nThis series is based on the merge of 'mt/parallel-checkout-part-1' and\n'master', so that it can use the \"brew cast\" fix and the latest security\nfix (both from master), to run the tests. (The merge is textually\nclean, but it needs a small semantic fix: the '#include \"entry.h\"'\naddition in builtin/stash.c).\n\nParallel-checkout-specific tests will be added in part-3.\n\n[1]: https://lore.kernel.org/git/cover.1604521275.git.matheus.bernardino@usp.br/\n\nMatheus Tavares (5):\n unpack-trees: add basic support for parallel checkout\n parallel-checkout: make it truly parallel\n parallel-checkout: add configuration options\n parallel-checkout: support progress displaying\n parallel-checkout: add design documentation\n\n .gitignore | 1 +\n Documentation/Makefile | 1 +\n Documentation/config/checkout.txt | 21 +\n Documentation/technical/parallel-checkout.txt | 262 ++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/checkout--helper.c | 142 ++++\n entry.c | 17 +-\n git.c | 2 +\n parallel-checkout.c | 624 ++++++++++++++++++\n parallel-checkout.h | 111 ++++\n unpack-trees.c | 19 +-\n 12 files changed, 1198 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/parallel-checkout.txt\n create mode 100644 builtin/checkout--helper.c\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\n-- \n2.30.1\n\n"},{"id":"419575","messageId":"8c83e92445b4131e9b8f8e2aa29b00717b257d13.1616015337.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1616015337.git.matheus.bernardino@usp.br","subject":"[PATCH 3/5] parallel-checkout: add configuration options","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-03-17T21:12:21Z","receivedAt":"2021-03-17T21:13:43Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make parallel checkout configurable by introducing two new settings:\ncheckout.workers and checkout.thresholdForParallelism. The first defines\nthe number of workers (where one means sequential checkout), and the\nsecond defines the minimum number of entries to attempt parallel\ncheckout.\n\nTo decide the default value for checkout.workers, the parallel version\nwas benchmarked during three operations in the linux repo, with cold\ncache: cloning v5.8, checking out v5.8 from v2.6.15 (checkout I) and\nchecking out v5.8 from v5.7 (checkout II). The four tables below show\nthe mean run times and standard deviations for 5 runs in: a local file\nsystem on SSD, a local file system on HDD, a Linux NFS server, and\nAmazon EFS (all on Linux). Each parallel checkout test was executed with\nthe number of workers that brings the best overall results in that\nenvironment.\n\nLocal SSD:\n Sequential 10 workers Speedup\nClone 8.805 s ± 0.043 s 3.564 s ± 0.041 s 2.47 ± 0.03\nCheckout I 9.678 s ± 0.057 s 4.486 s ± 0.050 s 2.16 ± 0.03\nCheckout II 5.034 s ± 0.072 s 3.021 s ± 0.038 s 1.67 ± 0.03\n\nLocal HDD:\n Sequential 10 workers Speedup\nClone 32.288 s ± 0.580 s 30.724 s ± 0.522 s 1.05 ± 0.03\nCheckout I 54.172 s ± 7.119 s 54.429 s ± 6.738 s 1.00 ± 0.18\nCheckout II 40.465 s ± 2.402 s 38.682 s ± 1.365 s 1.05 ± 0.07\n\nLinux NFS server (v4.1, on EBS, single availability zone):\n\n Sequential 32 workers Speedup\nClone 240.368 s ± 6.347 s 57.349 s ± 0.870 s 4.19 ± 0.13\nCheckout I 242.862 s ± 2.215 s 58.700 s ± 0.904 s 4.14 ± 0.07\nCheckout II 65.751 s ± 1.577 s 23.820 s ± 0.407 s 2.76 ± 0.08\n\nEFS (v4.1, replicated over multiple availability zones):\n\n Sequential 32 workers Speedup\nClone 922.321 s ± 2.274 s 210.453 s ± 3.412 s 4.38 ± 0.07\nCheckout I 1011.300 s ± 7.346 s 297.828 s ± 0.964 s 3.40 ± 0.03\nCheckout II 294.104 s ± 1.836 s 126.017 s ± 1.190 s 2.33 ± 0.03\n\nThe above benchmarks show that parallel checkout is most effective on\nrepositories located on an SSD or over a distributed file system. For\nlocal file systems on spinning disks, and/or older machines, the\nparallelism does not always bring a good performance. For this reason,\nthe default value for checkout.workers is one, a.k.a. sequential\ncheckout.\n\nTo decide the default value for checkout.thresholdForParallelism,\nanother benchmark was executed in the \"Local SSD\" setup, where parallel\ncheckout showed to be beneficial. This time, we compared the runtime of\na `git checkout -f`, with and without parallelism, after randomly\nremoving an increasing number of files from the Linux working tree. The\n\"sequential fallback\" column bellow corresponds to the executions where\ncheckout.workers was 10 but checkout.thresholdForParallelism was equal\nto the number of to-be-updated files plus one (so that we end up writing\nsequentially). Each test case was sampled 15 times, and each sample had\na randomly different set of files removed. Here are the results:\n\n sequential fallback 10 workers speedup\n10 files 772.3 ms ± 12.6 ms 769.0 ms ± 13.6 ms 1.00 ± 0.02\n20 files 780.5 ms ± 15.8 ms 775.2 ms ± 9.2 ms 1.01 ± 0.02\n50 files 806.2 ms ± 13.8 ms 767.4 ms ± 8.5 ms 1.05 ± 0.02\n100 files 833.7 ms ± 21.4 ms 750.5 ms ± 16.8 ms 1.11 ± 0.04\n200 files 897.6 ms ± 30.9 ms 730.5 ms ± 14.7 ms 1.23 ± 0.05\n500 files 1035.4 ms ± 48.0 ms 677.1 ms ± 22.3 ms 1.53 ± 0.09\n1000 files 1244.6 ms ± 35.6 ms 654.0 ms ± 38.3 ms 1.90 ± 0.12\n2000 files 1488.8 ms ± 53.4 ms 658.8 ms ± 23.8 ms 2.26 ± 0.12\n\nFrom the above numbers, 100 files seems to be a reasonable default value\nfor the threshold setting.\n\nNote: Up to 1000 files, we observe a drop in the execution time of the\nparallel code with an increase in the number of files. This is a rather\nodd behavior, but it was observed in multiple repetitions. Above 1000\nfiles, the execution time increases according to the number of files, as\none would expect.\n\nAbout the test environments: Local SSD tests were executed on an\ni7-7700HQ (4 cores with hyper-threading) running Manjaro Linux. Local\nHDD tests were executed on an Intel(R) Xeon(R) E3-1230 (also 4 cores\nwith hyper-threading), HDD Seagate Barracuda 7200.14 SATA 3.1, running\nDebian. NFS and EFS tests were executed on an Amazon EC2 c5n.xlarge\ninstance, with 4 vCPUs. The Linux NFS server was running on a m6g.large\ninstance with 2 vCPUSs and a 1 TB EBS GP2 volume. Before each timing,\nthe linux repository was removed (or checked out back to its previous\nstate), and `sync && sysctl vm.drop_caches=3` was executed.\n\nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Documentation/config/checkout.txt | 21 +++++++++++++++++++++\n parallel-checkout.c | 23 ++++++++++++++++++-----\n parallel-checkout.h | 9 +++++++--\n unpack-trees.c | 10 +++++++---\n 4 files changed, 53 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt\nindex 2cddf7b4b4..bfbca90f0e 100644\n--- a/Documentation/config/checkout.txt\n+++ b/Documentation/config/checkout.txt\n@@ -21,3 +21,24 @@ checkout.guess::\n \tProvides the default value for the `--guess` or `--no-guess`\n \toption in `git checkout` and `git switch`. See\n \tlinkgit:git-switch[1] and linkgit:git-checkout[1].\n+\n+checkout.workers::\n+\tThe number of parallel workers to use when updating the working tree.\n+\tThe default is one, i.e. sequential execution. If set to a value less\n+\tthan one, Git will use as many workers as the number of logical cores\n+\tavailable. This setting and `checkout.thresholdForParallelism` affect\n+\tall commands that perform checkout. E.g. checkout, clone, reset,\n+\tsparse-checkout, etc.\n++\n+Note: parallel checkout usually delivers better performance for repositories\n+located on SSDs or over NFS. For repositories on spinning disks and/or machines\n+with a small number of cores, the default sequential checkout often performs\n+better. The size and compression level of a repository might also influence how\n+well the parallel version performs.\n+\n+checkout.thresholdForParallelism::\n+\tWhen running parallel checkout with a small number of files, the cost\n+\tof subprocess spawning and inter-process communication might outweigh\n+\tthe parallelization gains. This setting allows to define the minimum\n+\tnumber of files for which parallel checkout should be attempted. The\n+\tdefault is 100.\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex df447aa3a6..92f3872653 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,9 +1,11 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n #include \"streaming.h\"\n+#include \"thread-utils.h\"\n \n struct pc_worker {\n \tstruct child_process cp;\n@@ -23,6 +25,19 @@ enum pc_status parallel_checkout_status(void)\n \treturn parallel_checkout.status;\n }\n \n+#define DEFAULT_THRESHOLD_FOR_PARALLELISM 100\n+\n+void get_parallel_checkout_configs(int *num_workers, int *threshold)\n+{\n+\tif (git_config_get_int(\"checkout.workers\", num_workers))\n+\t\t*num_workers = 1;\n+\telse if (*num_workers < 1)\n+\t\t*num_workers = online_cpus();\n+\n+\tif (git_config_get_int(\"checkout.thresholdForParallelism\", threshold))\n+\t\t*threshold = DEFAULT_THRESHOLD_FOR_PARALLELISM;\n+}\n+\n void init_parallel_checkout(void)\n {\n \tif (parallel_checkout.status != PC_UNINITIALIZED)\n@@ -554,11 +569,9 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n-#define DEFAULT_NUM_WORKERS 2\n-\n-int run_parallel_checkout(struct checkout *state)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n {\n-\tint ret, num_workers = DEFAULT_NUM_WORKERS;\n+\tint ret;\n \n \tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n@@ -568,7 +581,7 @@ int run_parallel_checkout(struct checkout *state)\n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\n \n-\tif (num_workers <= 1) {\n+\tif (num_workers <= 1 || parallel_checkout.nr < threshold) {\n \t\twrite_items_sequentially(state);\n \t} else {\n \t\tstruct pc_worker *workers = setup_workers(state, num_workers);\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 35e5e69a96..26f61ed2ac 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -17,6 +17,7 @@ enum pc_status {\n };\n \n enum pc_status parallel_checkout_status(void);\n+void get_parallel_checkout_configs(int *num_workers, int *threshold);\n \n /*\n * Put parallel checkout into the PC_ACCEPTING_ENTRIES state. Should be used\n@@ -31,8 +32,12 @@ void init_parallel_checkout(void);\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n \n-/* Write all the queued entries, returning 0 on success.*/\n-int run_parallel_checkout(struct checkout *state);\n+/*\n+ * Write all the queued entries, returning 0 on success. If the number of\n+ * entries is smaller than the specified threshold, the operation is performed\n+ * sequentially.\n+ */\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n \n /****************************************************************\n * Interface with checkout--helper\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex b9548de96a..8bc5061487 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -399,7 +399,7 @@ static int check_updates(struct unpack_trees_options *o,\n \tint errs = 0;\n \tstruct progress *progress;\n \tstruct checkout state = CHECKOUT_INIT;\n-\tint i;\n+\tint i, pc_workers, pc_threshold;\n \n \ttrace_performance_enter();\n \tstate.force = 1;\n@@ -465,8 +465,11 @@ static int check_updates(struct unpack_trees_options *o,\n \t\toid_array_clear(&to_fetch);\n \t}\n \n+\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n \tenable_delayed_checkout(&state);\n-\tinit_parallel_checkout();\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -480,7 +483,8 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n-\terrs |= run_parallel_checkout(&state);\n+\tif (pc_workers > 1)\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"419576","messageId":"0592740ec14c4ab72b8d46a7fecf1c66e7a497fd.1616015337.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1616015337.git.matheus.bernardino@usp.br","subject":"[PATCH 5/5] parallel-checkout: add design documentation","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-03-17T21:12:23Z","receivedAt":"2021-03-17T21:13:43Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Co-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Documentation/Makefile | 1 +\n Documentation/technical/parallel-checkout.txt | 262 ++++++++++++++++++\n 2 files changed, 263 insertions(+)\n create mode 100644 Documentation/technical/parallel-checkout.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 81d1bf7a04..af236927c9 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -90,6 +90,7 @@ TECH_DOCS += technical/multi-pack-index\n TECH_DOCS += technical/pack-format\n TECH_DOCS += technical/pack-heuristics\n TECH_DOCS += technical/pack-protocol\n+TECH_DOCS += technical/parallel-checkout\n TECH_DOCS += technical/partial-clone\n TECH_DOCS += technical/protocol-capabilities\n TECH_DOCS += technical/protocol-common\ndiff --git a/Documentation/technical/parallel-checkout.txt b/Documentation/technical/parallel-checkout.txt\nnew file mode 100644\nindex 0000000000..c7f6570f1c\n--- /dev/null\n+++ b/Documentation/technical/parallel-checkout.txt\n@@ -0,0 +1,262 @@\n+Parallel Checkout Design Notes\n+==============================\n+\n+The \"Parallel Checkout\" feature attempts to use multiple processes to\n+parallelize the work of uncompressing, smudging, and writing blobs\n+during a checkout operation. It can be used by all checkout-related\n+commands, such as `clone`, `checkout`, `reset`, `sparse-checkout`, and\n+others.\n+\n+These commands share the following basic structure:\n+\n+* Step 1: Read the current index file into memory.\n+\n+* Step 2: Modify the in-memory index based upon the command, and\n+ temporarily mark all cache entries that need to be updated.\n+\n+* Step 3: Populate the working tree to match the new candidate index.\n+ This includes iterating over all of the to-be-updated cache entries\n+ and delete, create, or overwrite the associated files in the working\n+ tree.\n+\n+* Step 4: Write the new index to disk.\n+\n+Step 3 is the focus of the \"parallel checkout\" effort described here.\n+It dominates the execution time for most of the above command types.\n+\n+Sequential Implementation\n+-------------------------\n+\n+For the purposes of discussion here, the current sequential\n+implementation of Step 3 has 3 layers:\n+\n+* Step 3a: `unpack-trees.c:check_updates()` contains a series of\n+ sequential loops iterating over the `cache_entry`'s array. The main\n+ loop in this function calls the next layer for each of the\n+ to-be-updated entries.\n+\n+* Step 3b: `entry.c:checkout_entry()` examines the existing working tree\n+ for file conflicts, collisions, and unsaved changes. It removes files\n+ and create leading directories as necessary. It calls the next layer\n+ for each entry to be written.\n+\n+* Step 3c: `entry.c:write_entry()` loads the blob into memory, smudges\n+ it if necessary, creates the file in the working tree, writes the\n+ smudged contents, calls `fstat()` or `lstat()`, and updates the\n+ associated `cache_entry` struct with the stat information gathered.\n+\n+It wouldn't be safe to perform Step 3b in parallel, as there could be\n+race conditions between file creations and removals. Instead, the\n+parallel checkout framework lets the sequential code handle Step 3b,\n+and use parallel workers to replace the sequential\n+`entry.c:write_entry()` calls from Step 3c.\n+\n+Rejected Multi-Threaded Solution\n+--------------------------------\n+\n+The most \"straightforward\" implementation would be to spread the set of\n+to-be-updated cache entries across multiple threads. But due to the\n+thread-unsafe functions in the ODB code, we would have to use locks to\n+coordinate the parallel operation. An early prototype of this solution\n+showed that the multi-threaded checkout would bring performance\n+improvements over the sequential code, but there was still too much lock\n+contention. A `perf` profiling indicated that around 20% of the runtime\n+during a local Linux clone (on an SSD) was spent in locking functions.\n+For this reason this approach was rejected in favor of using multiple\n+child processes, which led to a better performance.\n+\n+Multi-Process Solution\n+----------------------\n+\n+Parallel checkout alters the aforementioned Step 3 to use multiple\n+`checkout--helper` background processes to distribute the work. The\n+long-running worker processes are controlled by the foreground Git\n+command using the existing run-command API.\n+\n+Overview\n+~~~~~~~~\n+\n+Step 3b is only slightly altered; for each entry to be checked out, the\n+main process:\n+\n+* M1: Checks whether there is any untracked or unclean file in the\n+ working tree which would be overwritten by this entry, and decides\n+ whether to proceed (removing the file(s)) or not.\n+\n+* M2: Creates the leading directories.\n+\n+* M3: Loads the conversion attributes for the entry's path.\n+\n+* M4: Checks, based on the entry's type and conversion attributes,\n+ whether the entry is eligible for parallel checkout (more on this\n+ later). If it is eligible, enqueues the entry and the loaded\n+ attributes to later write the entry in parallel. If not, writes the\n+ entry right away, using the default sequential code.\n+\n+Note: we save the conversion attributes associated with each entry\n+because the workers don't have access to the main process' index state,\n+so they can't load the attributes by themselves (and the attributes are\n+needed to properly smudge the entry). Additionally, this has a positive\n+impact on performance as (1) we don't need to load the attributes twice\n+and (2) the attributes machinery is optimized to handle paths in\n+sequential order.\n+\n+After all entries have passed through the above steps, the main process\n+checks if the number of enqueued entries is sufficient to spread among\n+the workers. If not, it just writes them sequentially. Otherwise, it\n+spawns the workers and distributes the queued entries uniformly in\n+continuous chunks. This aims to minimize the chances of two workers\n+writing to the same directory simultaneously, which could increase lock\n+contention in the kernel.\n+\n+Then, for each assigned item, each worker:\n+\n+* W1: Checks if there is any non-directory file in the leading part of\n+ the entry's path or if there already exists a file at the entry' path.\n+ If so, mark the entry with `PC_ITEM_COLLIDED` and skip it (more on\n+ this later).\n+\n+* W2: Creates the file (with O_CREAT and O_EXCL).\n+\n+* W3: Loads the blob into memory (inflating and delta reconstructing\n+ it).\n+\n+* W4: Filters the blob.\n+\n+* W5: Writes the result to the file descriptor opened at W2.\n+\n+* W6: Calls `fstat()` or lstat()` on the just-written path, and sends\n+ the result back to the main process, together with the end status of\n+ the operation and the item's identification number.\n+\n+Note that steps W3 to W5 might actually be performed together, using the\n+streaming interface.\n+\n+Also note that the workers *never* remove any files. As mentioned\n+earlier, it is the responsibility of the main process to remove any\n+files that block the checkout operation (or abort it). This is crucial\n+to avoid race conditions and also to properly detect path collisions at\n+Step W1.\n+\n+After the workers finish writing the items and sending back the required\n+information, the main process handles the results in two steps:\n+\n+- First, it updates the in-memory index with the `lstat()` information\n+ sent by the workers. (This must be done first as this information\n+ might me required in the following step.)\n+\n+- Then it writes the items which collided on disk (i.e. items marked\n+ with `PC_ITEM_COLLIDED`). More on this below.\n+\n+Path Collisions\n+---------------\n+\n+Path collisions happen when two different paths correspond to the same\n+entry in the file system. E.g. the paths 'a' and 'A' would collide in a\n+case-insensitive file system.\n+\n+The sequential checkout deals with collisions in the same way that it\n+deals with files that were already present in the working tree before\n+checkout. Basically, it checks if the path that it wants to write\n+already exists on disk, makes sure the existing file doesn't have\n+unsaved data, and then overwrite it. (To be more pedantic: it deletes\n+the existing file and creates the new one.) So, if there are multiple\n+colliding files to be checked out, the sequential code will write each\n+one of them but only the last will actually survive on disk.\n+\n+Parallel checkout aims to reproduce the same behavior. However, we\n+cannot let the workers racily write to the same file on disk. Instead,\n+the workers detect when the entry that they want to check out would\n+collide with an existing file, and mark it with `PC_ITEM_COLLIDED`.\n+Later, the main process can sequentially feed these entries back to\n+`checkout_entry()` without the risk of race conditions. On clone, this\n+also has the effect of marking the colliding entries to later emit a\n+warning for the user, like the classic sequential checkout does.\n+\n+The workers are able to detect both collisions among the entries being\n+concurrently written and collisions among parallel-eligible and\n+ineligible entries. The general idea for collision detection is quite\n+straightforward: for each parallel-eligible entry, the main process must\n+remove all files that prevent this entry from being written (before\n+enqueueing it). This includes any non-directory file in the leading path\n+of the entry. Later, when a worker gets assigned the entry, it looks\n+again for the non-directories files and for an already existent file at\n+the entry's path. If any of these checks finds something, the worker\n+knows that there was a path collision.\n+\n+Because parallel checkout can distinguish path collisions from the case\n+where the file was already present in the working tree before checkout,\n+we could alternatively choose to skip the checkout of colliding entries.\n+However, each entry that doesn't get written would have NULL `lstat()`\n+fields on the index. This could cause performance penalties for\n+subsequent commands that need to refresh the index, as they would have\n+to go to the file system to see if the entry is dirty. Thus, if we have\n+N entries in a colliding group and we decide to write and `lstat()` only\n+one of them, every subsequent `git-status` will have to read, convert,\n+and hash the written file N - 1 times. By checking out all colliding\n+entries (like the sequential code does), we only pay the overhead once,\n+during checkout.\n+\n+Eligible Entries for Parallel Checkout\n+--------------------------------------\n+\n+As previously mentioned, not all entries passed to `checkout_entry()`\n+will be considered eligible for parallel checkout. More specifically, we\n+exclude:\n+\n+- Symbolic links; to avoid race conditions that, in combination with\n+ path collisions, could cause workers to write files at the wrong\n+ place. For example, if we were to concurrently check out a symlink\n+ 'a' -> 'b' and a regular file 'A/f' in a case-insensitive file system,\n+ we could potentially end up writing the file 'A/f' at 'a/f', due to a\n+ race condition.\n+\n+- Regular files that require external filters (either \"one shot\" filters\n+ or long-running process filters). These filters are black-boxes to Git\n+ and may have their own internal locking or non-concurrent assumptions.\n+ So it might not be safe to run multiple instances in parallel.\n++\n+Besides, long-running filters may use the delayed checkout feature to\n+postpone the return of some filtered blobs. The delayed checkout queue\n+and the parallel checkout queue are not compatible and should remain\n+separated.\n+\n+Ineligible entries are checked out by the classic sequential codepath\n+*before* spawning workers.\n+\n+Note: submodules's files are also eligible for parallel checkout (as\n+long as they don't fall into the two excluding categories mentioned\n+above). But since each submodule is checked out in its own child\n+process, we don't mix the superproject's and the submodules' files in\n+the same parallel checkout process or queue.\n+\n+The API\n+-------\n+\n+The parallel checkout API was designed with the goal to minimize changes\n+to the current users of the checkout machinery. This means that they\n+don't have to call a different function for sequential or parallel\n+checkout. As already mentioned, `checkout_entry()` will automatically\n+insert the given entry in the parallel checkout queue when this feature\n+is enabled and the entry is eligible; otherwise, it will just write the\n+entry right away, using the sequential code. In general, callers of the\n+parallel checkout API should look similar to this:\n+\n+----------------------------------------------\n+int pc_workers, pc_threshold, err = 0;\n+struct checkout state;\n+\n+get_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n+/*\n+ * This check is not strictly required, but it\n+ * should save some time in sequential mode.\n+ */\n+if (pc_workers > 1)\n+\tinit_parallel_checkout();\n+\n+for (each cache_entry ce to-be-updated)\n+\terr |= checkout_entry(ce, &state, NULL, NULL);\n+\n+err |= run_parallel_checkout(&state, pc_workers, pc_threshold, NULL, NULL);\n+----------------------------------------------\n-- \n2.30.1\n\n"},{"id":"419577","messageId":"f7432e15ddc04a2279cc52b87283f08a5e6bc08b.1616015337.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1616015337.git.matheus.bernardino@usp.br","subject":"[PATCH 4/5] parallel-checkout: support progress displaying","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-03-17T21:12:22Z","receivedAt":"2021-03-17T21:13:43Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Original-patch-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n parallel-checkout.c | 34 +++++++++++++++++++++++++++++++---\n parallel-checkout.h | 5 ++++-\n unpack-trees.c | 11 ++++++++---\n 3 files changed, 43 insertions(+), 7 deletions(-)\n\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 92f3872653..2ba0bbb50f 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -3,6 +3,7 @@\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n+#include \"progress.h\"\n #include \"run-command.h\"\n #include \"streaming.h\"\n #include \"thread-utils.h\"\n@@ -16,6 +17,8 @@ struct parallel_checkout {\n \tenum pc_status status;\n \tstruct parallel_checkout_item *items;\n \tsize_t nr, alloc;\n+\tstruct progress *progress;\n+\tunsigned int *progress_cnt;\n };\n \n static struct parallel_checkout parallel_checkout;\n@@ -131,6 +134,20 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \treturn 0;\n }\n \n+size_t pc_queue_size(void)\n+{\n+\treturn parallel_checkout.nr;\n+}\n+\n+static void advance_progress_meter(void)\n+{\n+\tif (parallel_checkout.progress) {\n+\t\t(*parallel_checkout.progress_cnt)++;\n+\t\tdisplay_progress(parallel_checkout.progress,\n+\t\t\t\t *parallel_checkout.progress_cnt);\n+\t}\n+}\n+\n static int handle_results(struct checkout *state)\n {\n \tint ret = 0;\n@@ -179,6 +196,7 @@ static int handle_results(struct checkout *state)\n \t\t\t */\n \t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n \t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tadvance_progress_meter();\n \t\t\tbreak;\n \t\tcase PC_ITEM_PENDING:\n \t\t\thave_pending = 1;\n@@ -506,6 +524,9 @@ static void parse_and_save_result(const char *line, int len,\n \tpc_item->status = res->status;\n \tif (st)\n \t\tpc_item->st = *st;\n+\n+\tif (res->status != PC_ITEM_COLLIDED)\n+\t\tadvance_progress_meter();\n }\n \n \n@@ -565,11 +586,16 @@ static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n \n-\tfor (i = 0; i < parallel_checkout.nr; i++)\n-\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\tif (pc_item->status != PC_ITEM_COLLIDED)\n+\t\t\tadvance_progress_meter();\n+\t}\n }\n \n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt)\n {\n \tint ret;\n \n@@ -577,6 +603,8 @@ int run_parallel_checkout(struct checkout *state, int num_workers, int threshold\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n+\tparallel_checkout.progress = progress;\n+\tparallel_checkout.progress_cnt = progress_cnt;\n \n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 26f61ed2ac..4114c6bda2 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -5,6 +5,7 @@\n \n struct cache_entry;\n struct checkout;\n+struct progress;\n \n /****************************************************************\n * Users of parallel checkout\n@@ -31,13 +32,15 @@ void init_parallel_checkout(void);\n * for later write and return 0.\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+size_t pc_queue_size(void);\n \n /*\n * Write all the queued entries, returning 0 on success. If the number of\n * entries is smaller than the specified threshold, the operation is performed\n * sequentially.\n */\n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt);\n \n /****************************************************************\n * Interface with checkout--helper\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8bc5061487..0e463870fd 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -474,17 +474,22 @@ static int check_updates(struct unpack_trees_options *o,\n \t\tstruct cache_entry *ce = index->cache[i];\n \n \t\tif (ce->ce_flags & CE_UPDATE) {\n+\t\t\tsize_t last_pc_queue_size = pc_queue_size();\n+\n \t\t\tif (ce->ce_flags & CE_WT_REMOVE)\n \t\t\t\tBUG(\"both update and delete flags are set on %s\",\n \t\t\t\t ce->name);\n-\t\t\tdisplay_progress(progress, ++cnt);\n \t\t\tce->ce_flags &= ~CE_UPDATE;\n \t\t\terrs |= checkout_entry(ce, &state, NULL, NULL);\n+\n+\t\t\tif (last_pc_queue_size == pc_queue_size())\n+\t\t\t\tdisplay_progress(progress, ++cnt);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tif (pc_workers > 1)\n-\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t\t progress, &cnt);\n+\tstop_progress(&progress);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"419578","messageId":"fca797698c78e2248bd2645bd0dce23ec4df35ae.1616015337.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1616015337.git.matheus.bernardino@usp.br","subject":"[PATCH 2/5] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-03-17T21:12:20Z","receivedAt":"2021-03-17T21:13:43Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Use multiple worker processes to distribute the queued entries and call\nwrite_pc_item() in parallel for them. The items are distributed\nuniformly in contiguous chunks. This minimizes the chances of two\nworkers writing to the same directory simultaneously, which could affect\nperformance due to lock contention in the kernel. Work stealing (or any\nother format of re-distribution) is not implemented yet.\n\nThe protocol between the main process and the workers is quite simple.\nThey exchange binary messages packed in pkt-line format, and use\nPKT-FLUSH to mark the end of input (from both sides). The main process\nstarts the communication by sending N pkt-lines, each corresponding to\nan item that needs to be written. These packets contain all the\nnecessary information to load, smudge, and write the blob associated\nwith each item. Then it waits for the worker to send back N pkt-lines\ncontaining the results for each item. The resulting packet must contain:\nthe identification number of the item that it refers to, the status of\nthe operation, and the lstat() data gathered after writing the file (iff\nthe operation was successful).\n\nFor now, checkout always uses a hardcoded value of 2 workers, only to\ndemonstrate that the parallel checkout framework correctly divides and\nwrites the queued entries. The next patch will add user configurations\nand define a more reasonable default, based on tests with the said\nsettings.\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/checkout--helper.c | 142 ++++++++++++++++++++\n git.c | 2 +\n parallel-checkout.c | 267 +++++++++++++++++++++++++++++++++----\n parallel-checkout.h | 73 +++++++++-\n 7 files changed, 460 insertions(+), 27 deletions(-)\n create mode 100644 builtin/checkout--helper.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 3dcdb6bb5a..26f8ddfc55 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -33,6 +33,7 @@\n /git-check-mailmap\n /git-check-ref-format\n /git-checkout\n+/git-checkout--helper\n /git-checkout-index\n /git-cherry\n /git-cherry-pick\ndiff --git a/Makefile b/Makefile\nindex f7b9ab49f9..29df386d8a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1054,6 +1054,7 @@ BUILTIN_OBJS += builtin/check-attr.o\n BUILTIN_OBJS += builtin/check-ignore.o\n BUILTIN_OBJS += builtin/check-mailmap.o\n BUILTIN_OBJS += builtin/check-ref-format.o\n+BUILTIN_OBJS += builtin/checkout--helper.o\n BUILTIN_OBJS += builtin/checkout-index.o\n BUILTIN_OBJS += builtin/checkout.o\n BUILTIN_OBJS += builtin/clean.o\ndiff --git a/builtin.h b/builtin.h\nindex b6ce981b73..a3a3f7abb1 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -123,6 +123,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix);\n int cmd_bundle(int argc, const char **argv, const char *prefix);\n int cmd_cat_file(int argc, const char **argv, const char *prefix);\n int cmd_checkout(int argc, const char **argv, const char *prefix);\n+int cmd_checkout__helper(int argc, const char **argv, const char *prefix);\n int cmd_checkout_index(int argc, const char **argv, const char *prefix);\n int cmd_check_attr(int argc, const char **argv, const char *prefix);\n int cmd_check_ignore(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/checkout--helper.c b/builtin/checkout--helper.c\nnew file mode 100644\nindex 0000000000..df9a6fab1e\n--- /dev/null\n+++ b/builtin/checkout--helper.c\n@@ -0,0 +1,142 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"parse-options.h\"\n+#include \"pkt-line.h\"\n+\n+static void packet_to_pc_item(char *line, int len,\n+\t\t\t struct parallel_checkout_item *pc_item)\n+{\n+\tstruct pc_item_fixed_portion *fixed_portion;\n+\tchar *encoding, *variant;\n+\n+\tif (len < sizeof(struct pc_item_fixed_portion))\n+\t\tBUG(\"checkout worker received too short item (got %dB, exp %dB)\",\n+\t\t len, (int)sizeof(struct pc_item_fixed_portion));\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)line;\n+\n+\tif (len - sizeof(struct pc_item_fixed_portion) !=\n+\t\tfixed_portion->name_len + fixed_portion->working_tree_encoding_len)\n+\t\tBUG(\"checkout worker received corrupted item\");\n+\n+\tvariant = line + sizeof(struct pc_item_fixed_portion);\n+\n+\t/*\n+\t * Note: the main process uses zero length to communicate that the\n+\t * encoding is NULL. There is no use case that requires sending an\n+\t * actual empty string, since convert_attrs() never sets\n+\t * ca.working_tree_enconding to \"\".\n+\t */\n+\tif (fixed_portion->working_tree_encoding_len) {\n+\t\tencoding = xmemdupz(variant,\n+\t\t\t\t fixed_portion->working_tree_encoding_len);\n+\t\tvariant += fixed_portion->working_tree_encoding_len;\n+\t} else {\n+\t\tencoding = NULL;\n+\t}\n+\n+\tmemset(pc_item, 0, sizeof(*pc_item));\n+\tpc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n+\tpc_item->ce->ce_namelen = fixed_portion->name_len;\n+\tpc_item->ce->ce_mode = fixed_portion->ce_mode;\n+\tmemcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen);\n+\toidcpy(&pc_item->ce->oid, &fixed_portion->oid);\n+\n+\tpc_item->id = fixed_portion->id;\n+\tpc_item->ca.crlf_action = fixed_portion->crlf_action;\n+\tpc_item->ca.ident = fixed_portion->ident;\n+\tpc_item->ca.working_tree_encoding = encoding;\n+}\n+\n+static void report_result(struct parallel_checkout_item *pc_item)\n+{\n+\tstruct pc_item_result res;\n+\tsize_t size;\n+\n+\tres.id = pc_item->id;\n+\tres.status = pc_item->status;\n+\n+\tif (pc_item->status == PC_ITEM_WRITTEN) {\n+\t\tres.st = pc_item->st;\n+\t\tsize = sizeof(res);\n+\t} else {\n+\t\tsize = PC_ITEM_RESULT_BASE_SIZE;\n+\t}\n+\n+\tpacket_write(1, (const char *)&res, size);\n+}\n+\n+/* Free the worker-side malloced data, but not pc_item itself. */\n+static void release_pc_item_data(struct parallel_checkout_item *pc_item)\n+{\n+\tfree((char *)pc_item->ca.working_tree_encoding);\n+\tdiscard_cache_entry(pc_item->ce);\n+}\n+\n+static void worker_loop(struct checkout *state)\n+{\n+\tstruct parallel_checkout_item *items = NULL;\n+\tsize_t i, nr = 0, alloc = 0;\n+\n+\twhile (1) {\n+\t\tint len;\n+\t\tchar *line = packet_read_line(0, &len);\n+\n+\t\tif (!line)\n+\t\t\tbreak;\n+\n+\t\tALLOC_GROW(items, nr + 1, alloc);\n+\t\tpacket_to_pc_item(line, len, &items[nr++]);\n+\t}\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = &items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\treport_result(pc_item);\n+\t\trelease_pc_item_data(pc_item);\n+\t}\n+\n+\tpacket_flush(1);\n+\n+\tfree(items);\n+}\n+\n+static const char * const checkout_helper_usage[] = {\n+\tN_(\"git checkout--helper []\"),\n+\tNULL\n+};\n+\n+int cmd_checkout__helper(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct checkout state = CHECKOUT_INIT;\n+\tstruct option checkout_helper_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &state.base_dir, N_(\"string\"),\n+\t\t\tN_(\"when creating files, prepend \")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(checkout_helper_usage,\n+\t\t\t\t checkout_helper_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix, checkout_helper_options,\n+\t\t\t checkout_helper_usage, 0);\n+\tif (argc > 0)\n+\t\tusage_with_options(checkout_helper_usage, checkout_helper_options);\n+\n+\tif (state.base_dir)\n+\t\tstate.base_dir_len = strlen(state.base_dir);\n+\n+\t/*\n+\t * Setting this on a worker won't actually update the index. We just\n+\t * need to tell the checkout machinery to lstat() the written entries,\n+\t * so that we can send this data back to the main process.\n+\t */\n+\tstate.refresh_cache = 1;\n+\n+\tworker_loop(&state);\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 9bc077a025..a9cd4b1e9a 100644\n--- a/git.c\n+++ b/git.c\n@@ -490,6 +490,8 @@ static struct cmd_struct commands[] = {\n \t{ \"check-mailmap\", cmd_check_mailmap, RUN_SETUP },\n \t{ \"check-ref-format\", cmd_check_ref_format, NO_PARSEOPT },\n \t{ \"checkout\", cmd_checkout, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"checkout--helper\", cmd_checkout__helper,\n+\t\tRUN_SETUP | NEED_WORK_TREE | SUPPORT_SUPER_PREFIX },\n \t{ \"checkout-index\", cmd_checkout_index,\n \t\tRUN_SETUP | NEED_WORK_TREE},\n \t{ \"cherry\", cmd_cherry, RUN_SETUP },\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 80a60eb2d3..df447aa3a6 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,28 +1,13 @@\n #include \"cache.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n+#include \"pkt-line.h\"\n+#include \"run-command.h\"\n #include \"streaming.h\"\n \n-enum pc_item_status {\n-\tPC_ITEM_PENDING = 0,\n-\tPC_ITEM_WRITTEN,\n-\t/*\n-\t * The entry could not be written because there was another file\n-\t * already present in its path or leading directories. Since\n-\t * checkout_entry_ca() removes such files from the working tree before\n-\t * enqueueing the entry for parallel checkout, it means that there was\n-\t * a path collision among the entries being written.\n-\t */\n-\tPC_ITEM_COLLIDED,\n-\tPC_ITEM_FAILED,\n-};\n-\n-struct parallel_checkout_item {\n-\t/* pointer to a istate->cache[] entry. Not owned by us. */\n-\tstruct cache_entry *ce;\n-\tstruct conv_attrs ca;\n-\tstruct stat st;\n-\tenum pc_item_status status;\n+struct pc_worker {\n+\tstruct child_process cp;\n+\tsize_t next_item_to_complete, nr_items_to_complete;\n };\n \n struct parallel_checkout {\n@@ -121,10 +106,12 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n \t\t parallel_checkout.alloc);\n \n-\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr];\n \tpc_item->ce = ce;\n \tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n \tpc_item->status = PC_ITEM_PENDING;\n+\tpc_item->id = parallel_checkout.nr;\n+\tparallel_checkout.nr++;\n \n \treturn 0;\n }\n@@ -237,7 +224,8 @@ static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n \t/*\n \t * checkout metadata is used to give context for external process\n \t * filters. Files requiring such filters are not eligible for parallel\n-\t * checkout, so pass NULL.\n+\t * checkout, so pass NULL. Note: if that changes, the metadata must also\n+\t * be passed from the main process to the workers.\n \t */\n \tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n \t\t\t\t\t new_blob, size, &buf, NULL);\n@@ -268,8 +256,8 @@ static int close_and_clear(int *fd)\n \treturn ret;\n }\n \n-static void write_pc_item(struct parallel_checkout_item *pc_item,\n-\t\t\t struct checkout *state)\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state)\n {\n \tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n \tint fd = -1, fstat_done = 0;\n@@ -343,6 +331,221 @@ static void write_pc_item(struct parallel_checkout_item *pc_item,\n \tstrbuf_release(&path);\n }\n \n+static void send_one_item(int fd, struct parallel_checkout_item *pc_item)\n+{\n+\tsize_t len_data;\n+\tchar *data, *variant;\n+\tstruct pc_item_fixed_portion *fixed_portion;\n+\tconst char *working_tree_encoding = pc_item->ca.working_tree_encoding;\n+\tsize_t name_len = pc_item->ce->ce_namelen;\n+\tsize_t working_tree_encoding_len = working_tree_encoding ?\n+\t\t\t\t\t strlen(working_tree_encoding) : 0;\n+\n+\tlen_data = sizeof(struct pc_item_fixed_portion) + name_len +\n+\t\t working_tree_encoding_len;\n+\n+\tdata = xcalloc(1, len_data);\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)data;\n+\tfixed_portion->id = pc_item->id;\n+\tfixed_portion->ce_mode = pc_item->ce->ce_mode;\n+\tfixed_portion->crlf_action = pc_item->ca.crlf_action;\n+\tfixed_portion->ident = pc_item->ca.ident;\n+\tfixed_portion->name_len = name_len;\n+\tfixed_portion->working_tree_encoding_len = working_tree_encoding_len;\n+\t/*\n+\t * We use hashcpy() instead of oidcpy() because the hash[] positions\n+\t * after `the_hash_algo->rawsz` might not be initialized. And Valgrind\n+\t * would complain about passing uninitialized bytes to a syscall\n+\t * (write(2)). There is no real harm in this case, but the warning could\n+\t * hinder the detection of actual errors.\n+\t */\n+\thashcpy(fixed_portion->oid.hash, pc_item->ce->oid.hash);\n+\n+\tvariant = data + sizeof(*fixed_portion);\n+\tif (working_tree_encoding_len) {\n+\t\tmemcpy(variant, working_tree_encoding, working_tree_encoding_len);\n+\t\tvariant += working_tree_encoding_len;\n+\t}\n+\tmemcpy(variant, pc_item->ce->name, name_len);\n+\n+\tpacket_write(fd, data, len_data);\n+\n+\tfree(data);\n+}\n+\n+static void send_batch(int fd, size_t start, size_t nr)\n+{\n+\tsize_t i;\n+\tfor (i = 0; i < nr; i++)\n+\t\tsend_one_item(fd, ¶llel_checkout.items[start + i]);\n+\tpacket_flush(fd);\n+}\n+\n+static struct pc_worker *setup_workers(struct checkout *state, int num_workers)\n+{\n+\tstruct pc_worker *workers;\n+\tint i, workers_with_one_extra_item;\n+\tsize_t base_batch_size, batch_beginning = 0;\n+\n+\tALLOC_ARRAY(workers, num_workers);\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->git_cmd = 1;\n+\t\tcp->in = -1;\n+\t\tcp->out = -1;\n+\t\tcp->clean_on_exit = 1;\n+\t\tstrvec_push(&cp->args, \"checkout--helper\");\n+\t\tif (state->base_dir_len)\n+\t\t\tstrvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n+\t\tif (start_command(cp))\n+\t\t\tdie(_(\"failed to spawn checkout worker\"));\n+\t}\n+\n+\tbase_batch_size = parallel_checkout.nr / num_workers;\n+\tworkers_with_one_extra_item = parallel_checkout.nr % num_workers;\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct pc_worker *worker = &workers[i];\n+\t\tsize_t batch_size = base_batch_size;\n+\n+\t\t/* distribute the extra work evenly */\n+\t\tif (i < workers_with_one_extra_item)\n+\t\t\tbatch_size++;\n+\n+\t\tsend_batch(worker->cp.in, batch_beginning, batch_size);\n+\t\tworker->next_item_to_complete = batch_beginning;\n+\t\tworker->nr_items_to_complete = batch_size;\n+\n+\t\tbatch_beginning += batch_size;\n+\t}\n+\n+\treturn workers;\n+}\n+\n+static void finish_workers(struct pc_worker *workers, int num_workers)\n+{\n+\tint i;\n+\n+\t/*\n+\t * Close pipes before calling finish_command() to let the workers\n+\t * exit asynchronously and avoid spending extra time on wait().\n+\t */\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\t\tif (cp->in >= 0)\n+\t\t\tclose(cp->in);\n+\t\tif (cp->out >= 0)\n+\t\t\tclose(cp->out);\n+\t}\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tif (finish_command(&workers[i].cp))\n+\t\t\terror(_(\"checkout worker %d finished with error\"), i);\n+\t}\n+\n+\tfree(workers);\n+}\n+\n+#define ASSERT_PC_ITEM_RESULT_SIZE(got, exp) \\\n+{ \\\n+\tif (got != exp) \\\n+\t\tBUG(\"corrupted result from checkout worker (got %dB, exp %dB)\", \\\n+\t\t got, exp); \\\n+} while(0)\n+\n+static void parse_and_save_result(const char *line, int len,\n+\t\t\t\t struct pc_worker *worker)\n+{\n+\tstruct pc_item_result *res;\n+\tstruct parallel_checkout_item *pc_item;\n+\tstruct stat *st = NULL;\n+\n+\tif (len < PC_ITEM_RESULT_BASE_SIZE)\n+\t\tBUG(\"too short result from checkout worker (got %dB, exp %dB)\",\n+\t\t len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\n+\tres = (struct pc_item_result *)line;\n+\n+\t/*\n+\t * Worker should send either the full result struct on success, or\n+\t * just the base (i.e. no stat data), otherwise.\n+\t */\n+\tif (res->status == PC_ITEM_WRITTEN) {\n+\t\tASSERT_PC_ITEM_RESULT_SIZE(len, (int)sizeof(struct pc_item_result));\n+\t\tst = &res->st;\n+\t} else {\n+\t\tASSERT_PC_ITEM_RESULT_SIZE(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\t}\n+\n+\tif (!worker->nr_items_to_complete || res->id != worker->next_item_to_complete)\n+\t\tBUG(\"checkout worker sent unexpected item id\");\n+\n+\tworker->next_item_to_complete++;\n+\tworker->nr_items_to_complete--;\n+\n+\tpc_item = ¶llel_checkout.items[res->id];\n+\tpc_item->status = res->status;\n+\tif (st)\n+\t\tpc_item->st = *st;\n+}\n+\n+\n+static void gather_results_from_workers(struct pc_worker *workers,\n+\t\t\t\t\tint num_workers)\n+{\n+\tint i, active_workers = num_workers;\n+\tstruct pollfd *pfds;\n+\n+\tCALLOC_ARRAY(pfds, num_workers);\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tpfds[i].fd = workers[i].cp.out;\n+\t\tpfds[i].events = POLLIN;\n+\t}\n+\n+\twhile (active_workers) {\n+\t\tint nr = poll(pfds, num_workers, -1);\n+\n+\t\tif (nr < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tdie_errno(\"failed to poll checkout workers\");\n+\t\t}\n+\n+\t\tfor (i = 0; i < num_workers && nr > 0; i++) {\n+\t\t\tstruct pc_worker *worker = &workers[i];\n+\t\t\tstruct pollfd *pfd = &pfds[i];\n+\n+\t\t\tif (!pfd->revents)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (pfd->revents & POLLIN) {\n+\t\t\t\tint len;\n+\t\t\t\tconst char *line = packet_read_line(pfd->fd, &len);\n+\n+\t\t\t\tif (!line) {\n+\t\t\t\t\tpfd->fd = -1;\n+\t\t\t\t\tactive_workers--;\n+\t\t\t\t} else {\n+\t\t\t\t\tparse_and_save_result(line, len, worker);\n+\t\t\t\t}\n+\t\t\t} else if (pfd->revents & POLLHUP) {\n+\t\t\t\tpfd->fd = -1;\n+\t\t\t\tactive_workers--;\n+\t\t\t} else if (pfd->revents & (POLLNVAL | POLLERR)) {\n+\t\t\t\tdie(_(\"error polling from checkout worker\"));\n+\t\t\t}\n+\n+\t\t\tnr--;\n+\t\t}\n+\t}\n+\n+\tfree(pfds);\n+}\n+\n static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n@@ -351,16 +554,28 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n+#define DEFAULT_NUM_WORKERS 2\n+\n int run_parallel_checkout(struct checkout *state)\n {\n-\tint ret;\n+\tint ret, num_workers = DEFAULT_NUM_WORKERS;\n \n \tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n \n-\twrite_items_sequentially(state);\n+\tif (parallel_checkout.nr < num_workers)\n+\t\tnum_workers = parallel_checkout.nr;\n+\n+\tif (num_workers <= 1) {\n+\t\twrite_items_sequentially(state);\n+\t} else {\n+\t\tstruct pc_worker *workers = setup_workers(state, num_workers);\n+\t\tgather_results_from_workers(workers, num_workers);\n+\t\tfinish_workers(workers, num_workers);\n+\t}\n+\n \tret = handle_results(state);\n \n \tfinish_parallel_checkout();\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 4ad2a519b3..35e5e69a96 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -1,9 +1,14 @@\n #ifndef PARALLEL_CHECKOUT_H\n #define PARALLEL_CHECKOUT_H\n \n+#include \"convert.h\"\n+\n struct cache_entry;\n struct checkout;\n-struct conv_attrs;\n+\n+/****************************************************************\n+ * Users of parallel checkout\n+ ****************************************************************/\n \n enum pc_status {\n \tPC_UNINITIALIZED = 0,\n@@ -29,4 +34,70 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n /* Write all the queued entries, returning 0 on success.*/\n int run_parallel_checkout(struct checkout *state);\n \n+/****************************************************************\n+ * Interface with checkout--helper\n+ ****************************************************************/\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/*\n+\t * In main process ce points to a istate->cache[] entry. Thus, it's not\n+\t * owned by us. In workers they own the memory, which *must be* released.\n+\t */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tsize_t id; /* position in parallel_checkout.items[] of main process */\n+\n+\t/* Output fields, sent from workers. */\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+/*\n+ * The fixed-size portion of `struct parallel_checkout_item` that is sent to the\n+ * workers. Following this will be 2 strings: ca.working_tree_encoding and\n+ * ce.name; These are NOT null terminated, since we have the size in the fixed\n+ * portion.\n+ *\n+ * Note that not all fields of conv_attrs and cache_entry are passed, only the\n+ * ones that will be required by the workers to smudge and write the entry.\n+ */\n+struct pc_item_fixed_portion {\n+\tsize_t id;\n+\tstruct object_id oid;\n+\tunsigned int ce_mode;\n+\tenum convert_crlf_action crlf_action;\n+\tint ident;\n+\tsize_t working_tree_encoding_len;\n+\tsize_t name_len;\n+};\n+\n+/*\n+ * The fields of `struct parallel_checkout_item` that are returned by the\n+ * workers. Note: `st` must be the last one, as it is omitted on error.\n+ */\n+struct pc_item_result {\n+\tsize_t id;\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+#define PC_ITEM_RESULT_BASE_SIZE offsetof(struct pc_item_result, st)\n+\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state);\n+\n #endif /* PARALLEL_CHECKOUT_H */\n-- \n2.30.1\n\n"},{"id":"419654","messageId":"xmqqft0srxjc.fsf@gitster.g","threadId":"55342","inReplyTo":"cover.1616015337.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 0/5] Parallel Checkout (part 2)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-18T20:56:23Z","receivedAt":"2021-03-18T20:57:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares writes:\n\n> This is the next step in the parallel checkout implementation. An\n> overview of the complete series can be seen at [1]. \n>\n> The last patch in this series adds a design doc, so it may help to\n> review it first. Also, there is no need to have any familiarity with\n> part-1, as this part doesn't have any semantic dependency with that.\n>\n> This series is based on the merge of 'mt/parallel-checkout-part-1' and\n> 'master', so that it can use the \"brew cast\" fix and the latest security\n> fix (both from master), to run the tests. (The merge is textually\n> clean, but it needs a small semantic fix: the '#include \"entry.h\"'\n> addition in builtin/stash.c).\n\nLet's redo part-1 on top of 'master' first without such a merge; it\nhas been out of 'next' so we can do so easily without wanting for\nthe tip of 'next' to get rewound.\n\n\n\n>\n> Parallel-checkout-specific tests will be added in part-3.\n>\n> [1]: https://lore.kernel.org/git/cover.1604521275.git.matheus.bernardino@usp.br/\n>\n> Matheus Tavares (5):\n> unpack-trees: add basic support for parallel checkout\n> parallel-checkout: make it truly parallel\n> parallel-checkout: add configuration options\n> parallel-checkout: support progress displaying\n> parallel-checkout: add design documentation\n>\n> .gitignore | 1 +\n> Documentation/Makefile | 1 +\n> Documentation/config/checkout.txt | 21 +\n> Documentation/technical/parallel-checkout.txt | 262 ++++++++\n> Makefile | 2 +\n> builtin.h | 1 +\n> builtin/checkout--helper.c | 142 ++++\n> entry.c | 17 +-\n> git.c | 2 +\n> parallel-checkout.c | 624 ++++++++++++++++++\n> parallel-checkout.h | 111 ++++\n> unpack-trees.c | 19 +-\n> 12 files changed, 1198 insertions(+), 5 deletions(-)\n> create mode 100644 Documentation/technical/parallel-checkout.txt\n> create mode 100644 builtin/checkout--helper.c\n> create mode 100644 parallel-checkout.c\n> create mode 100644 parallel-checkout.h\n"},{"id":"419679","messageId":"19fee535ee0ec5211f22b2b69a4a9804816c2322.1616122653.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"xmqqft0srxjc.fsf@gitster.g","subject":"Re: [PATCH 0/5] Parallel Checkout (part 2)","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-03-19T03:24:38Z","receivedAt":"2021-03-19T03:25:38Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Mar 18, 2021 at 5:56 PM Junio C Hamano wrote:\n>\n> Matheus Tavares writes:\n>\n> > This is the next step in the parallel checkout implementation. An\n> > overview of the complete series can be seen at [1].\n> >\n> > The last patch in this series adds a design doc, so it may help to\n> > review it first. Also, there is no need to have any familiarity with\n> > part-1, as this part doesn't have any semantic dependency with that.\n> >\n> > This series is based on the merge of 'mt/parallel-checkout-part-1' and\n> > 'master', so that it can use the \"brew cast\" fix and the latest security\n> > fix (both from master), to run the tests. (The merge is textually\n> > clean, but it needs a small semantic fix: the '#include \"entry.h\"'\n> > addition in builtin/stash.c).\n>\n> Let's redo part-1 on top of 'master' first without such a merge; it\n> has been out of 'next' so we can do so easily without wanting for\n> the tip of 'next' to get rewound.\n\nThanks!\n\nI saw you've added the \"entry.h\" inclusion that was missing at\nbuiltin/stash when merging this branch on 'seen'. However, now that\npart-1 is based on 'master', the branch is no longer buildable without\nthis fix. So could we perhaps squash this change directly into the\nrelevant commit in this series? (I'm asking this mainly to allow me to\nlater base part-3 directly on part-2; without having to base it on a\nmerge again.)\n\nIf you agree, here is a fixup! to be squashed into part-1, for\nconvenience:\n\n-- >8 --\nSubject: [PATCH] fixup! entry: extract a header file for entry.c functions\n\n---\n builtin/stash.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex ba774cce67..11f3ae3039 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -15,6 +15,7 @@\n #include \"log-tree.h\"\n #include \"diffcore.h\"\n #include \"exec-cmd.h\"\n+#include \"entry.h\"\n \n #define INCLUDE_ALL_FILES 2\n \n-- \n2.30.1\n\n"},{"id":"419739","messageId":"xmqqsg4qlpip.fsf@gitster.g","threadId":"55342","inReplyTo":"19fee535ee0ec5211f22b2b69a4a9804816c2322.1616122653.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 0/5] Parallel Checkout (part 2)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-19T22:58:22Z","receivedAt":"2021-03-19T22:59:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares writes:\n\n>> Let's redo part-1 on top of 'master' first without such a merge; it\n>> has been out of 'next' so we can do so easily without wanting for\n>> the tip of 'next' to get rewound.\n>\n> Thanks!\n>\n> I saw you've added the \"entry.h\" inclusion that was missing at\n> builtin/stash when merging this branch on 'seen'. However, now that\n> part-1 is based on 'master', the branch is no longer buildable without\n> this fix. So could we perhaps squash this change directly into the\n> relevant commit in this series?\n\nYeah, that was the kind of thing I had in mind when I suggested you\nto \"Let's redo part-1 on top of 'master'\".\n\nI'll mark the topic to be \"expecting a reroll\" for now, as I am deep\nin today's integration cycle.\n\nThanks.\n"},{"id":"420682","messageId":"CAP8UFD2eaaQEuytk6cPoYco2_dfdbsM3aAvJyYFqgYSfkZ487g@mail.gmail.com","threadId":"55342","inReplyTo":"a3383042e2ee6c454e024db7a9154dc979a5af9e.1616015337.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 1/5] unpack-trees: add basic support for parallel checkout","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-03-31T04:22:33Z","receivedAt":"2021-03-31T04:23:36Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Mar 17, 2021 at 10:12 PM Matheus Tavares\n wrote:\n\n> +struct parallel_checkout_item {\n> + /* pointer to a istate->cache[] entry. Not owned by us. */\n> + struct cache_entry *ce;\n> + struct conv_attrs ca;\n> + struct stat st;\n> + enum pc_item_status status;\n> +};\n> +\n> +struct parallel_checkout {\n> + enum pc_status status;\n> + struct parallel_checkout_item *items;\n> + size_t nr, alloc;\n> +};\n\nIt looks like this struct parallel_checkout contains what is called\nthe parallel checkout queue in many places. It would be a bit better\nif a name or at least a comment could make that clear.\n\n> +static struct parallel_checkout parallel_checkout;\n\n[...]\n\n> +static void finish_parallel_checkout(void)\n> +{\n> + if (parallel_checkout.status == PC_UNINITIALIZED)\n> + BUG(\"cannot finish parallel checkout: not initialized yet\");\n> +\n> + free(parallel_checkout.items);\n\nOk, it looks like the queue can indeed be freed like this as the items\ndon't need their fields to be freed.\n\n> + memset(¶llel_checkout, 0, sizeof(parallel_checkout));\n\nThat should work to reset it.\n\n> +}\n> +\n> +static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n> + const struct conv_attrs *ca)\n> +{\n> + enum conv_attrs_classification c;\n> +\n> + /*\n> + * Symlinks cannot be checked out in parallel as, in case of path\n> + * collision, they could racily replace leading directories of other\n> + * entries being checked out. Submodules are checked out in child\n> + * processes, which have their own parallel checkout queues.\n> + */\n> + if (!S_ISREG(ce->ce_mode))\n> + return 0;\n> +\n> + c = classify_conv_attrs(ca);\n> + switch (c) {\n> + case CA_CLASS_INCORE:\n> + return 1;\n> +\n> + case CA_CLASS_INCORE_FILTER:\n> + /*\n> + * It would be safe to allow concurrent instances of\n> + * single-file smudge filters, like rot13, but we should not\n> + * assume that all filters are parallel-process safe. So we\n> + * don't allow this.\n> + */\n> + return 0;\n> +\n> + case CA_CLASS_INCORE_PROCESS:\n> + /*\n> + * The parallel queue and the delayed queue are not compatible,\n> + * so they must be kept completely separated. And we can't tell\n> + * if a long-running process will delay its response without\n> + * actually asking it to perform the filtering. Therefore, this\n> + * type of filter is not allowed in parallel checkout.\n> + *\n> + * Furthermore, there should only be one instance of the\n> + * long-running process filter as we don't know how it is\n> + * managing its own concurrency. So, spreading the entries that\n> + * requisite such a filter among the parallel workers would\n> + * require a lot more inter-process communication. We would\n> + * probably have to designate a single process to interact with\n> + * the filter and send all the necessary data to it, for each\n> + * entry.\n> + */\n> + return 0;\n\nSo it looks like we don't process entries that need filtering. It's a\nbit disappointing as the commit message says:\n\n\"This new interface allows us to enqueue some of the entries being\nchecked out to later uncompress, smudge, and write them in parallel.\"\n\nSo it seems to say that entries could be smudged in parallel. Or maybe\nI am missing something?\n\n> + case CA_CLASS_STREAMABLE:\n> + return 1;\n> +\n> + default:\n> + BUG(\"unsupported conv_attrs classification '%d'\", c);\n> + }\n> +}\n\n[...]\n\n> +static int handle_results(struct checkout *state)\n> +{\n> + int ret = 0;\n> + size_t i;\n> + int have_pending = 0;\n> +\n> + /*\n> + * We first update the successfully written entries with the collected\n> + * stat() data, so that they can be found by mark_colliding_entries(),\n> + * in the next loop, when necessary.\n> + */\n> + for (i = 0; i < parallel_checkout.nr; i++) {\n> + struct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n> + if (pc_item->status == PC_ITEM_WRITTEN)\n> + update_ce_after_write(state, pc_item->ce, &pc_item->st);\n> + }\n> +\n> + for (i = 0; i < parallel_checkout.nr; i++) {\n> + struct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n> +\n> + switch(pc_item->status) {\n> + case PC_ITEM_WRITTEN:\n> + /* Already handled */\n> + break;\n> + case PC_ITEM_COLLIDED:\n> + /*\n> + * The entry could not be checked out due to a path\n> + * collision with another entry. Since there can only\n> + * be one entry of each colliding group on the disk, we\n> + * could skip trying to check out this one and move on.\n> + * However, this would leave the unwritten entries with\n> + * null stat() fields on the index, which could\n> + * potentially slow down subsequent operations that\n> + * require refreshing it: git would not be able to\n> + * trust st_size and would have to go to the filesystem\n> + * to see if the contents match (see ie_modified()).\n> + *\n> + * Instead, let's pay the overhead only once, now, and\n> + * call checkout_entry_ca() again for this file, to\n> + * have it's stat() data stored in the index. This also\n\ns/it's/its/\n\n> + * has the benefit of adding this entry and its\n> + * colliding pair to the collision report message.\n> + * Additionally, this overwriting behavior is consistent\n> + * with what the sequential checkout does, so it doesn't\n> + * add any extra overhead.\n> + */\n> + ret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n> + state, NULL, NULL);\n> + break;\n> + case PC_ITEM_PENDING:\n> + have_pending = 1;\n> + /* fall through */\n> + case PC_ITEM_FAILED:\n> + ret = -1;\n> + break;\n> + default:\n> + BUG(\"unknown checkout item status in parallel checkout\");\n> + }\n> + }\n> +\n> + if (have_pending)\n> + error(_(\"parallel checkout finished with pending entries\"));\n\nIs this an error that can actually happen? Or is it a bug? If this can\nactually happen what is the state of the working directory and what\ncan the user do? Start another checkout with an option to do it\nsequentially?\n\n> + return ret;\n> +}\n\n[...]\n\n> +static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n> + const char *path)\n> +{\n> + int ret;\n> + struct stream_filter *filter;\n> + struct strbuf buf = STRBUF_INIT;\n> + char *new_blob;\n> + unsigned long size;\n\nI thought that we shouldn't use unsigned long anymore for sizes, but\nunfortunately it looks like that's what read_blob_entry() expects.\n\n> + size_t newsize = 0;\n\nI don't think we need to initialize newsize. Also we could perhaps\ndeclare it closer to where it is used below.\n\n> + ssize_t wrote;\n> +\n> + /* Sanity check */\n> + assert(is_eligible_for_parallel_checkout(pc_item->ce, &pc_item->ca));\n> +\n> + filter = get_stream_filter_ca(&pc_item->ca, &pc_item->ce->oid);\n> + if (filter) {\n> + if (stream_blob_to_fd(fd, &pc_item->ce->oid, filter, 1)) {\n> + /* On error, reset fd to try writing without streaming */\n> + if (reset_fd(fd, path))\n> + return -1;\n> + } else {\n> + return 0;\n> + }\n> + }\n> +\n> + new_blob = read_blob_entry(pc_item->ce, &size);\n> + if (!new_blob)\n> + return error(\"unable to read sha1 file of %s (%s)\", path,\n\nWith the support of sha256, I guess we should avoid talking about\nsha1. Maybe: s/sha1/object/ Or maybe we could just say that we\ncouldn't read the object, as read_blob_entry() would perhaps read it\nfrom a pack-file and not from a loose file?\n\n> + oid_to_hex(&pc_item->ce->oid));\n> +\n> + /*\n> + * checkout metadata is used to give context for external process\n> + * filters. Files requiring such filters are not eligible for parallel\n> + * checkout, so pass NULL.\n> + */\n> + ret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n> + new_blob, size, &buf, NULL);\n> +\n> + if (ret) {\n> + free(new_blob);\n> + new_blob = strbuf_detach(&buf, &newsize);\n> + size = newsize;\n\n\"newsize\" seems to be used only here. It would be nice if we could get\nrid of it and pass \"&size\" to strbuf_detach(), but maybe we cannot\nbecause of the unsigned long/size_t type mismatch. At least we could\ndeclare \"newsize\" in this scope though.\n\n> + }\n> +\n> + wrote = write_in_full(fd, new_blob, size);\n> + free(new_blob);\n> + if (wrote < 0)\n> + return error(\"unable to write file %s\", path);\n> +\n> + return 0;\n> +}\n> +\n\n[...]\n\n> +static void write_pc_item(struct parallel_checkout_item *pc_item,\n> + struct checkout *state)\n> +{\n> + unsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n\nIt looks like we are using similar code in different places, so it's\nprobably ok, but maybe it would be nice one day to use a macro or an\ninline function for this.\n\n> + int fd = -1, fstat_done = 0;\n> + struct strbuf path = STRBUF_INIT;\n> + const char *dir_sep;\n"},{"id":"420684","messageId":"CAP8UFD0CmPmvK7a-+5fKk0e0=te_e7cCFf6_-vSFrX9COTBg-w@mail.gmail.com","threadId":"55342","inReplyTo":"fca797698c78e2248bd2645bd0dce23ec4df35ae.1616015337.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 2/5] parallel-checkout: make it truly parallel","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-03-31T04:32:38Z","receivedAt":"2021-03-31T04:33:24Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Mar 17, 2021 at 10:12 PM Matheus Tavares\n wrote:\n\n> diff --git a/.gitignore b/.gitignore\n> index 3dcdb6bb5a..26f8ddfc55 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -33,6 +33,7 @@\n> /git-check-mailmap\n> /git-check-ref-format\n> /git-checkout\n> +/git-checkout--helper\n\nI wonder if \"checkout--worker\" would be better than \"checkout--helper\".\n\n> /git-checkout-index\n> /git-cherry\n> /git-cherry-pick\n\n[...]\n\n> +#define ASSERT_PC_ITEM_RESULT_SIZE(got, exp) \\\n> +{ \\\n> + if (got != exp) \\\n> + BUG(\"corrupted result from checkout worker (got %dB, exp %dB)\", \\\n\nMaybe precompilers are smart enough to not replace the \"got\" and \"exp\"\nin the above string, but it might be a bit confusing for readers.\nAnway I wonder if this macro could just be a regular (possibly inline)\nfunction.\n\n> + got, exp); \\\n> +} while(0)\n\n> +static void parse_and_save_result(const char *line, int len,\n> + struct pc_worker *worker)\n> +{\n> + struct pc_item_result *res;\n> + struct parallel_checkout_item *pc_item;\n> + struct stat *st = NULL;\n> +\n> + if (len < PC_ITEM_RESULT_BASE_SIZE)\n> + BUG(\"too short result from checkout worker (got %dB, exp %dB)\",\n> + len, (int)PC_ITEM_RESULT_BASE_SIZE);\n> +\n> + res = (struct pc_item_result *)line;\n> +\n> + /*\n> + * Worker should send either the full result struct on success, or\n> + * just the base (i.e. no stat data), otherwise.\n> + */\n> + if (res->status == PC_ITEM_WRITTEN) {\n> + ASSERT_PC_ITEM_RESULT_SIZE(len, (int)sizeof(struct pc_item_result));\n> + st = &res->st;\n> + } else {\n> + ASSERT_PC_ITEM_RESULT_SIZE(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n> + }\n> +\n> + if (!worker->nr_items_to_complete || res->id != worker->next_item_to_complete)\n\nNit: maybe it could be useful to distinguish between these 2 potential\nbugs and have a specific BUG() for each one.\n\n> + BUG(\"checkout worker sent unexpected item id\");\n\n> + worker->next_item_to_complete++;\n> + worker->nr_items_to_complete--;\n> +\n> + pc_item = ¶llel_checkout.items[res->id];\n> + pc_item->status = res->status;\n> + if (st)\n> + pc_item->st = *st;\n> +}\n> +\n> +\n\nOne blank line is enough.\n\n> +static void gather_results_from_workers(struct pc_worker *workers,\n> + int num_workers)\n> +{\n> + int i, active_workers = num_workers;\n> + struct pollfd *pfds;\n> +\n> + CALLOC_ARRAY(pfds, num_workers);\n> + for (i = 0; i < num_workers; i++) {\n> + pfds[i].fd = workers[i].cp.out;\n> + pfds[i].events = POLLIN;\n> + }\n> +\n> + while (active_workers) {\n> + int nr = poll(pfds, num_workers, -1);\n> +\n> + if (nr < 0) {\n> + if (errno == EINTR)\n> + continue;\n> + die_errno(\"failed to poll checkout workers\");\n> + }\n> +\n> + for (i = 0; i < num_workers && nr > 0; i++) {\n\nIs it possible that nr is 0? If that happens, it looks like we would\nbe in an infinite `while (active_workers) { ... }` loop.\n\nActually in poll(2) there is: \"A value of 0 indicates that the call\ntimed out and no file descriptors were ready.\" So it seems that it\ncould, at least theorically, happen.\n\n> + struct pc_worker *worker = &workers[i];\n> + struct pollfd *pfd = &pfds[i];\n> +\n> + if (!pfd->revents)\n> + continue;\n> +\n> + if (pfd->revents & POLLIN) {\n> + int len;\n> + const char *line = packet_read_line(pfd->fd, &len);\n> +\n> + if (!line) {\n> + pfd->fd = -1;\n> + active_workers--;\n> + } else {\n> + parse_and_save_result(line, len, worker);\n> + }\n> + } else if (pfd->revents & POLLHUP) {\n> + pfd->fd = -1;\n> + active_workers--;\n> + } else if (pfd->revents & (POLLNVAL | POLLERR)) {\n> + die(_(\"error polling from checkout worker\"));\n> + }\n> +\n> + nr--;\n> + }\n> + }\n> +\n> + free(pfds);\n> +}\n\n[...]\n\n> +enum pc_item_status {\n> + PC_ITEM_PENDING = 0,\n> + PC_ITEM_WRITTEN,\n> + /*\n> + * The entry could not be written because there was another file\n> + * already present in its path or leading directories. Since\n> + * checkout_entry_ca() removes such files from the working tree before\n> + * enqueueing the entry for parallel checkout, it means that there was\n> + * a path collision among the entries being written.\n> + */\n> + PC_ITEM_COLLIDED,\n> + PC_ITEM_FAILED,\n> +};\n> +\n> +struct parallel_checkout_item {\n> + /*\n> + * In main process ce points to a istate->cache[] entry. Thus, it's not\n> + * owned by us. In workers they own the memory, which *must be* released.\n> + */\n> + struct cache_entry *ce;\n> + struct conv_attrs ca;\n> + size_t id; /* position in parallel_checkout.items[] of main process */\n> +\n> + /* Output fields, sent from workers. */\n> + enum pc_item_status status;\n> + struct stat st;\n> +};\n\nMaybe the previous patch could have declared both 'enum\npc_item_status' and 'struct parallel_checkout_item' here, in\nparallel-checkout.h, so that this patch wouldn't need to move them\nhere.\n"},{"id":"420685","messageId":"CAP8UFD3s3NUpi2eyPWFa5bL4rez1wNtj5-dUpv8dJLo2CKYu9A@mail.gmail.com","threadId":"55342","inReplyTo":"8c83e92445b4131e9b8f8e2aa29b00717b257d13.1616015337.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 3/5] parallel-checkout: add configuration options","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-03-31T04:33:43Z","receivedAt":"2021-03-31T04:35:05Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Mar 17, 2021 at 10:12 PM Matheus Tavares\n wrote:\n\n> The above benchmarks show that parallel checkout is most effective on\n> repositories located on an SSD or over a distributed file system. For\n> local file systems on spinning disks, and/or older machines, the\n> parallelism does not always bring a good performance. For this reason,\n> the default value for checkout.workers is one, a.k.a. sequential\n> checkout.\n\nI wonder how many people are still using HDD, and how many will still\nuse them in a few years.\n\nI think having 1 as the default value for checkout.workers might be\ngood for a while for backward compatibility and stability, while\npeople who are interested can test parallel checkout on different\nenvironments. But we might want, in a few releases, after some bugs,\nif any, have been fixed, to use a default, maybe 10, that will provide\nsignificant speedup for most people, and will justify the added\ncomplexity, especially as your numbers still show a small speedup for\nHDD when using 10.\n\n> To decide the default value for checkout.thresholdForParallelism,\n> another benchmark was executed in the \"Local SSD\" setup, where parallel\n> checkout showed to be beneficial. This time, we compared the runtime of\n> a `git checkout -f`, with and without parallelism, after randomly\n> removing an increasing number of files from the Linux working tree. The\n> \"sequential fallback\" column bellow corresponds to the executions where\n\ns/bellow/below/\n\n> checkout.workers was 10 but checkout.thresholdForParallelism was equal\n> to the number of to-be-updated files plus one (so that we end up writing\n> sequentially). Each test case was sampled 15 times, and each sample had\n> a randomly different set of files removed. Here are the results:\n>\n> sequential fallback 10 workers speedup\n> 10 files 772.3 ms ± 12.6 ms 769.0 ms ± 13.6 ms 1.00 ± 0.02\n> 20 files 780.5 ms ± 15.8 ms 775.2 ms ± 9.2 ms 1.01 ± 0.02\n> 50 files 806.2 ms ± 13.8 ms 767.4 ms ± 8.5 ms 1.05 ± 0.02\n> 100 files 833.7 ms ± 21.4 ms 750.5 ms ± 16.8 ms 1.11 ± 0.04\n> 200 files 897.6 ms ± 30.9 ms 730.5 ms ± 14.7 ms 1.23 ± 0.05\n> 500 files 1035.4 ms ± 48.0 ms 677.1 ms ± 22.3 ms 1.53 ± 0.09\n> 1000 files 1244.6 ms ± 35.6 ms 654.0 ms ± 38.3 ms 1.90 ± 0.12\n> 2000 files 1488.8 ms ± 53.4 ms 658.8 ms ± 23.8 ms 2.26 ± 0.12\n>\n> From the above numbers, 100 files seems to be a reasonable default value\n> for the threshold setting.\n>\n> Note: Up to 1000 files, we observe a drop in the execution time of the\n> parallel code with an increase in the number of files. This is a rather\n> odd behavior, but it was observed in multiple repetitions. Above 1000\n> files, the execution time increases according to the number of files, as\n> one would expect.\n>\n> About the test environments: Local SSD tests were executed on an\n> i7-7700HQ (4 cores with hyper-threading) running Manjaro Linux. Local\n> HDD tests were executed on an Intel(R) Xeon(R) E3-1230 (also 4 cores\n> with hyper-threading), HDD Seagate Barracuda 7200.14 SATA 3.1, running\n> Debian. NFS and EFS tests were executed on an Amazon EC2 c5n.xlarge\n> instance, with 4 vCPUs. The Linux NFS server was running on a m6g.large\n> instance with 2 vCPUSs and a 1 TB EBS GP2 volume. Before each timing,\n> the linux repository was removed (or checked out back to its previous\n> state), and `sync && sysctl vm.drop_caches=3` was executed.\n\nThanks for all these tests and details!\n\n> Co-authored-by: Jeff Hostetler \n> Signed-off-by: Matheus Tavares \n\n[...]\n\n> @@ -23,6 +25,19 @@ enum pc_status parallel_checkout_status(void)\n> return parallel_checkout.status;\n> }\n>\n> +#define DEFAULT_THRESHOLD_FOR_PARALLELISM 100\n\nUsing a \"static const int\" might be a bit better.\n\n> +void get_parallel_checkout_configs(int *num_workers, int *threshold)\n> +{\n> + if (git_config_get_int(\"checkout.workers\", num_workers))\n> + *num_workers = 1;\n\nI think it's better when an important default value like this \"1\" is\nmade more visible using a \"static const int\" or a \"#define\".\n\n> + else if (*num_workers < 1)\n> + *num_workers = online_cpus();\n> +\n> + if (git_config_get_int(\"checkout.thresholdForParallelism\", threshold))\n> + *threshold = DEFAULT_THRESHOLD_FOR_PARALLELISM;\n> +}\n> +\n> void init_parallel_checkout(void)\n> {\n> if (parallel_checkout.status != PC_UNINITIALIZED)\n> @@ -554,11 +569,9 @@ static void write_items_sequentially(struct checkout *state)\n> write_pc_item(¶llel_checkout.items[i], state);\n> }\n>\n> -#define DEFAULT_NUM_WORKERS 2\n\nAs I say above, it doesn't look like a good idea to have removed this.\n\n> -int run_parallel_checkout(struct checkout *state)\n> +int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n> {\n> - int ret, num_workers = DEFAULT_NUM_WORKERS;\n> + int ret;\n>\n> if (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n> BUG(\"cannot run parallel checkout: uninitialized or already running\");\n> @@ -568,7 +581,7 @@ int run_parallel_checkout(struct checkout *state)\n> if (parallel_checkout.nr < num_workers)\n> num_workers = parallel_checkout.nr;\n>\n> - if (num_workers <= 1) {\n> + if (num_workers <= 1 || parallel_checkout.nr < threshold) {\n\nHere we check the number of workers...\n\n> write_items_sequentially(state);\n> } else {\n> struct pc_worker *workers = setup_workers(state, num_workers);\n\n[...]\n\n> @@ -480,7 +483,8 @@ static int check_updates(struct unpack_trees_options *o,\n> }\n> }\n> stop_progress(&progress);\n> - errs |= run_parallel_checkout(&state);\n> + if (pc_workers > 1)\n\n...but the number of workers was already checked here.\n\n\n> + errs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n> errs |= finish_delayed_checkout(&state, NULL);\n> git_attr_set_direction(GIT_ATTR_CHECKIN);\n"},{"id":"420688","messageId":"CAP8UFD3XkU6H==pHCFLpchKrb9cNj8fFnmF4HmCG-SaYj8C-Vg@mail.gmail.com","threadId":"55342","inReplyTo":"0592740ec14c4ab72b8d46a7fecf1c66e7a497fd.1616015337.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 5/5] parallel-checkout: add design documentation","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-03-31T05:36:02Z","receivedAt":"2021-03-31T05:37:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Mar 17, 2021 at 10:12 PM Matheus Tavares\n wrote:\n\n> +For the purposes of discussion here, the current sequential\n> +implementation of Step 3 has 3 layers:\n\nYou refer to these layers as \"steps\" below, so you might want to use\n\"sub-steps\" instead of \"layers\".\n\n> +* Step 3a: `unpack-trees.c:check_updates()` contains a series of\n> + sequential loops iterating over the `cache_entry`'s array. The main\n> + loop in this function calls the next layer for each of the\n\nNot sure what \"layer\" means here. Does it mean Step 3b below? In this\ncase I would suggest using \"Step 3b\" instead of \"layer\".\n\n> + to-be-updated entries.\n> +\n> +* Step 3b: `entry.c:checkout_entry()` examines the existing working tree\n> + for file conflicts, collisions, and unsaved changes. It removes files\n> + and create leading directories as necessary. It calls the next layer\n\ns/create/creates/\n\nI guess the \"next layer\" is Step 3c below.\n\n> + for each entry to be written.\n> +\n> +* Step 3c: `entry.c:write_entry()` loads the blob into memory, smudges\n> + it if necessary, creates the file in the working tree, writes the\n> + smudged contents, calls `fstat()` or `lstat()`, and updates the\n> + associated `cache_entry` struct with the stat information gathered.\n> +\n> +It wouldn't be safe to perform Step 3b in parallel, as there could be\n> +race conditions between file creations and removals. Instead, the\n> +parallel checkout framework lets the sequential code handle Step 3b,\n> +and use parallel workers to replace the sequential\n> +`entry.c:write_entry()` calls from Step 3c.\n\nOk.\n\n> +Rejected Multi-Threaded Solution\n> +--------------------------------\n> +\n> +The most \"straightforward\" implementation would be to spread the set of\n> +to-be-updated cache entries across multiple threads. But due to the\n> +thread-unsafe functions in the ODB code, we would have to use locks to\n> +coordinate the parallel operation. An early prototype of this solution\n> +showed that the multi-threaded checkout would bring performance\n> +improvements over the sequential code, but there was still too much lock\n> +contention. A `perf` profiling indicated that around 20% of the runtime\n> +during a local Linux clone (on an SSD) was spent in locking functions.\n> +For this reason this approach was rejected in favor of using multiple\n> +child processes, which led to a better performance.\n\nNice explanation.\n\n> +Multi-Process Solution\n> +----------------------\n> +\n> +Parallel checkout alters the aforementioned Step 3 to use multiple\n> +`checkout--helper` background processes to distribute the work. The\n> +long-running worker processes are controlled by the foreground Git\n> +command using the existing run-command API.\n> +\n> +Overview\n> +~~~~~~~~\n> +\n> +Step 3b is only slightly altered; for each entry to be checked out, the\n> +main process:\n\nMaybe: s/main process:/main process performs the following steps:/\n\nIf you apply this suggestion, you may also want the following below:\n\ns/M1: Checks/M1: Check/\ns/and decides/and decide/\ns/M2: Creates/M2: Create/\n...\n\n> +* M1: Checks whether there is any untracked or unclean file in the\n> + working tree which would be overwritten by this entry, and decides\n> + whether to proceed (removing the file(s)) or not.\n> +\n> +* M2: Creates the leading directories.\n> +\n> +* M3: Loads the conversion attributes for the entry's path.\n> +\n> +* M4: Checks, based on the entry's type and conversion attributes,\n> + whether the entry is eligible for parallel checkout (more on this\n> + later). If it is eligible, enqueues the entry and the loaded\n> + attributes to later write the entry in parallel. If not, writes the\n> + entry right away, using the default sequential code.\n> +\n> +Note: we save the conversion attributes associated with each entry\n> +because the workers don't have access to the main process' index state,\n> +so they can't load the attributes by themselves (and the attributes are\n> +needed to properly smudge the entry). Additionally, this has a positive\n> +impact on performance as (1) we don't need to load the attributes twice\n> +and (2) the attributes machinery is optimized to handle paths in\n> +sequential order.\n\nNice!\n\n> +After all entries have passed through the above steps, the main process\n> +checks if the number of enqueued entries is sufficient to spread among\n> +the workers. If not, it just writes them sequentially. Otherwise, it\n> +spawns the workers and distributes the queued entries uniformly in\n> +continuous chunks. This aims to minimize the chances of two workers\n> +writing to the same directory simultaneously, which could increase lock\n> +contention in the kernel.\n> +\n> +Then, for each assigned item, each worker:\n> +\n> +* W1: Checks if there is any non-directory file in the leading part of\n> + the entry's path or if there already exists a file at the entry' path.\n> + If so, mark the entry with `PC_ITEM_COLLIDED` and skip it (more on\n> + this later).\n> +\n> +* W2: Creates the file (with O_CREAT and O_EXCL).\n> +\n> +* W3: Loads the blob into memory (inflating and delta reconstructing\n> + it).\n> +\n> +* W4: Filters the blob.\n\nNot sure what \"Filters\" means here. Is this related to the smudge filter?\n\n> +* W5: Writes the result to the file descriptor opened at W2.\n> +\n> +* W6: Calls `fstat()` or lstat()` on the just-written path, and sends\n> + the result back to the main process, together with the end status of\n> + the operation and the item's identification number.\n> +\n> +Note that steps W3 to W5 might actually be performed together, using the\n> +streaming interface.\n\nNot sure what \"performed together\" means here. Does it mean by a\nsingle function or set of functions?\n\n> +Also note that the workers *never* remove any files. As mentioned\n\nMaybe: s/any files/any file/\n\n> +earlier, it is the responsibility of the main process to remove any\n> +files that block the checkout operation (or abort it). This is crucial\n\nMaybe: s/files/file/ and s/block/blocks/ and s/abort/aborts/\n\n> +to avoid race conditions and also to properly detect path collisions at\n> +Step W1.\n> +\n> +After the workers finish writing the items and sending back the required\n> +information, the main process handles the results in two steps:\n> +\n> +- First, it updates the in-memory index with the `lstat()` information\n> + sent by the workers. (This must be done first as this information\n> + might me required in the following step.)\n> +\n> +- Then it writes the items which collided on disk (i.e. items marked\n> + with `PC_ITEM_COLLIDED`). More on this below.\n> +\n> +Path Collisions\n> +---------------\n> +\n> +Path collisions happen when two different paths correspond to the same\n> +entry in the file system. E.g. the paths 'a' and 'A' would collide in a\n> +case-insensitive file system.\n> +\n> +The sequential checkout deals with collisions in the same way that it\n> +deals with files that were already present in the working tree before\n> +checkout. Basically, it checks if the path that it wants to write\n> +already exists on disk, makes sure the existing file doesn't have\n> +unsaved data, and then overwrite it. (To be more pedantic: it deletes\n\ns/overwrite/overwrites/\n\n> +the existing file and creates the new one.) So, if there are multiple\n> +colliding files to be checked out, the sequential code will write each\n> +one of them but only the last will actually survive on disk.\n> +\n> +Parallel checkout aims to reproduce the same behavior. However, we\n> +cannot let the workers racily write to the same file on disk. Instead,\n> +the workers detect when the entry that they want to check out would\n> +collide with an existing file, and mark it with `PC_ITEM_COLLIDED`.\n> +Later, the main process can sequentially feed these entries back to\n> +`checkout_entry()` without the risk of race conditions. On clone, this\n> +also has the effect of marking the colliding entries to later emit a\n> +warning for the user, like the classic sequential checkout does.\n> +\n> +The workers are able to detect both collisions among the entries being\n> +concurrently written and collisions among parallel-eligible and\n> +ineligible entries. The general idea for collision detection is quite\n> +straightforward: for each parallel-eligible entry, the main process must\n> +remove all files that prevent this entry from being written (before\n> +enqueueing it). This includes any non-directory file in the leading path\n> +of the entry. Later, when a worker gets assigned the entry, it looks\n> +again for the non-directories files and for an already existent file at\n\nMaybe: s/existent/existing/\n\n> +the entry's path. If any of these checks finds something, the worker\n> +knows that there was a path collision.\n> +\n> +Because parallel checkout can distinguish path collisions from the case\n> +where the file was already present in the working tree before checkout,\n> +we could alternatively choose to skip the checkout of colliding entries.\n> +However, each entry that doesn't get written would have NULL `lstat()`\n> +fields on the index. This could cause performance penalties for\n> +subsequent commands that need to refresh the index, as they would have\n> +to go to the file system to see if the entry is dirty. Thus, if we have\n> +N entries in a colliding group and we decide to write and `lstat()` only\n> +one of them, every subsequent `git-status` will have to read, convert,\n> +and hash the written file N - 1 times. By checking out all colliding\n> +entries (like the sequential code does), we only pay the overhead once,\n> +during checkout.\n> +\n> +Eligible Entries for Parallel Checkout\n> +--------------------------------------\n> +\n> +As previously mentioned, not all entries passed to `checkout_entry()`\n> +will be considered eligible for parallel checkout. More specifically, we\n> +exclude:\n> +\n> +- Symbolic links; to avoid race conditions that, in combination with\n> + path collisions, could cause workers to write files at the wrong\n> + place. For example, if we were to concurrently check out a symlink\n> + 'a' -> 'b' and a regular file 'A/f' in a case-insensitive file system,\n> + we could potentially end up writing the file 'A/f' at 'a/f', due to a\n> + race condition.\n> +\n> +- Regular files that require external filters (either \"one shot\" filters\n> + or long-running process filters). These filters are black-boxes to Git\n> + and may have their own internal locking or non-concurrent assumptions.\n> + So it might not be safe to run multiple instances in parallel.\n> ++\n> +Besides, long-running filters may use the delayed checkout feature to\n> +postpone the return of some filtered blobs. The delayed checkout queue\n> +and the parallel checkout queue are not compatible and should remain\n> +separated.\n\nAre files that require some other internal filters eligible though?\n\n> +Ineligible entries are checked out by the classic sequential codepath\n> +*before* spawning workers.\n> +\n> +Note: submodules's files are also eligible for parallel checkout (as\n> +long as they don't fall into the two excluding categories mentioned\n> +above). But since each submodule is checked out in its own child\n> +process, we don't mix the superproject's and the submodules' files in\n> +the same parallel checkout process or queue.\n\nOk.\n\n> +The API\n> +-------\n> +\n> +The parallel checkout API was designed with the goal to minimize changes\n> +to the current users of the checkout machinery. This means that they\n> +don't have to call a different function for sequential or parallel\n> +checkout. As already mentioned, `checkout_entry()` will automatically\n> +insert the given entry in the parallel checkout queue when this feature\n> +is enabled and the entry is eligible; otherwise, it will just write the\n> +entry right away, using the sequential code. In general, callers of the\n> +parallel checkout API should look similar to this:\n> +\n> +----------------------------------------------\n> +int pc_workers, pc_threshold, err = 0;\n> +struct checkout state;\n> +\n> +get_parallel_checkout_configs(&pc_workers, &pc_threshold);\n> +\n> +/*\n> + * This check is not strictly required, but it\n> + * should save some time in sequential mode.\n> + */\n\nIt might be nice if this comment was also in front of the real code.\n\n> +if (pc_workers > 1)\n> + init_parallel_checkout();\n> +\n> +for (each cache_entry ce to-be-updated)\n> + err |= checkout_entry(ce, &state, NULL, NULL);\n> +\n> +err |= run_parallel_checkout(&state, pc_workers, pc_threshold, NULL, NULL);\n"},{"id":"420689","messageId":"CAP8UFD1stvx=2hBWyxmu75SXRzX-bHBfGr2jxWKgHdc85cfxRg@mail.gmail.com","threadId":"55342","inReplyTo":"cover.1616015337.git.matheus.bernardino@usp.br","subject":"Re: [PATCH 0/5] Parallel Checkout (part 2)","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-03-31T05:42:31Z","receivedAt":"2021-03-31T05:43:44Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Mar 17, 2021 at 10:12 PM Matheus Tavares\n wrote:\n>\n> This is the next step in the parallel checkout implementation. An\n> overview of the complete series can be seen at [1].\n>\n> The last patch in this series adds a design doc, so it may help to\n> review it first. Also, there is no need to have any familiarity with\n> part-1, as this part doesn't have any semantic dependency with that.\n>\n> This series is based on the merge of 'mt/parallel-checkout-part-1' and\n> 'master', so that it can use the \"brew cast\" fix and the latest security\n> fix (both from master), to run the tests. (The merge is textually\n> clean, but it needs a small semantic fix: the '#include \"entry.h\"'\n> addition in builtin/stash.c).\n\nI took a look and left a number of comments. Otherwise it looks good overall!\n\nThanks!\n"},{"id":"420882","messageId":"CAHd-oW7TYmbJAx9Jrzwsfb0zsdT=KmhumJnOJygk0yTWTW3i=Q@mail.gmail.com","threadId":"55342","inReplyTo":"CAP8UFD2eaaQEuytk6cPoYco2_dfdbsM3aAvJyYFqgYSfkZ487g@mail.gmail.com","subject":"Re: [PATCH 1/5] unpack-trees: add basic support for parallel checkout","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-02T14:39:30Z","receivedAt":"2021-04-02T14:39:47Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Mar 31, 2021 at 1:22 AM Christian Couder\n wrote:\n>\n> On Wed, Mar 17, 2021 at 10:12 PM Matheus Tavares\n> wrote:\n>\n> > +struct parallel_checkout_item {\n> > + /* pointer to a istate->cache[] entry. Not owned by us. */\n> > + struct cache_entry *ce;\n> > + struct conv_attrs ca;\n> > + struct stat st;\n> > + enum pc_item_status status;\n> > +};\n> > +\n> > +struct parallel_checkout {\n> > + enum pc_status status;\n> > + struct parallel_checkout_item *items;\n> > + size_t nr, alloc;\n> > +};\n>\n> It looks like this struct parallel_checkout contains what is called\n> the parallel checkout queue in many places. It would be a bit better\n> if a name or at least a comment could make that clear.\n\nRight, will do.\n\n> > + case CA_CLASS_INCORE_PROCESS:\n> > + /*\n> > + * The parallel queue and the delayed queue are not compatible,\n> > + * so they must be kept completely separated. And we can't tell\n> > + * if a long-running process will delay its response without\n> > + * actually asking it to perform the filtering. Therefore, this\n> > + * type of filter is not allowed in parallel checkout.\n> > + *\n> > + * Furthermore, there should only be one instance of the\n> > + * long-running process filter as we don't know how it is\n> > + * managing its own concurrency. So, spreading the entries that\n> > + * requisite such a filter among the parallel workers would\n> > + * require a lot more inter-process communication. We would\n> > + * probably have to designate a single process to interact with\n> > + * the filter and send all the necessary data to it, for each\n> > + * entry.\n> > + */\n> > + return 0;\n>\n> So it looks like we don't process entries that need filtering. It's a\n> bit disappointing as the commit message says:\n>\n> \"This new interface allows us to enqueue some of the entries being\n> checked out to later uncompress, smudge, and write them in parallel.\"\n>\n> So it seems to say that entries could be smudged in parallel. Or maybe\n> I am missing something?\n\nGood point, this part of the commit message is indeed misleading. Only\ninternal filters, like re-encoding and end-of-line conversions, can be\nperformed in parallel. I'll rephrase that sentence, thanks!\n\n> > +static int handle_results(struct checkout *state)\n[...]\n> > +\n> > + if (have_pending)\n> > + error(_(\"parallel checkout finished with pending entries\"));\n>\n> Is this an error that can actually happen? Or is it a bug? If this can\n> actually happen what is the state of the working directory [...] ?\n\nThis could happen both due to a bug or due to an actual error, e.g. if\none of the workers die()s or gets killed before finishing its work\nqueue. In these cases, the files associated with the unfinished items\nwill be missing from the working tree, and their index entries will\nhave null stat() information.\n\n> [...] and what\n> can the user do? Start another checkout with an option to do it\n> sequentially?\n\nHmm, it depends on what caused the error. For example, if the user\naccidentally killed one of the workers, starting another checkout\n(either parallel or sequential) should be able to create the missing\nfiles. But that might not be the case if the error was caused by a\nfatal condition in one of the workers which led it to die() (like a\npacket write failure or corrupted object).\n\nNevertheless, I think it might be interesting to rephrase the error\nmessage here to be more explicit about the outcome state. Perhaps even\nmention how many entries will be missing.\n\n> [...]\n>\n> > +static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n> > + const char *path)\n> > +{\n> > + int ret;\n> > + struct stream_filter *filter;\n> > + struct strbuf buf = STRBUF_INIT;\n> > + char *new_blob;\n> > + unsigned long size;\n>\n> I thought that we shouldn't use unsigned long anymore for sizes, but\n> unfortunately it looks like that's what read_blob_entry() expects.\n>\n> > + size_t newsize = 0;\n>\n> I don't think we need to initialize newsize. Also we could perhaps\n> declare it closer to where it is used below.\n>\n> > + ssize_t wrote;\n> > +\n> > + /* Sanity check */\n> > + assert(is_eligible_for_parallel_checkout(pc_item->ce, &pc_item->ca));\n> > +\n> > + filter = get_stream_filter_ca(&pc_item->ca, &pc_item->ce->oid);\n> > + if (filter) {\n> > + if (stream_blob_to_fd(fd, &pc_item->ce->oid, filter, 1)) {\n> > + /* On error, reset fd to try writing without streaming */\n> > + if (reset_fd(fd, path))\n> > + return -1;\n> > + } else {\n> > + return 0;\n> > + }\n> > + }\n> > +\n> > + new_blob = read_blob_entry(pc_item->ce, &size);\n> > + if (!new_blob)\n> > + return error(\"unable to read sha1 file of %s (%s)\", path,\n>\n> With the support of sha256, I guess we should avoid talking about\n> sha1. Maybe: s/sha1/object/ Or maybe we could just say that we\n> couldn't read the object, as read_blob_entry() would perhaps read it\n> from a pack-file and not from a loose file?\n\nGood point, thanks!\n\n> > + oid_to_hex(&pc_item->ce->oid));\n> > +\n> > + /*\n> > + * checkout metadata is used to give context for external process\n> > + * filters. Files requiring such filters are not eligible for parallel\n> > + * checkout, so pass NULL.\n> > + */\n> > + ret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n> > + new_blob, size, &buf, NULL);\n> > +\n> > + if (ret) {\n> > + free(new_blob);\n> > + new_blob = strbuf_detach(&buf, &newsize);\n> > + size = newsize;\n>\n> \"newsize\" seems to be used only here. It would be nice if we could get\n> rid of it and pass \"&size\" to strbuf_detach(), but maybe we cannot\n> because of the unsigned long/size_t type mismatch. At least we could\n> declare \"newsize\" in this scope though.\n\nSure, I'll move \"newsize\" down to this block and remove the\ninitialization, thanks.\n"},{"id":"420883","messageId":"CAHd-oW7+x17uf2AuCrdYWUv9Ln2bq+V4rMPKo3VoYe3ZouQjLA@mail.gmail.com","threadId":"55342","inReplyTo":"CAP8UFD0CmPmvK7a-+5fKk0e0=te_e7cCFf6_-vSFrX9COTBg-w@mail.gmail.com","subject":"Re: [PATCH 2/5] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-02T14:42:26Z","receivedAt":"2021-04-02T14:42:42Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Mar 31, 2021 at 1:32 AM Christian Couder\n wrote:\n>\n> On Wed, Mar 17, 2021 at 10:12 PM Matheus Tavares\n> wrote:\n>\n> > diff --git a/.gitignore b/.gitignore\n> > index 3dcdb6bb5a..26f8ddfc55 100644\n> > --- a/.gitignore\n> > +++ b/.gitignore\n> > @@ -33,6 +33,7 @@\n> > /git-check-mailmap\n> > /git-check-ref-format\n> > /git-checkout\n> > +/git-checkout--helper\n>\n> I wonder if \"checkout--worker\" would be better than \"checkout--helper\".\n\nYeah, good idea, I'll change that.\n\n> > /git-checkout-index\n> > /git-cherry\n> > /git-cherry-pick\n>\n> [...]\n>\n> > +#define ASSERT_PC_ITEM_RESULT_SIZE(got, exp) \\\n> > +{ \\\n> > + if (got != exp) \\\n> > + BUG(\"corrupted result from checkout worker (got %dB, exp %dB)\", \\\n>\n> Maybe precompilers are smart enough to not replace the \"got\" and \"exp\"\n> in the above string, but it might be a bit confusing for readers.\n> Anway I wonder if this macro could just be a regular (possibly inline)\n> function.\n\nOk, I will replace this with an inline function.\n\n> > + got, exp); \\\n> > +} while(0)\n>\n> > +static void parse_and_save_result(const char *line, int len,\n> > + struct pc_worker *worker)\n> > +{\n> > + struct pc_item_result *res;\n> > + struct parallel_checkout_item *pc_item;\n> > + struct stat *st = NULL;\n> > +\n> > + if (len < PC_ITEM_RESULT_BASE_SIZE)\n> > + BUG(\"too short result from checkout worker (got %dB, exp %dB)\",\n> > + len, (int)PC_ITEM_RESULT_BASE_SIZE);\n> > +\n> > + res = (struct pc_item_result *)line;\n> > +\n> > + /*\n> > + * Worker should send either the full result struct on success, or\n> > + * just the base (i.e. no stat data), otherwise.\n> > + */\n> > + if (res->status == PC_ITEM_WRITTEN) {\n> > + ASSERT_PC_ITEM_RESULT_SIZE(len, (int)sizeof(struct pc_item_result));\n> > + st = &res->st;\n> > + } else {\n> > + ASSERT_PC_ITEM_RESULT_SIZE(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n> > + }\n> > +\n> > + if (!worker->nr_items_to_complete || res->id != worker->next_item_to_complete)\n>\n> Nit: maybe it could be useful to distinguish between these 2 potential\n> bugs and have a specific BUG() for each one.\n\nRight, will do.\n\n> > +static void gather_results_from_workers(struct pc_worker *workers,\n> > + int num_workers)\n> > +{\n> > + int i, active_workers = num_workers;\n> > + struct pollfd *pfds;\n> > +\n> > + CALLOC_ARRAY(pfds, num_workers);\n> > + for (i = 0; i < num_workers; i++) {\n> > + pfds[i].fd = workers[i].cp.out;\n> > + pfds[i].events = POLLIN;\n> > + }\n> > +\n> > + while (active_workers) {\n> > + int nr = poll(pfds, num_workers, -1);\n> > +\n> > + if (nr < 0) {\n> > + if (errno == EINTR)\n> > + continue;\n> > + die_errno(\"failed to poll checkout workers\");\n> > + }\n> > +\n> > + for (i = 0; i < num_workers && nr > 0; i++) {\n>\n> Is it possible that nr is 0? If that happens, it looks like we would\n> be in an infinite `while (active_workers) { ... }` loop.\n>\n> Actually in poll(2) there is: \"A value of 0 indicates that the call\n> timed out and no file descriptors were ready.\" So it seems that it\n> could, at least theorically, happen.\n\nI think a 0 return might not be possible in this case because we call\npoll() with -1 as the timeout, which means \"infinite timeout\". So the\ncall should block until either an error occurs (negative return code)\nor there is a file descriptor available for reading (positive return\ncode).\n\n> > +enum pc_item_status {\n> > + PC_ITEM_PENDING = 0,\n> > + PC_ITEM_WRITTEN,\n> > + /*\n> > + * The entry could not be written because there was another file\n> > + * already present in its path or leading directories. Since\n> > + * checkout_entry_ca() removes such files from the working tree before\n> > + * enqueueing the entry for parallel checkout, it means that there was\n> > + * a path collision among the entries being written.\n> > + */\n> > + PC_ITEM_COLLIDED,\n> > + PC_ITEM_FAILED,\n> > +};\n> > +\n> > +struct parallel_checkout_item {\n> > + /*\n> > + * In main process ce points to a istate->cache[] entry. Thus, it's not\n> > + * owned by us. In workers they own the memory, which *must be* released.\n> > + */\n> > + struct cache_entry *ce;\n> > + struct conv_attrs ca;\n> > + size_t id; /* position in parallel_checkout.items[] of main process */\n> > +\n> > + /* Output fields, sent from workers. */\n> > + enum pc_item_status status;\n> > + struct stat st;\n> > +};\n>\n> Maybe the previous patch could have declared both 'enum\n> pc_item_status' and 'struct parallel_checkout_item' here, in\n> parallel-checkout.h, so that this patch wouldn't need to move them\n> here.\n\nYeah, while I was writing this patch I went back and forth on whether\nto declare these here from the start. But because I wanted to have the\n\"parallel-checkout users\" / \"checkout--helper interface\" division in\nparallel-checkout.h, I thought it would be better to move the structs\nto this header only after the checkout--helper was introduced.\n"},{"id":"420884","messageId":"CAHd-oW4jhoUz=4XZYC0HZPFTziyC1fw3DKQuR5rzSmmBgxiCCw@mail.gmail.com","threadId":"55342","inReplyTo":"CAP8UFD3s3NUpi2eyPWFa5bL4rez1wNtj5-dUpv8dJLo2CKYu9A@mail.gmail.com","subject":"Re: [PATCH 3/5] parallel-checkout: add configuration options","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-02T14:45:47Z","receivedAt":"2021-04-02T14:46:01Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Mar 31, 2021 at 1:33 AM Christian Couder\n wrote:\n>\n> On Wed, Mar 17, 2021 at 10:12 PM Matheus Tavares\n> wrote:\n>\n> > The above benchmarks show that parallel checkout is most effective on\n> > repositories located on an SSD or over a distributed file system. For\n> > local file systems on spinning disks, and/or older machines, the\n> > parallelism does not always bring a good performance. For this reason,\n> > the default value for checkout.workers is one, a.k.a. sequential\n> > checkout.\n>\n> I wonder how many people are still using HDD, and how many will still\n> use them in a few years.\n>\n> I think having 1 as the default value for checkout.workers might be\n> good for a while for backward compatibility and stability, while\n> people who are interested can test parallel checkout on different\n> environments. But we might want, in a few releases, after some bugs,\n> if any, have been fixed, to use a default, maybe 10, that will provide\n> significant speedup for most people, and will justify the added\n> complexity, especially as your numbers still show a small speedup for\n> HDD when using 10.\n\nYeah, I agree that it would be nice to have a better default in the\nnear future. Unfortunately, on some other HDD machines that I tested,\nparallel checkout was slower than the sequential version. So I think\nit may not be possible to enable parallelism by default now.\n\nNevertheless, I was also experimenting with some ideas to auto-detect\nif parallelism is efficient in a given environment/repo and\nauto-enable it if so. One interesting possibility suggested by Ævar\n[1] was to implement this as a git maintenance task, which could\nperiodically probe the system and tune the checkout settings\nappropriately.\n\n[1]: https://lore.kernel.org/git/87y2ixpvos.fsf@evledraar.gmail.com/\n\n> > @@ -23,6 +25,19 @@ enum pc_status parallel_checkout_status(void)\n> > return parallel_checkout.status;\n> > }\n> >\n> > +#define DEFAULT_THRESHOLD_FOR_PARALLELISM 100\n>\n> Using a \"static const int\" might be a bit better.\n\nOk, I will change that.\n\n> > +void get_parallel_checkout_configs(int *num_workers, int *threshold)\n> > +{\n> > + if (git_config_get_int(\"checkout.workers\", num_workers))\n> > + *num_workers = 1;\n>\n> I think it's better when an important default value like this \"1\" is\n> made more visible using a \"static const int\" or a \"#define\".\n\nWill do.\n\n> > @@ -568,7 +581,7 @@ int run_parallel_checkout(struct checkout *state)\n> > if (parallel_checkout.nr < num_workers)\n> > num_workers = parallel_checkout.nr;\n> >\n> > - if (num_workers <= 1) {\n> > + if (num_workers <= 1 || parallel_checkout.nr < threshold) {\n>\n> Here we check the number of workers...\n>\n> > write_items_sequentially(state);\n> > } else {\n> > struct pc_worker *workers = setup_workers(state, num_workers);\n>\n> [...]\n>\n> > @@ -480,7 +483,8 @@ static int check_updates(struct unpack_trees_options *o,\n> > }\n> > }\n> > stop_progress(&progress);\n> > - errs |= run_parallel_checkout(&state);\n> > + if (pc_workers > 1)\n> > + errs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n\n> ...but the number of workers was already checked here.\n\nThe re-checking at run_parallel_checkout() is important because\n`num_workers` might actually become <= 1 after the above check at\ncheck_updates(). This happens when there aren't enough enqueued\nentries for 2+ workers, so we fall back to sequential checkout. It\nalso kind of works as a safe-mechanism for the case where the\nrun_parallel_checkout() caller forgot to check if the user actually\nwants parallelism before calling the function.\n"},{"id":"421236","messageId":"cover.1617893234.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1616015337.git.matheus.bernardino@usp.br","subject":"[PATCH v2 0/5] Parallel Checkout (part 2)","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-08T16:16:59Z","receivedAt":"2021-04-08T16:17:15Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This is the next step in the parallel checkout implementation. As\nmt/parallel-checkout-part-1 is now on master, this round is based\ndirectly on master.\n\nChanges since v1:\n\nPatch 1:\n- Fixed misleading sentence in commit message which said that\n parallel-checkout can smudge entries in parallel (it can only run\n internal filters in parallel).\n- On write_pc_item_to_fd(), moved `newsize` declaration down to the\n scope where it is actually used and removed the initialization. \n- Also on write_pc_item_to_fd(), rephrased error message on\n read_blob_entry() to avoid mentioning 'sha1' and 'object file', which\n might not be the case there.\n- Fixed typo at handle_results(): s/it's/its/.\n- Added comment to 'struct parallel_checkout' to clarify what data we\n later refer as the \"parallel checkout queue\".\n- Quoted paths in error messages.\n\nPatch 2:\n- Renamed checkout--helper to checkout--worker.\n- Used packet_read() instead of packet_line_read() in both main process\n and workers. packet_line_read() is not suitable for binary data as it\n chomps a trailing newline (which could actually be part of the\n payload).\n- Added check for checkout items that cannot be sent in one single\n pkt-line. (Such items are very unlikely to appear in practice, but we\n cannot ignore them.)\n- Replaced macro that checks the pc_item_result size received from\n workers with an inline function.\n- At parse_and_save_result(), separated one BUG() that guards two\n different conditions into two BUG()s, to help debugging.\n- Suppressed error messages for finish_command() errors that don't\n come from a signal death. Such errors are already reported by the\n worker itself, so there is no need to repeat ourselves.\n- Ignored SIGPIPE when writing to workers so that we can get an EPIPE\n in the pkt-line machinery and die() with an error message instead of\n nothing.\n\nPatch 3:\n- Used `static const int` instead of #define for the default parallel\n checkout config values.\n\nPatch 5:\n- Fixed typos in design doc.\n- Clarified which types of filters can be applied in parallel.\n- Improved wording on the 'sequential implementation' section. In\n particular, avoided using the term \"layers\" to refer to the sub-steps.\n- Clarified sentence about the streaming interface and how it affects\n some of the steps performed by the workers.\n- Clarified that the main process only removes files with --force.\n\nAdditionally, the error() and die() messages from the previous round\nwere mostly not marked for translation, but there were 3 strings that\nwere accidentally marked. I'm not sure whether parallel-checkout\nmessages should be translated or not, since it's a low-level machinery\nthat can be used by both porcelain and plumbing commands. But because\nentry.c's messages are not localized, and parallel-checkout is kind of\nan extension of it, I decided to remove the three last _() occurrences\nin this round.\n\n\nMatheus Tavares (5):\n unpack-trees: add basic support for parallel checkout\n parallel-checkout: make it truly parallel\n parallel-checkout: add configuration options\n parallel-checkout: support progress displaying\n parallel-checkout: add design documentation\n\n .gitignore | 1 +\n Documentation/Makefile | 1 +\n Documentation/config/checkout.txt | 21 +\n Documentation/technical/parallel-checkout.txt | 268 +++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/checkout--worker.c | 145 ++++\n entry.c | 17 +-\n git.c | 2 +\n parallel-checkout.c | 658 ++++++++++++++++++\n parallel-checkout.h | 111 +++\n unpack-trees.c | 19 +-\n 12 files changed, 1241 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/parallel-checkout.txt\n create mode 100644 builtin/checkout--worker.c\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\nRange-diff against v1:\n1: a3383042e2 ! 1: 0506ac7159 unpack-trees: add basic support for parallel checkout\n @@ Commit message\n unpack-trees: add basic support for parallel checkout\n \n This new interface allows us to enqueue some of the entries being\n - checked out to later uncompress, smudge, and write them in parallel. For\n - now, the parallel checkout machinery is enabled by default and there is\n - no user configuration, but run_parallel_checkout() just writes the\n - queued entries in sequence (without spawning additional workers). The\n - next patch will actually implement the parallelism and, later, we will\n - make it configurable.\n + checked out to later uncompress them, apply in-process filters, and\n + write out the files in parallel. For now, the parallel checkout\n + machinery is enabled by default and there is no user configuration, but\n + run_parallel_checkout() just writes the queued entries in sequence\n + (without spawning additional workers). The next patch will actually\n + implement the parallelism and, later, we will make it configurable.\n \n Note that, to avoid potential data races, not all entries are eligible\n for parallel checkout. Also, paths that collide on disk (e.g.\n @@ parallel-checkout.c (new)\n +\n +struct parallel_checkout {\n +\tenum pc_status status;\n -+\tstruct parallel_checkout_item *items;\n ++\tstruct parallel_checkout_item *items; /* The parallel checkout queue. */\n +\tsize_t nr, alloc;\n +};\n +\n @@ parallel-checkout.c (new)\n +\t\t\t *\n +\t\t\t * Instead, let's pay the overhead only once, now, and\n +\t\t\t * call checkout_entry_ca() again for this file, to\n -+\t\t\t * have it's stat() data stored in the index. This also\n ++\t\t\t * have its stat() data stored in the index. This also\n +\t\t\t * has the benefit of adding this entry and its\n +\t\t\t * colliding pair to the collision report message.\n +\t\t\t * Additionally, this overwriting behavior is consistent\n @@ parallel-checkout.c (new)\n +\t}\n +\n +\tif (have_pending)\n -+\t\terror(_(\"parallel checkout finished with pending entries\"));\n ++\t\terror(\"parallel checkout finished with pending entries\");\n +\n +\treturn ret;\n +}\n @@ parallel-checkout.c (new)\n +static int reset_fd(int fd, const char *path)\n +{\n +\tif (lseek(fd, 0, SEEK_SET) != 0)\n -+\t\treturn error_errno(\"failed to rewind descriptor of %s\", path);\n ++\t\treturn error_errno(\"failed to rewind descriptor of '%s'\", path);\n +\tif (ftruncate(fd, 0))\n -+\t\treturn error_errno(\"failed to truncate file %s\", path);\n ++\t\treturn error_errno(\"failed to truncate file '%s'\", path);\n +\treturn 0;\n +}\n +\n @@ parallel-checkout.c (new)\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tchar *new_blob;\n +\tunsigned long size;\n -+\tsize_t newsize = 0;\n +\tssize_t wrote;\n +\n +\t/* Sanity check */\n @@ parallel-checkout.c (new)\n +\n +\tnew_blob = read_blob_entry(pc_item->ce, &size);\n +\tif (!new_blob)\n -+\t\treturn error(\"unable to read sha1 file of %s (%s)\", path,\n -+\t\t\t oid_to_hex(&pc_item->ce->oid));\n ++\t\treturn error(\"cannot read object %s '%s'\",\n ++\t\t\t oid_to_hex(&pc_item->ce->oid), path);\n +\n +\t/*\n +\t * checkout metadata is used to give context for external process\n @@ parallel-checkout.c (new)\n +\t\t\t\t\t new_blob, size, &buf, NULL);\n +\n +\tif (ret) {\n ++\t\tsize_t newsize;\n +\t\tfree(new_blob);\n +\t\tnew_blob = strbuf_detach(&buf, &newsize);\n +\t\tsize = newsize;\n @@ parallel-checkout.c (new)\n +\twrote = write_in_full(fd, new_blob, size);\n +\tfree(new_blob);\n +\tif (wrote < 0)\n -+\t\treturn error(\"unable to write file %s\", path);\n ++\t\treturn error(\"unable to write file '%s'\", path);\n +\n +\treturn 0;\n +}\n @@ parallel-checkout.c (new)\n +\t\t\t */\n +\t\t\tpc_item->status = PC_ITEM_COLLIDED;\n +\t\t} else {\n -+\t\t\terror_errno(\"failed to open file %s\", path.buf);\n ++\t\t\terror_errno(\"failed to open file '%s'\", path.buf);\n +\t\t\tpc_item->status = PC_ITEM_FAILED;\n +\t\t}\n +\t\tgoto out;\n @@ parallel-checkout.c (new)\n +\tfstat_done = fstat_checkout_output(fd, state, &pc_item->st);\n +\n +\tif (close_and_clear(&fd)) {\n -+\t\terror_errno(\"unable to close file %s\", path.buf);\n ++\t\terror_errno(\"unable to close file '%s'\", path.buf);\n +\t\tpc_item->status = PC_ITEM_FAILED;\n +\t\tgoto out;\n +\t}\n +\n +\tif (state->refresh_cache && !fstat_done && lstat(path.buf, &pc_item->st) < 0) {\n -+\t\terror_errno(\"unable to stat just-written file %s\", path.buf);\n ++\t\terror_errno(\"unable to stat just-written file '%s'\", path.buf);\n +\t\tpc_item->status = PC_ITEM_FAILED;\n +\t\tgoto out;\n +\t}\n2: fca797698c ! 2: 0d65b517bd parallel-checkout: make it truly parallel\n @@ .gitignore\n /git-check-mailmap\n /git-check-ref-format\n /git-checkout\n -+/git-checkout--helper\n ++/git-checkout--worker\n /git-checkout-index\n /git-cherry\n /git-cherry-pick\n @@ Makefile: BUILTIN_OBJS += builtin/check-attr.o\n BUILTIN_OBJS += builtin/check-ignore.o\n BUILTIN_OBJS += builtin/check-mailmap.o\n BUILTIN_OBJS += builtin/check-ref-format.o\n -+BUILTIN_OBJS += builtin/checkout--helper.o\n ++BUILTIN_OBJS += builtin/checkout--worker.o\n BUILTIN_OBJS += builtin/checkout-index.o\n BUILTIN_OBJS += builtin/checkout.o\n BUILTIN_OBJS += builtin/clean.o\n @@ builtin.h: int cmd_bugreport(int argc, const char **argv, const char *prefix);\n int cmd_bundle(int argc, const char **argv, const char *prefix);\n int cmd_cat_file(int argc, const char **argv, const char *prefix);\n int cmd_checkout(int argc, const char **argv, const char *prefix);\n -+int cmd_checkout__helper(int argc, const char **argv, const char *prefix);\n ++int cmd_checkout__worker(int argc, const char **argv, const char *prefix);\n int cmd_checkout_index(int argc, const char **argv, const char *prefix);\n int cmd_check_attr(int argc, const char **argv, const char *prefix);\n int cmd_check_ignore(int argc, const char **argv, const char *prefix);\n \n - ## builtin/checkout--helper.c (new) ##\n + ## builtin/checkout--worker.c (new) ##\n @@\n +#include \"builtin.h\"\n +#include \"config.h\"\n @@ builtin/checkout--helper.c (new)\n +#include \"parse-options.h\"\n +#include \"pkt-line.h\"\n +\n -+static void packet_to_pc_item(char *line, int len,\n ++static void packet_to_pc_item(const char *buffer, int len,\n +\t\t\t struct parallel_checkout_item *pc_item)\n +{\n -+\tstruct pc_item_fixed_portion *fixed_portion;\n -+\tchar *encoding, *variant;\n ++\tconst struct pc_item_fixed_portion *fixed_portion;\n ++\tconst char *variant;\n ++\tchar *encoding;\n +\n +\tif (len < sizeof(struct pc_item_fixed_portion))\n +\t\tBUG(\"checkout worker received too short item (got %dB, exp %dB)\",\n +\t\t len, (int)sizeof(struct pc_item_fixed_portion));\n +\n -+\tfixed_portion = (struct pc_item_fixed_portion *)line;\n ++\tfixed_portion = (struct pc_item_fixed_portion *)buffer;\n +\n +\tif (len - sizeof(struct pc_item_fixed_portion) !=\n +\t\tfixed_portion->name_len + fixed_portion->working_tree_encoding_len)\n +\t\tBUG(\"checkout worker received corrupted item\");\n +\n -+\tvariant = line + sizeof(struct pc_item_fixed_portion);\n ++\tvariant = buffer + sizeof(struct pc_item_fixed_portion);\n +\n +\t/*\n +\t * Note: the main process uses zero length to communicate that the\n @@ builtin/checkout--helper.c (new)\n +\tsize_t i, nr = 0, alloc = 0;\n +\n +\twhile (1) {\n -+\t\tint len;\n -+\t\tchar *line = packet_read_line(0, &len);\n ++\t\tint len = packet_read(0, NULL, NULL, packet_buffer,\n ++\t\t\t\t sizeof(packet_buffer), 0);\n +\n -+\t\tif (!line)\n ++\t\tif (len < 0)\n ++\t\t\tBUG(\"packet_read() returned negative value\");\n ++\t\telse if (!len)\n +\t\t\tbreak;\n +\n +\t\tALLOC_GROW(items, nr + 1, alloc);\n -+\t\tpacket_to_pc_item(line, len, &items[nr++]);\n ++\t\tpacket_to_pc_item(packet_buffer, len, &items[nr++]);\n +\t}\n +\n +\tfor (i = 0; i < nr; i++) {\n @@ builtin/checkout--helper.c (new)\n +\tfree(items);\n +}\n +\n -+static const char * const checkout_helper_usage[] = {\n -+\tN_(\"git checkout--helper []\"),\n ++static const char * const checkout_worker_usage[] = {\n ++\tN_(\"git checkout--worker []\"),\n +\tNULL\n +};\n +\n -+int cmd_checkout__helper(int argc, const char **argv, const char *prefix)\n ++int cmd_checkout__worker(int argc, const char **argv, const char *prefix)\n +{\n +\tstruct checkout state = CHECKOUT_INIT;\n -+\tstruct option checkout_helper_options[] = {\n ++\tstruct option checkout_worker_options[] = {\n +\t\tOPT_STRING(0, \"prefix\", &state.base_dir, N_(\"string\"),\n +\t\t\tN_(\"when creating files, prepend \")),\n +\t\tOPT_END()\n +\t};\n +\n +\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n -+\t\tusage_with_options(checkout_helper_usage,\n -+\t\t\t\t checkout_helper_options);\n ++\t\tusage_with_options(checkout_worker_usage,\n ++\t\t\t\t checkout_worker_options);\n +\n +\tgit_config(git_default_config, NULL);\n -+\targc = parse_options(argc, argv, prefix, checkout_helper_options,\n -+\t\t\t checkout_helper_usage, 0);\n ++\targc = parse_options(argc, argv, prefix, checkout_worker_options,\n ++\t\t\t checkout_worker_usage, 0);\n +\tif (argc > 0)\n -+\t\tusage_with_options(checkout_helper_usage, checkout_helper_options);\n ++\t\tusage_with_options(checkout_worker_usage, checkout_worker_options);\n +\n +\tif (state.base_dir)\n +\t\tstate.base_dir_len = strlen(state.base_dir);\n @@ git.c: static struct cmd_struct commands[] = {\n \t{ \"check-mailmap\", cmd_check_mailmap, RUN_SETUP },\n \t{ \"check-ref-format\", cmd_check_ref_format, NO_PARSEOPT },\n \t{ \"checkout\", cmd_checkout, RUN_SETUP | NEED_WORK_TREE },\n -+\t{ \"checkout--helper\", cmd_checkout__helper,\n ++\t{ \"checkout--worker\", cmd_checkout__worker,\n +\t\tRUN_SETUP | NEED_WORK_TREE | SUPPORT_SUPER_PREFIX },\n \t{ \"checkout-index\", cmd_checkout_index,\n \t\tRUN_SETUP | NEED_WORK_TREE},\n @@ parallel-checkout.c\n #include \"parallel-checkout.h\"\n +#include \"pkt-line.h\"\n +#include \"run-command.h\"\n ++#include \"sigchain.h\"\n #include \"streaming.h\"\n \n -enum pc_item_status {\n @@ parallel-checkout.c\n };\n \n struct parallel_checkout {\n +@@ parallel-checkout.c: static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n + \t\t\t\t\t const struct conv_attrs *ca)\n + {\n + \tenum conv_attrs_classification c;\n ++\tsize_t packed_item_size;\n + \n + \t/*\n + \t * Symlinks cannot be checked out in parallel as, in case of path\n +@@ parallel-checkout.c: static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n + \tif (!S_ISREG(ce->ce_mode))\n + \t\treturn 0;\n + \n ++\tpacked_item_size = sizeof(struct pc_item_fixed_portion) + ce->ce_namelen +\n ++\t\t(ca->working_tree_encoding ? strlen(ca->working_tree_encoding) : 0);\n ++\n ++\t/*\n ++\t * The amount of data we send to the workers per checkout item is\n ++\t * typically small (75~300B). So unless we find an insanely huge path\n ++\t * of 64KB, we should never reach the 65KB limit of one pkt-line. If\n ++\t * that does happen, we let the sequential code handle the item.\n ++\t */\n ++\tif (packed_item_size > LARGE_PACKET_DATA_MAX)\n ++\t\treturn 0;\n ++\n + \tc = classify_conv_attrs(ca);\n + \tswitch (c) {\n + \tcase CA_CLASS_INCORE:\n @@ parallel-checkout.c: int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n \t\t parallel_checkout.alloc);\n @@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc\n +\tsize_t working_tree_encoding_len = working_tree_encoding ?\n +\t\t\t\t\t strlen(working_tree_encoding) : 0;\n +\n ++\t/*\n ++\t * Any changes in the calculation of the message size must also be made\n ++\t * in is_eligible_for_parallel_checkout().\n ++\t */\n +\tlen_data = sizeof(struct pc_item_fixed_portion) + name_len +\n +\t\t working_tree_encoding_len;\n +\n @@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc\n +static void send_batch(int fd, size_t start, size_t nr)\n +{\n +\tsize_t i;\n ++\tsigchain_push(SIGPIPE, SIG_IGN);\n +\tfor (i = 0; i < nr; i++)\n +\t\tsend_one_item(fd, ¶llel_checkout.items[start + i]);\n +\tpacket_flush(fd);\n ++\tsigchain_pop(SIGPIPE);\n +}\n +\n +static struct pc_worker *setup_workers(struct checkout *state, int num_workers)\n @@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc\n +\t\tcp->in = -1;\n +\t\tcp->out = -1;\n +\t\tcp->clean_on_exit = 1;\n -+\t\tstrvec_push(&cp->args, \"checkout--helper\");\n ++\t\tstrvec_push(&cp->args, \"checkout--worker\");\n +\t\tif (state->base_dir_len)\n +\t\t\tstrvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n +\t\tif (start_command(cp))\n -+\t\t\tdie(_(\"failed to spawn checkout worker\"));\n ++\t\t\tdie(\"failed to spawn checkout worker\");\n +\t}\n +\n +\tbase_batch_size = parallel_checkout.nr / num_workers;\n @@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc\n +\t}\n +\n +\tfor (i = 0; i < num_workers; i++) {\n -+\t\tif (finish_command(&workers[i].cp))\n -+\t\t\terror(_(\"checkout worker %d finished with error\"), i);\n ++\t\tint rc = finish_command(&workers[i].cp);\n ++\t\tif (rc > 128) {\n ++\t\t\t/*\n ++\t\t\t * For a normal non-zero exit, the worker should have\n ++\t\t\t * already printed something useful to stderr. But a\n ++\t\t\t * death by signal should be mentioned to the user.\n ++\t\t\t */\n ++\t\t\terror(\"checkout worker %d died of signal %d\", i, rc - 128);\n ++\t\t}\n +\t}\n +\n +\tfree(workers);\n +}\n +\n -+#define ASSERT_PC_ITEM_RESULT_SIZE(got, exp) \\\n -+{ \\\n -+\tif (got != exp) \\\n -+\t\tBUG(\"corrupted result from checkout worker (got %dB, exp %dB)\", \\\n -+\t\t got, exp); \\\n -+} while(0)\n ++static inline void assert_pc_item_result_size(int got, int exp)\n ++{\n ++\tif (got != exp)\n ++\t\tBUG(\"wrong result size from checkout worker (got %dB, exp %dB)\",\n ++\t\t got, exp);\n ++}\n +\n -+static void parse_and_save_result(const char *line, int len,\n ++static void parse_and_save_result(const char *buffer, int len,\n +\t\t\t\t struct pc_worker *worker)\n +{\n +\tstruct pc_item_result *res;\n @@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc\n +\tstruct stat *st = NULL;\n +\n +\tif (len < PC_ITEM_RESULT_BASE_SIZE)\n -+\t\tBUG(\"too short result from checkout worker (got %dB, exp %dB)\",\n ++\t\tBUG(\"too short result from checkout worker (got %dB, exp >=%dB)\",\n +\t\t len, (int)PC_ITEM_RESULT_BASE_SIZE);\n +\n -+\tres = (struct pc_item_result *)line;\n ++\tres = (struct pc_item_result *)buffer;\n +\n +\t/*\n +\t * Worker should send either the full result struct on success, or\n +\t * just the base (i.e. no stat data), otherwise.\n +\t */\n +\tif (res->status == PC_ITEM_WRITTEN) {\n -+\t\tASSERT_PC_ITEM_RESULT_SIZE(len, (int)sizeof(struct pc_item_result));\n ++\t\tassert_pc_item_result_size(len, (int)sizeof(struct pc_item_result));\n +\t\tst = &res->st;\n +\t} else {\n -+\t\tASSERT_PC_ITEM_RESULT_SIZE(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n ++\t\tassert_pc_item_result_size(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n +\t}\n +\n -+\tif (!worker->nr_items_to_complete || res->id != worker->next_item_to_complete)\n -+\t\tBUG(\"checkout worker sent unexpected item id\");\n ++\tif (!worker->nr_items_to_complete)\n ++\t\tBUG(\"received result from supposedly finished checkout worker\");\n ++\tif (res->id != worker->next_item_to_complete)\n ++\t\tBUG(\"unexpected item id from checkout worker (got %\"PRIuMAX\", exp %\"PRIuMAX\")\",\n ++\t\t (uintmax_t)res->id, (uintmax_t)worker->next_item_to_complete);\n +\n +\tworker->next_item_to_complete++;\n +\tworker->nr_items_to_complete--;\n @@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc\n +\t\tpc_item->st = *st;\n +}\n +\n -+\n +static void gather_results_from_workers(struct pc_worker *workers,\n +\t\t\t\t\tint num_workers)\n +{\n @@ parallel-checkout.c: static void write_pc_item(struct parallel_checkout_item *pc\n +\t\t\t\tcontinue;\n +\n +\t\t\tif (pfd->revents & POLLIN) {\n -+\t\t\t\tint len;\n -+\t\t\t\tconst char *line = packet_read_line(pfd->fd, &len);\n ++\t\t\t\tint len = packet_read(pfd->fd, NULL, NULL,\n ++\t\t\t\t\t\t packet_buffer,\n ++\t\t\t\t\t\t sizeof(packet_buffer), 0);\n +\n -+\t\t\t\tif (!line) {\n ++\t\t\t\tif (len < 0) {\n ++\t\t\t\t\tBUG(\"packet_read() returned negative value\");\n ++\t\t\t\t} else if (!len) {\n +\t\t\t\t\tpfd->fd = -1;\n +\t\t\t\t\tactive_workers--;\n +\t\t\t\t} else {\n -+\t\t\t\t\tparse_and_save_result(line, len, worker);\n ++\t\t\t\t\tparse_and_save_result(packet_buffer,\n ++\t\t\t\t\t\t\t len, worker);\n +\t\t\t\t}\n +\t\t\t} else if (pfd->revents & POLLHUP) {\n +\t\t\t\tpfd->fd = -1;\n +\t\t\t\tactive_workers--;\n +\t\t\t} else if (pfd->revents & (POLLNVAL | POLLERR)) {\n -+\t\t\t\tdie(_(\"error polling from checkout worker\"));\n ++\t\t\t\tdie(\"error polling from checkout worker\");\n +\t\t\t}\n +\n +\t\t\tnr--;\n @@ parallel-checkout.c: static void write_items_sequentially(struct checkout *state\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n -+#define DEFAULT_NUM_WORKERS 2\n ++static const int DEFAULT_NUM_WORKERS = 2;\n +\n int run_parallel_checkout(struct checkout *state)\n {\n @@ parallel-checkout.h: int enqueue_checkout(struct cache_entry *ce, struct conv_at\n int run_parallel_checkout(struct checkout *state);\n \n +/****************************************************************\n -+ * Interface with checkout--helper\n ++ * Interface with checkout--worker\n + ****************************************************************/\n +\n +enum pc_item_status {\n3: 8c83e92445 ! 3: 6ea057f9c5 parallel-checkout: add configuration options\n @@ Commit message\n checkout showed to be beneficial. This time, we compared the runtime of\n a `git checkout -f`, with and without parallelism, after randomly\n removing an increasing number of files from the Linux working tree. The\n - \"sequential fallback\" column bellow corresponds to the executions where\n + \"sequential fallback\" column below corresponds to the executions where\n checkout.workers was 10 but checkout.thresholdForParallelism was equal\n to the number of to-be-updated files plus one (so that we end up writing\n sequentially). Each test case was sampled 15 times, and each sample had\n @@ parallel-checkout.c\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n + #include \"sigchain.h\"\n #include \"streaming.h\"\n +#include \"thread-utils.h\"\n \n @@ parallel-checkout.c: enum pc_status parallel_checkout_status(void)\n \treturn parallel_checkout.status;\n }\n \n -+#define DEFAULT_THRESHOLD_FOR_PARALLELISM 100\n ++static const int DEFAULT_THRESHOLD_FOR_PARALLELISM = 100;\n ++static const int DEFAULT_NUM_WORKERS = 1;\n +\n +void get_parallel_checkout_configs(int *num_workers, int *threshold)\n +{\n +\tif (git_config_get_int(\"checkout.workers\", num_workers))\n -+\t\t*num_workers = 1;\n ++\t\t*num_workers = DEFAULT_NUM_WORKERS;\n +\telse if (*num_workers < 1)\n +\t\t*num_workers = online_cpus();\n +\n @@ parallel-checkout.c: static void write_items_sequentially(struct checkout *state\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n --#define DEFAULT_NUM_WORKERS 2\n +-static const int DEFAULT_NUM_WORKERS = 2;\n -\n -int run_parallel_checkout(struct checkout *state)\n +int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n @@ parallel-checkout.h: void init_parallel_checkout(void);\n +int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n \n /****************************************************************\n - * Interface with checkout--helper\n + * Interface with checkout--worker\n \n ## unpack-trees.c ##\n @@ unpack-trees.c: static int check_updates(struct unpack_trees_options *o,\n4: f7432e15dd ! 4: 0ac4bee67e parallel-checkout: support progress displaying\n @@ parallel-checkout.c\n #include \"pkt-line.h\"\n +#include \"progress.h\"\n #include \"run-command.h\"\n + #include \"sigchain.h\"\n #include \"streaming.h\"\n - #include \"thread-utils.h\"\n @@ parallel-checkout.c: struct parallel_checkout {\n \tenum pc_status status;\n - \tstruct parallel_checkout_item *items;\n + \tstruct parallel_checkout_item *items; /* The parallel checkout queue. */\n \tsize_t nr, alloc;\n +\tstruct progress *progress;\n +\tunsigned int *progress_cnt;\n @@ parallel-checkout.c: static int handle_results(struct checkout *state)\n \t\t\tbreak;\n \t\tcase PC_ITEM_PENDING:\n \t\t\thave_pending = 1;\n -@@ parallel-checkout.c: static void parse_and_save_result(const char *line, int len,\n +@@ parallel-checkout.c: static void parse_and_save_result(const char *buffer, int len,\n \tpc_item->status = res->status;\n \tif (st)\n \t\tpc_item->st = *st;\n @@ parallel-checkout.c: static void parse_and_save_result(const char *line, int len\n +\t\tadvance_progress_meter();\n }\n \n - \n + static void gather_results_from_workers(struct pc_worker *workers,\n @@ parallel-checkout.c: static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n @@ parallel-checkout.h: void init_parallel_checkout(void);\n +\t\t\t struct progress *progress, unsigned int *progress_cnt);\n \n /****************************************************************\n - * Interface with checkout--helper\n + * Interface with checkout--worker\n \n ## unpack-trees.c ##\n @@ unpack-trees.c: static int check_updates(struct unpack_trees_options *o,\n5: 0592740ec1 ! 5: 087f8bdf35 parallel-checkout: add design documentation\n @@ Documentation/technical/parallel-checkout.txt (new)\n +==============================\n +\n +The \"Parallel Checkout\" feature attempts to use multiple processes to\n -+parallelize the work of uncompressing, smudging, and writing blobs\n -+during a checkout operation. It can be used by all checkout-related\n -+commands, such as `clone`, `checkout`, `reset`, `sparse-checkout`, and\n -+others.\n ++parallelize the work of uncompressing the blobs, applying in-core\n ++filters, and writing the resulting contents to the working tree during a\n ++checkout operation. It can be used by all checkout-related commands,\n ++such as `clone`, `checkout`, `reset`, `sparse-checkout`, and others.\n +\n +These commands share the following basic structure:\n +\n @@ Documentation/technical/parallel-checkout.txt (new)\n +-------------------------\n +\n +For the purposes of discussion here, the current sequential\n -+implementation of Step 3 has 3 layers:\n ++implementation of Step 3 is divided in 3 parts, each one implemented in\n ++its own function:\n +\n +* Step 3a: `unpack-trees.c:check_updates()` contains a series of\n + sequential loops iterating over the `cache_entry`'s array. The main\n -+ loop in this function calls the next layer for each of the\n ++ loop in this function calls the Step 3b function for each of the\n + to-be-updated entries.\n +\n +* Step 3b: `entry.c:checkout_entry()` examines the existing working tree\n + for file conflicts, collisions, and unsaved changes. It removes files\n -+ and create leading directories as necessary. It calls the next layer\n -+ for each entry to be written.\n ++ and creates leading directories as necessary. It calls the Step 3c\n ++ function for each entry to be written.\n +\n +* Step 3c: `entry.c:write_entry()` loads the blob into memory, smudges\n + it if necessary, creates the file in the working tree, writes the\n @@ Documentation/technical/parallel-checkout.txt (new)\n +----------------------\n +\n +Parallel checkout alters the aforementioned Step 3 to use multiple\n -+`checkout--helper` background processes to distribute the work. The\n ++`checkout--worker` background processes to distribute the work. The\n +long-running worker processes are controlled by the foreground Git\n +command using the existing run-command API.\n +\n @@ Documentation/technical/parallel-checkout.txt (new)\n +~~~~~~~~\n +\n +Step 3b is only slightly altered; for each entry to be checked out, the\n -+main process:\n ++main process performs the following steps:\n +\n -+* M1: Checks whether there is any untracked or unclean file in the\n -+ working tree which would be overwritten by this entry, and decides\n ++* M1: Check whether there is any untracked or unclean file in the\n ++ working tree which would be overwritten by this entry, and decide\n + whether to proceed (removing the file(s)) or not.\n +\n -+* M2: Creates the leading directories.\n ++* M2: Create the leading directories.\n +\n -+* M3: Loads the conversion attributes for the entry's path.\n ++* M3: Load the conversion attributes for the entry's path.\n +\n -+* M4: Checks, based on the entry's type and conversion attributes,\n ++* M4: Check, based on the entry's type and conversion attributes,\n + whether the entry is eligible for parallel checkout (more on this\n -+ later). If it is eligible, enqueues the entry and the loaded\n -+ attributes to later write the entry in parallel. If not, writes the\n ++ later). If it is eligible, enqueue the entry and the loaded\n ++ attributes to later write the entry in parallel. If not, write the\n + entry right away, using the default sequential code.\n +\n +Note: we save the conversion attributes associated with each entry\n @@ Documentation/technical/parallel-checkout.txt (new)\n +* W3: Loads the blob into memory (inflating and delta reconstructing\n + it).\n +\n -+* W4: Filters the blob.\n ++* W4: Applies any required in-process filter, like end-of-line\n ++ conversion and re-encoding.\n +\n +* W5: Writes the result to the file descriptor opened at W2.\n +\n @@ Documentation/technical/parallel-checkout.txt (new)\n + the result back to the main process, together with the end status of\n + the operation and the item's identification number.\n +\n -+Note that steps W3 to W5 might actually be performed together, using the\n -+streaming interface.\n ++Note that, when possible, steps W3 to W5 are delegated to the streaming\n ++machinery, removing the need to keep the entire blob in memory.\n +\n -+Also note that the workers *never* remove any files. As mentioned\n -+earlier, it is the responsibility of the main process to remove any\n -+files that block the checkout operation (or abort it). This is crucial\n -+to avoid race conditions and also to properly detect path collisions at\n -+Step W1.\n ++Also note that the workers *never* remove any file. As mentioned\n ++earlier, it is the responsibility of the main process to remove any file\n ++that blocks the checkout operation (or abort if removing the file(s)\n ++would cause data loss and the user didn't ask to `--force`). This is\n ++crucial to avoid race conditions and also to properly detect path\n ++collisions at Step W1.\n +\n +After the workers finish writing the items and sending back the required\n +information, the main process handles the results in two steps:\n @@ Documentation/technical/parallel-checkout.txt (new)\n +deals with files that were already present in the working tree before\n +checkout. Basically, it checks if the path that it wants to write\n +already exists on disk, makes sure the existing file doesn't have\n -+unsaved data, and then overwrite it. (To be more pedantic: it deletes\n ++unsaved data, and then overwrites it. (To be more pedantic: it deletes\n +the existing file and creates the new one.) So, if there are multiple\n +colliding files to be checked out, the sequential code will write each\n +one of them but only the last will actually survive on disk.\n @@ Documentation/technical/parallel-checkout.txt (new)\n +remove all files that prevent this entry from being written (before\n +enqueueing it). This includes any non-directory file in the leading path\n +of the entry. Later, when a worker gets assigned the entry, it looks\n -+again for the non-directories files and for an already existent file at\n ++again for the non-directories files and for an already existing file at\n +the entry's path. If any of these checks finds something, the worker\n +knows that there was a path collision.\n +\n @@ Documentation/technical/parallel-checkout.txt (new)\n +postpone the return of some filtered blobs. The delayed checkout queue\n +and the parallel checkout queue are not compatible and should remain\n +separated.\n +++\n ++Note: regular files that only require internal filters, like end-of-line\n ++conversion and re-encoding, are eligible for parallel checkout.\n +\n +Ineligible entries are checked out by the classic sequential codepath\n +*before* spawning workers.\n +\n +Note: submodules's files are also eligible for parallel checkout (as\n -+long as they don't fall into the two excluding categories mentioned\n ++long as they don't fall into any of the excluding categories mentioned\n +above). But since each submodule is checked out in its own child\n +process, we don't mix the superproject's and the submodules' files in\n +the same parallel checkout process or queue.\n-- \n2.30.1\n\n"},{"id":"421237","messageId":"0506ac7159d3a372d743c10c47e1604689276d4b.1617893234.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1617893234.git.matheus.bernardino@usp.br","subject":"[PATCH v2 1/5] unpack-trees: add basic support for parallel checkout","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-08T16:17:00Z","receivedAt":"2021-04-08T16:17:25Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This new interface allows us to enqueue some of the entries being\nchecked out to later uncompress them, apply in-process filters, and\nwrite out the files in parallel. For now, the parallel checkout\nmachinery is enabled by default and there is no user configuration, but\nrun_parallel_checkout() just writes the queued entries in sequence\n(without spawning additional workers). The next patch will actually\nimplement the parallelism and, later, we will make it configurable.\n\nNote that, to avoid potential data races, not all entries are eligible\nfor parallel checkout. Also, paths that collide on disk (e.g.\ncase-sensitive paths in case-insensitive file systems), are detected by\nthe parallel checkout code and skipped, so that they can be safely\nsequentially handled later. The collision detection works like the\nfollowing:\n\n- If the collision was at basename (e.g. 'a/b' and 'a/B'), the framework\n detects it by looking for EEXIST and EISDIR errors after an\n open(O_CREAT | O_EXCL) failure.\n\n- If the collision was at dirname (e.g. 'a/b' and 'A'), it is detected\n at the has_dirs_only_path() check, which is done for the leading path\n of each item in the parallel checkout queue.\n\nBoth verifications rely on the fact that, before enqueueing an entry for\nparallel checkout, checkout_entry() makes sure that there is no file at\nthe entry's path and that its leading components are all real\ndirectories. So, any later change in these conditions indicates that\nthere was a collision (either between two parallel-eligible entries or\nbetween an eligible and an ineligible one).\n\nAfter all parallel-eligible entries have been processed, the collided\n(and thus, skipped) entries are sequentially fed to checkout_entry()\nagain. This is similar to the way the current code deals with\ncollisions, overwriting the previously checked out entries with the\nsubsequent ones. The only difference is that, since we no longer create\nthe files in the same order that they appear on index, we are not able\nto determine which of the colliding entries will survive on disk (for\nthe classic code, it is always the last entry).\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Makefile | 1 +\n entry.c | 17 +-\n parallel-checkout.c | 368 ++++++++++++++++++++++++++++++++++++++++++++\n parallel-checkout.h | 32 ++++\n unpack-trees.c | 6 +-\n 5 files changed, 421 insertions(+), 3 deletions(-)\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\ndiff --git a/Makefile b/Makefile\nindex a6a73c5741..99734cc2ea 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -946,6 +946,7 @@ LIB_OBJS += pack-revindex.o\n LIB_OBJS += pack-write.o\n LIB_OBJS += packfile.o\n LIB_OBJS += pager.o\n+LIB_OBJS += parallel-checkout.o\n LIB_OBJS += parse-options-cb.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\ndiff --git a/entry.c b/entry.c\nindex 2dc94ba5cc..d7ed38aa40 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -7,6 +7,7 @@\n #include \"progress.h\"\n #include \"fsmonitor.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -426,8 +427,17 @@ static void mark_colliding_entries(const struct checkout *state,\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-\t\tif (dup == ce)\n-\t\t\tbreak;\n+\t\tif (dup == ce) {\n+\t\t\t/*\n+\t\t\t * Parallel checkout doesn't create the files in index\n+\t\t\t * order. So the other side of the collision may appear\n+\t\t\t * after the given cache_entry in the array.\n+\t\t\t */\n+\t\t\tif (parallel_checkout_status() == PC_RUNNING)\n+\t\t\t\tcontinue;\n+\t\t\telse\n+\t\t\t\tbreak;\n+\t\t}\n \n \t\tif (dup->ce_flags & (CE_MATCHED | CE_VALID | CE_SKIP_WORKTREE))\n \t\t\tcontinue;\n@@ -536,6 +546,9 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\tca = &ca_buf;\n \t}\n \n+\tif (!enqueue_checkout(ce, ca))\n+\t\treturn 0;\n+\n \treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nnew file mode 100644\nindex 0000000000..80c839837b\n--- /dev/null\n+++ b/parallel-checkout.c\n@@ -0,0 +1,368 @@\n+#include \"cache.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"streaming.h\"\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/* pointer to a istate->cache[] entry. Not owned by us. */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tstruct stat st;\n+\tenum pc_item_status status;\n+};\n+\n+struct parallel_checkout {\n+\tenum pc_status status;\n+\tstruct parallel_checkout_item *items; /* The parallel checkout queue. */\n+\tsize_t nr, alloc;\n+};\n+\n+static struct parallel_checkout parallel_checkout;\n+\n+enum pc_status parallel_checkout_status(void)\n+{\n+\treturn parallel_checkout.status;\n+}\n+\n+void init_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status != PC_UNINITIALIZED)\n+\t\tBUG(\"parallel checkout already initialized\");\n+\n+\tparallel_checkout.status = PC_ACCEPTING_ENTRIES;\n+}\n+\n+static void finish_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status == PC_UNINITIALIZED)\n+\t\tBUG(\"cannot finish parallel checkout: not initialized yet\");\n+\n+\tfree(parallel_checkout.items);\n+\tmemset(¶llel_checkout, 0, sizeof(parallel_checkout));\n+}\n+\n+static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n+\t\t\t\t\t const struct conv_attrs *ca)\n+{\n+\tenum conv_attrs_classification c;\n+\n+\t/*\n+\t * Symlinks cannot be checked out in parallel as, in case of path\n+\t * collision, they could racily replace leading directories of other\n+\t * entries being checked out. Submodules are checked out in child\n+\t * processes, which have their own parallel checkout queues.\n+\t */\n+\tif (!S_ISREG(ce->ce_mode))\n+\t\treturn 0;\n+\n+\tc = classify_conv_attrs(ca);\n+\tswitch (c) {\n+\tcase CA_CLASS_INCORE:\n+\t\treturn 1;\n+\n+\tcase CA_CLASS_INCORE_FILTER:\n+\t\t/*\n+\t\t * It would be safe to allow concurrent instances of\n+\t\t * single-file smudge filters, like rot13, but we should not\n+\t\t * assume that all filters are parallel-process safe. So we\n+\t\t * don't allow this.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_INCORE_PROCESS:\n+\t\t/*\n+\t\t * The parallel queue and the delayed queue are not compatible,\n+\t\t * so they must be kept completely separated. And we can't tell\n+\t\t * if a long-running process will delay its response without\n+\t\t * actually asking it to perform the filtering. Therefore, this\n+\t\t * type of filter is not allowed in parallel checkout.\n+\t\t *\n+\t\t * Furthermore, there should only be one instance of the\n+\t\t * long-running process filter as we don't know how it is\n+\t\t * managing its own concurrency. So, spreading the entries that\n+\t\t * requisite such a filter among the parallel workers would\n+\t\t * require a lot more inter-process communication. We would\n+\t\t * probably have to designate a single process to interact with\n+\t\t * the filter and send all the necessary data to it, for each\n+\t\t * entry.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_STREAMABLE:\n+\t\treturn 1;\n+\n+\tdefault:\n+\t\tBUG(\"unsupported conv_attrs classification '%d'\", c);\n+\t}\n+}\n+\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n+{\n+\tstruct parallel_checkout_item *pc_item;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES ||\n+\t !is_eligible_for_parallel_checkout(ce, ca))\n+\t\treturn -1;\n+\n+\tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n+\t\t parallel_checkout.alloc);\n+\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item->ce = ce;\n+\tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n+\tpc_item->status = PC_ITEM_PENDING;\n+\n+\treturn 0;\n+}\n+\n+static int handle_results(struct checkout *state)\n+{\n+\tint ret = 0;\n+\tsize_t i;\n+\tint have_pending = 0;\n+\n+\t/*\n+\t * We first update the successfully written entries with the collected\n+\t * stat() data, so that they can be found by mark_colliding_entries(),\n+\t * in the next loop, when necessary.\n+\t */\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\tif (pc_item->status == PC_ITEM_WRITTEN)\n+\t\t\tupdate_ce_after_write(state, pc_item->ce, &pc_item->st);\n+\t}\n+\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\n+\t\tswitch(pc_item->status) {\n+\t\tcase PC_ITEM_WRITTEN:\n+\t\t\t/* Already handled */\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_COLLIDED:\n+\t\t\t/*\n+\t\t\t * The entry could not be checked out due to a path\n+\t\t\t * collision with another entry. Since there can only\n+\t\t\t * be one entry of each colliding group on the disk, we\n+\t\t\t * could skip trying to check out this one and move on.\n+\t\t\t * However, this would leave the unwritten entries with\n+\t\t\t * null stat() fields on the index, which could\n+\t\t\t * potentially slow down subsequent operations that\n+\t\t\t * require refreshing it: git would not be able to\n+\t\t\t * trust st_size and would have to go to the filesystem\n+\t\t\t * to see if the contents match (see ie_modified()).\n+\t\t\t *\n+\t\t\t * Instead, let's pay the overhead only once, now, and\n+\t\t\t * call checkout_entry_ca() again for this file, to\n+\t\t\t * have its stat() data stored in the index. This also\n+\t\t\t * has the benefit of adding this entry and its\n+\t\t\t * colliding pair to the collision report message.\n+\t\t\t * Additionally, this overwriting behavior is consistent\n+\t\t\t * with what the sequential checkout does, so it doesn't\n+\t\t\t * add any extra overhead.\n+\t\t\t */\n+\t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n+\t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_PENDING:\n+\t\t\thave_pending = 1;\n+\t\t\t/* fall through */\n+\t\tcase PC_ITEM_FAILED:\n+\t\t\tret = -1;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unknown checkout item status in parallel checkout\");\n+\t\t}\n+\t}\n+\n+\tif (have_pending)\n+\t\terror(\"parallel checkout finished with pending entries\");\n+\n+\treturn ret;\n+}\n+\n+static int reset_fd(int fd, const char *path)\n+{\n+\tif (lseek(fd, 0, SEEK_SET) != 0)\n+\t\treturn error_errno(\"failed to rewind descriptor of '%s'\", path);\n+\tif (ftruncate(fd, 0))\n+\t\treturn error_errno(\"failed to truncate file '%s'\", path);\n+\treturn 0;\n+}\n+\n+static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n+\t\t\t const char *path)\n+{\n+\tint ret;\n+\tstruct stream_filter *filter;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *new_blob;\n+\tunsigned long size;\n+\tssize_t wrote;\n+\n+\t/* Sanity check */\n+\tassert(is_eligible_for_parallel_checkout(pc_item->ce, &pc_item->ca));\n+\n+\tfilter = get_stream_filter_ca(&pc_item->ca, &pc_item->ce->oid);\n+\tif (filter) {\n+\t\tif (stream_blob_to_fd(fd, &pc_item->ce->oid, filter, 1)) {\n+\t\t\t/* On error, reset fd to try writing without streaming */\n+\t\t\tif (reset_fd(fd, path))\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tnew_blob = read_blob_entry(pc_item->ce, &size);\n+\tif (!new_blob)\n+\t\treturn error(\"cannot read object %s '%s'\",\n+\t\t\t oid_to_hex(&pc_item->ce->oid), path);\n+\n+\t/*\n+\t * checkout metadata is used to give context for external process\n+\t * filters. Files requiring such filters are not eligible for parallel\n+\t * checkout, so pass NULL.\n+\t */\n+\tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n+\t\t\t\t\t new_blob, size, &buf, NULL);\n+\n+\tif (ret) {\n+\t\tsize_t newsize;\n+\t\tfree(new_blob);\n+\t\tnew_blob = strbuf_detach(&buf, &newsize);\n+\t\tsize = newsize;\n+\t}\n+\n+\twrote = write_in_full(fd, new_blob, size);\n+\tfree(new_blob);\n+\tif (wrote < 0)\n+\t\treturn error(\"unable to write file '%s'\", path);\n+\n+\treturn 0;\n+}\n+\n+static int close_and_clear(int *fd)\n+{\n+\tint ret = 0;\n+\n+\tif (*fd >= 0) {\n+\t\tret = close(*fd);\n+\t\t*fd = -1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t\t struct checkout *state)\n+{\n+\tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n+\tint fd = -1, fstat_done = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tconst char *dir_sep;\n+\n+\tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n+\tstrbuf_add(&path, pc_item->ce->name, pc_item->ce->ce_namelen);\n+\n+\tdir_sep = find_last_dir_sep(path.buf);\n+\n+\t/*\n+\t * The leading dirs should have been already created by now. But, in\n+\t * case of path collisions, one of the dirs could have been replaced by\n+\t * a symlink (checked out after we enqueued this entry for parallel\n+\t * checkout). Thus, we must check the leading dirs again.\n+\t */\n+\tif (dir_sep && !has_dirs_only_path(path.buf, dir_sep - path.buf,\n+\t\t\t\t\t state->base_dir_len)) {\n+\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\tgoto out;\n+\t}\n+\n+\tfd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, mode);\n+\n+\tif (fd < 0) {\n+\t\tif (errno == EEXIST || errno == EISDIR) {\n+\t\t\t/*\n+\t\t\t * Errors which probably represent a path collision.\n+\t\t\t * Suppress the error message and mark the item to be\n+\t\t\t * retried later, sequentially. ENOTDIR and ENOENT are\n+\t\t\t * also interesting, but the above has_dirs_only_path()\n+\t\t\t * call should have already caught these cases.\n+\t\t\t */\n+\t\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\t} else {\n+\t\t\terror_errno(\"failed to open file '%s'\", path.buf);\n+\t\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\t}\n+\t\tgoto out;\n+\t}\n+\n+\tif (write_pc_item_to_fd(pc_item, fd, path.buf)) {\n+\t\t/* Error was already reported. */\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tfstat_done = fstat_checkout_output(fd, state, &pc_item->st);\n+\n+\tif (close_and_clear(&fd)) {\n+\t\terror_errno(\"unable to close file '%s'\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tif (state->refresh_cache && !fstat_done && lstat(path.buf, &pc_item->st) < 0) {\n+\t\terror_errno(\"unable to stat just-written file '%s'\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tpc_item->status = PC_ITEM_WRITTEN;\n+\n+out:\n+\t/*\n+\t * No need to check close() return at this point. Either fd is already\n+\t * closed, or we are on an error path.\n+\t */\n+\tclose_and_clear(&fd);\n+\tstrbuf_release(&path);\n+}\n+\n+static void write_items_sequentially(struct checkout *state)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < parallel_checkout.nr; i++)\n+\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+}\n+\n+int run_parallel_checkout(struct checkout *state)\n+{\n+\tint ret;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n+\t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n+\n+\tparallel_checkout.status = PC_RUNNING;\n+\n+\twrite_items_sequentially(state);\n+\tret = handle_results(state);\n+\n+\tfinish_parallel_checkout();\n+\treturn ret;\n+}\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nnew file mode 100644\nindex 0000000000..4ad2a519b3\n--- /dev/null\n+++ b/parallel-checkout.h\n@@ -0,0 +1,32 @@\n+#ifndef PARALLEL_CHECKOUT_H\n+#define PARALLEL_CHECKOUT_H\n+\n+struct cache_entry;\n+struct checkout;\n+struct conv_attrs;\n+\n+enum pc_status {\n+\tPC_UNINITIALIZED = 0,\n+\tPC_ACCEPTING_ENTRIES,\n+\tPC_RUNNING,\n+};\n+\n+enum pc_status parallel_checkout_status(void);\n+\n+/*\n+ * Put parallel checkout into the PC_ACCEPTING_ENTRIES state. Should be used\n+ * only when in the PC_UNINITIALIZED state.\n+ */\n+void init_parallel_checkout(void);\n+\n+/*\n+ * Return -1 if parallel checkout is currently not accepting entries or if the\n+ * entry is not eligible for parallel checkout. Otherwise, enqueue the entry\n+ * for later write and return 0.\n+ */\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+\n+/* Write all the queued entries, returning 0 on success.*/\n+int run_parallel_checkout(struct checkout *state);\n+\n+#endif /* PARALLEL_CHECKOUT_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8a1afbc1e4..f0430d458d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -17,6 +17,7 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -441,7 +442,6 @@ static int check_updates(struct unpack_trees_options *o,\n \tif (should_update_submodules())\n \t\tload_gitmodules_file(index, &state);\n \n-\tenable_delayed_checkout(&state);\n \tif (has_promisor_remote()) {\n \t\t/*\n \t\t * Prefetch the objects that are to be checked out in the loop\n@@ -464,6 +464,9 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t\t\t\t to_fetch.oid, to_fetch.nr);\n \t\toid_array_clear(&to_fetch);\n \t}\n+\n+\tenable_delayed_checkout(&state);\n+\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -477,6 +480,7 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n+\terrs |= run_parallel_checkout(&state);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"421238","messageId":"0d65b517bd3db5074b8ad6b51d25b6c488ff0774.1617893234.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1617893234.git.matheus.bernardino@usp.br","subject":"[PATCH v2 2/5] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-08T16:17:01Z","receivedAt":"2021-04-08T16:17:26Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Use multiple worker processes to distribute the queued entries and call\nwrite_pc_item() in parallel for them. The items are distributed\nuniformly in contiguous chunks. This minimizes the chances of two\nworkers writing to the same directory simultaneously, which could affect\nperformance due to lock contention in the kernel. Work stealing (or any\nother format of re-distribution) is not implemented yet.\n\nThe protocol between the main process and the workers is quite simple.\nThey exchange binary messages packed in pkt-line format, and use\nPKT-FLUSH to mark the end of input (from both sides). The main process\nstarts the communication by sending N pkt-lines, each corresponding to\nan item that needs to be written. These packets contain all the\nnecessary information to load, smudge, and write the blob associated\nwith each item. Then it waits for the worker to send back N pkt-lines\ncontaining the results for each item. The resulting packet must contain:\nthe identification number of the item that it refers to, the status of\nthe operation, and the lstat() data gathered after writing the file (iff\nthe operation was successful).\n\nFor now, checkout always uses a hardcoded value of 2 workers, only to\ndemonstrate that the parallel checkout framework correctly divides and\nwrites the queued entries. The next patch will add user configurations\nand define a more reasonable default, based on tests with the said\nsettings.\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/checkout--worker.c | 145 ++++++++++++++++++\n git.c | 2 +\n parallel-checkout.c | 300 +++++++++++++++++++++++++++++++++----\n parallel-checkout.h | 73 ++++++++-\n 7 files changed, 496 insertions(+), 27 deletions(-)\n create mode 100644 builtin/checkout--worker.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 3dcdb6bb5a..96c794b1c7 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -33,6 +33,7 @@\n /git-check-mailmap\n /git-check-ref-format\n /git-checkout\n+/git-checkout--worker\n /git-checkout-index\n /git-cherry\n /git-cherry-pick\ndiff --git a/Makefile b/Makefile\nindex 99734cc2ea..84e15f58a0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1062,6 +1062,7 @@ BUILTIN_OBJS += builtin/check-attr.o\n BUILTIN_OBJS += builtin/check-ignore.o\n BUILTIN_OBJS += builtin/check-mailmap.o\n BUILTIN_OBJS += builtin/check-ref-format.o\n+BUILTIN_OBJS += builtin/checkout--worker.o\n BUILTIN_OBJS += builtin/checkout-index.o\n BUILTIN_OBJS += builtin/checkout.o\n BUILTIN_OBJS += builtin/clean.o\ndiff --git a/builtin.h b/builtin.h\nindex b6ce981b73..16ecd5586f 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -123,6 +123,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix);\n int cmd_bundle(int argc, const char **argv, const char *prefix);\n int cmd_cat_file(int argc, const char **argv, const char *prefix);\n int cmd_checkout(int argc, const char **argv, const char *prefix);\n+int cmd_checkout__worker(int argc, const char **argv, const char *prefix);\n int cmd_checkout_index(int argc, const char **argv, const char *prefix);\n int cmd_check_attr(int argc, const char **argv, const char *prefix);\n int cmd_check_ignore(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/checkout--worker.c b/builtin/checkout--worker.c\nnew file mode 100644\nindex 0000000000..31e0de2f7e\n--- /dev/null\n+++ b/builtin/checkout--worker.c\n@@ -0,0 +1,145 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"parse-options.h\"\n+#include \"pkt-line.h\"\n+\n+static void packet_to_pc_item(const char *buffer, int len,\n+\t\t\t struct parallel_checkout_item *pc_item)\n+{\n+\tconst struct pc_item_fixed_portion *fixed_portion;\n+\tconst char *variant;\n+\tchar *encoding;\n+\n+\tif (len < sizeof(struct pc_item_fixed_portion))\n+\t\tBUG(\"checkout worker received too short item (got %dB, exp %dB)\",\n+\t\t len, (int)sizeof(struct pc_item_fixed_portion));\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)buffer;\n+\n+\tif (len - sizeof(struct pc_item_fixed_portion) !=\n+\t\tfixed_portion->name_len + fixed_portion->working_tree_encoding_len)\n+\t\tBUG(\"checkout worker received corrupted item\");\n+\n+\tvariant = buffer + sizeof(struct pc_item_fixed_portion);\n+\n+\t/*\n+\t * Note: the main process uses zero length to communicate that the\n+\t * encoding is NULL. There is no use case that requires sending an\n+\t * actual empty string, since convert_attrs() never sets\n+\t * ca.working_tree_enconding to \"\".\n+\t */\n+\tif (fixed_portion->working_tree_encoding_len) {\n+\t\tencoding = xmemdupz(variant,\n+\t\t\t\t fixed_portion->working_tree_encoding_len);\n+\t\tvariant += fixed_portion->working_tree_encoding_len;\n+\t} else {\n+\t\tencoding = NULL;\n+\t}\n+\n+\tmemset(pc_item, 0, sizeof(*pc_item));\n+\tpc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n+\tpc_item->ce->ce_namelen = fixed_portion->name_len;\n+\tpc_item->ce->ce_mode = fixed_portion->ce_mode;\n+\tmemcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen);\n+\toidcpy(&pc_item->ce->oid, &fixed_portion->oid);\n+\n+\tpc_item->id = fixed_portion->id;\n+\tpc_item->ca.crlf_action = fixed_portion->crlf_action;\n+\tpc_item->ca.ident = fixed_portion->ident;\n+\tpc_item->ca.working_tree_encoding = encoding;\n+}\n+\n+static void report_result(struct parallel_checkout_item *pc_item)\n+{\n+\tstruct pc_item_result res;\n+\tsize_t size;\n+\n+\tres.id = pc_item->id;\n+\tres.status = pc_item->status;\n+\n+\tif (pc_item->status == PC_ITEM_WRITTEN) {\n+\t\tres.st = pc_item->st;\n+\t\tsize = sizeof(res);\n+\t} else {\n+\t\tsize = PC_ITEM_RESULT_BASE_SIZE;\n+\t}\n+\n+\tpacket_write(1, (const char *)&res, size);\n+}\n+\n+/* Free the worker-side malloced data, but not pc_item itself. */\n+static void release_pc_item_data(struct parallel_checkout_item *pc_item)\n+{\n+\tfree((char *)pc_item->ca.working_tree_encoding);\n+\tdiscard_cache_entry(pc_item->ce);\n+}\n+\n+static void worker_loop(struct checkout *state)\n+{\n+\tstruct parallel_checkout_item *items = NULL;\n+\tsize_t i, nr = 0, alloc = 0;\n+\n+\twhile (1) {\n+\t\tint len = packet_read(0, NULL, NULL, packet_buffer,\n+\t\t\t\t sizeof(packet_buffer), 0);\n+\n+\t\tif (len < 0)\n+\t\t\tBUG(\"packet_read() returned negative value\");\n+\t\telse if (!len)\n+\t\t\tbreak;\n+\n+\t\tALLOC_GROW(items, nr + 1, alloc);\n+\t\tpacket_to_pc_item(packet_buffer, len, &items[nr++]);\n+\t}\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = &items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\treport_result(pc_item);\n+\t\trelease_pc_item_data(pc_item);\n+\t}\n+\n+\tpacket_flush(1);\n+\n+\tfree(items);\n+}\n+\n+static const char * const checkout_worker_usage[] = {\n+\tN_(\"git checkout--worker []\"),\n+\tNULL\n+};\n+\n+int cmd_checkout__worker(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct checkout state = CHECKOUT_INIT;\n+\tstruct option checkout_worker_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &state.base_dir, N_(\"string\"),\n+\t\t\tN_(\"when creating files, prepend \")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(checkout_worker_usage,\n+\t\t\t\t checkout_worker_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix, checkout_worker_options,\n+\t\t\t checkout_worker_usage, 0);\n+\tif (argc > 0)\n+\t\tusage_with_options(checkout_worker_usage, checkout_worker_options);\n+\n+\tif (state.base_dir)\n+\t\tstate.base_dir_len = strlen(state.base_dir);\n+\n+\t/*\n+\t * Setting this on a worker won't actually update the index. We just\n+\t * need to tell the checkout machinery to lstat() the written entries,\n+\t * so that we can send this data back to the main process.\n+\t */\n+\tstate.refresh_cache = 1;\n+\n+\tworker_loop(&state);\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 9bc077a025..532b5b4136 100644\n--- a/git.c\n+++ b/git.c\n@@ -490,6 +490,8 @@ static struct cmd_struct commands[] = {\n \t{ \"check-mailmap\", cmd_check_mailmap, RUN_SETUP },\n \t{ \"check-ref-format\", cmd_check_ref_format, NO_PARSEOPT },\n \t{ \"checkout\", cmd_checkout, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"checkout--worker\", cmd_checkout__worker,\n+\t\tRUN_SETUP | NEED_WORK_TREE | SUPPORT_SUPER_PREFIX },\n \t{ \"checkout-index\", cmd_checkout_index,\n \t\tRUN_SETUP | NEED_WORK_TREE},\n \t{ \"cherry\", cmd_cherry, RUN_SETUP },\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 80c839837b..41c301bbda 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,28 +1,14 @@\n #include \"cache.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n+#include \"pkt-line.h\"\n+#include \"run-command.h\"\n+#include \"sigchain.h\"\n #include \"streaming.h\"\n \n-enum pc_item_status {\n-\tPC_ITEM_PENDING = 0,\n-\tPC_ITEM_WRITTEN,\n-\t/*\n-\t * The entry could not be written because there was another file\n-\t * already present in its path or leading directories. Since\n-\t * checkout_entry_ca() removes such files from the working tree before\n-\t * enqueueing the entry for parallel checkout, it means that there was\n-\t * a path collision among the entries being written.\n-\t */\n-\tPC_ITEM_COLLIDED,\n-\tPC_ITEM_FAILED,\n-};\n-\n-struct parallel_checkout_item {\n-\t/* pointer to a istate->cache[] entry. Not owned by us. */\n-\tstruct cache_entry *ce;\n-\tstruct conv_attrs ca;\n-\tstruct stat st;\n-\tenum pc_item_status status;\n+struct pc_worker {\n+\tstruct child_process cp;\n+\tsize_t next_item_to_complete, nr_items_to_complete;\n };\n \n struct parallel_checkout {\n@@ -59,6 +45,7 @@ static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n \t\t\t\t\t const struct conv_attrs *ca)\n {\n \tenum conv_attrs_classification c;\n+\tsize_t packed_item_size;\n \n \t/*\n \t * Symlinks cannot be checked out in parallel as, in case of path\n@@ -69,6 +56,18 @@ static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n \tif (!S_ISREG(ce->ce_mode))\n \t\treturn 0;\n \n+\tpacked_item_size = sizeof(struct pc_item_fixed_portion) + ce->ce_namelen +\n+\t\t(ca->working_tree_encoding ? strlen(ca->working_tree_encoding) : 0);\n+\n+\t/*\n+\t * The amount of data we send to the workers per checkout item is\n+\t * typically small (75~300B). So unless we find an insanely huge path\n+\t * of 64KB, we should never reach the 65KB limit of one pkt-line. If\n+\t * that does happen, we let the sequential code handle the item.\n+\t */\n+\tif (packed_item_size > LARGE_PACKET_DATA_MAX)\n+\t\treturn 0;\n+\n \tc = classify_conv_attrs(ca);\n \tswitch (c) {\n \tcase CA_CLASS_INCORE:\n@@ -121,10 +120,12 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n \t\t parallel_checkout.alloc);\n \n-\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr];\n \tpc_item->ce = ce;\n \tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n \tpc_item->status = PC_ITEM_PENDING;\n+\tpc_item->id = parallel_checkout.nr;\n+\tparallel_checkout.nr++;\n \n \treturn 0;\n }\n@@ -236,7 +237,8 @@ static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n \t/*\n \t * checkout metadata is used to give context for external process\n \t * filters. Files requiring such filters are not eligible for parallel\n-\t * checkout, so pass NULL.\n+\t * checkout, so pass NULL. Note: if that changes, the metadata must also\n+\t * be passed from the main process to the workers.\n \t */\n \tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n \t\t\t\t\t new_blob, size, &buf, NULL);\n@@ -268,8 +270,8 @@ static int close_and_clear(int *fd)\n \treturn ret;\n }\n \n-static void write_pc_item(struct parallel_checkout_item *pc_item,\n-\t\t\t struct checkout *state)\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state)\n {\n \tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n \tint fd = -1, fstat_done = 0;\n@@ -343,6 +345,240 @@ static void write_pc_item(struct parallel_checkout_item *pc_item,\n \tstrbuf_release(&path);\n }\n \n+static void send_one_item(int fd, struct parallel_checkout_item *pc_item)\n+{\n+\tsize_t len_data;\n+\tchar *data, *variant;\n+\tstruct pc_item_fixed_portion *fixed_portion;\n+\tconst char *working_tree_encoding = pc_item->ca.working_tree_encoding;\n+\tsize_t name_len = pc_item->ce->ce_namelen;\n+\tsize_t working_tree_encoding_len = working_tree_encoding ?\n+\t\t\t\t\t strlen(working_tree_encoding) : 0;\n+\n+\t/*\n+\t * Any changes in the calculation of the message size must also be made\n+\t * in is_eligible_for_parallel_checkout().\n+\t */\n+\tlen_data = sizeof(struct pc_item_fixed_portion) + name_len +\n+\t\t working_tree_encoding_len;\n+\n+\tdata = xcalloc(1, len_data);\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)data;\n+\tfixed_portion->id = pc_item->id;\n+\tfixed_portion->ce_mode = pc_item->ce->ce_mode;\n+\tfixed_portion->crlf_action = pc_item->ca.crlf_action;\n+\tfixed_portion->ident = pc_item->ca.ident;\n+\tfixed_portion->name_len = name_len;\n+\tfixed_portion->working_tree_encoding_len = working_tree_encoding_len;\n+\t/*\n+\t * We use hashcpy() instead of oidcpy() because the hash[] positions\n+\t * after `the_hash_algo->rawsz` might not be initialized. And Valgrind\n+\t * would complain about passing uninitialized bytes to a syscall\n+\t * (write(2)). There is no real harm in this case, but the warning could\n+\t * hinder the detection of actual errors.\n+\t */\n+\thashcpy(fixed_portion->oid.hash, pc_item->ce->oid.hash);\n+\n+\tvariant = data + sizeof(*fixed_portion);\n+\tif (working_tree_encoding_len) {\n+\t\tmemcpy(variant, working_tree_encoding, working_tree_encoding_len);\n+\t\tvariant += working_tree_encoding_len;\n+\t}\n+\tmemcpy(variant, pc_item->ce->name, name_len);\n+\n+\tpacket_write(fd, data, len_data);\n+\n+\tfree(data);\n+}\n+\n+static void send_batch(int fd, size_t start, size_t nr)\n+{\n+\tsize_t i;\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tfor (i = 0; i < nr; i++)\n+\t\tsend_one_item(fd, ¶llel_checkout.items[start + i]);\n+\tpacket_flush(fd);\n+\tsigchain_pop(SIGPIPE);\n+}\n+\n+static struct pc_worker *setup_workers(struct checkout *state, int num_workers)\n+{\n+\tstruct pc_worker *workers;\n+\tint i, workers_with_one_extra_item;\n+\tsize_t base_batch_size, batch_beginning = 0;\n+\n+\tALLOC_ARRAY(workers, num_workers);\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->git_cmd = 1;\n+\t\tcp->in = -1;\n+\t\tcp->out = -1;\n+\t\tcp->clean_on_exit = 1;\n+\t\tstrvec_push(&cp->args, \"checkout--worker\");\n+\t\tif (state->base_dir_len)\n+\t\t\tstrvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n+\t\tif (start_command(cp))\n+\t\t\tdie(\"failed to spawn checkout worker\");\n+\t}\n+\n+\tbase_batch_size = parallel_checkout.nr / num_workers;\n+\tworkers_with_one_extra_item = parallel_checkout.nr % num_workers;\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct pc_worker *worker = &workers[i];\n+\t\tsize_t batch_size = base_batch_size;\n+\n+\t\t/* distribute the extra work evenly */\n+\t\tif (i < workers_with_one_extra_item)\n+\t\t\tbatch_size++;\n+\n+\t\tsend_batch(worker->cp.in, batch_beginning, batch_size);\n+\t\tworker->next_item_to_complete = batch_beginning;\n+\t\tworker->nr_items_to_complete = batch_size;\n+\n+\t\tbatch_beginning += batch_size;\n+\t}\n+\n+\treturn workers;\n+}\n+\n+static void finish_workers(struct pc_worker *workers, int num_workers)\n+{\n+\tint i;\n+\n+\t/*\n+\t * Close pipes before calling finish_command() to let the workers\n+\t * exit asynchronously and avoid spending extra time on wait().\n+\t */\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\t\tif (cp->in >= 0)\n+\t\t\tclose(cp->in);\n+\t\tif (cp->out >= 0)\n+\t\t\tclose(cp->out);\n+\t}\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tint rc = finish_command(&workers[i].cp);\n+\t\tif (rc > 128) {\n+\t\t\t/*\n+\t\t\t * For a normal non-zero exit, the worker should have\n+\t\t\t * already printed something useful to stderr. But a\n+\t\t\t * death by signal should be mentioned to the user.\n+\t\t\t */\n+\t\t\terror(\"checkout worker %d died of signal %d\", i, rc - 128);\n+\t\t}\n+\t}\n+\n+\tfree(workers);\n+}\n+\n+static inline void assert_pc_item_result_size(int got, int exp)\n+{\n+\tif (got != exp)\n+\t\tBUG(\"wrong result size from checkout worker (got %dB, exp %dB)\",\n+\t\t got, exp);\n+}\n+\n+static void parse_and_save_result(const char *buffer, int len,\n+\t\t\t\t struct pc_worker *worker)\n+{\n+\tstruct pc_item_result *res;\n+\tstruct parallel_checkout_item *pc_item;\n+\tstruct stat *st = NULL;\n+\n+\tif (len < PC_ITEM_RESULT_BASE_SIZE)\n+\t\tBUG(\"too short result from checkout worker (got %dB, exp >=%dB)\",\n+\t\t len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\n+\tres = (struct pc_item_result *)buffer;\n+\n+\t/*\n+\t * Worker should send either the full result struct on success, or\n+\t * just the base (i.e. no stat data), otherwise.\n+\t */\n+\tif (res->status == PC_ITEM_WRITTEN) {\n+\t\tassert_pc_item_result_size(len, (int)sizeof(struct pc_item_result));\n+\t\tst = &res->st;\n+\t} else {\n+\t\tassert_pc_item_result_size(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\t}\n+\n+\tif (!worker->nr_items_to_complete)\n+\t\tBUG(\"received result from supposedly finished checkout worker\");\n+\tif (res->id != worker->next_item_to_complete)\n+\t\tBUG(\"unexpected item id from checkout worker (got %\"PRIuMAX\", exp %\"PRIuMAX\")\",\n+\t\t (uintmax_t)res->id, (uintmax_t)worker->next_item_to_complete);\n+\n+\tworker->next_item_to_complete++;\n+\tworker->nr_items_to_complete--;\n+\n+\tpc_item = ¶llel_checkout.items[res->id];\n+\tpc_item->status = res->status;\n+\tif (st)\n+\t\tpc_item->st = *st;\n+}\n+\n+static void gather_results_from_workers(struct pc_worker *workers,\n+\t\t\t\t\tint num_workers)\n+{\n+\tint i, active_workers = num_workers;\n+\tstruct pollfd *pfds;\n+\n+\tCALLOC_ARRAY(pfds, num_workers);\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tpfds[i].fd = workers[i].cp.out;\n+\t\tpfds[i].events = POLLIN;\n+\t}\n+\n+\twhile (active_workers) {\n+\t\tint nr = poll(pfds, num_workers, -1);\n+\n+\t\tif (nr < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tdie_errno(\"failed to poll checkout workers\");\n+\t\t}\n+\n+\t\tfor (i = 0; i < num_workers && nr > 0; i++) {\n+\t\t\tstruct pc_worker *worker = &workers[i];\n+\t\t\tstruct pollfd *pfd = &pfds[i];\n+\n+\t\t\tif (!pfd->revents)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (pfd->revents & POLLIN) {\n+\t\t\t\tint len = packet_read(pfd->fd, NULL, NULL,\n+\t\t\t\t\t\t packet_buffer,\n+\t\t\t\t\t\t sizeof(packet_buffer), 0);\n+\n+\t\t\t\tif (len < 0) {\n+\t\t\t\t\tBUG(\"packet_read() returned negative value\");\n+\t\t\t\t} else if (!len) {\n+\t\t\t\t\tpfd->fd = -1;\n+\t\t\t\t\tactive_workers--;\n+\t\t\t\t} else {\n+\t\t\t\t\tparse_and_save_result(packet_buffer,\n+\t\t\t\t\t\t\t len, worker);\n+\t\t\t\t}\n+\t\t\t} else if (pfd->revents & POLLHUP) {\n+\t\t\t\tpfd->fd = -1;\n+\t\t\t\tactive_workers--;\n+\t\t\t} else if (pfd->revents & (POLLNVAL | POLLERR)) {\n+\t\t\t\tdie(\"error polling from checkout worker\");\n+\t\t\t}\n+\n+\t\t\tnr--;\n+\t\t}\n+\t}\n+\n+\tfree(pfds);\n+}\n+\n static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n@@ -351,16 +587,28 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n+static const int DEFAULT_NUM_WORKERS = 2;\n+\n int run_parallel_checkout(struct checkout *state)\n {\n-\tint ret;\n+\tint ret, num_workers = DEFAULT_NUM_WORKERS;\n \n \tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n \n-\twrite_items_sequentially(state);\n+\tif (parallel_checkout.nr < num_workers)\n+\t\tnum_workers = parallel_checkout.nr;\n+\n+\tif (num_workers <= 1) {\n+\t\twrite_items_sequentially(state);\n+\t} else {\n+\t\tstruct pc_worker *workers = setup_workers(state, num_workers);\n+\t\tgather_results_from_workers(workers, num_workers);\n+\t\tfinish_workers(workers, num_workers);\n+\t}\n+\n \tret = handle_results(state);\n \n \tfinish_parallel_checkout();\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 4ad2a519b3..ec58716519 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -1,9 +1,14 @@\n #ifndef PARALLEL_CHECKOUT_H\n #define PARALLEL_CHECKOUT_H\n \n+#include \"convert.h\"\n+\n struct cache_entry;\n struct checkout;\n-struct conv_attrs;\n+\n+/****************************************************************\n+ * Users of parallel checkout\n+ ****************************************************************/\n \n enum pc_status {\n \tPC_UNINITIALIZED = 0,\n@@ -29,4 +34,70 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n /* Write all the queued entries, returning 0 on success.*/\n int run_parallel_checkout(struct checkout *state);\n \n+/****************************************************************\n+ * Interface with checkout--worker\n+ ****************************************************************/\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/*\n+\t * In main process ce points to a istate->cache[] entry. Thus, it's not\n+\t * owned by us. In workers they own the memory, which *must be* released.\n+\t */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tsize_t id; /* position in parallel_checkout.items[] of main process */\n+\n+\t/* Output fields, sent from workers. */\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+/*\n+ * The fixed-size portion of `struct parallel_checkout_item` that is sent to the\n+ * workers. Following this will be 2 strings: ca.working_tree_encoding and\n+ * ce.name; These are NOT null terminated, since we have the size in the fixed\n+ * portion.\n+ *\n+ * Note that not all fields of conv_attrs and cache_entry are passed, only the\n+ * ones that will be required by the workers to smudge and write the entry.\n+ */\n+struct pc_item_fixed_portion {\n+\tsize_t id;\n+\tstruct object_id oid;\n+\tunsigned int ce_mode;\n+\tenum convert_crlf_action crlf_action;\n+\tint ident;\n+\tsize_t working_tree_encoding_len;\n+\tsize_t name_len;\n+};\n+\n+/*\n+ * The fields of `struct parallel_checkout_item` that are returned by the\n+ * workers. Note: `st` must be the last one, as it is omitted on error.\n+ */\n+struct pc_item_result {\n+\tsize_t id;\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+#define PC_ITEM_RESULT_BASE_SIZE offsetof(struct pc_item_result, st)\n+\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state);\n+\n #endif /* PARALLEL_CHECKOUT_H */\n-- \n2.30.1\n\n"},{"id":"421239","messageId":"6ea057f9c555882d8aa67bc9a2952f7d5a79548d.1617893234.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1617893234.git.matheus.bernardino@usp.br","subject":"[PATCH v2 3/5] parallel-checkout: add configuration options","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-08T16:17:02Z","receivedAt":"2021-04-08T16:17:29Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make parallel checkout configurable by introducing two new settings:\ncheckout.workers and checkout.thresholdForParallelism. The first defines\nthe number of workers (where one means sequential checkout), and the\nsecond defines the minimum number of entries to attempt parallel\ncheckout.\n\nTo decide the default value for checkout.workers, the parallel version\nwas benchmarked during three operations in the linux repo, with cold\ncache: cloning v5.8, checking out v5.8 from v2.6.15 (checkout I) and\nchecking out v5.8 from v5.7 (checkout II). The four tables below show\nthe mean run times and standard deviations for 5 runs in: a local file\nsystem on SSD, a local file system on HDD, a Linux NFS server, and\nAmazon EFS (all on Linux). Each parallel checkout test was executed with\nthe number of workers that brings the best overall results in that\nenvironment.\n\nLocal SSD:\n Sequential 10 workers Speedup\nClone 8.805 s ± 0.043 s 3.564 s ± 0.041 s 2.47 ± 0.03\nCheckout I 9.678 s ± 0.057 s 4.486 s ± 0.050 s 2.16 ± 0.03\nCheckout II 5.034 s ± 0.072 s 3.021 s ± 0.038 s 1.67 ± 0.03\n\nLocal HDD:\n Sequential 10 workers Speedup\nClone 32.288 s ± 0.580 s 30.724 s ± 0.522 s 1.05 ± 0.03\nCheckout I 54.172 s ± 7.119 s 54.429 s ± 6.738 s 1.00 ± 0.18\nCheckout II 40.465 s ± 2.402 s 38.682 s ± 1.365 s 1.05 ± 0.07\n\nLinux NFS server (v4.1, on EBS, single availability zone):\n\n Sequential 32 workers Speedup\nClone 240.368 s ± 6.347 s 57.349 s ± 0.870 s 4.19 ± 0.13\nCheckout I 242.862 s ± 2.215 s 58.700 s ± 0.904 s 4.14 ± 0.07\nCheckout II 65.751 s ± 1.577 s 23.820 s ± 0.407 s 2.76 ± 0.08\n\nEFS (v4.1, replicated over multiple availability zones):\n\n Sequential 32 workers Speedup\nClone 922.321 s ± 2.274 s 210.453 s ± 3.412 s 4.38 ± 0.07\nCheckout I 1011.300 s ± 7.346 s 297.828 s ± 0.964 s 3.40 ± 0.03\nCheckout II 294.104 s ± 1.836 s 126.017 s ± 1.190 s 2.33 ± 0.03\n\nThe above benchmarks show that parallel checkout is most effective on\nrepositories located on an SSD or over a distributed file system. For\nlocal file systems on spinning disks, and/or older machines, the\nparallelism does not always bring a good performance. For this reason,\nthe default value for checkout.workers is one, a.k.a. sequential\ncheckout.\n\nTo decide the default value for checkout.thresholdForParallelism,\nanother benchmark was executed in the \"Local SSD\" setup, where parallel\ncheckout showed to be beneficial. This time, we compared the runtime of\na `git checkout -f`, with and without parallelism, after randomly\nremoving an increasing number of files from the Linux working tree. The\n\"sequential fallback\" column below corresponds to the executions where\ncheckout.workers was 10 but checkout.thresholdForParallelism was equal\nto the number of to-be-updated files plus one (so that we end up writing\nsequentially). Each test case was sampled 15 times, and each sample had\na randomly different set of files removed. Here are the results:\n\n sequential fallback 10 workers speedup\n10 files 772.3 ms ± 12.6 ms 769.0 ms ± 13.6 ms 1.00 ± 0.02\n20 files 780.5 ms ± 15.8 ms 775.2 ms ± 9.2 ms 1.01 ± 0.02\n50 files 806.2 ms ± 13.8 ms 767.4 ms ± 8.5 ms 1.05 ± 0.02\n100 files 833.7 ms ± 21.4 ms 750.5 ms ± 16.8 ms 1.11 ± 0.04\n200 files 897.6 ms ± 30.9 ms 730.5 ms ± 14.7 ms 1.23 ± 0.05\n500 files 1035.4 ms ± 48.0 ms 677.1 ms ± 22.3 ms 1.53 ± 0.09\n1000 files 1244.6 ms ± 35.6 ms 654.0 ms ± 38.3 ms 1.90 ± 0.12\n2000 files 1488.8 ms ± 53.4 ms 658.8 ms ± 23.8 ms 2.26 ± 0.12\n\nFrom the above numbers, 100 files seems to be a reasonable default value\nfor the threshold setting.\n\nNote: Up to 1000 files, we observe a drop in the execution time of the\nparallel code with an increase in the number of files. This is a rather\nodd behavior, but it was observed in multiple repetitions. Above 1000\nfiles, the execution time increases according to the number of files, as\none would expect.\n\nAbout the test environments: Local SSD tests were executed on an\ni7-7700HQ (4 cores with hyper-threading) running Manjaro Linux. Local\nHDD tests were executed on an Intel(R) Xeon(R) E3-1230 (also 4 cores\nwith hyper-threading), HDD Seagate Barracuda 7200.14 SATA 3.1, running\nDebian. NFS and EFS tests were executed on an Amazon EC2 c5n.xlarge\ninstance, with 4 vCPUs. The Linux NFS server was running on a m6g.large\ninstance with 2 vCPUSs and a 1 TB EBS GP2 volume. Before each timing,\nthe linux repository was removed (or checked out back to its previous\nstate), and `sync && sysctl vm.drop_caches=3` was executed.\n\nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Documentation/config/checkout.txt | 21 +++++++++++++++++++++\n parallel-checkout.c | 24 +++++++++++++++++++-----\n parallel-checkout.h | 9 +++++++--\n unpack-trees.c | 10 +++++++---\n 4 files changed, 54 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt\nindex 2cddf7b4b4..bfbca90f0e 100644\n--- a/Documentation/config/checkout.txt\n+++ b/Documentation/config/checkout.txt\n@@ -21,3 +21,24 @@ checkout.guess::\n \tProvides the default value for the `--guess` or `--no-guess`\n \toption in `git checkout` and `git switch`. See\n \tlinkgit:git-switch[1] and linkgit:git-checkout[1].\n+\n+checkout.workers::\n+\tThe number of parallel workers to use when updating the working tree.\n+\tThe default is one, i.e. sequential execution. If set to a value less\n+\tthan one, Git will use as many workers as the number of logical cores\n+\tavailable. This setting and `checkout.thresholdForParallelism` affect\n+\tall commands that perform checkout. E.g. checkout, clone, reset,\n+\tsparse-checkout, etc.\n++\n+Note: parallel checkout usually delivers better performance for repositories\n+located on SSDs or over NFS. For repositories on spinning disks and/or machines\n+with a small number of cores, the default sequential checkout often performs\n+better. The size and compression level of a repository might also influence how\n+well the parallel version performs.\n+\n+checkout.thresholdForParallelism::\n+\tWhen running parallel checkout with a small number of files, the cost\n+\tof subprocess spawning and inter-process communication might outweigh\n+\tthe parallelization gains. This setting allows to define the minimum\n+\tnumber of files for which parallel checkout should be attempted. The\n+\tdefault is 100.\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 41c301bbda..d6a0f31664 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,10 +1,12 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"streaming.h\"\n+#include \"thread-utils.h\"\n \n struct pc_worker {\n \tstruct child_process cp;\n@@ -24,6 +26,20 @@ enum pc_status parallel_checkout_status(void)\n \treturn parallel_checkout.status;\n }\n \n+static const int DEFAULT_THRESHOLD_FOR_PARALLELISM = 100;\n+static const int DEFAULT_NUM_WORKERS = 1;\n+\n+void get_parallel_checkout_configs(int *num_workers, int *threshold)\n+{\n+\tif (git_config_get_int(\"checkout.workers\", num_workers))\n+\t\t*num_workers = DEFAULT_NUM_WORKERS;\n+\telse if (*num_workers < 1)\n+\t\t*num_workers = online_cpus();\n+\n+\tif (git_config_get_int(\"checkout.thresholdForParallelism\", threshold))\n+\t\t*threshold = DEFAULT_THRESHOLD_FOR_PARALLELISM;\n+}\n+\n void init_parallel_checkout(void)\n {\n \tif (parallel_checkout.status != PC_UNINITIALIZED)\n@@ -587,11 +603,9 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n-static const int DEFAULT_NUM_WORKERS = 2;\n-\n-int run_parallel_checkout(struct checkout *state)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n {\n-\tint ret, num_workers = DEFAULT_NUM_WORKERS;\n+\tint ret;\n \n \tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n@@ -601,7 +615,7 @@ int run_parallel_checkout(struct checkout *state)\n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\n \n-\tif (num_workers <= 1) {\n+\tif (num_workers <= 1 || parallel_checkout.nr < threshold) {\n \t\twrite_items_sequentially(state);\n \t} else {\n \t\tstruct pc_worker *workers = setup_workers(state, num_workers);\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex ec58716519..2a68ab954d 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -17,6 +17,7 @@ enum pc_status {\n };\n \n enum pc_status parallel_checkout_status(void);\n+void get_parallel_checkout_configs(int *num_workers, int *threshold);\n \n /*\n * Put parallel checkout into the PC_ACCEPTING_ENTRIES state. Should be used\n@@ -31,8 +32,12 @@ void init_parallel_checkout(void);\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n \n-/* Write all the queued entries, returning 0 on success.*/\n-int run_parallel_checkout(struct checkout *state);\n+/*\n+ * Write all the queued entries, returning 0 on success. If the number of\n+ * entries is smaller than the specified threshold, the operation is performed\n+ * sequentially.\n+ */\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n \n /****************************************************************\n * Interface with checkout--worker\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f0430d458d..0669748f21 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -399,7 +399,7 @@ static int check_updates(struct unpack_trees_options *o,\n \tint errs = 0;\n \tstruct progress *progress;\n \tstruct checkout state = CHECKOUT_INIT;\n-\tint i;\n+\tint i, pc_workers, pc_threshold;\n \n \ttrace_performance_enter();\n \tstate.force = 1;\n@@ -465,8 +465,11 @@ static int check_updates(struct unpack_trees_options *o,\n \t\toid_array_clear(&to_fetch);\n \t}\n \n+\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n \tenable_delayed_checkout(&state);\n-\tinit_parallel_checkout();\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -480,7 +483,8 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n-\terrs |= run_parallel_checkout(&state);\n+\tif (pc_workers > 1)\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"421240","messageId":"0ac4bee67e5ed43ba87e7beb3f119f11d90706fd.1617893234.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1617893234.git.matheus.bernardino@usp.br","subject":"[PATCH v2 4/5] parallel-checkout: support progress displaying","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-08T16:17:03Z","receivedAt":"2021-04-08T16:17:31Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Original-patch-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n parallel-checkout.c | 34 +++++++++++++++++++++++++++++++---\n parallel-checkout.h | 5 ++++-\n unpack-trees.c | 11 ++++++++---\n 3 files changed, 43 insertions(+), 7 deletions(-)\n\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex d6a0f31664..d09ad8a7c6 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -3,6 +3,7 @@\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n+#include \"progress.h\"\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"streaming.h\"\n@@ -17,6 +18,8 @@ struct parallel_checkout {\n \tenum pc_status status;\n \tstruct parallel_checkout_item *items; /* The parallel checkout queue. */\n \tsize_t nr, alloc;\n+\tstruct progress *progress;\n+\tunsigned int *progress_cnt;\n };\n \n static struct parallel_checkout parallel_checkout;\n@@ -146,6 +149,20 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \treturn 0;\n }\n \n+size_t pc_queue_size(void)\n+{\n+\treturn parallel_checkout.nr;\n+}\n+\n+static void advance_progress_meter(void)\n+{\n+\tif (parallel_checkout.progress) {\n+\t\t(*parallel_checkout.progress_cnt)++;\n+\t\tdisplay_progress(parallel_checkout.progress,\n+\t\t\t\t *parallel_checkout.progress_cnt);\n+\t}\n+}\n+\n static int handle_results(struct checkout *state)\n {\n \tint ret = 0;\n@@ -194,6 +211,7 @@ static int handle_results(struct checkout *state)\n \t\t\t */\n \t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n \t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tadvance_progress_meter();\n \t\t\tbreak;\n \t\tcase PC_ITEM_PENDING:\n \t\t\thave_pending = 1;\n@@ -537,6 +555,9 @@ static void parse_and_save_result(const char *buffer, int len,\n \tpc_item->status = res->status;\n \tif (st)\n \t\tpc_item->st = *st;\n+\n+\tif (res->status != PC_ITEM_COLLIDED)\n+\t\tadvance_progress_meter();\n }\n \n static void gather_results_from_workers(struct pc_worker *workers,\n@@ -599,11 +620,16 @@ static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n \n-\tfor (i = 0; i < parallel_checkout.nr; i++)\n-\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\tif (pc_item->status != PC_ITEM_COLLIDED)\n+\t\t\tadvance_progress_meter();\n+\t}\n }\n \n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt)\n {\n \tint ret;\n \n@@ -611,6 +637,8 @@ int run_parallel_checkout(struct checkout *state, int num_workers, int threshold\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n+\tparallel_checkout.progress = progress;\n+\tparallel_checkout.progress_cnt = progress_cnt;\n \n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 2a68ab954d..80f539bcb7 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -5,6 +5,7 @@\n \n struct cache_entry;\n struct checkout;\n+struct progress;\n \n /****************************************************************\n * Users of parallel checkout\n@@ -31,13 +32,15 @@ void init_parallel_checkout(void);\n * for later write and return 0.\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+size_t pc_queue_size(void);\n \n /*\n * Write all the queued entries, returning 0 on success. If the number of\n * entries is smaller than the specified threshold, the operation is performed\n * sequentially.\n */\n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt);\n \n /****************************************************************\n * Interface with checkout--worker\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 0669748f21..4b77e52c6b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -474,17 +474,22 @@ static int check_updates(struct unpack_trees_options *o,\n \t\tstruct cache_entry *ce = index->cache[i];\n \n \t\tif (ce->ce_flags & CE_UPDATE) {\n+\t\t\tsize_t last_pc_queue_size = pc_queue_size();\n+\n \t\t\tif (ce->ce_flags & CE_WT_REMOVE)\n \t\t\t\tBUG(\"both update and delete flags are set on %s\",\n \t\t\t\t ce->name);\n-\t\t\tdisplay_progress(progress, ++cnt);\n \t\t\tce->ce_flags &= ~CE_UPDATE;\n \t\t\terrs |= checkout_entry(ce, &state, NULL, NULL);\n+\n+\t\t\tif (last_pc_queue_size == pc_queue_size())\n+\t\t\t\tdisplay_progress(progress, ++cnt);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tif (pc_workers > 1)\n-\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t\t progress, &cnt);\n+\tstop_progress(&progress);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"421241","messageId":"087f8bdf35f65114505fd830f787159411d86467.1617893234.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1617893234.git.matheus.bernardino@usp.br","subject":"[PATCH v2 5/5] parallel-checkout: add design documentation","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-08T16:17:04Z","receivedAt":"2021-04-08T16:17:38Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Co-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Documentation/Makefile | 1 +\n Documentation/technical/parallel-checkout.txt | 268 ++++++++++++++++++\n 2 files changed, 269 insertions(+)\n create mode 100644 Documentation/technical/parallel-checkout.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 81d1bf7a04..af236927c9 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -90,6 +90,7 @@ TECH_DOCS += technical/multi-pack-index\n TECH_DOCS += technical/pack-format\n TECH_DOCS += technical/pack-heuristics\n TECH_DOCS += technical/pack-protocol\n+TECH_DOCS += technical/parallel-checkout\n TECH_DOCS += technical/partial-clone\n TECH_DOCS += technical/protocol-capabilities\n TECH_DOCS += technical/protocol-common\ndiff --git a/Documentation/technical/parallel-checkout.txt b/Documentation/technical/parallel-checkout.txt\nnew file mode 100644\nindex 0000000000..b8703eae49\n--- /dev/null\n+++ b/Documentation/technical/parallel-checkout.txt\n@@ -0,0 +1,268 @@\n+Parallel Checkout Design Notes\n+==============================\n+\n+The \"Parallel Checkout\" feature attempts to use multiple processes to\n+parallelize the work of uncompressing the blobs, applying in-core\n+filters, and writing the resulting contents to the working tree during a\n+checkout operation. It can be used by all checkout-related commands,\n+such as `clone`, `checkout`, `reset`, `sparse-checkout`, and others.\n+\n+These commands share the following basic structure:\n+\n+* Step 1: Read the current index file into memory.\n+\n+* Step 2: Modify the in-memory index based upon the command, and\n+ temporarily mark all cache entries that need to be updated.\n+\n+* Step 3: Populate the working tree to match the new candidate index.\n+ This includes iterating over all of the to-be-updated cache entries\n+ and delete, create, or overwrite the associated files in the working\n+ tree.\n+\n+* Step 4: Write the new index to disk.\n+\n+Step 3 is the focus of the \"parallel checkout\" effort described here.\n+It dominates the execution time for most of the above command types.\n+\n+Sequential Implementation\n+-------------------------\n+\n+For the purposes of discussion here, the current sequential\n+implementation of Step 3 is divided in 3 parts, each one implemented in\n+its own function:\n+\n+* Step 3a: `unpack-trees.c:check_updates()` contains a series of\n+ sequential loops iterating over the `cache_entry`'s array. The main\n+ loop in this function calls the Step 3b function for each of the\n+ to-be-updated entries.\n+\n+* Step 3b: `entry.c:checkout_entry()` examines the existing working tree\n+ for file conflicts, collisions, and unsaved changes. It removes files\n+ and creates leading directories as necessary. It calls the Step 3c\n+ function for each entry to be written.\n+\n+* Step 3c: `entry.c:write_entry()` loads the blob into memory, smudges\n+ it if necessary, creates the file in the working tree, writes the\n+ smudged contents, calls `fstat()` or `lstat()`, and updates the\n+ associated `cache_entry` struct with the stat information gathered.\n+\n+It wouldn't be safe to perform Step 3b in parallel, as there could be\n+race conditions between file creations and removals. Instead, the\n+parallel checkout framework lets the sequential code handle Step 3b,\n+and use parallel workers to replace the sequential\n+`entry.c:write_entry()` calls from Step 3c.\n+\n+Rejected Multi-Threaded Solution\n+--------------------------------\n+\n+The most \"straightforward\" implementation would be to spread the set of\n+to-be-updated cache entries across multiple threads. But due to the\n+thread-unsafe functions in the ODB code, we would have to use locks to\n+coordinate the parallel operation. An early prototype of this solution\n+showed that the multi-threaded checkout would bring performance\n+improvements over the sequential code, but there was still too much lock\n+contention. A `perf` profiling indicated that around 20% of the runtime\n+during a local Linux clone (on an SSD) was spent in locking functions.\n+For this reason this approach was rejected in favor of using multiple\n+child processes, which led to a better performance.\n+\n+Multi-Process Solution\n+----------------------\n+\n+Parallel checkout alters the aforementioned Step 3 to use multiple\n+`checkout--worker` background processes to distribute the work. The\n+long-running worker processes are controlled by the foreground Git\n+command using the existing run-command API.\n+\n+Overview\n+~~~~~~~~\n+\n+Step 3b is only slightly altered; for each entry to be checked out, the\n+main process performs the following steps:\n+\n+* M1: Check whether there is any untracked or unclean file in the\n+ working tree which would be overwritten by this entry, and decide\n+ whether to proceed (removing the file(s)) or not.\n+\n+* M2: Create the leading directories.\n+\n+* M3: Load the conversion attributes for the entry's path.\n+\n+* M4: Check, based on the entry's type and conversion attributes,\n+ whether the entry is eligible for parallel checkout (more on this\n+ later). If it is eligible, enqueue the entry and the loaded\n+ attributes to later write the entry in parallel. If not, write the\n+ entry right away, using the default sequential code.\n+\n+Note: we save the conversion attributes associated with each entry\n+because the workers don't have access to the main process' index state,\n+so they can't load the attributes by themselves (and the attributes are\n+needed to properly smudge the entry). Additionally, this has a positive\n+impact on performance as (1) we don't need to load the attributes twice\n+and (2) the attributes machinery is optimized to handle paths in\n+sequential order.\n+\n+After all entries have passed through the above steps, the main process\n+checks if the number of enqueued entries is sufficient to spread among\n+the workers. If not, it just writes them sequentially. Otherwise, it\n+spawns the workers and distributes the queued entries uniformly in\n+continuous chunks. This aims to minimize the chances of two workers\n+writing to the same directory simultaneously, which could increase lock\n+contention in the kernel.\n+\n+Then, for each assigned item, each worker:\n+\n+* W1: Checks if there is any non-directory file in the leading part of\n+ the entry's path or if there already exists a file at the entry' path.\n+ If so, mark the entry with `PC_ITEM_COLLIDED` and skip it (more on\n+ this later).\n+\n+* W2: Creates the file (with O_CREAT and O_EXCL).\n+\n+* W3: Loads the blob into memory (inflating and delta reconstructing\n+ it).\n+\n+* W4: Applies any required in-process filter, like end-of-line\n+ conversion and re-encoding.\n+\n+* W5: Writes the result to the file descriptor opened at W2.\n+\n+* W6: Calls `fstat()` or lstat()` on the just-written path, and sends\n+ the result back to the main process, together with the end status of\n+ the operation and the item's identification number.\n+\n+Note that, when possible, steps W3 to W5 are delegated to the streaming\n+machinery, removing the need to keep the entire blob in memory.\n+\n+Also note that the workers *never* remove any file. As mentioned\n+earlier, it is the responsibility of the main process to remove any file\n+that blocks the checkout operation (or abort if removing the file(s)\n+would cause data loss and the user didn't ask to `--force`). This is\n+crucial to avoid race conditions and also to properly detect path\n+collisions at Step W1.\n+\n+After the workers finish writing the items and sending back the required\n+information, the main process handles the results in two steps:\n+\n+- First, it updates the in-memory index with the `lstat()` information\n+ sent by the workers. (This must be done first as this information\n+ might me required in the following step.)\n+\n+- Then it writes the items which collided on disk (i.e. items marked\n+ with `PC_ITEM_COLLIDED`). More on this below.\n+\n+Path Collisions\n+---------------\n+\n+Path collisions happen when two different paths correspond to the same\n+entry in the file system. E.g. the paths 'a' and 'A' would collide in a\n+case-insensitive file system.\n+\n+The sequential checkout deals with collisions in the same way that it\n+deals with files that were already present in the working tree before\n+checkout. Basically, it checks if the path that it wants to write\n+already exists on disk, makes sure the existing file doesn't have\n+unsaved data, and then overwrites it. (To be more pedantic: it deletes\n+the existing file and creates the new one.) So, if there are multiple\n+colliding files to be checked out, the sequential code will write each\n+one of them but only the last will actually survive on disk.\n+\n+Parallel checkout aims to reproduce the same behavior. However, we\n+cannot let the workers racily write to the same file on disk. Instead,\n+the workers detect when the entry that they want to check out would\n+collide with an existing file, and mark it with `PC_ITEM_COLLIDED`.\n+Later, the main process can sequentially feed these entries back to\n+`checkout_entry()` without the risk of race conditions. On clone, this\n+also has the effect of marking the colliding entries to later emit a\n+warning for the user, like the classic sequential checkout does.\n+\n+The workers are able to detect both collisions among the entries being\n+concurrently written and collisions among parallel-eligible and\n+ineligible entries. The general idea for collision detection is quite\n+straightforward: for each parallel-eligible entry, the main process must\n+remove all files that prevent this entry from being written (before\n+enqueueing it). This includes any non-directory file in the leading path\n+of the entry. Later, when a worker gets assigned the entry, it looks\n+again for the non-directories files and for an already existing file at\n+the entry's path. If any of these checks finds something, the worker\n+knows that there was a path collision.\n+\n+Because parallel checkout can distinguish path collisions from the case\n+where the file was already present in the working tree before checkout,\n+we could alternatively choose to skip the checkout of colliding entries.\n+However, each entry that doesn't get written would have NULL `lstat()`\n+fields on the index. This could cause performance penalties for\n+subsequent commands that need to refresh the index, as they would have\n+to go to the file system to see if the entry is dirty. Thus, if we have\n+N entries in a colliding group and we decide to write and `lstat()` only\n+one of them, every subsequent `git-status` will have to read, convert,\n+and hash the written file N - 1 times. By checking out all colliding\n+entries (like the sequential code does), we only pay the overhead once,\n+during checkout.\n+\n+Eligible Entries for Parallel Checkout\n+--------------------------------------\n+\n+As previously mentioned, not all entries passed to `checkout_entry()`\n+will be considered eligible for parallel checkout. More specifically, we\n+exclude:\n+\n+- Symbolic links; to avoid race conditions that, in combination with\n+ path collisions, could cause workers to write files at the wrong\n+ place. For example, if we were to concurrently check out a symlink\n+ 'a' -> 'b' and a regular file 'A/f' in a case-insensitive file system,\n+ we could potentially end up writing the file 'A/f' at 'a/f', due to a\n+ race condition.\n+\n+- Regular files that require external filters (either \"one shot\" filters\n+ or long-running process filters). These filters are black-boxes to Git\n+ and may have their own internal locking or non-concurrent assumptions.\n+ So it might not be safe to run multiple instances in parallel.\n++\n+Besides, long-running filters may use the delayed checkout feature to\n+postpone the return of some filtered blobs. The delayed checkout queue\n+and the parallel checkout queue are not compatible and should remain\n+separated.\n++\n+Note: regular files that only require internal filters, like end-of-line\n+conversion and re-encoding, are eligible for parallel checkout.\n+\n+Ineligible entries are checked out by the classic sequential codepath\n+*before* spawning workers.\n+\n+Note: submodules's files are also eligible for parallel checkout (as\n+long as they don't fall into any of the excluding categories mentioned\n+above). But since each submodule is checked out in its own child\n+process, we don't mix the superproject's and the submodules' files in\n+the same parallel checkout process or queue.\n+\n+The API\n+-------\n+\n+The parallel checkout API was designed with the goal to minimize changes\n+to the current users of the checkout machinery. This means that they\n+don't have to call a different function for sequential or parallel\n+checkout. As already mentioned, `checkout_entry()` will automatically\n+insert the given entry in the parallel checkout queue when this feature\n+is enabled and the entry is eligible; otherwise, it will just write the\n+entry right away, using the sequential code. In general, callers of the\n+parallel checkout API should look similar to this:\n+\n+----------------------------------------------\n+int pc_workers, pc_threshold, err = 0;\n+struct checkout state;\n+\n+get_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n+/*\n+ * This check is not strictly required, but it\n+ * should save some time in sequential mode.\n+ */\n+if (pc_workers > 1)\n+\tinit_parallel_checkout();\n+\n+for (each cache_entry ce to-be-updated)\n+\terr |= checkout_entry(ce, &state, NULL, NULL);\n+\n+err |= run_parallel_checkout(&state, pc_workers, pc_threshold, NULL, NULL);\n+----------------------------------------------\n-- \n2.30.1\n\n"},{"id":"421259","messageId":"xmqqtuogsgen.fsf@gitster.g","threadId":"55342","inReplyTo":"cover.1617893234.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v2 0/5] Parallel Checkout (part 2)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-08T19:52:16Z","receivedAt":"2021-04-08T19:52:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares writes:\n\n> This is the next step in the parallel checkout implementation. As\n> mt/parallel-checkout-part-1 is now on master, this round is based\n> directly on master.\n\nThanks, will replace.\n"},{"id":"422127","messageId":"xmqqpmytkis8.fsf@gitster.g","threadId":"55342","inReplyTo":"cover.1617893234.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v2 0/5] Parallel Checkout (part 2)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-16T21:43:03Z","receivedAt":"2021-04-16T21:43:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matheus Tavares writes:\n\n> This is the next step in the parallel checkout implementation. As\n> mt/parallel-checkout-part-1 is now on master, this round is based\n> directly on master.\n>\n> Changes since v1:\n\nNow, I do not think we've seen any response to these messages.\n\nIt seems that review comments for the earlier round cf.\n\n\nhave been addressed? Should we merge it down to 'next' now?\n\n"},{"id":"422186","messageId":"CAHd-oW4qdVTQyEeUB5_78iDoK-QWMNXOZ5cN9HuONYASvw6zpA@mail.gmail.com","threadId":"55342","inReplyTo":"xmqqpmytkis8.fsf@gitster.g","subject":"Re: [PATCH v2 0/5] Parallel Checkout (part 2)","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-17T19:57:03Z","receivedAt":"2021-04-17T19:57:19Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Fri, Apr 16, 2021 at 6:43 PM Junio C Hamano wrote:\n>\n> Matheus Tavares writes:\n>\n> > This is the next step in the parallel checkout implementation. As\n> > mt/parallel-checkout-part-1 is now on master, this round is based\n> > directly on master.\n> >\n> > Changes since v1:\n>\n> Now, I do not think we've seen any response to these messages.\n>\n> It seems that review comments for the earlier round cf.\n> \n>\n> have been addressed? Should we merge it down to 'next' now?\n\nYes, I think I've addressed all the comments from the previous round.\nThere is just a minor change that I made locally yesterday, so I'll\nreroll the series with it now. Nonetheless, the change is quite small\nand trivial, so I think the rerolled version can be merged to 'next'.\n"},{"id":"422220","messageId":"cover.1618790794.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1617893234.git.matheus.bernardino@usp.br","subject":"[PATCH v3 0/5] Parallel Checkout (part 2)","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T00:14:52Z","receivedAt":"2021-04-19T00:15:08Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Three small changes since the last version: \n\n- In `write_pc_item_to_fd()`, renamed variable 'new_blob' to just 'blob'.\n\n- In `write_pc_item_to_fd()`, used `ce->name` instead of `path` when\n reporting failure to read the object.\n\n- Made `write_pc_item()` unlink() the file on a `write_pc_item_to_fd()`\n error. The reasoning for this is:\n\n Unlike sequential checkout, the parallel workers create each file\n *before* reading and converting the associated blob, so that they can\n detect path collisions as soon as possible in the process. This is\n important both to avoid unnecessary work reading and filtering\n objects that are not going to be written in parallel, and also to\n avoid reporting errors for colliding entries at the wrong time. These\n entries will be handled in a sequential phase later, and that will be\n the correct time to print any errors regarding their checkout.\n\n However, these logistics make us leave an empty file behind when we\n find an error after creating the file, e.g. a missing object. The\n added unlink() fix this case while maintaining the other important\n mechanics. The design doc was also adjusted to mention this.\n\nMatheus Tavares (5):\n unpack-trees: add basic support for parallel checkout\n parallel-checkout: make it truly parallel\n parallel-checkout: add configuration options\n parallel-checkout: support progress displaying\n parallel-checkout: add design documentation\n\n .gitignore | 1 +\n Documentation/Makefile | 1 +\n Documentation/config/checkout.txt | 21 +\n Documentation/technical/parallel-checkout.txt | 271 ++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/checkout--worker.c | 145 ++++\n entry.c | 17 +-\n git.c | 2 +\n parallel-checkout.c | 655 ++++++++++++++++++\n parallel-checkout.h | 111 +++\n unpack-trees.c | 19 +-\n 12 files changed, 1241 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/parallel-checkout.txt\n create mode 100644 builtin/checkout--worker.c\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\nRange-diff against v2:\n1: 0506ac7159 ! 1: 7096822c14 unpack-trees: add basic support for parallel checkout\n @@ parallel-checkout.c (new)\n +\tint ret;\n +\tstruct stream_filter *filter;\n +\tstruct strbuf buf = STRBUF_INIT;\n -+\tchar *new_blob;\n ++\tchar *blob;\n +\tunsigned long size;\n +\tssize_t wrote;\n +\n @@ parallel-checkout.c (new)\n +\t\t}\n +\t}\n +\n -+\tnew_blob = read_blob_entry(pc_item->ce, &size);\n -+\tif (!new_blob)\n ++\tblob = read_blob_entry(pc_item->ce, &size);\n ++\tif (!blob)\n +\t\treturn error(\"cannot read object %s '%s'\",\n -+\t\t\t oid_to_hex(&pc_item->ce->oid), path);\n ++\t\t\t oid_to_hex(&pc_item->ce->oid), pc_item->ce->name);\n +\n +\t/*\n +\t * checkout metadata is used to give context for external process\n @@ parallel-checkout.c (new)\n +\t * checkout, so pass NULL.\n +\t */\n +\tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n -+\t\t\t\t\t new_blob, size, &buf, NULL);\n ++\t\t\t\t\t blob, size, &buf, NULL);\n +\n +\tif (ret) {\n +\t\tsize_t newsize;\n -+\t\tfree(new_blob);\n -+\t\tnew_blob = strbuf_detach(&buf, &newsize);\n ++\t\tfree(blob);\n ++\t\tblob = strbuf_detach(&buf, &newsize);\n +\t\tsize = newsize;\n +\t}\n +\n -+\twrote = write_in_full(fd, new_blob, size);\n -+\tfree(new_blob);\n ++\twrote = write_in_full(fd, blob, size);\n ++\tfree(blob);\n +\tif (wrote < 0)\n +\t\treturn error(\"unable to write file '%s'\", path);\n +\n @@ parallel-checkout.c (new)\n +\tif (write_pc_item_to_fd(pc_item, fd, path.buf)) {\n +\t\t/* Error was already reported. */\n +\t\tpc_item->status = PC_ITEM_FAILED;\n ++\t\tclose_and_clear(&fd);\n ++\t\tunlink(path.buf);\n +\t\tgoto out;\n +\t}\n +\n @@ parallel-checkout.c (new)\n +\tpc_item->status = PC_ITEM_WRITTEN;\n +\n +out:\n -+\t/*\n -+\t * No need to check close() return at this point. Either fd is already\n -+\t * closed, or we are on an error path.\n -+\t */\n -+\tclose_and_clear(&fd);\n +\tstrbuf_release(&path);\n +}\n +\n2: 0d65b517bd ! 2: 4526516ea0 parallel-checkout: make it truly parallel\n @@ parallel-checkout.c: static int write_pc_item_to_fd(struct parallel_checkout_ite\n +\t * be passed from the main process to the workers.\n \t */\n \tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n - \t\t\t\t\t new_blob, size, &buf, NULL);\n + \t\t\t\t\t blob, size, &buf, NULL);\n @@ parallel-checkout.c: static int close_and_clear(int *fd)\n \treturn ret;\n }\n3: 6ea057f9c5 = 3: ad165c0637 parallel-checkout: add configuration options\n4: 0ac4bee67e = 4: cf9e28dc0e parallel-checkout: support progress displaying\n5: 087f8bdf35 ! 5: 415d4114aa parallel-checkout: add design documentation\n @@ Documentation/technical/parallel-checkout.txt (new)\n +Note that, when possible, steps W3 to W5 are delegated to the streaming\n +machinery, removing the need to keep the entire blob in memory.\n +\n -+Also note that the workers *never* remove any file. As mentioned\n -+earlier, it is the responsibility of the main process to remove any file\n -+that blocks the checkout operation (or abort if removing the file(s)\n -+would cause data loss and the user didn't ask to `--force`). This is\n -+crucial to avoid race conditions and also to properly detect path\n -+collisions at Step W1.\n ++If the worker fails to read the blob or to write it to the working tree,\n ++it removes the created file to avoid leaving empty files behind. This is\n ++the *only* time a worker is allowed to remove a file.\n ++\n ++As mentioned earlier, it is the responsibility of the main process to\n ++remove any file that blocks the checkout operation (or abort if the\n ++removal(s) would cause data loss and the user didn't ask to `--force`).\n ++This is crucial to avoid race conditions and also to properly detect\n ++path collisions at Step W1.\n +\n +After the workers finish writing the items and sending back the required\n +information, the main process handles the results in two steps:\n-- \n2.30.1\n\n"},{"id":"422221","messageId":"7096822c143ed4aed66e811227740610f4d6ac8a.1618790794.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618790794.git.matheus.bernardino@usp.br","subject":"[PATCH v3 1/5] unpack-trees: add basic support for parallel checkout","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T00:14:53Z","receivedAt":"2021-04-19T00:15:11Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This new interface allows us to enqueue some of the entries being\nchecked out to later uncompress them, apply in-process filters, and\nwrite out the files in parallel. For now, the parallel checkout\nmachinery is enabled by default and there is no user configuration, but\nrun_parallel_checkout() just writes the queued entries in sequence\n(without spawning additional workers). The next patch will actually\nimplement the parallelism and, later, we will make it configurable.\n\nNote that, to avoid potential data races, not all entries are eligible\nfor parallel checkout. Also, paths that collide on disk (e.g.\ncase-sensitive paths in case-insensitive file systems), are detected by\nthe parallel checkout code and skipped, so that they can be safely\nsequentially handled later. The collision detection works like the\nfollowing:\n\n- If the collision was at basename (e.g. 'a/b' and 'a/B'), the framework\n detects it by looking for EEXIST and EISDIR errors after an\n open(O_CREAT | O_EXCL) failure.\n\n- If the collision was at dirname (e.g. 'a/b' and 'A'), it is detected\n at the has_dirs_only_path() check, which is done for the leading path\n of each item in the parallel checkout queue.\n\nBoth verifications rely on the fact that, before enqueueing an entry for\nparallel checkout, checkout_entry() makes sure that there is no file at\nthe entry's path and that its leading components are all real\ndirectories. So, any later change in these conditions indicates that\nthere was a collision (either between two parallel-eligible entries or\nbetween an eligible and an ineligible one).\n\nAfter all parallel-eligible entries have been processed, the collided\n(and thus, skipped) entries are sequentially fed to checkout_entry()\nagain. This is similar to the way the current code deals with\ncollisions, overwriting the previously checked out entries with the\nsubsequent ones. The only difference is that, since we no longer create\nthe files in the same order that they appear on index, we are not able\nto determine which of the colliding entries will survive on disk (for\nthe classic code, it is always the last entry).\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Makefile | 1 +\n entry.c | 17 ++-\n parallel-checkout.c | 365 ++++++++++++++++++++++++++++++++++++++++++++\n parallel-checkout.h | 32 ++++\n unpack-trees.c | 6 +-\n 5 files changed, 418 insertions(+), 3 deletions(-)\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\ndiff --git a/Makefile b/Makefile\nindex a6a73c5741..99734cc2ea 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -946,6 +946,7 @@ LIB_OBJS += pack-revindex.o\n LIB_OBJS += pack-write.o\n LIB_OBJS += packfile.o\n LIB_OBJS += pager.o\n+LIB_OBJS += parallel-checkout.o\n LIB_OBJS += parse-options-cb.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\ndiff --git a/entry.c b/entry.c\nindex 2dc94ba5cc..d7ed38aa40 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -7,6 +7,7 @@\n #include \"progress.h\"\n #include \"fsmonitor.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -426,8 +427,17 @@ static void mark_colliding_entries(const struct checkout *state,\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-\t\tif (dup == ce)\n-\t\t\tbreak;\n+\t\tif (dup == ce) {\n+\t\t\t/*\n+\t\t\t * Parallel checkout doesn't create the files in index\n+\t\t\t * order. So the other side of the collision may appear\n+\t\t\t * after the given cache_entry in the array.\n+\t\t\t */\n+\t\t\tif (parallel_checkout_status() == PC_RUNNING)\n+\t\t\t\tcontinue;\n+\t\t\telse\n+\t\t\t\tbreak;\n+\t\t}\n \n \t\tif (dup->ce_flags & (CE_MATCHED | CE_VALID | CE_SKIP_WORKTREE))\n \t\t\tcontinue;\n@@ -536,6 +546,9 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\tca = &ca_buf;\n \t}\n \n+\tif (!enqueue_checkout(ce, ca))\n+\t\treturn 0;\n+\n \treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nnew file mode 100644\nindex 0000000000..590e2a3046\n--- /dev/null\n+++ b/parallel-checkout.c\n@@ -0,0 +1,365 @@\n+#include \"cache.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"streaming.h\"\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/* pointer to a istate->cache[] entry. Not owned by us. */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tstruct stat st;\n+\tenum pc_item_status status;\n+};\n+\n+struct parallel_checkout {\n+\tenum pc_status status;\n+\tstruct parallel_checkout_item *items; /* The parallel checkout queue. */\n+\tsize_t nr, alloc;\n+};\n+\n+static struct parallel_checkout parallel_checkout;\n+\n+enum pc_status parallel_checkout_status(void)\n+{\n+\treturn parallel_checkout.status;\n+}\n+\n+void init_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status != PC_UNINITIALIZED)\n+\t\tBUG(\"parallel checkout already initialized\");\n+\n+\tparallel_checkout.status = PC_ACCEPTING_ENTRIES;\n+}\n+\n+static void finish_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status == PC_UNINITIALIZED)\n+\t\tBUG(\"cannot finish parallel checkout: not initialized yet\");\n+\n+\tfree(parallel_checkout.items);\n+\tmemset(¶llel_checkout, 0, sizeof(parallel_checkout));\n+}\n+\n+static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n+\t\t\t\t\t const struct conv_attrs *ca)\n+{\n+\tenum conv_attrs_classification c;\n+\n+\t/*\n+\t * Symlinks cannot be checked out in parallel as, in case of path\n+\t * collision, they could racily replace leading directories of other\n+\t * entries being checked out. Submodules are checked out in child\n+\t * processes, which have their own parallel checkout queues.\n+\t */\n+\tif (!S_ISREG(ce->ce_mode))\n+\t\treturn 0;\n+\n+\tc = classify_conv_attrs(ca);\n+\tswitch (c) {\n+\tcase CA_CLASS_INCORE:\n+\t\treturn 1;\n+\n+\tcase CA_CLASS_INCORE_FILTER:\n+\t\t/*\n+\t\t * It would be safe to allow concurrent instances of\n+\t\t * single-file smudge filters, like rot13, but we should not\n+\t\t * assume that all filters are parallel-process safe. So we\n+\t\t * don't allow this.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_INCORE_PROCESS:\n+\t\t/*\n+\t\t * The parallel queue and the delayed queue are not compatible,\n+\t\t * so they must be kept completely separated. And we can't tell\n+\t\t * if a long-running process will delay its response without\n+\t\t * actually asking it to perform the filtering. Therefore, this\n+\t\t * type of filter is not allowed in parallel checkout.\n+\t\t *\n+\t\t * Furthermore, there should only be one instance of the\n+\t\t * long-running process filter as we don't know how it is\n+\t\t * managing its own concurrency. So, spreading the entries that\n+\t\t * requisite such a filter among the parallel workers would\n+\t\t * require a lot more inter-process communication. We would\n+\t\t * probably have to designate a single process to interact with\n+\t\t * the filter and send all the necessary data to it, for each\n+\t\t * entry.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_STREAMABLE:\n+\t\treturn 1;\n+\n+\tdefault:\n+\t\tBUG(\"unsupported conv_attrs classification '%d'\", c);\n+\t}\n+}\n+\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n+{\n+\tstruct parallel_checkout_item *pc_item;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES ||\n+\t !is_eligible_for_parallel_checkout(ce, ca))\n+\t\treturn -1;\n+\n+\tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n+\t\t parallel_checkout.alloc);\n+\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item->ce = ce;\n+\tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n+\tpc_item->status = PC_ITEM_PENDING;\n+\n+\treturn 0;\n+}\n+\n+static int handle_results(struct checkout *state)\n+{\n+\tint ret = 0;\n+\tsize_t i;\n+\tint have_pending = 0;\n+\n+\t/*\n+\t * We first update the successfully written entries with the collected\n+\t * stat() data, so that they can be found by mark_colliding_entries(),\n+\t * in the next loop, when necessary.\n+\t */\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\tif (pc_item->status == PC_ITEM_WRITTEN)\n+\t\t\tupdate_ce_after_write(state, pc_item->ce, &pc_item->st);\n+\t}\n+\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\n+\t\tswitch(pc_item->status) {\n+\t\tcase PC_ITEM_WRITTEN:\n+\t\t\t/* Already handled */\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_COLLIDED:\n+\t\t\t/*\n+\t\t\t * The entry could not be checked out due to a path\n+\t\t\t * collision with another entry. Since there can only\n+\t\t\t * be one entry of each colliding group on the disk, we\n+\t\t\t * could skip trying to check out this one and move on.\n+\t\t\t * However, this would leave the unwritten entries with\n+\t\t\t * null stat() fields on the index, which could\n+\t\t\t * potentially slow down subsequent operations that\n+\t\t\t * require refreshing it: git would not be able to\n+\t\t\t * trust st_size and would have to go to the filesystem\n+\t\t\t * to see if the contents match (see ie_modified()).\n+\t\t\t *\n+\t\t\t * Instead, let's pay the overhead only once, now, and\n+\t\t\t * call checkout_entry_ca() again for this file, to\n+\t\t\t * have its stat() data stored in the index. This also\n+\t\t\t * has the benefit of adding this entry and its\n+\t\t\t * colliding pair to the collision report message.\n+\t\t\t * Additionally, this overwriting behavior is consistent\n+\t\t\t * with what the sequential checkout does, so it doesn't\n+\t\t\t * add any extra overhead.\n+\t\t\t */\n+\t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n+\t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_PENDING:\n+\t\t\thave_pending = 1;\n+\t\t\t/* fall through */\n+\t\tcase PC_ITEM_FAILED:\n+\t\t\tret = -1;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unknown checkout item status in parallel checkout\");\n+\t\t}\n+\t}\n+\n+\tif (have_pending)\n+\t\terror(\"parallel checkout finished with pending entries\");\n+\n+\treturn ret;\n+}\n+\n+static int reset_fd(int fd, const char *path)\n+{\n+\tif (lseek(fd, 0, SEEK_SET) != 0)\n+\t\treturn error_errno(\"failed to rewind descriptor of '%s'\", path);\n+\tif (ftruncate(fd, 0))\n+\t\treturn error_errno(\"failed to truncate file '%s'\", path);\n+\treturn 0;\n+}\n+\n+static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n+\t\t\t const char *path)\n+{\n+\tint ret;\n+\tstruct stream_filter *filter;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *blob;\n+\tunsigned long size;\n+\tssize_t wrote;\n+\n+\t/* Sanity check */\n+\tassert(is_eligible_for_parallel_checkout(pc_item->ce, &pc_item->ca));\n+\n+\tfilter = get_stream_filter_ca(&pc_item->ca, &pc_item->ce->oid);\n+\tif (filter) {\n+\t\tif (stream_blob_to_fd(fd, &pc_item->ce->oid, filter, 1)) {\n+\t\t\t/* On error, reset fd to try writing without streaming */\n+\t\t\tif (reset_fd(fd, path))\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tblob = read_blob_entry(pc_item->ce, &size);\n+\tif (!blob)\n+\t\treturn error(\"cannot read object %s '%s'\",\n+\t\t\t oid_to_hex(&pc_item->ce->oid), pc_item->ce->name);\n+\n+\t/*\n+\t * checkout metadata is used to give context for external process\n+\t * filters. Files requiring such filters are not eligible for parallel\n+\t * checkout, so pass NULL.\n+\t */\n+\tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n+\t\t\t\t\t blob, size, &buf, NULL);\n+\n+\tif (ret) {\n+\t\tsize_t newsize;\n+\t\tfree(blob);\n+\t\tblob = strbuf_detach(&buf, &newsize);\n+\t\tsize = newsize;\n+\t}\n+\n+\twrote = write_in_full(fd, blob, size);\n+\tfree(blob);\n+\tif (wrote < 0)\n+\t\treturn error(\"unable to write file '%s'\", path);\n+\n+\treturn 0;\n+}\n+\n+static int close_and_clear(int *fd)\n+{\n+\tint ret = 0;\n+\n+\tif (*fd >= 0) {\n+\t\tret = close(*fd);\n+\t\t*fd = -1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t\t struct checkout *state)\n+{\n+\tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n+\tint fd = -1, fstat_done = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tconst char *dir_sep;\n+\n+\tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n+\tstrbuf_add(&path, pc_item->ce->name, pc_item->ce->ce_namelen);\n+\n+\tdir_sep = find_last_dir_sep(path.buf);\n+\n+\t/*\n+\t * The leading dirs should have been already created by now. But, in\n+\t * case of path collisions, one of the dirs could have been replaced by\n+\t * a symlink (checked out after we enqueued this entry for parallel\n+\t * checkout). Thus, we must check the leading dirs again.\n+\t */\n+\tif (dir_sep && !has_dirs_only_path(path.buf, dir_sep - path.buf,\n+\t\t\t\t\t state->base_dir_len)) {\n+\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\tgoto out;\n+\t}\n+\n+\tfd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, mode);\n+\n+\tif (fd < 0) {\n+\t\tif (errno == EEXIST || errno == EISDIR) {\n+\t\t\t/*\n+\t\t\t * Errors which probably represent a path collision.\n+\t\t\t * Suppress the error message and mark the item to be\n+\t\t\t * retried later, sequentially. ENOTDIR and ENOENT are\n+\t\t\t * also interesting, but the above has_dirs_only_path()\n+\t\t\t * call should have already caught these cases.\n+\t\t\t */\n+\t\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\t} else {\n+\t\t\terror_errno(\"failed to open file '%s'\", path.buf);\n+\t\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\t}\n+\t\tgoto out;\n+\t}\n+\n+\tif (write_pc_item_to_fd(pc_item, fd, path.buf)) {\n+\t\t/* Error was already reported. */\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tclose_and_clear(&fd);\n+\t\tunlink(path.buf);\n+\t\tgoto out;\n+\t}\n+\n+\tfstat_done = fstat_checkout_output(fd, state, &pc_item->st);\n+\n+\tif (close_and_clear(&fd)) {\n+\t\terror_errno(\"unable to close file '%s'\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tif (state->refresh_cache && !fstat_done && lstat(path.buf, &pc_item->st) < 0) {\n+\t\terror_errno(\"unable to stat just-written file '%s'\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tpc_item->status = PC_ITEM_WRITTEN;\n+\n+out:\n+\tstrbuf_release(&path);\n+}\n+\n+static void write_items_sequentially(struct checkout *state)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < parallel_checkout.nr; i++)\n+\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+}\n+\n+int run_parallel_checkout(struct checkout *state)\n+{\n+\tint ret;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n+\t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n+\n+\tparallel_checkout.status = PC_RUNNING;\n+\n+\twrite_items_sequentially(state);\n+\tret = handle_results(state);\n+\n+\tfinish_parallel_checkout();\n+\treturn ret;\n+}\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nnew file mode 100644\nindex 0000000000..4ad2a519b3\n--- /dev/null\n+++ b/parallel-checkout.h\n@@ -0,0 +1,32 @@\n+#ifndef PARALLEL_CHECKOUT_H\n+#define PARALLEL_CHECKOUT_H\n+\n+struct cache_entry;\n+struct checkout;\n+struct conv_attrs;\n+\n+enum pc_status {\n+\tPC_UNINITIALIZED = 0,\n+\tPC_ACCEPTING_ENTRIES,\n+\tPC_RUNNING,\n+};\n+\n+enum pc_status parallel_checkout_status(void);\n+\n+/*\n+ * Put parallel checkout into the PC_ACCEPTING_ENTRIES state. Should be used\n+ * only when in the PC_UNINITIALIZED state.\n+ */\n+void init_parallel_checkout(void);\n+\n+/*\n+ * Return -1 if parallel checkout is currently not accepting entries or if the\n+ * entry is not eligible for parallel checkout. Otherwise, enqueue the entry\n+ * for later write and return 0.\n+ */\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+\n+/* Write all the queued entries, returning 0 on success.*/\n+int run_parallel_checkout(struct checkout *state);\n+\n+#endif /* PARALLEL_CHECKOUT_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8a1afbc1e4..f0430d458d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -17,6 +17,7 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -441,7 +442,6 @@ static int check_updates(struct unpack_trees_options *o,\n \tif (should_update_submodules())\n \t\tload_gitmodules_file(index, &state);\n \n-\tenable_delayed_checkout(&state);\n \tif (has_promisor_remote()) {\n \t\t/*\n \t\t * Prefetch the objects that are to be checked out in the loop\n@@ -464,6 +464,9 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t\t\t\t to_fetch.oid, to_fetch.nr);\n \t\toid_array_clear(&to_fetch);\n \t}\n+\n+\tenable_delayed_checkout(&state);\n+\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -477,6 +480,7 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n+\terrs |= run_parallel_checkout(&state);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"422222","messageId":"4526516ea0e2c7897758eb54f0d56bfe49cc9f89.1618790794.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618790794.git.matheus.bernardino@usp.br","subject":"[PATCH v3 2/5] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T00:14:54Z","receivedAt":"2021-04-19T00:15:12Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Use multiple worker processes to distribute the queued entries and call\nwrite_pc_item() in parallel for them. The items are distributed\nuniformly in contiguous chunks. This minimizes the chances of two\nworkers writing to the same directory simultaneously, which could affect\nperformance due to lock contention in the kernel. Work stealing (or any\nother format of re-distribution) is not implemented yet.\n\nThe protocol between the main process and the workers is quite simple.\nThey exchange binary messages packed in pkt-line format, and use\nPKT-FLUSH to mark the end of input (from both sides). The main process\nstarts the communication by sending N pkt-lines, each corresponding to\nan item that needs to be written. These packets contain all the\nnecessary information to load, smudge, and write the blob associated\nwith each item. Then it waits for the worker to send back N pkt-lines\ncontaining the results for each item. The resulting packet must contain:\nthe identification number of the item that it refers to, the status of\nthe operation, and the lstat() data gathered after writing the file (iff\nthe operation was successful).\n\nFor now, checkout always uses a hardcoded value of 2 workers, only to\ndemonstrate that the parallel checkout framework correctly divides and\nwrites the queued entries. The next patch will add user configurations\nand define a more reasonable default, based on tests with the said\nsettings.\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/checkout--worker.c | 145 ++++++++++++++++++\n git.c | 2 +\n parallel-checkout.c | 300 +++++++++++++++++++++++++++++++++----\n parallel-checkout.h | 73 ++++++++-\n 7 files changed, 496 insertions(+), 27 deletions(-)\n create mode 100644 builtin/checkout--worker.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 3dcdb6bb5a..96c794b1c7 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -33,6 +33,7 @@\n /git-check-mailmap\n /git-check-ref-format\n /git-checkout\n+/git-checkout--worker\n /git-checkout-index\n /git-cherry\n /git-cherry-pick\ndiff --git a/Makefile b/Makefile\nindex 99734cc2ea..84e15f58a0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1062,6 +1062,7 @@ BUILTIN_OBJS += builtin/check-attr.o\n BUILTIN_OBJS += builtin/check-ignore.o\n BUILTIN_OBJS += builtin/check-mailmap.o\n BUILTIN_OBJS += builtin/check-ref-format.o\n+BUILTIN_OBJS += builtin/checkout--worker.o\n BUILTIN_OBJS += builtin/checkout-index.o\n BUILTIN_OBJS += builtin/checkout.o\n BUILTIN_OBJS += builtin/clean.o\ndiff --git a/builtin.h b/builtin.h\nindex b6ce981b73..16ecd5586f 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -123,6 +123,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix);\n int cmd_bundle(int argc, const char **argv, const char *prefix);\n int cmd_cat_file(int argc, const char **argv, const char *prefix);\n int cmd_checkout(int argc, const char **argv, const char *prefix);\n+int cmd_checkout__worker(int argc, const char **argv, const char *prefix);\n int cmd_checkout_index(int argc, const char **argv, const char *prefix);\n int cmd_check_attr(int argc, const char **argv, const char *prefix);\n int cmd_check_ignore(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/checkout--worker.c b/builtin/checkout--worker.c\nnew file mode 100644\nindex 0000000000..31e0de2f7e\n--- /dev/null\n+++ b/builtin/checkout--worker.c\n@@ -0,0 +1,145 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"parse-options.h\"\n+#include \"pkt-line.h\"\n+\n+static void packet_to_pc_item(const char *buffer, int len,\n+\t\t\t struct parallel_checkout_item *pc_item)\n+{\n+\tconst struct pc_item_fixed_portion *fixed_portion;\n+\tconst char *variant;\n+\tchar *encoding;\n+\n+\tif (len < sizeof(struct pc_item_fixed_portion))\n+\t\tBUG(\"checkout worker received too short item (got %dB, exp %dB)\",\n+\t\t len, (int)sizeof(struct pc_item_fixed_portion));\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)buffer;\n+\n+\tif (len - sizeof(struct pc_item_fixed_portion) !=\n+\t\tfixed_portion->name_len + fixed_portion->working_tree_encoding_len)\n+\t\tBUG(\"checkout worker received corrupted item\");\n+\n+\tvariant = buffer + sizeof(struct pc_item_fixed_portion);\n+\n+\t/*\n+\t * Note: the main process uses zero length to communicate that the\n+\t * encoding is NULL. There is no use case that requires sending an\n+\t * actual empty string, since convert_attrs() never sets\n+\t * ca.working_tree_enconding to \"\".\n+\t */\n+\tif (fixed_portion->working_tree_encoding_len) {\n+\t\tencoding = xmemdupz(variant,\n+\t\t\t\t fixed_portion->working_tree_encoding_len);\n+\t\tvariant += fixed_portion->working_tree_encoding_len;\n+\t} else {\n+\t\tencoding = NULL;\n+\t}\n+\n+\tmemset(pc_item, 0, sizeof(*pc_item));\n+\tpc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n+\tpc_item->ce->ce_namelen = fixed_portion->name_len;\n+\tpc_item->ce->ce_mode = fixed_portion->ce_mode;\n+\tmemcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen);\n+\toidcpy(&pc_item->ce->oid, &fixed_portion->oid);\n+\n+\tpc_item->id = fixed_portion->id;\n+\tpc_item->ca.crlf_action = fixed_portion->crlf_action;\n+\tpc_item->ca.ident = fixed_portion->ident;\n+\tpc_item->ca.working_tree_encoding = encoding;\n+}\n+\n+static void report_result(struct parallel_checkout_item *pc_item)\n+{\n+\tstruct pc_item_result res;\n+\tsize_t size;\n+\n+\tres.id = pc_item->id;\n+\tres.status = pc_item->status;\n+\n+\tif (pc_item->status == PC_ITEM_WRITTEN) {\n+\t\tres.st = pc_item->st;\n+\t\tsize = sizeof(res);\n+\t} else {\n+\t\tsize = PC_ITEM_RESULT_BASE_SIZE;\n+\t}\n+\n+\tpacket_write(1, (const char *)&res, size);\n+}\n+\n+/* Free the worker-side malloced data, but not pc_item itself. */\n+static void release_pc_item_data(struct parallel_checkout_item *pc_item)\n+{\n+\tfree((char *)pc_item->ca.working_tree_encoding);\n+\tdiscard_cache_entry(pc_item->ce);\n+}\n+\n+static void worker_loop(struct checkout *state)\n+{\n+\tstruct parallel_checkout_item *items = NULL;\n+\tsize_t i, nr = 0, alloc = 0;\n+\n+\twhile (1) {\n+\t\tint len = packet_read(0, NULL, NULL, packet_buffer,\n+\t\t\t\t sizeof(packet_buffer), 0);\n+\n+\t\tif (len < 0)\n+\t\t\tBUG(\"packet_read() returned negative value\");\n+\t\telse if (!len)\n+\t\t\tbreak;\n+\n+\t\tALLOC_GROW(items, nr + 1, alloc);\n+\t\tpacket_to_pc_item(packet_buffer, len, &items[nr++]);\n+\t}\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = &items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\treport_result(pc_item);\n+\t\trelease_pc_item_data(pc_item);\n+\t}\n+\n+\tpacket_flush(1);\n+\n+\tfree(items);\n+}\n+\n+static const char * const checkout_worker_usage[] = {\n+\tN_(\"git checkout--worker []\"),\n+\tNULL\n+};\n+\n+int cmd_checkout__worker(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct checkout state = CHECKOUT_INIT;\n+\tstruct option checkout_worker_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &state.base_dir, N_(\"string\"),\n+\t\t\tN_(\"when creating files, prepend \")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(checkout_worker_usage,\n+\t\t\t\t checkout_worker_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix, checkout_worker_options,\n+\t\t\t checkout_worker_usage, 0);\n+\tif (argc > 0)\n+\t\tusage_with_options(checkout_worker_usage, checkout_worker_options);\n+\n+\tif (state.base_dir)\n+\t\tstate.base_dir_len = strlen(state.base_dir);\n+\n+\t/*\n+\t * Setting this on a worker won't actually update the index. We just\n+\t * need to tell the checkout machinery to lstat() the written entries,\n+\t * so that we can send this data back to the main process.\n+\t */\n+\tstate.refresh_cache = 1;\n+\n+\tworker_loop(&state);\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 9bc077a025..532b5b4136 100644\n--- a/git.c\n+++ b/git.c\n@@ -490,6 +490,8 @@ static struct cmd_struct commands[] = {\n \t{ \"check-mailmap\", cmd_check_mailmap, RUN_SETUP },\n \t{ \"check-ref-format\", cmd_check_ref_format, NO_PARSEOPT },\n \t{ \"checkout\", cmd_checkout, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"checkout--worker\", cmd_checkout__worker,\n+\t\tRUN_SETUP | NEED_WORK_TREE | SUPPORT_SUPER_PREFIX },\n \t{ \"checkout-index\", cmd_checkout_index,\n \t\tRUN_SETUP | NEED_WORK_TREE},\n \t{ \"cherry\", cmd_cherry, RUN_SETUP },\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 590e2a3046..836154fec6 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,28 +1,14 @@\n #include \"cache.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n+#include \"pkt-line.h\"\n+#include \"run-command.h\"\n+#include \"sigchain.h\"\n #include \"streaming.h\"\n \n-enum pc_item_status {\n-\tPC_ITEM_PENDING = 0,\n-\tPC_ITEM_WRITTEN,\n-\t/*\n-\t * The entry could not be written because there was another file\n-\t * already present in its path or leading directories. Since\n-\t * checkout_entry_ca() removes such files from the working tree before\n-\t * enqueueing the entry for parallel checkout, it means that there was\n-\t * a path collision among the entries being written.\n-\t */\n-\tPC_ITEM_COLLIDED,\n-\tPC_ITEM_FAILED,\n-};\n-\n-struct parallel_checkout_item {\n-\t/* pointer to a istate->cache[] entry. Not owned by us. */\n-\tstruct cache_entry *ce;\n-\tstruct conv_attrs ca;\n-\tstruct stat st;\n-\tenum pc_item_status status;\n+struct pc_worker {\n+\tstruct child_process cp;\n+\tsize_t next_item_to_complete, nr_items_to_complete;\n };\n \n struct parallel_checkout {\n@@ -59,6 +45,7 @@ static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n \t\t\t\t\t const struct conv_attrs *ca)\n {\n \tenum conv_attrs_classification c;\n+\tsize_t packed_item_size;\n \n \t/*\n \t * Symlinks cannot be checked out in parallel as, in case of path\n@@ -69,6 +56,18 @@ static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n \tif (!S_ISREG(ce->ce_mode))\n \t\treturn 0;\n \n+\tpacked_item_size = sizeof(struct pc_item_fixed_portion) + ce->ce_namelen +\n+\t\t(ca->working_tree_encoding ? strlen(ca->working_tree_encoding) : 0);\n+\n+\t/*\n+\t * The amount of data we send to the workers per checkout item is\n+\t * typically small (75~300B). So unless we find an insanely huge path\n+\t * of 64KB, we should never reach the 65KB limit of one pkt-line. If\n+\t * that does happen, we let the sequential code handle the item.\n+\t */\n+\tif (packed_item_size > LARGE_PACKET_DATA_MAX)\n+\t\treturn 0;\n+\n \tc = classify_conv_attrs(ca);\n \tswitch (c) {\n \tcase CA_CLASS_INCORE:\n@@ -121,10 +120,12 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n \t\t parallel_checkout.alloc);\n \n-\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr];\n \tpc_item->ce = ce;\n \tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n \tpc_item->status = PC_ITEM_PENDING;\n+\tpc_item->id = parallel_checkout.nr;\n+\tparallel_checkout.nr++;\n \n \treturn 0;\n }\n@@ -236,7 +237,8 @@ static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n \t/*\n \t * checkout metadata is used to give context for external process\n \t * filters. Files requiring such filters are not eligible for parallel\n-\t * checkout, so pass NULL.\n+\t * checkout, so pass NULL. Note: if that changes, the metadata must also\n+\t * be passed from the main process to the workers.\n \t */\n \tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n \t\t\t\t\t blob, size, &buf, NULL);\n@@ -268,8 +270,8 @@ static int close_and_clear(int *fd)\n \treturn ret;\n }\n \n-static void write_pc_item(struct parallel_checkout_item *pc_item,\n-\t\t\t struct checkout *state)\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state)\n {\n \tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n \tint fd = -1, fstat_done = 0;\n@@ -340,6 +342,240 @@ static void write_pc_item(struct parallel_checkout_item *pc_item,\n \tstrbuf_release(&path);\n }\n \n+static void send_one_item(int fd, struct parallel_checkout_item *pc_item)\n+{\n+\tsize_t len_data;\n+\tchar *data, *variant;\n+\tstruct pc_item_fixed_portion *fixed_portion;\n+\tconst char *working_tree_encoding = pc_item->ca.working_tree_encoding;\n+\tsize_t name_len = pc_item->ce->ce_namelen;\n+\tsize_t working_tree_encoding_len = working_tree_encoding ?\n+\t\t\t\t\t strlen(working_tree_encoding) : 0;\n+\n+\t/*\n+\t * Any changes in the calculation of the message size must also be made\n+\t * in is_eligible_for_parallel_checkout().\n+\t */\n+\tlen_data = sizeof(struct pc_item_fixed_portion) + name_len +\n+\t\t working_tree_encoding_len;\n+\n+\tdata = xcalloc(1, len_data);\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)data;\n+\tfixed_portion->id = pc_item->id;\n+\tfixed_portion->ce_mode = pc_item->ce->ce_mode;\n+\tfixed_portion->crlf_action = pc_item->ca.crlf_action;\n+\tfixed_portion->ident = pc_item->ca.ident;\n+\tfixed_portion->name_len = name_len;\n+\tfixed_portion->working_tree_encoding_len = working_tree_encoding_len;\n+\t/*\n+\t * We use hashcpy() instead of oidcpy() because the hash[] positions\n+\t * after `the_hash_algo->rawsz` might not be initialized. And Valgrind\n+\t * would complain about passing uninitialized bytes to a syscall\n+\t * (write(2)). There is no real harm in this case, but the warning could\n+\t * hinder the detection of actual errors.\n+\t */\n+\thashcpy(fixed_portion->oid.hash, pc_item->ce->oid.hash);\n+\n+\tvariant = data + sizeof(*fixed_portion);\n+\tif (working_tree_encoding_len) {\n+\t\tmemcpy(variant, working_tree_encoding, working_tree_encoding_len);\n+\t\tvariant += working_tree_encoding_len;\n+\t}\n+\tmemcpy(variant, pc_item->ce->name, name_len);\n+\n+\tpacket_write(fd, data, len_data);\n+\n+\tfree(data);\n+}\n+\n+static void send_batch(int fd, size_t start, size_t nr)\n+{\n+\tsize_t i;\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tfor (i = 0; i < nr; i++)\n+\t\tsend_one_item(fd, ¶llel_checkout.items[start + i]);\n+\tpacket_flush(fd);\n+\tsigchain_pop(SIGPIPE);\n+}\n+\n+static struct pc_worker *setup_workers(struct checkout *state, int num_workers)\n+{\n+\tstruct pc_worker *workers;\n+\tint i, workers_with_one_extra_item;\n+\tsize_t base_batch_size, batch_beginning = 0;\n+\n+\tALLOC_ARRAY(workers, num_workers);\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->git_cmd = 1;\n+\t\tcp->in = -1;\n+\t\tcp->out = -1;\n+\t\tcp->clean_on_exit = 1;\n+\t\tstrvec_push(&cp->args, \"checkout--worker\");\n+\t\tif (state->base_dir_len)\n+\t\t\tstrvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n+\t\tif (start_command(cp))\n+\t\t\tdie(\"failed to spawn checkout worker\");\n+\t}\n+\n+\tbase_batch_size = parallel_checkout.nr / num_workers;\n+\tworkers_with_one_extra_item = parallel_checkout.nr % num_workers;\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct pc_worker *worker = &workers[i];\n+\t\tsize_t batch_size = base_batch_size;\n+\n+\t\t/* distribute the extra work evenly */\n+\t\tif (i < workers_with_one_extra_item)\n+\t\t\tbatch_size++;\n+\n+\t\tsend_batch(worker->cp.in, batch_beginning, batch_size);\n+\t\tworker->next_item_to_complete = batch_beginning;\n+\t\tworker->nr_items_to_complete = batch_size;\n+\n+\t\tbatch_beginning += batch_size;\n+\t}\n+\n+\treturn workers;\n+}\n+\n+static void finish_workers(struct pc_worker *workers, int num_workers)\n+{\n+\tint i;\n+\n+\t/*\n+\t * Close pipes before calling finish_command() to let the workers\n+\t * exit asynchronously and avoid spending extra time on wait().\n+\t */\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\t\tif (cp->in >= 0)\n+\t\t\tclose(cp->in);\n+\t\tif (cp->out >= 0)\n+\t\t\tclose(cp->out);\n+\t}\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tint rc = finish_command(&workers[i].cp);\n+\t\tif (rc > 128) {\n+\t\t\t/*\n+\t\t\t * For a normal non-zero exit, the worker should have\n+\t\t\t * already printed something useful to stderr. But a\n+\t\t\t * death by signal should be mentioned to the user.\n+\t\t\t */\n+\t\t\terror(\"checkout worker %d died of signal %d\", i, rc - 128);\n+\t\t}\n+\t}\n+\n+\tfree(workers);\n+}\n+\n+static inline void assert_pc_item_result_size(int got, int exp)\n+{\n+\tif (got != exp)\n+\t\tBUG(\"wrong result size from checkout worker (got %dB, exp %dB)\",\n+\t\t got, exp);\n+}\n+\n+static void parse_and_save_result(const char *buffer, int len,\n+\t\t\t\t struct pc_worker *worker)\n+{\n+\tstruct pc_item_result *res;\n+\tstruct parallel_checkout_item *pc_item;\n+\tstruct stat *st = NULL;\n+\n+\tif (len < PC_ITEM_RESULT_BASE_SIZE)\n+\t\tBUG(\"too short result from checkout worker (got %dB, exp >=%dB)\",\n+\t\t len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\n+\tres = (struct pc_item_result *)buffer;\n+\n+\t/*\n+\t * Worker should send either the full result struct on success, or\n+\t * just the base (i.e. no stat data), otherwise.\n+\t */\n+\tif (res->status == PC_ITEM_WRITTEN) {\n+\t\tassert_pc_item_result_size(len, (int)sizeof(struct pc_item_result));\n+\t\tst = &res->st;\n+\t} else {\n+\t\tassert_pc_item_result_size(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\t}\n+\n+\tif (!worker->nr_items_to_complete)\n+\t\tBUG(\"received result from supposedly finished checkout worker\");\n+\tif (res->id != worker->next_item_to_complete)\n+\t\tBUG(\"unexpected item id from checkout worker (got %\"PRIuMAX\", exp %\"PRIuMAX\")\",\n+\t\t (uintmax_t)res->id, (uintmax_t)worker->next_item_to_complete);\n+\n+\tworker->next_item_to_complete++;\n+\tworker->nr_items_to_complete--;\n+\n+\tpc_item = ¶llel_checkout.items[res->id];\n+\tpc_item->status = res->status;\n+\tif (st)\n+\t\tpc_item->st = *st;\n+}\n+\n+static void gather_results_from_workers(struct pc_worker *workers,\n+\t\t\t\t\tint num_workers)\n+{\n+\tint i, active_workers = num_workers;\n+\tstruct pollfd *pfds;\n+\n+\tCALLOC_ARRAY(pfds, num_workers);\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tpfds[i].fd = workers[i].cp.out;\n+\t\tpfds[i].events = POLLIN;\n+\t}\n+\n+\twhile (active_workers) {\n+\t\tint nr = poll(pfds, num_workers, -1);\n+\n+\t\tif (nr < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tdie_errno(\"failed to poll checkout workers\");\n+\t\t}\n+\n+\t\tfor (i = 0; i < num_workers && nr > 0; i++) {\n+\t\t\tstruct pc_worker *worker = &workers[i];\n+\t\t\tstruct pollfd *pfd = &pfds[i];\n+\n+\t\t\tif (!pfd->revents)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (pfd->revents & POLLIN) {\n+\t\t\t\tint len = packet_read(pfd->fd, NULL, NULL,\n+\t\t\t\t\t\t packet_buffer,\n+\t\t\t\t\t\t sizeof(packet_buffer), 0);\n+\n+\t\t\t\tif (len < 0) {\n+\t\t\t\t\tBUG(\"packet_read() returned negative value\");\n+\t\t\t\t} else if (!len) {\n+\t\t\t\t\tpfd->fd = -1;\n+\t\t\t\t\tactive_workers--;\n+\t\t\t\t} else {\n+\t\t\t\t\tparse_and_save_result(packet_buffer,\n+\t\t\t\t\t\t\t len, worker);\n+\t\t\t\t}\n+\t\t\t} else if (pfd->revents & POLLHUP) {\n+\t\t\t\tpfd->fd = -1;\n+\t\t\t\tactive_workers--;\n+\t\t\t} else if (pfd->revents & (POLLNVAL | POLLERR)) {\n+\t\t\t\tdie(\"error polling from checkout worker\");\n+\t\t\t}\n+\n+\t\t\tnr--;\n+\t\t}\n+\t}\n+\n+\tfree(pfds);\n+}\n+\n static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n@@ -348,16 +584,28 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n+static const int DEFAULT_NUM_WORKERS = 2;\n+\n int run_parallel_checkout(struct checkout *state)\n {\n-\tint ret;\n+\tint ret, num_workers = DEFAULT_NUM_WORKERS;\n \n \tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n \n-\twrite_items_sequentially(state);\n+\tif (parallel_checkout.nr < num_workers)\n+\t\tnum_workers = parallel_checkout.nr;\n+\n+\tif (num_workers <= 1) {\n+\t\twrite_items_sequentially(state);\n+\t} else {\n+\t\tstruct pc_worker *workers = setup_workers(state, num_workers);\n+\t\tgather_results_from_workers(workers, num_workers);\n+\t\tfinish_workers(workers, num_workers);\n+\t}\n+\n \tret = handle_results(state);\n \n \tfinish_parallel_checkout();\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 4ad2a519b3..ec58716519 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -1,9 +1,14 @@\n #ifndef PARALLEL_CHECKOUT_H\n #define PARALLEL_CHECKOUT_H\n \n+#include \"convert.h\"\n+\n struct cache_entry;\n struct checkout;\n-struct conv_attrs;\n+\n+/****************************************************************\n+ * Users of parallel checkout\n+ ****************************************************************/\n \n enum pc_status {\n \tPC_UNINITIALIZED = 0,\n@@ -29,4 +34,70 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n /* Write all the queued entries, returning 0 on success.*/\n int run_parallel_checkout(struct checkout *state);\n \n+/****************************************************************\n+ * Interface with checkout--worker\n+ ****************************************************************/\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/*\n+\t * In main process ce points to a istate->cache[] entry. Thus, it's not\n+\t * owned by us. In workers they own the memory, which *must be* released.\n+\t */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tsize_t id; /* position in parallel_checkout.items[] of main process */\n+\n+\t/* Output fields, sent from workers. */\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+/*\n+ * The fixed-size portion of `struct parallel_checkout_item` that is sent to the\n+ * workers. Following this will be 2 strings: ca.working_tree_encoding and\n+ * ce.name; These are NOT null terminated, since we have the size in the fixed\n+ * portion.\n+ *\n+ * Note that not all fields of conv_attrs and cache_entry are passed, only the\n+ * ones that will be required by the workers to smudge and write the entry.\n+ */\n+struct pc_item_fixed_portion {\n+\tsize_t id;\n+\tstruct object_id oid;\n+\tunsigned int ce_mode;\n+\tenum convert_crlf_action crlf_action;\n+\tint ident;\n+\tsize_t working_tree_encoding_len;\n+\tsize_t name_len;\n+};\n+\n+/*\n+ * The fields of `struct parallel_checkout_item` that are returned by the\n+ * workers. Note: `st` must be the last one, as it is omitted on error.\n+ */\n+struct pc_item_result {\n+\tsize_t id;\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+#define PC_ITEM_RESULT_BASE_SIZE offsetof(struct pc_item_result, st)\n+\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state);\n+\n #endif /* PARALLEL_CHECKOUT_H */\n-- \n2.30.1\n\n"},{"id":"422223","messageId":"ad165c0637b0244df1851f10386fe87df2a70368.1618790794.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618790794.git.matheus.bernardino@usp.br","subject":"[PATCH v3 3/5] parallel-checkout: add configuration options","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T00:14:55Z","receivedAt":"2021-04-19T00:15:14Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make parallel checkout configurable by introducing two new settings:\ncheckout.workers and checkout.thresholdForParallelism. The first defines\nthe number of workers (where one means sequential checkout), and the\nsecond defines the minimum number of entries to attempt parallel\ncheckout.\n\nTo decide the default value for checkout.workers, the parallel version\nwas benchmarked during three operations in the linux repo, with cold\ncache: cloning v5.8, checking out v5.8 from v2.6.15 (checkout I) and\nchecking out v5.8 from v5.7 (checkout II). The four tables below show\nthe mean run times and standard deviations for 5 runs in: a local file\nsystem on SSD, a local file system on HDD, a Linux NFS server, and\nAmazon EFS (all on Linux). Each parallel checkout test was executed with\nthe number of workers that brings the best overall results in that\nenvironment.\n\nLocal SSD:\n Sequential 10 workers Speedup\nClone 8.805 s ± 0.043 s 3.564 s ± 0.041 s 2.47 ± 0.03\nCheckout I 9.678 s ± 0.057 s 4.486 s ± 0.050 s 2.16 ± 0.03\nCheckout II 5.034 s ± 0.072 s 3.021 s ± 0.038 s 1.67 ± 0.03\n\nLocal HDD:\n Sequential 10 workers Speedup\nClone 32.288 s ± 0.580 s 30.724 s ± 0.522 s 1.05 ± 0.03\nCheckout I 54.172 s ± 7.119 s 54.429 s ± 6.738 s 1.00 ± 0.18\nCheckout II 40.465 s ± 2.402 s 38.682 s ± 1.365 s 1.05 ± 0.07\n\nLinux NFS server (v4.1, on EBS, single availability zone):\n\n Sequential 32 workers Speedup\nClone 240.368 s ± 6.347 s 57.349 s ± 0.870 s 4.19 ± 0.13\nCheckout I 242.862 s ± 2.215 s 58.700 s ± 0.904 s 4.14 ± 0.07\nCheckout II 65.751 s ± 1.577 s 23.820 s ± 0.407 s 2.76 ± 0.08\n\nEFS (v4.1, replicated over multiple availability zones):\n\n Sequential 32 workers Speedup\nClone 922.321 s ± 2.274 s 210.453 s ± 3.412 s 4.38 ± 0.07\nCheckout I 1011.300 s ± 7.346 s 297.828 s ± 0.964 s 3.40 ± 0.03\nCheckout II 294.104 s ± 1.836 s 126.017 s ± 1.190 s 2.33 ± 0.03\n\nThe above benchmarks show that parallel checkout is most effective on\nrepositories located on an SSD or over a distributed file system. For\nlocal file systems on spinning disks, and/or older machines, the\nparallelism does not always bring a good performance. For this reason,\nthe default value for checkout.workers is one, a.k.a. sequential\ncheckout.\n\nTo decide the default value for checkout.thresholdForParallelism,\nanother benchmark was executed in the \"Local SSD\" setup, where parallel\ncheckout showed to be beneficial. This time, we compared the runtime of\na `git checkout -f`, with and without parallelism, after randomly\nremoving an increasing number of files from the Linux working tree. The\n\"sequential fallback\" column below corresponds to the executions where\ncheckout.workers was 10 but checkout.thresholdForParallelism was equal\nto the number of to-be-updated files plus one (so that we end up writing\nsequentially). Each test case was sampled 15 times, and each sample had\na randomly different set of files removed. Here are the results:\n\n sequential fallback 10 workers speedup\n10 files 772.3 ms ± 12.6 ms 769.0 ms ± 13.6 ms 1.00 ± 0.02\n20 files 780.5 ms ± 15.8 ms 775.2 ms ± 9.2 ms 1.01 ± 0.02\n50 files 806.2 ms ± 13.8 ms 767.4 ms ± 8.5 ms 1.05 ± 0.02\n100 files 833.7 ms ± 21.4 ms 750.5 ms ± 16.8 ms 1.11 ± 0.04\n200 files 897.6 ms ± 30.9 ms 730.5 ms ± 14.7 ms 1.23 ± 0.05\n500 files 1035.4 ms ± 48.0 ms 677.1 ms ± 22.3 ms 1.53 ± 0.09\n1000 files 1244.6 ms ± 35.6 ms 654.0 ms ± 38.3 ms 1.90 ± 0.12\n2000 files 1488.8 ms ± 53.4 ms 658.8 ms ± 23.8 ms 2.26 ± 0.12\n\nFrom the above numbers, 100 files seems to be a reasonable default value\nfor the threshold setting.\n\nNote: Up to 1000 files, we observe a drop in the execution time of the\nparallel code with an increase in the number of files. This is a rather\nodd behavior, but it was observed in multiple repetitions. Above 1000\nfiles, the execution time increases according to the number of files, as\none would expect.\n\nAbout the test environments: Local SSD tests were executed on an\ni7-7700HQ (4 cores with hyper-threading) running Manjaro Linux. Local\nHDD tests were executed on an Intel(R) Xeon(R) E3-1230 (also 4 cores\nwith hyper-threading), HDD Seagate Barracuda 7200.14 SATA 3.1, running\nDebian. NFS and EFS tests were executed on an Amazon EC2 c5n.xlarge\ninstance, with 4 vCPUs. The Linux NFS server was running on a m6g.large\ninstance with 2 vCPUSs and a 1 TB EBS GP2 volume. Before each timing,\nthe linux repository was removed (or checked out back to its previous\nstate), and `sync && sysctl vm.drop_caches=3` was executed.\n\nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Documentation/config/checkout.txt | 21 +++++++++++++++++++++\n parallel-checkout.c | 24 +++++++++++++++++++-----\n parallel-checkout.h | 9 +++++++--\n unpack-trees.c | 10 +++++++---\n 4 files changed, 54 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt\nindex 2cddf7b4b4..bfbca90f0e 100644\n--- a/Documentation/config/checkout.txt\n+++ b/Documentation/config/checkout.txt\n@@ -21,3 +21,24 @@ checkout.guess::\n \tProvides the default value for the `--guess` or `--no-guess`\n \toption in `git checkout` and `git switch`. See\n \tlinkgit:git-switch[1] and linkgit:git-checkout[1].\n+\n+checkout.workers::\n+\tThe number of parallel workers to use when updating the working tree.\n+\tThe default is one, i.e. sequential execution. If set to a value less\n+\tthan one, Git will use as many workers as the number of logical cores\n+\tavailable. This setting and `checkout.thresholdForParallelism` affect\n+\tall commands that perform checkout. E.g. checkout, clone, reset,\n+\tsparse-checkout, etc.\n++\n+Note: parallel checkout usually delivers better performance for repositories\n+located on SSDs or over NFS. For repositories on spinning disks and/or machines\n+with a small number of cores, the default sequential checkout often performs\n+better. The size and compression level of a repository might also influence how\n+well the parallel version performs.\n+\n+checkout.thresholdForParallelism::\n+\tWhen running parallel checkout with a small number of files, the cost\n+\tof subprocess spawning and inter-process communication might outweigh\n+\tthe parallelization gains. This setting allows to define the minimum\n+\tnumber of files for which parallel checkout should be attempted. The\n+\tdefault is 100.\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 836154fec6..3cc2028861 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,10 +1,12 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"streaming.h\"\n+#include \"thread-utils.h\"\n \n struct pc_worker {\n \tstruct child_process cp;\n@@ -24,6 +26,20 @@ enum pc_status parallel_checkout_status(void)\n \treturn parallel_checkout.status;\n }\n \n+static const int DEFAULT_THRESHOLD_FOR_PARALLELISM = 100;\n+static const int DEFAULT_NUM_WORKERS = 1;\n+\n+void get_parallel_checkout_configs(int *num_workers, int *threshold)\n+{\n+\tif (git_config_get_int(\"checkout.workers\", num_workers))\n+\t\t*num_workers = DEFAULT_NUM_WORKERS;\n+\telse if (*num_workers < 1)\n+\t\t*num_workers = online_cpus();\n+\n+\tif (git_config_get_int(\"checkout.thresholdForParallelism\", threshold))\n+\t\t*threshold = DEFAULT_THRESHOLD_FOR_PARALLELISM;\n+}\n+\n void init_parallel_checkout(void)\n {\n \tif (parallel_checkout.status != PC_UNINITIALIZED)\n@@ -584,11 +600,9 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n-static const int DEFAULT_NUM_WORKERS = 2;\n-\n-int run_parallel_checkout(struct checkout *state)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n {\n-\tint ret, num_workers = DEFAULT_NUM_WORKERS;\n+\tint ret;\n \n \tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n@@ -598,7 +612,7 @@ int run_parallel_checkout(struct checkout *state)\n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\n \n-\tif (num_workers <= 1) {\n+\tif (num_workers <= 1 || parallel_checkout.nr < threshold) {\n \t\twrite_items_sequentially(state);\n \t} else {\n \t\tstruct pc_worker *workers = setup_workers(state, num_workers);\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex ec58716519..2a68ab954d 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -17,6 +17,7 @@ enum pc_status {\n };\n \n enum pc_status parallel_checkout_status(void);\n+void get_parallel_checkout_configs(int *num_workers, int *threshold);\n \n /*\n * Put parallel checkout into the PC_ACCEPTING_ENTRIES state. Should be used\n@@ -31,8 +32,12 @@ void init_parallel_checkout(void);\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n \n-/* Write all the queued entries, returning 0 on success.*/\n-int run_parallel_checkout(struct checkout *state);\n+/*\n+ * Write all the queued entries, returning 0 on success. If the number of\n+ * entries is smaller than the specified threshold, the operation is performed\n+ * sequentially.\n+ */\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n \n /****************************************************************\n * Interface with checkout--worker\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f0430d458d..0669748f21 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -399,7 +399,7 @@ static int check_updates(struct unpack_trees_options *o,\n \tint errs = 0;\n \tstruct progress *progress;\n \tstruct checkout state = CHECKOUT_INIT;\n-\tint i;\n+\tint i, pc_workers, pc_threshold;\n \n \ttrace_performance_enter();\n \tstate.force = 1;\n@@ -465,8 +465,11 @@ static int check_updates(struct unpack_trees_options *o,\n \t\toid_array_clear(&to_fetch);\n \t}\n \n+\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n \tenable_delayed_checkout(&state);\n-\tinit_parallel_checkout();\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -480,7 +483,8 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n-\terrs |= run_parallel_checkout(&state);\n+\tif (pc_workers > 1)\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"422224","messageId":"cf9e28dc0eddfc8e36a64b9738c94bb790349092.1618790794.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618790794.git.matheus.bernardino@usp.br","subject":"[PATCH v3 4/5] parallel-checkout: support progress displaying","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T00:14:56Z","receivedAt":"2021-04-19T00:15:16Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Original-patch-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n parallel-checkout.c | 34 +++++++++++++++++++++++++++++++---\n parallel-checkout.h | 5 ++++-\n unpack-trees.c | 11 ++++++++---\n 3 files changed, 43 insertions(+), 7 deletions(-)\n\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 3cc2028861..09e8b10a35 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -3,6 +3,7 @@\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n+#include \"progress.h\"\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"streaming.h\"\n@@ -17,6 +18,8 @@ struct parallel_checkout {\n \tenum pc_status status;\n \tstruct parallel_checkout_item *items; /* The parallel checkout queue. */\n \tsize_t nr, alloc;\n+\tstruct progress *progress;\n+\tunsigned int *progress_cnt;\n };\n \n static struct parallel_checkout parallel_checkout;\n@@ -146,6 +149,20 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \treturn 0;\n }\n \n+size_t pc_queue_size(void)\n+{\n+\treturn parallel_checkout.nr;\n+}\n+\n+static void advance_progress_meter(void)\n+{\n+\tif (parallel_checkout.progress) {\n+\t\t(*parallel_checkout.progress_cnt)++;\n+\t\tdisplay_progress(parallel_checkout.progress,\n+\t\t\t\t *parallel_checkout.progress_cnt);\n+\t}\n+}\n+\n static int handle_results(struct checkout *state)\n {\n \tint ret = 0;\n@@ -194,6 +211,7 @@ static int handle_results(struct checkout *state)\n \t\t\t */\n \t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n \t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tadvance_progress_meter();\n \t\t\tbreak;\n \t\tcase PC_ITEM_PENDING:\n \t\t\thave_pending = 1;\n@@ -534,6 +552,9 @@ static void parse_and_save_result(const char *buffer, int len,\n \tpc_item->status = res->status;\n \tif (st)\n \t\tpc_item->st = *st;\n+\n+\tif (res->status != PC_ITEM_COLLIDED)\n+\t\tadvance_progress_meter();\n }\n \n static void gather_results_from_workers(struct pc_worker *workers,\n@@ -596,11 +617,16 @@ static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n \n-\tfor (i = 0; i < parallel_checkout.nr; i++)\n-\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\tif (pc_item->status != PC_ITEM_COLLIDED)\n+\t\t\tadvance_progress_meter();\n+\t}\n }\n \n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt)\n {\n \tint ret;\n \n@@ -608,6 +634,8 @@ int run_parallel_checkout(struct checkout *state, int num_workers, int threshold\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n+\tparallel_checkout.progress = progress;\n+\tparallel_checkout.progress_cnt = progress_cnt;\n \n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 2a68ab954d..80f539bcb7 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -5,6 +5,7 @@\n \n struct cache_entry;\n struct checkout;\n+struct progress;\n \n /****************************************************************\n * Users of parallel checkout\n@@ -31,13 +32,15 @@ void init_parallel_checkout(void);\n * for later write and return 0.\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+size_t pc_queue_size(void);\n \n /*\n * Write all the queued entries, returning 0 on success. If the number of\n * entries is smaller than the specified threshold, the operation is performed\n * sequentially.\n */\n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt);\n \n /****************************************************************\n * Interface with checkout--worker\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 0669748f21..4b77e52c6b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -474,17 +474,22 @@ static int check_updates(struct unpack_trees_options *o,\n \t\tstruct cache_entry *ce = index->cache[i];\n \n \t\tif (ce->ce_flags & CE_UPDATE) {\n+\t\t\tsize_t last_pc_queue_size = pc_queue_size();\n+\n \t\t\tif (ce->ce_flags & CE_WT_REMOVE)\n \t\t\t\tBUG(\"both update and delete flags are set on %s\",\n \t\t\t\t ce->name);\n-\t\t\tdisplay_progress(progress, ++cnt);\n \t\t\tce->ce_flags &= ~CE_UPDATE;\n \t\t\terrs |= checkout_entry(ce, &state, NULL, NULL);\n+\n+\t\t\tif (last_pc_queue_size == pc_queue_size())\n+\t\t\t\tdisplay_progress(progress, ++cnt);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tif (pc_workers > 1)\n-\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t\t progress, &cnt);\n+\tstop_progress(&progress);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"422225","messageId":"415d4114aadbdba30567a73e8d70f3d5795bfb4e.1618790794.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618790794.git.matheus.bernardino@usp.br","subject":"[PATCH v3 5/5] parallel-checkout: add design documentation","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T00:14:57Z","receivedAt":"2021-04-19T00:15:18Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Co-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Documentation/Makefile | 1 +\n Documentation/technical/parallel-checkout.txt | 271 ++++++++++++++++++\n 2 files changed, 272 insertions(+)\n create mode 100644 Documentation/technical/parallel-checkout.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 81d1bf7a04..af236927c9 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -90,6 +90,7 @@ TECH_DOCS += technical/multi-pack-index\n TECH_DOCS += technical/pack-format\n TECH_DOCS += technical/pack-heuristics\n TECH_DOCS += technical/pack-protocol\n+TECH_DOCS += technical/parallel-checkout\n TECH_DOCS += technical/partial-clone\n TECH_DOCS += technical/protocol-capabilities\n TECH_DOCS += technical/protocol-common\ndiff --git a/Documentation/technical/parallel-checkout.txt b/Documentation/technical/parallel-checkout.txt\nnew file mode 100644\nindex 0000000000..fddb0ed4fd\n--- /dev/null\n+++ b/Documentation/technical/parallel-checkout.txt\n@@ -0,0 +1,271 @@\n+Parallel Checkout Design Notes\n+==============================\n+\n+The \"Parallel Checkout\" feature attempts to use multiple processes to\n+parallelize the work of uncompressing the blobs, applying in-core\n+filters, and writing the resulting contents to the working tree during a\n+checkout operation. It can be used by all checkout-related commands,\n+such as `clone`, `checkout`, `reset`, `sparse-checkout`, and others.\n+\n+These commands share the following basic structure:\n+\n+* Step 1: Read the current index file into memory.\n+\n+* Step 2: Modify the in-memory index based upon the command, and\n+ temporarily mark all cache entries that need to be updated.\n+\n+* Step 3: Populate the working tree to match the new candidate index.\n+ This includes iterating over all of the to-be-updated cache entries\n+ and delete, create, or overwrite the associated files in the working\n+ tree.\n+\n+* Step 4: Write the new index to disk.\n+\n+Step 3 is the focus of the \"parallel checkout\" effort described here.\n+It dominates the execution time for most of the above command types.\n+\n+Sequential Implementation\n+-------------------------\n+\n+For the purposes of discussion here, the current sequential\n+implementation of Step 3 is divided in 3 parts, each one implemented in\n+its own function:\n+\n+* Step 3a: `unpack-trees.c:check_updates()` contains a series of\n+ sequential loops iterating over the `cache_entry`'s array. The main\n+ loop in this function calls the Step 3b function for each of the\n+ to-be-updated entries.\n+\n+* Step 3b: `entry.c:checkout_entry()` examines the existing working tree\n+ for file conflicts, collisions, and unsaved changes. It removes files\n+ and creates leading directories as necessary. It calls the Step 3c\n+ function for each entry to be written.\n+\n+* Step 3c: `entry.c:write_entry()` loads the blob into memory, smudges\n+ it if necessary, creates the file in the working tree, writes the\n+ smudged contents, calls `fstat()` or `lstat()`, and updates the\n+ associated `cache_entry` struct with the stat information gathered.\n+\n+It wouldn't be safe to perform Step 3b in parallel, as there could be\n+race conditions between file creations and removals. Instead, the\n+parallel checkout framework lets the sequential code handle Step 3b,\n+and use parallel workers to replace the sequential\n+`entry.c:write_entry()` calls from Step 3c.\n+\n+Rejected Multi-Threaded Solution\n+--------------------------------\n+\n+The most \"straightforward\" implementation would be to spread the set of\n+to-be-updated cache entries across multiple threads. But due to the\n+thread-unsafe functions in the ODB code, we would have to use locks to\n+coordinate the parallel operation. An early prototype of this solution\n+showed that the multi-threaded checkout would bring performance\n+improvements over the sequential code, but there was still too much lock\n+contention. A `perf` profiling indicated that around 20% of the runtime\n+during a local Linux clone (on an SSD) was spent in locking functions.\n+For this reason this approach was rejected in favor of using multiple\n+child processes, which led to a better performance.\n+\n+Multi-Process Solution\n+----------------------\n+\n+Parallel checkout alters the aforementioned Step 3 to use multiple\n+`checkout--worker` background processes to distribute the work. The\n+long-running worker processes are controlled by the foreground Git\n+command using the existing run-command API.\n+\n+Overview\n+~~~~~~~~\n+\n+Step 3b is only slightly altered; for each entry to be checked out, the\n+main process performs the following steps:\n+\n+* M1: Check whether there is any untracked or unclean file in the\n+ working tree which would be overwritten by this entry, and decide\n+ whether to proceed (removing the file(s)) or not.\n+\n+* M2: Create the leading directories.\n+\n+* M3: Load the conversion attributes for the entry's path.\n+\n+* M4: Check, based on the entry's type and conversion attributes,\n+ whether the entry is eligible for parallel checkout (more on this\n+ later). If it is eligible, enqueue the entry and the loaded\n+ attributes to later write the entry in parallel. If not, write the\n+ entry right away, using the default sequential code.\n+\n+Note: we save the conversion attributes associated with each entry\n+because the workers don't have access to the main process' index state,\n+so they can't load the attributes by themselves (and the attributes are\n+needed to properly smudge the entry). Additionally, this has a positive\n+impact on performance as (1) we don't need to load the attributes twice\n+and (2) the attributes machinery is optimized to handle paths in\n+sequential order.\n+\n+After all entries have passed through the above steps, the main process\n+checks if the number of enqueued entries is sufficient to spread among\n+the workers. If not, it just writes them sequentially. Otherwise, it\n+spawns the workers and distributes the queued entries uniformly in\n+continuous chunks. This aims to minimize the chances of two workers\n+writing to the same directory simultaneously, which could increase lock\n+contention in the kernel.\n+\n+Then, for each assigned item, each worker:\n+\n+* W1: Checks if there is any non-directory file in the leading part of\n+ the entry's path or if there already exists a file at the entry' path.\n+ If so, mark the entry with `PC_ITEM_COLLIDED` and skip it (more on\n+ this later).\n+\n+* W2: Creates the file (with O_CREAT and O_EXCL).\n+\n+* W3: Loads the blob into memory (inflating and delta reconstructing\n+ it).\n+\n+* W4: Applies any required in-process filter, like end-of-line\n+ conversion and re-encoding.\n+\n+* W5: Writes the result to the file descriptor opened at W2.\n+\n+* W6: Calls `fstat()` or lstat()` on the just-written path, and sends\n+ the result back to the main process, together with the end status of\n+ the operation and the item's identification number.\n+\n+Note that, when possible, steps W3 to W5 are delegated to the streaming\n+machinery, removing the need to keep the entire blob in memory.\n+\n+If the worker fails to read the blob or to write it to the working tree,\n+it removes the created file to avoid leaving empty files behind. This is\n+the *only* time a worker is allowed to remove a file.\n+\n+As mentioned earlier, it is the responsibility of the main process to\n+remove any file that blocks the checkout operation (or abort if the\n+removal(s) would cause data loss and the user didn't ask to `--force`).\n+This is crucial to avoid race conditions and also to properly detect\n+path collisions at Step W1.\n+\n+After the workers finish writing the items and sending back the required\n+information, the main process handles the results in two steps:\n+\n+- First, it updates the in-memory index with the `lstat()` information\n+ sent by the workers. (This must be done first as this information\n+ might me required in the following step.)\n+\n+- Then it writes the items which collided on disk (i.e. items marked\n+ with `PC_ITEM_COLLIDED`). More on this below.\n+\n+Path Collisions\n+---------------\n+\n+Path collisions happen when two different paths correspond to the same\n+entry in the file system. E.g. the paths 'a' and 'A' would collide in a\n+case-insensitive file system.\n+\n+The sequential checkout deals with collisions in the same way that it\n+deals with files that were already present in the working tree before\n+checkout. Basically, it checks if the path that it wants to write\n+already exists on disk, makes sure the existing file doesn't have\n+unsaved data, and then overwrites it. (To be more pedantic: it deletes\n+the existing file and creates the new one.) So, if there are multiple\n+colliding files to be checked out, the sequential code will write each\n+one of them but only the last will actually survive on disk.\n+\n+Parallel checkout aims to reproduce the same behavior. However, we\n+cannot let the workers racily write to the same file on disk. Instead,\n+the workers detect when the entry that they want to check out would\n+collide with an existing file, and mark it with `PC_ITEM_COLLIDED`.\n+Later, the main process can sequentially feed these entries back to\n+`checkout_entry()` without the risk of race conditions. On clone, this\n+also has the effect of marking the colliding entries to later emit a\n+warning for the user, like the classic sequential checkout does.\n+\n+The workers are able to detect both collisions among the entries being\n+concurrently written and collisions among parallel-eligible and\n+ineligible entries. The general idea for collision detection is quite\n+straightforward: for each parallel-eligible entry, the main process must\n+remove all files that prevent this entry from being written (before\n+enqueueing it). This includes any non-directory file in the leading path\n+of the entry. Later, when a worker gets assigned the entry, it looks\n+again for the non-directories files and for an already existing file at\n+the entry's path. If any of these checks finds something, the worker\n+knows that there was a path collision.\n+\n+Because parallel checkout can distinguish path collisions from the case\n+where the file was already present in the working tree before checkout,\n+we could alternatively choose to skip the checkout of colliding entries.\n+However, each entry that doesn't get written would have NULL `lstat()`\n+fields on the index. This could cause performance penalties for\n+subsequent commands that need to refresh the index, as they would have\n+to go to the file system to see if the entry is dirty. Thus, if we have\n+N entries in a colliding group and we decide to write and `lstat()` only\n+one of them, every subsequent `git-status` will have to read, convert,\n+and hash the written file N - 1 times. By checking out all colliding\n+entries (like the sequential code does), we only pay the overhead once,\n+during checkout.\n+\n+Eligible Entries for Parallel Checkout\n+--------------------------------------\n+\n+As previously mentioned, not all entries passed to `checkout_entry()`\n+will be considered eligible for parallel checkout. More specifically, we\n+exclude:\n+\n+- Symbolic links; to avoid race conditions that, in combination with\n+ path collisions, could cause workers to write files at the wrong\n+ place. For example, if we were to concurrently check out a symlink\n+ 'a' -> 'b' and a regular file 'A/f' in a case-insensitive file system,\n+ we could potentially end up writing the file 'A/f' at 'a/f', due to a\n+ race condition.\n+\n+- Regular files that require external filters (either \"one shot\" filters\n+ or long-running process filters). These filters are black-boxes to Git\n+ and may have their own internal locking or non-concurrent assumptions.\n+ So it might not be safe to run multiple instances in parallel.\n++\n+Besides, long-running filters may use the delayed checkout feature to\n+postpone the return of some filtered blobs. The delayed checkout queue\n+and the parallel checkout queue are not compatible and should remain\n+separated.\n++\n+Note: regular files that only require internal filters, like end-of-line\n+conversion and re-encoding, are eligible for parallel checkout.\n+\n+Ineligible entries are checked out by the classic sequential codepath\n+*before* spawning workers.\n+\n+Note: submodules's files are also eligible for parallel checkout (as\n+long as they don't fall into any of the excluding categories mentioned\n+above). But since each submodule is checked out in its own child\n+process, we don't mix the superproject's and the submodules' files in\n+the same parallel checkout process or queue.\n+\n+The API\n+-------\n+\n+The parallel checkout API was designed with the goal to minimize changes\n+to the current users of the checkout machinery. This means that they\n+don't have to call a different function for sequential or parallel\n+checkout. As already mentioned, `checkout_entry()` will automatically\n+insert the given entry in the parallel checkout queue when this feature\n+is enabled and the entry is eligible; otherwise, it will just write the\n+entry right away, using the sequential code. In general, callers of the\n+parallel checkout API should look similar to this:\n+\n+----------------------------------------------\n+int pc_workers, pc_threshold, err = 0;\n+struct checkout state;\n+\n+get_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n+/*\n+ * This check is not strictly required, but it\n+ * should save some time in sequential mode.\n+ */\n+if (pc_workers > 1)\n+\tinit_parallel_checkout();\n+\n+for (each cache_entry ce to-be-updated)\n+\terr |= checkout_entry(ce, &state, NULL, NULL);\n+\n+err |= run_parallel_checkout(&state, pc_workers, pc_threshold, NULL, NULL);\n+----------------------------------------------\n-- \n2.30.1\n\n"},{"id":"422239","messageId":"CAP8UFD1K6aB-py4O5uSKOwyCMg21+6mVo0qMMRVhEwf2KF2M2g@mail.gmail.com","threadId":"55342","inReplyTo":"415d4114aadbdba30567a73e8d70f3d5795bfb4e.1618790794.git.matheus.bernardino@usp.br","subject":"Re: [PATCH v3 5/5] parallel-checkout: add design documentation","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-04-19T09:36:28Z","receivedAt":"2021-04-19T09:36:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Apr 19, 2021 at 2:15 AM Matheus Tavares\n wrote:\n>\n> Co-authored-by: Jeff Hostetler \n> Signed-off-by: Matheus Tavares \n> ---\n> Documentation/Makefile | 1 +\n> Documentation/technical/parallel-checkout.txt | 271 ++++++++++++++++++\n> 2 files changed, 272 insertions(+)\n> create mode 100644 Documentation/technical/parallel-checkout.txt\n>\n> diff --git a/Documentation/Makefile b/Documentation/Makefile\n> index 81d1bf7a04..af236927c9 100644\n> --- a/Documentation/Makefile\n> +++ b/Documentation/Makefile\n> @@ -90,6 +90,7 @@ TECH_DOCS += technical/multi-pack-index\n> TECH_DOCS += technical/pack-format\n> TECH_DOCS += technical/pack-heuristics\n> TECH_DOCS += technical/pack-protocol\n> +TECH_DOCS += technical/parallel-checkout\n> TECH_DOCS += technical/partial-clone\n> TECH_DOCS += technical/protocol-capabilities\n> TECH_DOCS += technical/protocol-common\n> diff --git a/Documentation/technical/parallel-checkout.txt b/Documentation/technical/parallel-checkout.txt\n> new file mode 100644\n> index 0000000000..fddb0ed4fd\n> --- /dev/null\n> +++ b/Documentation/technical/parallel-checkout.txt\n> @@ -0,0 +1,271 @@\n> +Parallel Checkout Design Notes\n> +==============================\n> +\n> +The \"Parallel Checkout\" feature attempts to use multiple processes to\n> +parallelize the work of uncompressing the blobs, applying in-core\n> +filters, and writing the resulting contents to the working tree during a\n> +checkout operation. It can be used by all checkout-related commands,\n> +such as `clone`, `checkout`, `reset`, `sparse-checkout`, and others.\n> +\n> +These commands share the following basic structure:\n> +\n> +* Step 1: Read the current index file into memory.\n> +\n> +* Step 2: Modify the in-memory index based upon the command, and\n> + temporarily mark all cache entries that need to be updated.\n> +\n> +* Step 3: Populate the working tree to match the new candidate index.\n> + This includes iterating over all of the to-be-updated cache entries\n> + and delete, create, or overwrite the associated files in the working\n> + tree.\n> +\n> +* Step 4: Write the new index to disk.\n> +\n> +Step 3 is the focus of the \"parallel checkout\" effort described here.\n> +It dominates the execution time for most of the above command types.\n\nMaybe: s/command types/checkout-related commands/\n\n> +Sequential Implementation\n> +-------------------------\n> +\n> +For the purposes of discussion here, the current sequential\n> +implementation of Step 3 is divided in 3 parts, each one implemented in\n> +its own function:\n> +\n> +* Step 3a: `unpack-trees.c:check_updates()` contains a series of\n> + sequential loops iterating over the `cache_entry`'s array. The main\n> + loop in this function calls the Step 3b function for each of the\n> + to-be-updated entries.\n> +\n> +* Step 3b: `entry.c:checkout_entry()` examines the existing working tree\n> + for file conflicts, collisions, and unsaved changes. It removes files\n> + and creates leading directories as necessary. It calls the Step 3c\n> + function for each entry to be written.\n> +\n> +* Step 3c: `entry.c:write_entry()` loads the blob into memory, smudges\n> + it if necessary, creates the file in the working tree, writes the\n> + smudged contents, calls `fstat()` or `lstat()`, and updates the\n> + associated `cache_entry` struct with the stat information gathered.\n> +\n> +It wouldn't be safe to perform Step 3b in parallel, as there could be\n> +race conditions between file creations and removals. Instead, the\n> +parallel checkout framework lets the sequential code handle Step 3b,\n> +and use parallel workers to replace the sequential\n\ns/use/uses/\n\n> +`entry.c:write_entry()` calls from Step 3c.\n> +\n> +Rejected Multi-Threaded Solution\n> +--------------------------------\n> +\n> +The most \"straightforward\" implementation would be to spread the set of\n> +to-be-updated cache entries across multiple threads. But due to the\n> +thread-unsafe functions in the ODB code, we would have to use locks to\n> +coordinate the parallel operation. An early prototype of this solution\n> +showed that the multi-threaded checkout would bring performance\n> +improvements over the sequential code, but there was still too much lock\n> +contention. A `perf` profiling indicated that around 20% of the runtime\n> +during a local Linux clone (on an SSD) was spent in locking functions.\n> +For this reason this approach was rejected in favor of using multiple\n> +child processes, which led to a better performance.\n> +\n> +Multi-Process Solution\n> +----------------------\n> +\n> +Parallel checkout alters the aforementioned Step 3 to use multiple\n> +`checkout--worker` background processes to distribute the work. The\n> +long-running worker processes are controlled by the foreground Git\n> +command using the existing run-command API.\n> +\n> +Overview\n> +~~~~~~~~\n> +\n> +Step 3b is only slightly altered; for each entry to be checked out, the\n> +main process performs the following steps:\n> +\n> +* M1: Check whether there is any untracked or unclean file in the\n> + working tree which would be overwritten by this entry, and decide\n> + whether to proceed (removing the file(s)) or not.\n> +\n> +* M2: Create the leading directories.\n> +\n> +* M3: Load the conversion attributes for the entry's path.\n> +\n> +* M4: Check, based on the entry's type and conversion attributes,\n> + whether the entry is eligible for parallel checkout (more on this\n> + later). If it is eligible, enqueue the entry and the loaded\n> + attributes to later write the entry in parallel. If not, write the\n> + entry right away, using the default sequential code.\n> +\n> +Note: we save the conversion attributes associated with each entry\n> +because the workers don't have access to the main process' index state,\n> +so they can't load the attributes by themselves (and the attributes are\n> +needed to properly smudge the entry). Additionally, this has a positive\n> +impact on performance as (1) we don't need to load the attributes twice\n> +and (2) the attributes machinery is optimized to handle paths in\n> +sequential order.\n> +\n> +After all entries have passed through the above steps, the main process\n> +checks if the number of enqueued entries is sufficient to spread among\n> +the workers. If not, it just writes them sequentially. Otherwise, it\n> +spawns the workers and distributes the queued entries uniformly in\n> +continuous chunks. This aims to minimize the chances of two workers\n> +writing to the same directory simultaneously, which could increase lock\n> +contention in the kernel.\n> +\n> +Then, for each assigned item, each worker:\n> +\n> +* W1: Checks if there is any non-directory file in the leading part of\n> + the entry's path or if there already exists a file at the entry' path.\n> + If so, mark the entry with `PC_ITEM_COLLIDED` and skip it (more on\n> + this later).\n> +\n> +* W2: Creates the file (with O_CREAT and O_EXCL).\n> +\n> +* W3: Loads the blob into memory (inflating and delta reconstructing\n> + it).\n> +\n> +* W4: Applies any required in-process filter, like end-of-line\n> + conversion and re-encoding.\n> +\n> +* W5: Writes the result to the file descriptor opened at W2.\n> +\n> +* W6: Calls `fstat()` or lstat()` on the just-written path, and sends\n> + the result back to the main process, together with the end status of\n> + the operation and the item's identification number.\n> +\n> +Note that, when possible, steps W3 to W5 are delegated to the streaming\n> +machinery, removing the need to keep the entire blob in memory.\n> +\n> +If the worker fails to read the blob or to write it to the working tree,\n> +it removes the created file to avoid leaving empty files behind. This is\n> +the *only* time a worker is allowed to remove a file.\n> +\n> +As mentioned earlier, it is the responsibility of the main process to\n> +remove any file that blocks the checkout operation (or abort if the\n> +removal(s) would cause data loss and the user didn't ask to `--force`).\n> +This is crucial to avoid race conditions and also to properly detect\n> +path collisions at Step W1.\n> +\n> +After the workers finish writing the items and sending back the required\n> +information, the main process handles the results in two steps:\n> +\n> +- First, it updates the in-memory index with the `lstat()` information\n> + sent by the workers. (This must be done first as this information\n> + might me required in the following step.)\n> +\n> +- Then it writes the items which collided on disk (i.e. items marked\n> + with `PC_ITEM_COLLIDED`). More on this below.\n> +\n> +Path Collisions\n> +---------------\n> +\n> +Path collisions happen when two different paths correspond to the same\n> +entry in the file system. E.g. the paths 'a' and 'A' would collide in a\n> +case-insensitive file system.\n> +\n> +The sequential checkout deals with collisions in the same way that it\n> +deals with files that were already present in the working tree before\n> +checkout. Basically, it checks if the path that it wants to write\n> +already exists on disk, makes sure the existing file doesn't have\n> +unsaved data, and then overwrites it. (To be more pedantic: it deletes\n> +the existing file and creates the new one.) So, if there are multiple\n> +colliding files to be checked out, the sequential code will write each\n> +one of them but only the last will actually survive on disk.\n> +\n> +Parallel checkout aims to reproduce the same behavior. However, we\n> +cannot let the workers racily write to the same file on disk. Instead,\n> +the workers detect when the entry that they want to check out would\n> +collide with an existing file, and mark it with `PC_ITEM_COLLIDED`.\n> +Later, the main process can sequentially feed these entries back to\n> +`checkout_entry()` without the risk of race conditions. On clone, this\n> +also has the effect of marking the colliding entries to later emit a\n> +warning for the user, like the classic sequential checkout does.\n> +\n> +The workers are able to detect both collisions among the entries being\n> +concurrently written and collisions among parallel-eligible and\n> +ineligible entries.\n\nI am not sure I understand the above correctly. Does this mean that if\nthere are 2 parallel-ineligible entries that collide the workers will\ndetect that? How is that possible if the parallel-ineligible entries\nare not processed by the workers?\n\nOr maybe s/among/between/ would make things clearer. For example:\n\n\"The workers are able to detect both collisions among entries being\nconcurrently written and collisions between a parallel-eligible entry and\none or more ineligible entries.\"\n\n> The general idea for collision detection is quite\n> +straightforward: for each parallel-eligible entry, the main process must\n> +remove all files that prevent this entry from being written (before\n> +enqueueing it). This includes any non-directory file in the leading path\n> +of the entry. Later, when a worker gets assigned the entry, it looks\n> +again for the non-directories files and for an already existing file at\n> +the entry's path. If any of these checks finds something, the worker\n> +knows that there was a path collision.\n> +\n> +Because parallel checkout can distinguish path collisions from the case\n> +where the file was already present in the working tree before checkout,\n> +we could alternatively choose to skip the checkout of colliding entries.\n> +However, each entry that doesn't get written would have NULL `lstat()`\n> +fields on the index. This could cause performance penalties for\n> +subsequent commands that need to refresh the index, as they would have\n> +to go to the file system to see if the entry is dirty. Thus, if we have\n> +N entries in a colliding group and we decide to write and `lstat()` only\n> +one of them, every subsequent `git-status` will have to read, convert,\n> +and hash the written file N - 1 times. By checking out all colliding\n> +entries (like the sequential code does), we only pay the overhead once,\n> +during checkout.\n> +\n> +Eligible Entries for Parallel Checkout\n> +--------------------------------------\n> +\n> +As previously mentioned, not all entries passed to `checkout_entry()`\n> +will be considered eligible for parallel checkout. More specifically, we\n> +exclude:\n> +\n> +- Symbolic links; to avoid race conditions that, in combination with\n> + path collisions, could cause workers to write files at the wrong\n> + place. For example, if we were to concurrently check out a symlink\n> + 'a' -> 'b' and a regular file 'A/f' in a case-insensitive file system,\n> + we could potentially end up writing the file 'A/f' at 'a/f', due to a\n> + race condition.\n> +\n> +- Regular files that require external filters (either \"one shot\" filters\n> + or long-running process filters). These filters are black-boxes to Git\n> + and may have their own internal locking or non-concurrent assumptions.\n> + So it might not be safe to run multiple instances in parallel.\n> ++\n> +Besides, long-running filters may use the delayed checkout feature to\n> +postpone the return of some filtered blobs. The delayed checkout queue\n> +and the parallel checkout queue are not compatible and should remain\n> +separated.\n\nMaybe: s/separated/separate/\n\n> +Note: regular files that only require internal filters, like end-of-line\n> +conversion and re-encoding, are eligible for parallel checkout.\n> +\n> +Ineligible entries are checked out by the classic sequential codepath\n> +*before* spawning workers.\n> +\n> +Note: submodules's files are also eligible for parallel checkout (as\n> +long as they don't fall into any of the excluding categories mentioned\n> +above). But since each submodule is checked out in its own child\n> +process, we don't mix the superproject's and the submodules' files in\n> +the same parallel checkout process or queue.\n> +\n> +The API\n> +-------\n> +\n> +The parallel checkout API was designed with the goal to minimize changes\n\ns/to minimize/of minimizing/\n\n> +to the current users of the checkout machinery. This means that they\n> +don't have to call a different function for sequential or parallel\n> +checkout. As already mentioned, `checkout_entry()` will automatically\n> +insert the given entry in the parallel checkout queue when this feature\n> +is enabled and the entry is eligible; otherwise, it will just write the\n> +entry right away, using the sequential code. In general, callers of the\n> +parallel checkout API should look similar to this:\n"},{"id":"422240","messageId":"CAP8UFD1=a8BOd-jFz3ByfHGiLZ=7uZn7BjrO9Tqd9C56nLG6Ew@mail.gmail.com","threadId":"55342","inReplyTo":"xmqqpmytkis8.fsf@gitster.g","subject":"Re: [PATCH v2 0/5] Parallel Checkout (part 2)","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-04-19T09:41:34Z","receivedAt":"2021-04-19T09:41:59Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Apr 16, 2021 at 11:43 PM Junio C Hamano wrote:\n>\n> Matheus Tavares writes:\n>\n> > This is the next step in the parallel checkout implementation. As\n> > mt/parallel-checkout-part-1 is now on master, this round is based\n> > directly on master.\n> >\n> > Changes since v1:\n>\n> Now, I do not think we've seen any response to these messages.\n>\n> It seems that review comments for the earlier round cf.\n> \n>\n> have been addressed? Should we merge it down to 'next' now?\n\nYeah, my concerns in v1 were all addressed. I took another look at v3\nand found only small nits in the documentation patch. For what it's\nworth, with or without the small nits addressed, the v3 has my:\n\nReviewed-by: Christian Couder \n"},{"id":"422329","messageId":"cover.1618861380.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618790794.git.matheus.bernardino@usp.br","subject":"[PATCH v4 0/5] Parallel Checkout (part 2)","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T19:53:30Z","receivedAt":"2021-04-19T19:53:46Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This version is almost identical to v3, but the last patch incorporates\nthe typo fixes and other rewording suggestions Christian made about the\ndesign doc on the last round.\n\nI decided to remove the sentence about step 3 dominating the execution\ntime as that's not always the case on e.g. a non-local clone or\nsparse-checkout.\n\nMatheus Tavares (5):\n unpack-trees: add basic support for parallel checkout\n parallel-checkout: make it truly parallel\n parallel-checkout: add configuration options\n parallel-checkout: support progress displaying\n parallel-checkout: add design documentation\n\n .gitignore | 1 +\n Documentation/Makefile | 1 +\n Documentation/config/checkout.txt | 21 +\n Documentation/technical/parallel-checkout.txt | 270 ++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/checkout--worker.c | 145 ++++\n entry.c | 17 +-\n git.c | 2 +\n parallel-checkout.c | 655 ++++++++++++++++++\n parallel-checkout.h | 111 +++\n unpack-trees.c | 19 +-\n 12 files changed, 1240 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/technical/parallel-checkout.txt\n create mode 100644 builtin/checkout--worker.c\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\nRange-diff against v3:\n1: 7096822c14 = 1: 7096822c14 unpack-trees: add basic support for parallel checkout\n2: 4526516ea0 = 2: 4526516ea0 parallel-checkout: make it truly parallel\n3: ad165c0637 = 3: ad165c0637 parallel-checkout: add configuration options\n4: cf9e28dc0e = 4: cf9e28dc0e parallel-checkout: support progress displaying\n5: 415d4114aa ! 5: fd929f072c parallel-checkout: add design documentation\n @@ Documentation/technical/parallel-checkout.txt (new)\n +* Step 4: Write the new index to disk.\n +\n +Step 3 is the focus of the \"parallel checkout\" effort described here.\n -+It dominates the execution time for most of the above command types.\n +\n +Sequential Implementation\n +-------------------------\n @@ Documentation/technical/parallel-checkout.txt (new)\n +It wouldn't be safe to perform Step 3b in parallel, as there could be\n +race conditions between file creations and removals. Instead, the\n +parallel checkout framework lets the sequential code handle Step 3b,\n -+and use parallel workers to replace the sequential\n ++and uses parallel workers to replace the sequential\n +`entry.c:write_entry()` calls from Step 3c.\n +\n +Rejected Multi-Threaded Solution\n @@ Documentation/technical/parallel-checkout.txt (new)\n +warning for the user, like the classic sequential checkout does.\n +\n +The workers are able to detect both collisions among the entries being\n -+concurrently written and collisions among parallel-eligible and\n -+ineligible entries. The general idea for collision detection is quite\n -+straightforward: for each parallel-eligible entry, the main process must\n -+remove all files that prevent this entry from being written (before\n -+enqueueing it). This includes any non-directory file in the leading path\n -+of the entry. Later, when a worker gets assigned the entry, it looks\n -+again for the non-directories files and for an already existing file at\n -+the entry's path. If any of these checks finds something, the worker\n -+knows that there was a path collision.\n ++concurrently written and collisions between a parallel-eligible entry\n ++and an ineligible entry. The general idea for collision detection is\n ++quite straightforward: for each parallel-eligible entry, the main\n ++process must remove all files that prevent this entry from being written\n ++(before enqueueing it). This includes any non-directory file in the\n ++leading path of the entry. Later, when a worker gets assigned the entry,\n ++it looks again for the non-directories files and for an already existing\n ++file at the entry's path. If any of these checks finds something, the\n ++worker knows that there was a path collision.\n +\n +Because parallel checkout can distinguish path collisions from the case\n +where the file was already present in the working tree before checkout,\n @@ Documentation/technical/parallel-checkout.txt (new)\n +Besides, long-running filters may use the delayed checkout feature to\n +postpone the return of some filtered blobs. The delayed checkout queue\n +and the parallel checkout queue are not compatible and should remain\n -+separated.\n ++separate.\n ++\n +Note: regular files that only require internal filters, like end-of-line\n +conversion and re-encoding, are eligible for parallel checkout.\n @@ Documentation/technical/parallel-checkout.txt (new)\n +The API\n +-------\n +\n -+The parallel checkout API was designed with the goal to minimize changes\n -+to the current users of the checkout machinery. This means that they\n -+don't have to call a different function for sequential or parallel\n ++The parallel checkout API was designed with the goal of minimizing\n ++changes to the current users of the checkout machinery. This means that\n ++they don't have to call a different function for sequential or parallel\n +checkout. As already mentioned, `checkout_entry()` will automatically\n +insert the given entry in the parallel checkout queue when this feature\n +is enabled and the entry is eligible; otherwise, it will just write the\n-- \n2.30.1\n\n"},{"id":"422330","messageId":"7096822c143ed4aed66e811227740610f4d6ac8a.1618861380.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618861380.git.matheus.bernardino@usp.br","subject":"[PATCH v4 1/5] unpack-trees: add basic support for parallel checkout","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T19:53:31Z","receivedAt":"2021-04-19T19:53:49Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"This new interface allows us to enqueue some of the entries being\nchecked out to later uncompress them, apply in-process filters, and\nwrite out the files in parallel. For now, the parallel checkout\nmachinery is enabled by default and there is no user configuration, but\nrun_parallel_checkout() just writes the queued entries in sequence\n(without spawning additional workers). The next patch will actually\nimplement the parallelism and, later, we will make it configurable.\n\nNote that, to avoid potential data races, not all entries are eligible\nfor parallel checkout. Also, paths that collide on disk (e.g.\ncase-sensitive paths in case-insensitive file systems), are detected by\nthe parallel checkout code and skipped, so that they can be safely\nsequentially handled later. The collision detection works like the\nfollowing:\n\n- If the collision was at basename (e.g. 'a/b' and 'a/B'), the framework\n detects it by looking for EEXIST and EISDIR errors after an\n open(O_CREAT | O_EXCL) failure.\n\n- If the collision was at dirname (e.g. 'a/b' and 'A'), it is detected\n at the has_dirs_only_path() check, which is done for the leading path\n of each item in the parallel checkout queue.\n\nBoth verifications rely on the fact that, before enqueueing an entry for\nparallel checkout, checkout_entry() makes sure that there is no file at\nthe entry's path and that its leading components are all real\ndirectories. So, any later change in these conditions indicates that\nthere was a collision (either between two parallel-eligible entries or\nbetween an eligible and an ineligible one).\n\nAfter all parallel-eligible entries have been processed, the collided\n(and thus, skipped) entries are sequentially fed to checkout_entry()\nagain. This is similar to the way the current code deals with\ncollisions, overwriting the previously checked out entries with the\nsubsequent ones. The only difference is that, since we no longer create\nthe files in the same order that they appear on index, we are not able\nto determine which of the colliding entries will survive on disk (for\nthe classic code, it is always the last entry).\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Makefile | 1 +\n entry.c | 17 ++-\n parallel-checkout.c | 365 ++++++++++++++++++++++++++++++++++++++++++++\n parallel-checkout.h | 32 ++++\n unpack-trees.c | 6 +-\n 5 files changed, 418 insertions(+), 3 deletions(-)\n create mode 100644 parallel-checkout.c\n create mode 100644 parallel-checkout.h\n\ndiff --git a/Makefile b/Makefile\nindex a6a73c5741..99734cc2ea 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -946,6 +946,7 @@ LIB_OBJS += pack-revindex.o\n LIB_OBJS += pack-write.o\n LIB_OBJS += packfile.o\n LIB_OBJS += pager.o\n+LIB_OBJS += parallel-checkout.o\n LIB_OBJS += parse-options-cb.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += patch-delta.o\ndiff --git a/entry.c b/entry.c\nindex 2dc94ba5cc..d7ed38aa40 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -7,6 +7,7 @@\n #include \"progress.h\"\n #include \"fsmonitor.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -426,8 +427,17 @@ static void mark_colliding_entries(const struct checkout *state,\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-\t\tif (dup == ce)\n-\t\t\tbreak;\n+\t\tif (dup == ce) {\n+\t\t\t/*\n+\t\t\t * Parallel checkout doesn't create the files in index\n+\t\t\t * order. So the other side of the collision may appear\n+\t\t\t * after the given cache_entry in the array.\n+\t\t\t */\n+\t\t\tif (parallel_checkout_status() == PC_RUNNING)\n+\t\t\t\tcontinue;\n+\t\t\telse\n+\t\t\t\tbreak;\n+\t\t}\n \n \t\tif (dup->ce_flags & (CE_MATCHED | CE_VALID | CE_SKIP_WORKTREE))\n \t\t\tcontinue;\n@@ -536,6 +546,9 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,\n \t\tca = &ca_buf;\n \t}\n \n+\tif (!enqueue_checkout(ce, ca))\n+\t\treturn 0;\n+\n \treturn write_entry(ce, path.buf, ca, state, 0);\n }\n \ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nnew file mode 100644\nindex 0000000000..590e2a3046\n--- /dev/null\n+++ b/parallel-checkout.c\n@@ -0,0 +1,365 @@\n+#include \"cache.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"streaming.h\"\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/* pointer to a istate->cache[] entry. Not owned by us. */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tstruct stat st;\n+\tenum pc_item_status status;\n+};\n+\n+struct parallel_checkout {\n+\tenum pc_status status;\n+\tstruct parallel_checkout_item *items; /* The parallel checkout queue. */\n+\tsize_t nr, alloc;\n+};\n+\n+static struct parallel_checkout parallel_checkout;\n+\n+enum pc_status parallel_checkout_status(void)\n+{\n+\treturn parallel_checkout.status;\n+}\n+\n+void init_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status != PC_UNINITIALIZED)\n+\t\tBUG(\"parallel checkout already initialized\");\n+\n+\tparallel_checkout.status = PC_ACCEPTING_ENTRIES;\n+}\n+\n+static void finish_parallel_checkout(void)\n+{\n+\tif (parallel_checkout.status == PC_UNINITIALIZED)\n+\t\tBUG(\"cannot finish parallel checkout: not initialized yet\");\n+\n+\tfree(parallel_checkout.items);\n+\tmemset(¶llel_checkout, 0, sizeof(parallel_checkout));\n+}\n+\n+static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n+\t\t\t\t\t const struct conv_attrs *ca)\n+{\n+\tenum conv_attrs_classification c;\n+\n+\t/*\n+\t * Symlinks cannot be checked out in parallel as, in case of path\n+\t * collision, they could racily replace leading directories of other\n+\t * entries being checked out. Submodules are checked out in child\n+\t * processes, which have their own parallel checkout queues.\n+\t */\n+\tif (!S_ISREG(ce->ce_mode))\n+\t\treturn 0;\n+\n+\tc = classify_conv_attrs(ca);\n+\tswitch (c) {\n+\tcase CA_CLASS_INCORE:\n+\t\treturn 1;\n+\n+\tcase CA_CLASS_INCORE_FILTER:\n+\t\t/*\n+\t\t * It would be safe to allow concurrent instances of\n+\t\t * single-file smudge filters, like rot13, but we should not\n+\t\t * assume that all filters are parallel-process safe. So we\n+\t\t * don't allow this.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_INCORE_PROCESS:\n+\t\t/*\n+\t\t * The parallel queue and the delayed queue are not compatible,\n+\t\t * so they must be kept completely separated. And we can't tell\n+\t\t * if a long-running process will delay its response without\n+\t\t * actually asking it to perform the filtering. Therefore, this\n+\t\t * type of filter is not allowed in parallel checkout.\n+\t\t *\n+\t\t * Furthermore, there should only be one instance of the\n+\t\t * long-running process filter as we don't know how it is\n+\t\t * managing its own concurrency. So, spreading the entries that\n+\t\t * requisite such a filter among the parallel workers would\n+\t\t * require a lot more inter-process communication. We would\n+\t\t * probably have to designate a single process to interact with\n+\t\t * the filter and send all the necessary data to it, for each\n+\t\t * entry.\n+\t\t */\n+\t\treturn 0;\n+\n+\tcase CA_CLASS_STREAMABLE:\n+\t\treturn 1;\n+\n+\tdefault:\n+\t\tBUG(\"unsupported conv_attrs classification '%d'\", c);\n+\t}\n+}\n+\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n+{\n+\tstruct parallel_checkout_item *pc_item;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES ||\n+\t !is_eligible_for_parallel_checkout(ce, ca))\n+\t\treturn -1;\n+\n+\tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n+\t\t parallel_checkout.alloc);\n+\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item->ce = ce;\n+\tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n+\tpc_item->status = PC_ITEM_PENDING;\n+\n+\treturn 0;\n+}\n+\n+static int handle_results(struct checkout *state)\n+{\n+\tint ret = 0;\n+\tsize_t i;\n+\tint have_pending = 0;\n+\n+\t/*\n+\t * We first update the successfully written entries with the collected\n+\t * stat() data, so that they can be found by mark_colliding_entries(),\n+\t * in the next loop, when necessary.\n+\t */\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\tif (pc_item->status == PC_ITEM_WRITTEN)\n+\t\t\tupdate_ce_after_write(state, pc_item->ce, &pc_item->st);\n+\t}\n+\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\n+\t\tswitch(pc_item->status) {\n+\t\tcase PC_ITEM_WRITTEN:\n+\t\t\t/* Already handled */\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_COLLIDED:\n+\t\t\t/*\n+\t\t\t * The entry could not be checked out due to a path\n+\t\t\t * collision with another entry. Since there can only\n+\t\t\t * be one entry of each colliding group on the disk, we\n+\t\t\t * could skip trying to check out this one and move on.\n+\t\t\t * However, this would leave the unwritten entries with\n+\t\t\t * null stat() fields on the index, which could\n+\t\t\t * potentially slow down subsequent operations that\n+\t\t\t * require refreshing it: git would not be able to\n+\t\t\t * trust st_size and would have to go to the filesystem\n+\t\t\t * to see if the contents match (see ie_modified()).\n+\t\t\t *\n+\t\t\t * Instead, let's pay the overhead only once, now, and\n+\t\t\t * call checkout_entry_ca() again for this file, to\n+\t\t\t * have its stat() data stored in the index. This also\n+\t\t\t * has the benefit of adding this entry and its\n+\t\t\t * colliding pair to the collision report message.\n+\t\t\t * Additionally, this overwriting behavior is consistent\n+\t\t\t * with what the sequential checkout does, so it doesn't\n+\t\t\t * add any extra overhead.\n+\t\t\t */\n+\t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n+\t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tbreak;\n+\t\tcase PC_ITEM_PENDING:\n+\t\t\thave_pending = 1;\n+\t\t\t/* fall through */\n+\t\tcase PC_ITEM_FAILED:\n+\t\t\tret = -1;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unknown checkout item status in parallel checkout\");\n+\t\t}\n+\t}\n+\n+\tif (have_pending)\n+\t\terror(\"parallel checkout finished with pending entries\");\n+\n+\treturn ret;\n+}\n+\n+static int reset_fd(int fd, const char *path)\n+{\n+\tif (lseek(fd, 0, SEEK_SET) != 0)\n+\t\treturn error_errno(\"failed to rewind descriptor of '%s'\", path);\n+\tif (ftruncate(fd, 0))\n+\t\treturn error_errno(\"failed to truncate file '%s'\", path);\n+\treturn 0;\n+}\n+\n+static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n+\t\t\t const char *path)\n+{\n+\tint ret;\n+\tstruct stream_filter *filter;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *blob;\n+\tunsigned long size;\n+\tssize_t wrote;\n+\n+\t/* Sanity check */\n+\tassert(is_eligible_for_parallel_checkout(pc_item->ce, &pc_item->ca));\n+\n+\tfilter = get_stream_filter_ca(&pc_item->ca, &pc_item->ce->oid);\n+\tif (filter) {\n+\t\tif (stream_blob_to_fd(fd, &pc_item->ce->oid, filter, 1)) {\n+\t\t\t/* On error, reset fd to try writing without streaming */\n+\t\t\tif (reset_fd(fd, path))\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tblob = read_blob_entry(pc_item->ce, &size);\n+\tif (!blob)\n+\t\treturn error(\"cannot read object %s '%s'\",\n+\t\t\t oid_to_hex(&pc_item->ce->oid), pc_item->ce->name);\n+\n+\t/*\n+\t * checkout metadata is used to give context for external process\n+\t * filters. Files requiring such filters are not eligible for parallel\n+\t * checkout, so pass NULL.\n+\t */\n+\tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n+\t\t\t\t\t blob, size, &buf, NULL);\n+\n+\tif (ret) {\n+\t\tsize_t newsize;\n+\t\tfree(blob);\n+\t\tblob = strbuf_detach(&buf, &newsize);\n+\t\tsize = newsize;\n+\t}\n+\n+\twrote = write_in_full(fd, blob, size);\n+\tfree(blob);\n+\tif (wrote < 0)\n+\t\treturn error(\"unable to write file '%s'\", path);\n+\n+\treturn 0;\n+}\n+\n+static int close_and_clear(int *fd)\n+{\n+\tint ret = 0;\n+\n+\tif (*fd >= 0) {\n+\t\tret = close(*fd);\n+\t\t*fd = -1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t\t struct checkout *state)\n+{\n+\tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n+\tint fd = -1, fstat_done = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tconst char *dir_sep;\n+\n+\tstrbuf_add(&path, state->base_dir, state->base_dir_len);\n+\tstrbuf_add(&path, pc_item->ce->name, pc_item->ce->ce_namelen);\n+\n+\tdir_sep = find_last_dir_sep(path.buf);\n+\n+\t/*\n+\t * The leading dirs should have been already created by now. But, in\n+\t * case of path collisions, one of the dirs could have been replaced by\n+\t * a symlink (checked out after we enqueued this entry for parallel\n+\t * checkout). Thus, we must check the leading dirs again.\n+\t */\n+\tif (dir_sep && !has_dirs_only_path(path.buf, dir_sep - path.buf,\n+\t\t\t\t\t state->base_dir_len)) {\n+\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\tgoto out;\n+\t}\n+\n+\tfd = open(path.buf, O_WRONLY | O_CREAT | O_EXCL, mode);\n+\n+\tif (fd < 0) {\n+\t\tif (errno == EEXIST || errno == EISDIR) {\n+\t\t\t/*\n+\t\t\t * Errors which probably represent a path collision.\n+\t\t\t * Suppress the error message and mark the item to be\n+\t\t\t * retried later, sequentially. ENOTDIR and ENOENT are\n+\t\t\t * also interesting, but the above has_dirs_only_path()\n+\t\t\t * call should have already caught these cases.\n+\t\t\t */\n+\t\t\tpc_item->status = PC_ITEM_COLLIDED;\n+\t\t} else {\n+\t\t\terror_errno(\"failed to open file '%s'\", path.buf);\n+\t\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\t}\n+\t\tgoto out;\n+\t}\n+\n+\tif (write_pc_item_to_fd(pc_item, fd, path.buf)) {\n+\t\t/* Error was already reported. */\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tclose_and_clear(&fd);\n+\t\tunlink(path.buf);\n+\t\tgoto out;\n+\t}\n+\n+\tfstat_done = fstat_checkout_output(fd, state, &pc_item->st);\n+\n+\tif (close_and_clear(&fd)) {\n+\t\terror_errno(\"unable to close file '%s'\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tif (state->refresh_cache && !fstat_done && lstat(path.buf, &pc_item->st) < 0) {\n+\t\terror_errno(\"unable to stat just-written file '%s'\", path.buf);\n+\t\tpc_item->status = PC_ITEM_FAILED;\n+\t\tgoto out;\n+\t}\n+\n+\tpc_item->status = PC_ITEM_WRITTEN;\n+\n+out:\n+\tstrbuf_release(&path);\n+}\n+\n+static void write_items_sequentially(struct checkout *state)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < parallel_checkout.nr; i++)\n+\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+}\n+\n+int run_parallel_checkout(struct checkout *state)\n+{\n+\tint ret;\n+\n+\tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n+\t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n+\n+\tparallel_checkout.status = PC_RUNNING;\n+\n+\twrite_items_sequentially(state);\n+\tret = handle_results(state);\n+\n+\tfinish_parallel_checkout();\n+\treturn ret;\n+}\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nnew file mode 100644\nindex 0000000000..4ad2a519b3\n--- /dev/null\n+++ b/parallel-checkout.h\n@@ -0,0 +1,32 @@\n+#ifndef PARALLEL_CHECKOUT_H\n+#define PARALLEL_CHECKOUT_H\n+\n+struct cache_entry;\n+struct checkout;\n+struct conv_attrs;\n+\n+enum pc_status {\n+\tPC_UNINITIALIZED = 0,\n+\tPC_ACCEPTING_ENTRIES,\n+\tPC_RUNNING,\n+};\n+\n+enum pc_status parallel_checkout_status(void);\n+\n+/*\n+ * Put parallel checkout into the PC_ACCEPTING_ENTRIES state. Should be used\n+ * only when in the PC_UNINITIALIZED state.\n+ */\n+void init_parallel_checkout(void);\n+\n+/*\n+ * Return -1 if parallel checkout is currently not accepting entries or if the\n+ * entry is not eligible for parallel checkout. Otherwise, enqueue the entry\n+ * for later write and return 0.\n+ */\n+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+\n+/* Write all the queued entries, returning 0 on success.*/\n+int run_parallel_checkout(struct checkout *state);\n+\n+#endif /* PARALLEL_CHECKOUT_H */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8a1afbc1e4..f0430d458d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -17,6 +17,7 @@\n #include \"object-store.h\"\n #include \"promisor-remote.h\"\n #include \"entry.h\"\n+#include \"parallel-checkout.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -441,7 +442,6 @@ static int check_updates(struct unpack_trees_options *o,\n \tif (should_update_submodules())\n \t\tload_gitmodules_file(index, &state);\n \n-\tenable_delayed_checkout(&state);\n \tif (has_promisor_remote()) {\n \t\t/*\n \t\t * Prefetch the objects that are to be checked out in the loop\n@@ -464,6 +464,9 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t\t\t\t to_fetch.oid, to_fetch.nr);\n \t\toid_array_clear(&to_fetch);\n \t}\n+\n+\tenable_delayed_checkout(&state);\n+\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -477,6 +480,7 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n+\terrs |= run_parallel_checkout(&state);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"422331","messageId":"4526516ea0e2c7897758eb54f0d56bfe49cc9f89.1618861380.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618861380.git.matheus.bernardino@usp.br","subject":"[PATCH v4 2/5] parallel-checkout: make it truly parallel","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T19:53:32Z","receivedAt":"2021-04-19T19:53:50Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Use multiple worker processes to distribute the queued entries and call\nwrite_pc_item() in parallel for them. The items are distributed\nuniformly in contiguous chunks. This minimizes the chances of two\nworkers writing to the same directory simultaneously, which could affect\nperformance due to lock contention in the kernel. Work stealing (or any\nother format of re-distribution) is not implemented yet.\n\nThe protocol between the main process and the workers is quite simple.\nThey exchange binary messages packed in pkt-line format, and use\nPKT-FLUSH to mark the end of input (from both sides). The main process\nstarts the communication by sending N pkt-lines, each corresponding to\nan item that needs to be written. These packets contain all the\nnecessary information to load, smudge, and write the blob associated\nwith each item. Then it waits for the worker to send back N pkt-lines\ncontaining the results for each item. The resulting packet must contain:\nthe identification number of the item that it refers to, the status of\nthe operation, and the lstat() data gathered after writing the file (iff\nthe operation was successful).\n\nFor now, checkout always uses a hardcoded value of 2 workers, only to\ndemonstrate that the parallel checkout framework correctly divides and\nwrites the queued entries. The next patch will add user configurations\nand define a more reasonable default, based on tests with the said\nsettings.\n\nCo-authored-by: Nguyễn Thái Ngọc Duy \nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n .gitignore | 1 +\n Makefile | 1 +\n builtin.h | 1 +\n builtin/checkout--worker.c | 145 ++++++++++++++++++\n git.c | 2 +\n parallel-checkout.c | 300 +++++++++++++++++++++++++++++++++----\n parallel-checkout.h | 73 ++++++++-\n 7 files changed, 496 insertions(+), 27 deletions(-)\n create mode 100644 builtin/checkout--worker.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 3dcdb6bb5a..96c794b1c7 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -33,6 +33,7 @@\n /git-check-mailmap\n /git-check-ref-format\n /git-checkout\n+/git-checkout--worker\n /git-checkout-index\n /git-cherry\n /git-cherry-pick\ndiff --git a/Makefile b/Makefile\nindex 99734cc2ea..84e15f58a0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1062,6 +1062,7 @@ BUILTIN_OBJS += builtin/check-attr.o\n BUILTIN_OBJS += builtin/check-ignore.o\n BUILTIN_OBJS += builtin/check-mailmap.o\n BUILTIN_OBJS += builtin/check-ref-format.o\n+BUILTIN_OBJS += builtin/checkout--worker.o\n BUILTIN_OBJS += builtin/checkout-index.o\n BUILTIN_OBJS += builtin/checkout.o\n BUILTIN_OBJS += builtin/clean.o\ndiff --git a/builtin.h b/builtin.h\nindex b6ce981b73..16ecd5586f 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -123,6 +123,7 @@ int cmd_bugreport(int argc, const char **argv, const char *prefix);\n int cmd_bundle(int argc, const char **argv, const char *prefix);\n int cmd_cat_file(int argc, const char **argv, const char *prefix);\n int cmd_checkout(int argc, const char **argv, const char *prefix);\n+int cmd_checkout__worker(int argc, const char **argv, const char *prefix);\n int cmd_checkout_index(int argc, const char **argv, const char *prefix);\n int cmd_check_attr(int argc, const char **argv, const char *prefix);\n int cmd_check_ignore(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/checkout--worker.c b/builtin/checkout--worker.c\nnew file mode 100644\nindex 0000000000..31e0de2f7e\n--- /dev/null\n+++ b/builtin/checkout--worker.c\n@@ -0,0 +1,145 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"entry.h\"\n+#include \"parallel-checkout.h\"\n+#include \"parse-options.h\"\n+#include \"pkt-line.h\"\n+\n+static void packet_to_pc_item(const char *buffer, int len,\n+\t\t\t struct parallel_checkout_item *pc_item)\n+{\n+\tconst struct pc_item_fixed_portion *fixed_portion;\n+\tconst char *variant;\n+\tchar *encoding;\n+\n+\tif (len < sizeof(struct pc_item_fixed_portion))\n+\t\tBUG(\"checkout worker received too short item (got %dB, exp %dB)\",\n+\t\t len, (int)sizeof(struct pc_item_fixed_portion));\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)buffer;\n+\n+\tif (len - sizeof(struct pc_item_fixed_portion) !=\n+\t\tfixed_portion->name_len + fixed_portion->working_tree_encoding_len)\n+\t\tBUG(\"checkout worker received corrupted item\");\n+\n+\tvariant = buffer + sizeof(struct pc_item_fixed_portion);\n+\n+\t/*\n+\t * Note: the main process uses zero length to communicate that the\n+\t * encoding is NULL. There is no use case that requires sending an\n+\t * actual empty string, since convert_attrs() never sets\n+\t * ca.working_tree_enconding to \"\".\n+\t */\n+\tif (fixed_portion->working_tree_encoding_len) {\n+\t\tencoding = xmemdupz(variant,\n+\t\t\t\t fixed_portion->working_tree_encoding_len);\n+\t\tvariant += fixed_portion->working_tree_encoding_len;\n+\t} else {\n+\t\tencoding = NULL;\n+\t}\n+\n+\tmemset(pc_item, 0, sizeof(*pc_item));\n+\tpc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len);\n+\tpc_item->ce->ce_namelen = fixed_portion->name_len;\n+\tpc_item->ce->ce_mode = fixed_portion->ce_mode;\n+\tmemcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen);\n+\toidcpy(&pc_item->ce->oid, &fixed_portion->oid);\n+\n+\tpc_item->id = fixed_portion->id;\n+\tpc_item->ca.crlf_action = fixed_portion->crlf_action;\n+\tpc_item->ca.ident = fixed_portion->ident;\n+\tpc_item->ca.working_tree_encoding = encoding;\n+}\n+\n+static void report_result(struct parallel_checkout_item *pc_item)\n+{\n+\tstruct pc_item_result res;\n+\tsize_t size;\n+\n+\tres.id = pc_item->id;\n+\tres.status = pc_item->status;\n+\n+\tif (pc_item->status == PC_ITEM_WRITTEN) {\n+\t\tres.st = pc_item->st;\n+\t\tsize = sizeof(res);\n+\t} else {\n+\t\tsize = PC_ITEM_RESULT_BASE_SIZE;\n+\t}\n+\n+\tpacket_write(1, (const char *)&res, size);\n+}\n+\n+/* Free the worker-side malloced data, but not pc_item itself. */\n+static void release_pc_item_data(struct parallel_checkout_item *pc_item)\n+{\n+\tfree((char *)pc_item->ca.working_tree_encoding);\n+\tdiscard_cache_entry(pc_item->ce);\n+}\n+\n+static void worker_loop(struct checkout *state)\n+{\n+\tstruct parallel_checkout_item *items = NULL;\n+\tsize_t i, nr = 0, alloc = 0;\n+\n+\twhile (1) {\n+\t\tint len = packet_read(0, NULL, NULL, packet_buffer,\n+\t\t\t\t sizeof(packet_buffer), 0);\n+\n+\t\tif (len < 0)\n+\t\t\tBUG(\"packet_read() returned negative value\");\n+\t\telse if (!len)\n+\t\t\tbreak;\n+\n+\t\tALLOC_GROW(items, nr + 1, alloc);\n+\t\tpacket_to_pc_item(packet_buffer, len, &items[nr++]);\n+\t}\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = &items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\treport_result(pc_item);\n+\t\trelease_pc_item_data(pc_item);\n+\t}\n+\n+\tpacket_flush(1);\n+\n+\tfree(items);\n+}\n+\n+static const char * const checkout_worker_usage[] = {\n+\tN_(\"git checkout--worker []\"),\n+\tNULL\n+};\n+\n+int cmd_checkout__worker(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct checkout state = CHECKOUT_INIT;\n+\tstruct option checkout_worker_options[] = {\n+\t\tOPT_STRING(0, \"prefix\", &state.base_dir, N_(\"string\"),\n+\t\t\tN_(\"when creating files, prepend \")),\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(checkout_worker_usage,\n+\t\t\t\t checkout_worker_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix, checkout_worker_options,\n+\t\t\t checkout_worker_usage, 0);\n+\tif (argc > 0)\n+\t\tusage_with_options(checkout_worker_usage, checkout_worker_options);\n+\n+\tif (state.base_dir)\n+\t\tstate.base_dir_len = strlen(state.base_dir);\n+\n+\t/*\n+\t * Setting this on a worker won't actually update the index. We just\n+\t * need to tell the checkout machinery to lstat() the written entries,\n+\t * so that we can send this data back to the main process.\n+\t */\n+\tstate.refresh_cache = 1;\n+\n+\tworker_loop(&state);\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 9bc077a025..532b5b4136 100644\n--- a/git.c\n+++ b/git.c\n@@ -490,6 +490,8 @@ static struct cmd_struct commands[] = {\n \t{ \"check-mailmap\", cmd_check_mailmap, RUN_SETUP },\n \t{ \"check-ref-format\", cmd_check_ref_format, NO_PARSEOPT },\n \t{ \"checkout\", cmd_checkout, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"checkout--worker\", cmd_checkout__worker,\n+\t\tRUN_SETUP | NEED_WORK_TREE | SUPPORT_SUPER_PREFIX },\n \t{ \"checkout-index\", cmd_checkout_index,\n \t\tRUN_SETUP | NEED_WORK_TREE},\n \t{ \"cherry\", cmd_cherry, RUN_SETUP },\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 590e2a3046..836154fec6 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,28 +1,14 @@\n #include \"cache.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n+#include \"pkt-line.h\"\n+#include \"run-command.h\"\n+#include \"sigchain.h\"\n #include \"streaming.h\"\n \n-enum pc_item_status {\n-\tPC_ITEM_PENDING = 0,\n-\tPC_ITEM_WRITTEN,\n-\t/*\n-\t * The entry could not be written because there was another file\n-\t * already present in its path or leading directories. Since\n-\t * checkout_entry_ca() removes such files from the working tree before\n-\t * enqueueing the entry for parallel checkout, it means that there was\n-\t * a path collision among the entries being written.\n-\t */\n-\tPC_ITEM_COLLIDED,\n-\tPC_ITEM_FAILED,\n-};\n-\n-struct parallel_checkout_item {\n-\t/* pointer to a istate->cache[] entry. Not owned by us. */\n-\tstruct cache_entry *ce;\n-\tstruct conv_attrs ca;\n-\tstruct stat st;\n-\tenum pc_item_status status;\n+struct pc_worker {\n+\tstruct child_process cp;\n+\tsize_t next_item_to_complete, nr_items_to_complete;\n };\n \n struct parallel_checkout {\n@@ -59,6 +45,7 @@ static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n \t\t\t\t\t const struct conv_attrs *ca)\n {\n \tenum conv_attrs_classification c;\n+\tsize_t packed_item_size;\n \n \t/*\n \t * Symlinks cannot be checked out in parallel as, in case of path\n@@ -69,6 +56,18 @@ static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,\n \tif (!S_ISREG(ce->ce_mode))\n \t\treturn 0;\n \n+\tpacked_item_size = sizeof(struct pc_item_fixed_portion) + ce->ce_namelen +\n+\t\t(ca->working_tree_encoding ? strlen(ca->working_tree_encoding) : 0);\n+\n+\t/*\n+\t * The amount of data we send to the workers per checkout item is\n+\t * typically small (75~300B). So unless we find an insanely huge path\n+\t * of 64KB, we should never reach the 65KB limit of one pkt-line. If\n+\t * that does happen, we let the sequential code handle the item.\n+\t */\n+\tif (packed_item_size > LARGE_PACKET_DATA_MAX)\n+\t\treturn 0;\n+\n \tc = classify_conv_attrs(ca);\n \tswitch (c) {\n \tcase CA_CLASS_INCORE:\n@@ -121,10 +120,12 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \tALLOC_GROW(parallel_checkout.items, parallel_checkout.nr + 1,\n \t\t parallel_checkout.alloc);\n \n-\tpc_item = ¶llel_checkout.items[parallel_checkout.nr++];\n+\tpc_item = ¶llel_checkout.items[parallel_checkout.nr];\n \tpc_item->ce = ce;\n \tmemcpy(&pc_item->ca, ca, sizeof(pc_item->ca));\n \tpc_item->status = PC_ITEM_PENDING;\n+\tpc_item->id = parallel_checkout.nr;\n+\tparallel_checkout.nr++;\n \n \treturn 0;\n }\n@@ -236,7 +237,8 @@ static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,\n \t/*\n \t * checkout metadata is used to give context for external process\n \t * filters. Files requiring such filters are not eligible for parallel\n-\t * checkout, so pass NULL.\n+\t * checkout, so pass NULL. Note: if that changes, the metadata must also\n+\t * be passed from the main process to the workers.\n \t */\n \tret = convert_to_working_tree_ca(&pc_item->ca, pc_item->ce->name,\n \t\t\t\t\t blob, size, &buf, NULL);\n@@ -268,8 +270,8 @@ static int close_and_clear(int *fd)\n \treturn ret;\n }\n \n-static void write_pc_item(struct parallel_checkout_item *pc_item,\n-\t\t\t struct checkout *state)\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state)\n {\n \tunsigned int mode = (pc_item->ce->ce_mode & 0100) ? 0777 : 0666;\n \tint fd = -1, fstat_done = 0;\n@@ -340,6 +342,240 @@ static void write_pc_item(struct parallel_checkout_item *pc_item,\n \tstrbuf_release(&path);\n }\n \n+static void send_one_item(int fd, struct parallel_checkout_item *pc_item)\n+{\n+\tsize_t len_data;\n+\tchar *data, *variant;\n+\tstruct pc_item_fixed_portion *fixed_portion;\n+\tconst char *working_tree_encoding = pc_item->ca.working_tree_encoding;\n+\tsize_t name_len = pc_item->ce->ce_namelen;\n+\tsize_t working_tree_encoding_len = working_tree_encoding ?\n+\t\t\t\t\t strlen(working_tree_encoding) : 0;\n+\n+\t/*\n+\t * Any changes in the calculation of the message size must also be made\n+\t * in is_eligible_for_parallel_checkout().\n+\t */\n+\tlen_data = sizeof(struct pc_item_fixed_portion) + name_len +\n+\t\t working_tree_encoding_len;\n+\n+\tdata = xcalloc(1, len_data);\n+\n+\tfixed_portion = (struct pc_item_fixed_portion *)data;\n+\tfixed_portion->id = pc_item->id;\n+\tfixed_portion->ce_mode = pc_item->ce->ce_mode;\n+\tfixed_portion->crlf_action = pc_item->ca.crlf_action;\n+\tfixed_portion->ident = pc_item->ca.ident;\n+\tfixed_portion->name_len = name_len;\n+\tfixed_portion->working_tree_encoding_len = working_tree_encoding_len;\n+\t/*\n+\t * We use hashcpy() instead of oidcpy() because the hash[] positions\n+\t * after `the_hash_algo->rawsz` might not be initialized. And Valgrind\n+\t * would complain about passing uninitialized bytes to a syscall\n+\t * (write(2)). There is no real harm in this case, but the warning could\n+\t * hinder the detection of actual errors.\n+\t */\n+\thashcpy(fixed_portion->oid.hash, pc_item->ce->oid.hash);\n+\n+\tvariant = data + sizeof(*fixed_portion);\n+\tif (working_tree_encoding_len) {\n+\t\tmemcpy(variant, working_tree_encoding, working_tree_encoding_len);\n+\t\tvariant += working_tree_encoding_len;\n+\t}\n+\tmemcpy(variant, pc_item->ce->name, name_len);\n+\n+\tpacket_write(fd, data, len_data);\n+\n+\tfree(data);\n+}\n+\n+static void send_batch(int fd, size_t start, size_t nr)\n+{\n+\tsize_t i;\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\tfor (i = 0; i < nr; i++)\n+\t\tsend_one_item(fd, ¶llel_checkout.items[start + i]);\n+\tpacket_flush(fd);\n+\tsigchain_pop(SIGPIPE);\n+}\n+\n+static struct pc_worker *setup_workers(struct checkout *state, int num_workers)\n+{\n+\tstruct pc_worker *workers;\n+\tint i, workers_with_one_extra_item;\n+\tsize_t base_batch_size, batch_beginning = 0;\n+\n+\tALLOC_ARRAY(workers, num_workers);\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\n+\t\tchild_process_init(cp);\n+\t\tcp->git_cmd = 1;\n+\t\tcp->in = -1;\n+\t\tcp->out = -1;\n+\t\tcp->clean_on_exit = 1;\n+\t\tstrvec_push(&cp->args, \"checkout--worker\");\n+\t\tif (state->base_dir_len)\n+\t\t\tstrvec_pushf(&cp->args, \"--prefix=%s\", state->base_dir);\n+\t\tif (start_command(cp))\n+\t\t\tdie(\"failed to spawn checkout worker\");\n+\t}\n+\n+\tbase_batch_size = parallel_checkout.nr / num_workers;\n+\tworkers_with_one_extra_item = parallel_checkout.nr % num_workers;\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct pc_worker *worker = &workers[i];\n+\t\tsize_t batch_size = base_batch_size;\n+\n+\t\t/* distribute the extra work evenly */\n+\t\tif (i < workers_with_one_extra_item)\n+\t\t\tbatch_size++;\n+\n+\t\tsend_batch(worker->cp.in, batch_beginning, batch_size);\n+\t\tworker->next_item_to_complete = batch_beginning;\n+\t\tworker->nr_items_to_complete = batch_size;\n+\n+\t\tbatch_beginning += batch_size;\n+\t}\n+\n+\treturn workers;\n+}\n+\n+static void finish_workers(struct pc_worker *workers, int num_workers)\n+{\n+\tint i;\n+\n+\t/*\n+\t * Close pipes before calling finish_command() to let the workers\n+\t * exit asynchronously and avoid spending extra time on wait().\n+\t */\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tstruct child_process *cp = &workers[i].cp;\n+\t\tif (cp->in >= 0)\n+\t\t\tclose(cp->in);\n+\t\tif (cp->out >= 0)\n+\t\t\tclose(cp->out);\n+\t}\n+\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tint rc = finish_command(&workers[i].cp);\n+\t\tif (rc > 128) {\n+\t\t\t/*\n+\t\t\t * For a normal non-zero exit, the worker should have\n+\t\t\t * already printed something useful to stderr. But a\n+\t\t\t * death by signal should be mentioned to the user.\n+\t\t\t */\n+\t\t\terror(\"checkout worker %d died of signal %d\", i, rc - 128);\n+\t\t}\n+\t}\n+\n+\tfree(workers);\n+}\n+\n+static inline void assert_pc_item_result_size(int got, int exp)\n+{\n+\tif (got != exp)\n+\t\tBUG(\"wrong result size from checkout worker (got %dB, exp %dB)\",\n+\t\t got, exp);\n+}\n+\n+static void parse_and_save_result(const char *buffer, int len,\n+\t\t\t\t struct pc_worker *worker)\n+{\n+\tstruct pc_item_result *res;\n+\tstruct parallel_checkout_item *pc_item;\n+\tstruct stat *st = NULL;\n+\n+\tif (len < PC_ITEM_RESULT_BASE_SIZE)\n+\t\tBUG(\"too short result from checkout worker (got %dB, exp >=%dB)\",\n+\t\t len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\n+\tres = (struct pc_item_result *)buffer;\n+\n+\t/*\n+\t * Worker should send either the full result struct on success, or\n+\t * just the base (i.e. no stat data), otherwise.\n+\t */\n+\tif (res->status == PC_ITEM_WRITTEN) {\n+\t\tassert_pc_item_result_size(len, (int)sizeof(struct pc_item_result));\n+\t\tst = &res->st;\n+\t} else {\n+\t\tassert_pc_item_result_size(len, (int)PC_ITEM_RESULT_BASE_SIZE);\n+\t}\n+\n+\tif (!worker->nr_items_to_complete)\n+\t\tBUG(\"received result from supposedly finished checkout worker\");\n+\tif (res->id != worker->next_item_to_complete)\n+\t\tBUG(\"unexpected item id from checkout worker (got %\"PRIuMAX\", exp %\"PRIuMAX\")\",\n+\t\t (uintmax_t)res->id, (uintmax_t)worker->next_item_to_complete);\n+\n+\tworker->next_item_to_complete++;\n+\tworker->nr_items_to_complete--;\n+\n+\tpc_item = ¶llel_checkout.items[res->id];\n+\tpc_item->status = res->status;\n+\tif (st)\n+\t\tpc_item->st = *st;\n+}\n+\n+static void gather_results_from_workers(struct pc_worker *workers,\n+\t\t\t\t\tint num_workers)\n+{\n+\tint i, active_workers = num_workers;\n+\tstruct pollfd *pfds;\n+\n+\tCALLOC_ARRAY(pfds, num_workers);\n+\tfor (i = 0; i < num_workers; i++) {\n+\t\tpfds[i].fd = workers[i].cp.out;\n+\t\tpfds[i].events = POLLIN;\n+\t}\n+\n+\twhile (active_workers) {\n+\t\tint nr = poll(pfds, num_workers, -1);\n+\n+\t\tif (nr < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tdie_errno(\"failed to poll checkout workers\");\n+\t\t}\n+\n+\t\tfor (i = 0; i < num_workers && nr > 0; i++) {\n+\t\t\tstruct pc_worker *worker = &workers[i];\n+\t\t\tstruct pollfd *pfd = &pfds[i];\n+\n+\t\t\tif (!pfd->revents)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (pfd->revents & POLLIN) {\n+\t\t\t\tint len = packet_read(pfd->fd, NULL, NULL,\n+\t\t\t\t\t\t packet_buffer,\n+\t\t\t\t\t\t sizeof(packet_buffer), 0);\n+\n+\t\t\t\tif (len < 0) {\n+\t\t\t\t\tBUG(\"packet_read() returned negative value\");\n+\t\t\t\t} else if (!len) {\n+\t\t\t\t\tpfd->fd = -1;\n+\t\t\t\t\tactive_workers--;\n+\t\t\t\t} else {\n+\t\t\t\t\tparse_and_save_result(packet_buffer,\n+\t\t\t\t\t\t\t len, worker);\n+\t\t\t\t}\n+\t\t\t} else if (pfd->revents & POLLHUP) {\n+\t\t\t\tpfd->fd = -1;\n+\t\t\t\tactive_workers--;\n+\t\t\t} else if (pfd->revents & (POLLNVAL | POLLERR)) {\n+\t\t\t\tdie(\"error polling from checkout worker\");\n+\t\t\t}\n+\n+\t\t\tnr--;\n+\t\t}\n+\t}\n+\n+\tfree(pfds);\n+}\n+\n static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n@@ -348,16 +584,28 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n+static const int DEFAULT_NUM_WORKERS = 2;\n+\n int run_parallel_checkout(struct checkout *state)\n {\n-\tint ret;\n+\tint ret, num_workers = DEFAULT_NUM_WORKERS;\n \n \tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n \n-\twrite_items_sequentially(state);\n+\tif (parallel_checkout.nr < num_workers)\n+\t\tnum_workers = parallel_checkout.nr;\n+\n+\tif (num_workers <= 1) {\n+\t\twrite_items_sequentially(state);\n+\t} else {\n+\t\tstruct pc_worker *workers = setup_workers(state, num_workers);\n+\t\tgather_results_from_workers(workers, num_workers);\n+\t\tfinish_workers(workers, num_workers);\n+\t}\n+\n \tret = handle_results(state);\n \n \tfinish_parallel_checkout();\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 4ad2a519b3..ec58716519 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -1,9 +1,14 @@\n #ifndef PARALLEL_CHECKOUT_H\n #define PARALLEL_CHECKOUT_H\n \n+#include \"convert.h\"\n+\n struct cache_entry;\n struct checkout;\n-struct conv_attrs;\n+\n+/****************************************************************\n+ * Users of parallel checkout\n+ ****************************************************************/\n \n enum pc_status {\n \tPC_UNINITIALIZED = 0,\n@@ -29,4 +34,70 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n /* Write all the queued entries, returning 0 on success.*/\n int run_parallel_checkout(struct checkout *state);\n \n+/****************************************************************\n+ * Interface with checkout--worker\n+ ****************************************************************/\n+\n+enum pc_item_status {\n+\tPC_ITEM_PENDING = 0,\n+\tPC_ITEM_WRITTEN,\n+\t/*\n+\t * The entry could not be written because there was another file\n+\t * already present in its path or leading directories. Since\n+\t * checkout_entry_ca() removes such files from the working tree before\n+\t * enqueueing the entry for parallel checkout, it means that there was\n+\t * a path collision among the entries being written.\n+\t */\n+\tPC_ITEM_COLLIDED,\n+\tPC_ITEM_FAILED,\n+};\n+\n+struct parallel_checkout_item {\n+\t/*\n+\t * In main process ce points to a istate->cache[] entry. Thus, it's not\n+\t * owned by us. In workers they own the memory, which *must be* released.\n+\t */\n+\tstruct cache_entry *ce;\n+\tstruct conv_attrs ca;\n+\tsize_t id; /* position in parallel_checkout.items[] of main process */\n+\n+\t/* Output fields, sent from workers. */\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+/*\n+ * The fixed-size portion of `struct parallel_checkout_item` that is sent to the\n+ * workers. Following this will be 2 strings: ca.working_tree_encoding and\n+ * ce.name; These are NOT null terminated, since we have the size in the fixed\n+ * portion.\n+ *\n+ * Note that not all fields of conv_attrs and cache_entry are passed, only the\n+ * ones that will be required by the workers to smudge and write the entry.\n+ */\n+struct pc_item_fixed_portion {\n+\tsize_t id;\n+\tstruct object_id oid;\n+\tunsigned int ce_mode;\n+\tenum convert_crlf_action crlf_action;\n+\tint ident;\n+\tsize_t working_tree_encoding_len;\n+\tsize_t name_len;\n+};\n+\n+/*\n+ * The fields of `struct parallel_checkout_item` that are returned by the\n+ * workers. Note: `st` must be the last one, as it is omitted on error.\n+ */\n+struct pc_item_result {\n+\tsize_t id;\n+\tenum pc_item_status status;\n+\tstruct stat st;\n+};\n+\n+#define PC_ITEM_RESULT_BASE_SIZE offsetof(struct pc_item_result, st)\n+\n+void write_pc_item(struct parallel_checkout_item *pc_item,\n+\t\t struct checkout *state);\n+\n #endif /* PARALLEL_CHECKOUT_H */\n-- \n2.30.1\n\n"},{"id":"422332","messageId":"ad165c0637b0244df1851f10386fe87df2a70368.1618861380.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618861380.git.matheus.bernardino@usp.br","subject":"[PATCH v4 3/5] parallel-checkout: add configuration options","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T19:53:33Z","receivedAt":"2021-04-19T19:53:52Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Make parallel checkout configurable by introducing two new settings:\ncheckout.workers and checkout.thresholdForParallelism. The first defines\nthe number of workers (where one means sequential checkout), and the\nsecond defines the minimum number of entries to attempt parallel\ncheckout.\n\nTo decide the default value for checkout.workers, the parallel version\nwas benchmarked during three operations in the linux repo, with cold\ncache: cloning v5.8, checking out v5.8 from v2.6.15 (checkout I) and\nchecking out v5.8 from v5.7 (checkout II). The four tables below show\nthe mean run times and standard deviations for 5 runs in: a local file\nsystem on SSD, a local file system on HDD, a Linux NFS server, and\nAmazon EFS (all on Linux). Each parallel checkout test was executed with\nthe number of workers that brings the best overall results in that\nenvironment.\n\nLocal SSD:\n Sequential 10 workers Speedup\nClone 8.805 s ± 0.043 s 3.564 s ± 0.041 s 2.47 ± 0.03\nCheckout I 9.678 s ± 0.057 s 4.486 s ± 0.050 s 2.16 ± 0.03\nCheckout II 5.034 s ± 0.072 s 3.021 s ± 0.038 s 1.67 ± 0.03\n\nLocal HDD:\n Sequential 10 workers Speedup\nClone 32.288 s ± 0.580 s 30.724 s ± 0.522 s 1.05 ± 0.03\nCheckout I 54.172 s ± 7.119 s 54.429 s ± 6.738 s 1.00 ± 0.18\nCheckout II 40.465 s ± 2.402 s 38.682 s ± 1.365 s 1.05 ± 0.07\n\nLinux NFS server (v4.1, on EBS, single availability zone):\n\n Sequential 32 workers Speedup\nClone 240.368 s ± 6.347 s 57.349 s ± 0.870 s 4.19 ± 0.13\nCheckout I 242.862 s ± 2.215 s 58.700 s ± 0.904 s 4.14 ± 0.07\nCheckout II 65.751 s ± 1.577 s 23.820 s ± 0.407 s 2.76 ± 0.08\n\nEFS (v4.1, replicated over multiple availability zones):\n\n Sequential 32 workers Speedup\nClone 922.321 s ± 2.274 s 210.453 s ± 3.412 s 4.38 ± 0.07\nCheckout I 1011.300 s ± 7.346 s 297.828 s ± 0.964 s 3.40 ± 0.03\nCheckout II 294.104 s ± 1.836 s 126.017 s ± 1.190 s 2.33 ± 0.03\n\nThe above benchmarks show that parallel checkout is most effective on\nrepositories located on an SSD or over a distributed file system. For\nlocal file systems on spinning disks, and/or older machines, the\nparallelism does not always bring a good performance. For this reason,\nthe default value for checkout.workers is one, a.k.a. sequential\ncheckout.\n\nTo decide the default value for checkout.thresholdForParallelism,\nanother benchmark was executed in the \"Local SSD\" setup, where parallel\ncheckout showed to be beneficial. This time, we compared the runtime of\na `git checkout -f`, with and without parallelism, after randomly\nremoving an increasing number of files from the Linux working tree. The\n\"sequential fallback\" column below corresponds to the executions where\ncheckout.workers was 10 but checkout.thresholdForParallelism was equal\nto the number of to-be-updated files plus one (so that we end up writing\nsequentially). Each test case was sampled 15 times, and each sample had\na randomly different set of files removed. Here are the results:\n\n sequential fallback 10 workers speedup\n10 files 772.3 ms ± 12.6 ms 769.0 ms ± 13.6 ms 1.00 ± 0.02\n20 files 780.5 ms ± 15.8 ms 775.2 ms ± 9.2 ms 1.01 ± 0.02\n50 files 806.2 ms ± 13.8 ms 767.4 ms ± 8.5 ms 1.05 ± 0.02\n100 files 833.7 ms ± 21.4 ms 750.5 ms ± 16.8 ms 1.11 ± 0.04\n200 files 897.6 ms ± 30.9 ms 730.5 ms ± 14.7 ms 1.23 ± 0.05\n500 files 1035.4 ms ± 48.0 ms 677.1 ms ± 22.3 ms 1.53 ± 0.09\n1000 files 1244.6 ms ± 35.6 ms 654.0 ms ± 38.3 ms 1.90 ± 0.12\n2000 files 1488.8 ms ± 53.4 ms 658.8 ms ± 23.8 ms 2.26 ± 0.12\n\nFrom the above numbers, 100 files seems to be a reasonable default value\nfor the threshold setting.\n\nNote: Up to 1000 files, we observe a drop in the execution time of the\nparallel code with an increase in the number of files. This is a rather\nodd behavior, but it was observed in multiple repetitions. Above 1000\nfiles, the execution time increases according to the number of files, as\none would expect.\n\nAbout the test environments: Local SSD tests were executed on an\ni7-7700HQ (4 cores with hyper-threading) running Manjaro Linux. Local\nHDD tests were executed on an Intel(R) Xeon(R) E3-1230 (also 4 cores\nwith hyper-threading), HDD Seagate Barracuda 7200.14 SATA 3.1, running\nDebian. NFS and EFS tests were executed on an Amazon EC2 c5n.xlarge\ninstance, with 4 vCPUs. The Linux NFS server was running on a m6g.large\ninstance with 2 vCPUSs and a 1 TB EBS GP2 volume. Before each timing,\nthe linux repository was removed (or checked out back to its previous\nstate), and `sync && sysctl vm.drop_caches=3` was executed.\n\nCo-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Documentation/config/checkout.txt | 21 +++++++++++++++++++++\n parallel-checkout.c | 24 +++++++++++++++++++-----\n parallel-checkout.h | 9 +++++++--\n unpack-trees.c | 10 +++++++---\n 4 files changed, 54 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config/checkout.txt b/Documentation/config/checkout.txt\nindex 2cddf7b4b4..bfbca90f0e 100644\n--- a/Documentation/config/checkout.txt\n+++ b/Documentation/config/checkout.txt\n@@ -21,3 +21,24 @@ checkout.guess::\n \tProvides the default value for the `--guess` or `--no-guess`\n \toption in `git checkout` and `git switch`. See\n \tlinkgit:git-switch[1] and linkgit:git-checkout[1].\n+\n+checkout.workers::\n+\tThe number of parallel workers to use when updating the working tree.\n+\tThe default is one, i.e. sequential execution. If set to a value less\n+\tthan one, Git will use as many workers as the number of logical cores\n+\tavailable. This setting and `checkout.thresholdForParallelism` affect\n+\tall commands that perform checkout. E.g. checkout, clone, reset,\n+\tsparse-checkout, etc.\n++\n+Note: parallel checkout usually delivers better performance for repositories\n+located on SSDs or over NFS. For repositories on spinning disks and/or machines\n+with a small number of cores, the default sequential checkout often performs\n+better. The size and compression level of a repository might also influence how\n+well the parallel version performs.\n+\n+checkout.thresholdForParallelism::\n+\tWhen running parallel checkout with a small number of files, the cost\n+\tof subprocess spawning and inter-process communication might outweigh\n+\tthe parallelization gains. This setting allows to define the minimum\n+\tnumber of files for which parallel checkout should be attempted. The\n+\tdefault is 100.\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 836154fec6..3cc2028861 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -1,10 +1,12 @@\n #include \"cache.h\"\n+#include \"config.h\"\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"streaming.h\"\n+#include \"thread-utils.h\"\n \n struct pc_worker {\n \tstruct child_process cp;\n@@ -24,6 +26,20 @@ enum pc_status parallel_checkout_status(void)\n \treturn parallel_checkout.status;\n }\n \n+static const int DEFAULT_THRESHOLD_FOR_PARALLELISM = 100;\n+static const int DEFAULT_NUM_WORKERS = 1;\n+\n+void get_parallel_checkout_configs(int *num_workers, int *threshold)\n+{\n+\tif (git_config_get_int(\"checkout.workers\", num_workers))\n+\t\t*num_workers = DEFAULT_NUM_WORKERS;\n+\telse if (*num_workers < 1)\n+\t\t*num_workers = online_cpus();\n+\n+\tif (git_config_get_int(\"checkout.thresholdForParallelism\", threshold))\n+\t\t*threshold = DEFAULT_THRESHOLD_FOR_PARALLELISM;\n+}\n+\n void init_parallel_checkout(void)\n {\n \tif (parallel_checkout.status != PC_UNINITIALIZED)\n@@ -584,11 +600,9 @@ static void write_items_sequentially(struct checkout *state)\n \t\twrite_pc_item(¶llel_checkout.items[i], state);\n }\n \n-static const int DEFAULT_NUM_WORKERS = 2;\n-\n-int run_parallel_checkout(struct checkout *state)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n {\n-\tint ret, num_workers = DEFAULT_NUM_WORKERS;\n+\tint ret;\n \n \tif (parallel_checkout.status != PC_ACCEPTING_ENTRIES)\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n@@ -598,7 +612,7 @@ int run_parallel_checkout(struct checkout *state)\n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\n \n-\tif (num_workers <= 1) {\n+\tif (num_workers <= 1 || parallel_checkout.nr < threshold) {\n \t\twrite_items_sequentially(state);\n \t} else {\n \t\tstruct pc_worker *workers = setup_workers(state, num_workers);\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex ec58716519..2a68ab954d 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -17,6 +17,7 @@ enum pc_status {\n };\n \n enum pc_status parallel_checkout_status(void);\n+void get_parallel_checkout_configs(int *num_workers, int *threshold);\n \n /*\n * Put parallel checkout into the PC_ACCEPTING_ENTRIES state. Should be used\n@@ -31,8 +32,12 @@ void init_parallel_checkout(void);\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n \n-/* Write all the queued entries, returning 0 on success.*/\n-int run_parallel_checkout(struct checkout *state);\n+/*\n+ * Write all the queued entries, returning 0 on success. If the number of\n+ * entries is smaller than the specified threshold, the operation is performed\n+ * sequentially.\n+ */\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n \n /****************************************************************\n * Interface with checkout--worker\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f0430d458d..0669748f21 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -399,7 +399,7 @@ static int check_updates(struct unpack_trees_options *o,\n \tint errs = 0;\n \tstruct progress *progress;\n \tstruct checkout state = CHECKOUT_INIT;\n-\tint i;\n+\tint i, pc_workers, pc_threshold;\n \n \ttrace_performance_enter();\n \tstate.force = 1;\n@@ -465,8 +465,11 @@ static int check_updates(struct unpack_trees_options *o,\n \t\toid_array_clear(&to_fetch);\n \t}\n \n+\tget_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n \tenable_delayed_checkout(&state);\n-\tinit_parallel_checkout();\n+\tif (pc_workers > 1)\n+\t\tinit_parallel_checkout();\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n@@ -480,7 +483,8 @@ static int check_updates(struct unpack_trees_options *o,\n \t\t}\n \t}\n \tstop_progress(&progress);\n-\terrs |= run_parallel_checkout(&state);\n+\tif (pc_workers > 1)\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"422333","messageId":"cf9e28dc0eddfc8e36a64b9738c94bb790349092.1618861380.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618861380.git.matheus.bernardino@usp.br","subject":"[PATCH v4 4/5] parallel-checkout: support progress displaying","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T19:53:34Z","receivedAt":"2021-04-19T19:53:55Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Original-patch-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Matheus Tavares \n---\n parallel-checkout.c | 34 +++++++++++++++++++++++++++++++---\n parallel-checkout.h | 5 ++++-\n unpack-trees.c | 11 ++++++++---\n 3 files changed, 43 insertions(+), 7 deletions(-)\n\ndiff --git a/parallel-checkout.c b/parallel-checkout.c\nindex 3cc2028861..09e8b10a35 100644\n--- a/parallel-checkout.c\n+++ b/parallel-checkout.c\n@@ -3,6 +3,7 @@\n #include \"entry.h\"\n #include \"parallel-checkout.h\"\n #include \"pkt-line.h\"\n+#include \"progress.h\"\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"streaming.h\"\n@@ -17,6 +18,8 @@ struct parallel_checkout {\n \tenum pc_status status;\n \tstruct parallel_checkout_item *items; /* The parallel checkout queue. */\n \tsize_t nr, alloc;\n+\tstruct progress *progress;\n+\tunsigned int *progress_cnt;\n };\n \n static struct parallel_checkout parallel_checkout;\n@@ -146,6 +149,20 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)\n \treturn 0;\n }\n \n+size_t pc_queue_size(void)\n+{\n+\treturn parallel_checkout.nr;\n+}\n+\n+static void advance_progress_meter(void)\n+{\n+\tif (parallel_checkout.progress) {\n+\t\t(*parallel_checkout.progress_cnt)++;\n+\t\tdisplay_progress(parallel_checkout.progress,\n+\t\t\t\t *parallel_checkout.progress_cnt);\n+\t}\n+}\n+\n static int handle_results(struct checkout *state)\n {\n \tint ret = 0;\n@@ -194,6 +211,7 @@ static int handle_results(struct checkout *state)\n \t\t\t */\n \t\t\tret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,\n \t\t\t\t\t\t state, NULL, NULL);\n+\t\t\tadvance_progress_meter();\n \t\t\tbreak;\n \t\tcase PC_ITEM_PENDING:\n \t\t\thave_pending = 1;\n@@ -534,6 +552,9 @@ static void parse_and_save_result(const char *buffer, int len,\n \tpc_item->status = res->status;\n \tif (st)\n \t\tpc_item->st = *st;\n+\n+\tif (res->status != PC_ITEM_COLLIDED)\n+\t\tadvance_progress_meter();\n }\n \n static void gather_results_from_workers(struct pc_worker *workers,\n@@ -596,11 +617,16 @@ static void write_items_sequentially(struct checkout *state)\n {\n \tsize_t i;\n \n-\tfor (i = 0; i < parallel_checkout.nr; i++)\n-\t\twrite_pc_item(¶llel_checkout.items[i], state);\n+\tfor (i = 0; i < parallel_checkout.nr; i++) {\n+\t\tstruct parallel_checkout_item *pc_item = ¶llel_checkout.items[i];\n+\t\twrite_pc_item(pc_item, state);\n+\t\tif (pc_item->status != PC_ITEM_COLLIDED)\n+\t\t\tadvance_progress_meter();\n+\t}\n }\n \n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold)\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt)\n {\n \tint ret;\n \n@@ -608,6 +634,8 @@ int run_parallel_checkout(struct checkout *state, int num_workers, int threshold\n \t\tBUG(\"cannot run parallel checkout: uninitialized or already running\");\n \n \tparallel_checkout.status = PC_RUNNING;\n+\tparallel_checkout.progress = progress;\n+\tparallel_checkout.progress_cnt = progress_cnt;\n \n \tif (parallel_checkout.nr < num_workers)\n \t\tnum_workers = parallel_checkout.nr;\ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 2a68ab954d..80f539bcb7 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -5,6 +5,7 @@\n \n struct cache_entry;\n struct checkout;\n+struct progress;\n \n /****************************************************************\n * Users of parallel checkout\n@@ -31,13 +32,15 @@ void init_parallel_checkout(void);\n * for later write and return 0.\n */\n int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);\n+size_t pc_queue_size(void);\n \n /*\n * Write all the queued entries, returning 0 on success. If the number of\n * entries is smaller than the specified threshold, the operation is performed\n * sequentially.\n */\n-int run_parallel_checkout(struct checkout *state, int num_workers, int threshold);\n+int run_parallel_checkout(struct checkout *state, int num_workers, int threshold,\n+\t\t\t struct progress *progress, unsigned int *progress_cnt);\n \n /****************************************************************\n * Interface with checkout--worker\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 0669748f21..4b77e52c6b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -474,17 +474,22 @@ static int check_updates(struct unpack_trees_options *o,\n \t\tstruct cache_entry *ce = index->cache[i];\n \n \t\tif (ce->ce_flags & CE_UPDATE) {\n+\t\t\tsize_t last_pc_queue_size = pc_queue_size();\n+\n \t\t\tif (ce->ce_flags & CE_WT_REMOVE)\n \t\t\t\tBUG(\"both update and delete flags are set on %s\",\n \t\t\t\t ce->name);\n-\t\t\tdisplay_progress(progress, ++cnt);\n \t\t\tce->ce_flags &= ~CE_UPDATE;\n \t\t\terrs |= checkout_entry(ce, &state, NULL, NULL);\n+\n+\t\t\tif (last_pc_queue_size == pc_queue_size())\n+\t\t\t\tdisplay_progress(progress, ++cnt);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tif (pc_workers > 1)\n-\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold);\n+\t\terrs |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n+\t\t\t\t\t progress, &cnt);\n+\tstop_progress(&progress);\n \terrs |= finish_delayed_checkout(&state, NULL);\n \tgit_attr_set_direction(GIT_ATTR_CHECKIN);\n \n-- \n2.30.1\n\n"},{"id":"422334","messageId":"fd929f072c040f738212ec769df7cb632356f8d1.1618861380.git.matheus.bernardino@usp.br","threadId":"55342","inReplyTo":"cover.1618861380.git.matheus.bernardino@usp.br","subject":"[PATCH v4 5/5] parallel-checkout: add design documentation","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-19T19:53:35Z","receivedAt":"2021-04-19T19:53:56Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Co-authored-by: Jeff Hostetler \nSigned-off-by: Matheus Tavares \n---\n Documentation/Makefile | 1 +\n Documentation/technical/parallel-checkout.txt | 270 ++++++++++++++++++\n 2 files changed, 271 insertions(+)\n create mode 100644 Documentation/technical/parallel-checkout.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 81d1bf7a04..af236927c9 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -90,6 +90,7 @@ TECH_DOCS += technical/multi-pack-index\n TECH_DOCS += technical/pack-format\n TECH_DOCS += technical/pack-heuristics\n TECH_DOCS += technical/pack-protocol\n+TECH_DOCS += technical/parallel-checkout\n TECH_DOCS += technical/partial-clone\n TECH_DOCS += technical/protocol-capabilities\n TECH_DOCS += technical/protocol-common\ndiff --git a/Documentation/technical/parallel-checkout.txt b/Documentation/technical/parallel-checkout.txt\nnew file mode 100644\nindex 0000000000..e790258a1a\n--- /dev/null\n+++ b/Documentation/technical/parallel-checkout.txt\n@@ -0,0 +1,270 @@\n+Parallel Checkout Design Notes\n+==============================\n+\n+The \"Parallel Checkout\" feature attempts to use multiple processes to\n+parallelize the work of uncompressing the blobs, applying in-core\n+filters, and writing the resulting contents to the working tree during a\n+checkout operation. It can be used by all checkout-related commands,\n+such as `clone`, `checkout`, `reset`, `sparse-checkout`, and others.\n+\n+These commands share the following basic structure:\n+\n+* Step 1: Read the current index file into memory.\n+\n+* Step 2: Modify the in-memory index based upon the command, and\n+ temporarily mark all cache entries that need to be updated.\n+\n+* Step 3: Populate the working tree to match the new candidate index.\n+ This includes iterating over all of the to-be-updated cache entries\n+ and delete, create, or overwrite the associated files in the working\n+ tree.\n+\n+* Step 4: Write the new index to disk.\n+\n+Step 3 is the focus of the \"parallel checkout\" effort described here.\n+\n+Sequential Implementation\n+-------------------------\n+\n+For the purposes of discussion here, the current sequential\n+implementation of Step 3 is divided in 3 parts, each one implemented in\n+its own function:\n+\n+* Step 3a: `unpack-trees.c:check_updates()` contains a series of\n+ sequential loops iterating over the `cache_entry`'s array. The main\n+ loop in this function calls the Step 3b function for each of the\n+ to-be-updated entries.\n+\n+* Step 3b: `entry.c:checkout_entry()` examines the existing working tree\n+ for file conflicts, collisions, and unsaved changes. It removes files\n+ and creates leading directories as necessary. It calls the Step 3c\n+ function for each entry to be written.\n+\n+* Step 3c: `entry.c:write_entry()` loads the blob into memory, smudges\n+ it if necessary, creates the file in the working tree, writes the\n+ smudged contents, calls `fstat()` or `lstat()`, and updates the\n+ associated `cache_entry` struct with the stat information gathered.\n+\n+It wouldn't be safe to perform Step 3b in parallel, as there could be\n+race conditions between file creations and removals. Instead, the\n+parallel checkout framework lets the sequential code handle Step 3b,\n+and uses parallel workers to replace the sequential\n+`entry.c:write_entry()` calls from Step 3c.\n+\n+Rejected Multi-Threaded Solution\n+--------------------------------\n+\n+The most \"straightforward\" implementation would be to spread the set of\n+to-be-updated cache entries across multiple threads. But due to the\n+thread-unsafe functions in the ODB code, we would have to use locks to\n+coordinate the parallel operation. An early prototype of this solution\n+showed that the multi-threaded checkout would bring performance\n+improvements over the sequential code, but there was still too much lock\n+contention. A `perf` profiling indicated that around 20% of the runtime\n+during a local Linux clone (on an SSD) was spent in locking functions.\n+For this reason this approach was rejected in favor of using multiple\n+child processes, which led to a better performance.\n+\n+Multi-Process Solution\n+----------------------\n+\n+Parallel checkout alters the aforementioned Step 3 to use multiple\n+`checkout--worker` background processes to distribute the work. The\n+long-running worker processes are controlled by the foreground Git\n+command using the existing run-command API.\n+\n+Overview\n+~~~~~~~~\n+\n+Step 3b is only slightly altered; for each entry to be checked out, the\n+main process performs the following steps:\n+\n+* M1: Check whether there is any untracked or unclean file in the\n+ working tree which would be overwritten by this entry, and decide\n+ whether to proceed (removing the file(s)) or not.\n+\n+* M2: Create the leading directories.\n+\n+* M3: Load the conversion attributes for the entry's path.\n+\n+* M4: Check, based on the entry's type and conversion attributes,\n+ whether the entry is eligible for parallel checkout (more on this\n+ later). If it is eligible, enqueue the entry and the loaded\n+ attributes to later write the entry in parallel. If not, write the\n+ entry right away, using the default sequential code.\n+\n+Note: we save the conversion attributes associated with each entry\n+because the workers don't have access to the main process' index state,\n+so they can't load the attributes by themselves (and the attributes are\n+needed to properly smudge the entry). Additionally, this has a positive\n+impact on performance as (1) we don't need to load the attributes twice\n+and (2) the attributes machinery is optimized to handle paths in\n+sequential order.\n+\n+After all entries have passed through the above steps, the main process\n+checks if the number of enqueued entries is sufficient to spread among\n+the workers. If not, it just writes them sequentially. Otherwise, it\n+spawns the workers and distributes the queued entries uniformly in\n+continuous chunks. This aims to minimize the chances of two workers\n+writing to the same directory simultaneously, which could increase lock\n+contention in the kernel.\n+\n+Then, for each assigned item, each worker:\n+\n+* W1: Checks if there is any non-directory file in the leading part of\n+ the entry's path or if there already exists a file at the entry' path.\n+ If so, mark the entry with `PC_ITEM_COLLIDED` and skip it (more on\n+ this later).\n+\n+* W2: Creates the file (with O_CREAT and O_EXCL).\n+\n+* W3: Loads the blob into memory (inflating and delta reconstructing\n+ it).\n+\n+* W4: Applies any required in-process filter, like end-of-line\n+ conversion and re-encoding.\n+\n+* W5: Writes the result to the file descriptor opened at W2.\n+\n+* W6: Calls `fstat()` or lstat()` on the just-written path, and sends\n+ the result back to the main process, together with the end status of\n+ the operation and the item's identification number.\n+\n+Note that, when possible, steps W3 to W5 are delegated to the streaming\n+machinery, removing the need to keep the entire blob in memory.\n+\n+If the worker fails to read the blob or to write it to the working tree,\n+it removes the created file to avoid leaving empty files behind. This is\n+the *only* time a worker is allowed to remove a file.\n+\n+As mentioned earlier, it is the responsibility of the main process to\n+remove any file that blocks the checkout operation (or abort if the\n+removal(s) would cause data loss and the user didn't ask to `--force`).\n+This is crucial to avoid race conditions and also to properly detect\n+path collisions at Step W1.\n+\n+After the workers finish writing the items and sending back the required\n+information, the main process handles the results in two steps:\n+\n+- First, it updates the in-memory index with the `lstat()` information\n+ sent by the workers. (This must be done first as this information\n+ might me required in the following step.)\n+\n+- Then it writes the items which collided on disk (i.e. items marked\n+ with `PC_ITEM_COLLIDED`). More on this below.\n+\n+Path Collisions\n+---------------\n+\n+Path collisions happen when two different paths correspond to the same\n+entry in the file system. E.g. the paths 'a' and 'A' would collide in a\n+case-insensitive file system.\n+\n+The sequential checkout deals with collisions in the same way that it\n+deals with files that were already present in the working tree before\n+checkout. Basically, it checks if the path that it wants to write\n+already exists on disk, makes sure the existing file doesn't have\n+unsaved data, and then overwrites it. (To be more pedantic: it deletes\n+the existing file and creates the new one.) So, if there are multiple\n+colliding files to be checked out, the sequential code will write each\n+one of them but only the last will actually survive on disk.\n+\n+Parallel checkout aims to reproduce the same behavior. However, we\n+cannot let the workers racily write to the same file on disk. Instead,\n+the workers detect when the entry that they want to check out would\n+collide with an existing file, and mark it with `PC_ITEM_COLLIDED`.\n+Later, the main process can sequentially feed these entries back to\n+`checkout_entry()` without the risk of race conditions. On clone, this\n+also has the effect of marking the colliding entries to later emit a\n+warning for the user, like the classic sequential checkout does.\n+\n+The workers are able to detect both collisions among the entries being\n+concurrently written and collisions between a parallel-eligible entry\n+and an ineligible entry. The general idea for collision detection is\n+quite straightforward: for each parallel-eligible entry, the main\n+process must remove all files that prevent this entry from being written\n+(before enqueueing it). This includes any non-directory file in the\n+leading path of the entry. Later, when a worker gets assigned the entry,\n+it looks again for the non-directories files and for an already existing\n+file at the entry's path. If any of these checks finds something, the\n+worker knows that there was a path collision.\n+\n+Because parallel checkout can distinguish path collisions from the case\n+where the file was already present in the working tree before checkout,\n+we could alternatively choose to skip the checkout of colliding entries.\n+However, each entry that doesn't get written would have NULL `lstat()`\n+fields on the index. This could cause performance penalties for\n+subsequent commands that need to refresh the index, as they would have\n+to go to the file system to see if the entry is dirty. Thus, if we have\n+N entries in a colliding group and we decide to write and `lstat()` only\n+one of them, every subsequent `git-status` will have to read, convert,\n+and hash the written file N - 1 times. By checking out all colliding\n+entries (like the sequential code does), we only pay the overhead once,\n+during checkout.\n+\n+Eligible Entries for Parallel Checkout\n+--------------------------------------\n+\n+As previously mentioned, not all entries passed to `checkout_entry()`\n+will be considered eligible for parallel checkout. More specifically, we\n+exclude:\n+\n+- Symbolic links; to avoid race conditions that, in combination with\n+ path collisions, could cause workers to write files at the wrong\n+ place. For example, if we were to concurrently check out a symlink\n+ 'a' -> 'b' and a regular file 'A/f' in a case-insensitive file system,\n+ we could potentially end up writing the file 'A/f' at 'a/f', due to a\n+ race condition.\n+\n+- Regular files that require external filters (either \"one shot\" filters\n+ or long-running process filters). These filters are black-boxes to Git\n+ and may have their own internal locking or non-concurrent assumptions.\n+ So it might not be safe to run multiple instances in parallel.\n++\n+Besides, long-running filters may use the delayed checkout feature to\n+postpone the return of some filtered blobs. The delayed checkout queue\n+and the parallel checkout queue are not compatible and should remain\n+separate.\n++\n+Note: regular files that only require internal filters, like end-of-line\n+conversion and re-encoding, are eligible for parallel checkout.\n+\n+Ineligible entries are checked out by the classic sequential codepath\n+*before* spawning workers.\n+\n+Note: submodules's files are also eligible for parallel checkout (as\n+long as they don't fall into any of the excluding categories mentioned\n+above). But since each submodule is checked out in its own child\n+process, we don't mix the superproject's and the submodules' files in\n+the same parallel checkout process or queue.\n+\n+The API\n+-------\n+\n+The parallel checkout API was designed with the goal of minimizing\n+changes to the current users of the checkout machinery. This means that\n+they don't have to call a different function for sequential or parallel\n+checkout. As already mentioned, `checkout_entry()` will automatically\n+insert the given entry in the parallel checkout queue when this feature\n+is enabled and the entry is eligible; otherwise, it will just write the\n+entry right away, using the sequential code. In general, callers of the\n+parallel checkout API should look similar to this:\n+\n+----------------------------------------------\n+int pc_workers, pc_threshold, err = 0;\n+struct checkout state;\n+\n+get_parallel_checkout_configs(&pc_workers, &pc_threshold);\n+\n+/*\n+ * This check is not strictly required, but it\n+ * should save some time in sequential mode.\n+ */\n+if (pc_workers > 1)\n+\tinit_parallel_checkout();\n+\n+for (each cache_entry ce to-be-updated)\n+\terr |= checkout_entry(ce, &state, NULL, NULL);\n+\n+err |= run_parallel_checkout(&state, pc_workers, pc_threshold, NULL, NULL);\n+----------------------------------------------\n-- \n2.30.1\n\n"}]}