git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 6/6] diff-lib: parallelize run_diff_files for submodules

From
Calvin Wan <calvinwan@google.com>
Date
Nov 3, 2022, 21:16 UTC
Message-ID
<CAFySSZDgo7Ny+u4cvi7D-+Xx4jSZqKDsH_OKgtpQvKTJ7Fbc5w@mail.gmail.com>
In-Reply-To
<221021.86pmemeyiu.gmgdl@evledraar.gmail.com>
Show 7 quoted lines
> And, I may be missing something, but later in that function we do:
>
>         if (repo_read_index(r) < 0)
>                 die(_("index file corrupt"));
>
> Do we need to read the index there if we're just invoking a "status"
> command, isn't it reading it for us & reporting back?
You're correct I don't need to read the index there
Show 30 quoted lines
> > +static struct status_task *
> > +get_status_task_from_index(struct submodule_parallel_status *sps,
> > +                        struct strbuf *err)
> > +{
> > +     for (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {
> > +             struct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;
> > +             const struct cache_entry *ce = util->ce;
> > +             struct status_task *task;
> > +             struct strbuf buf = STRBUF_INIT;
> > +             const char *git_dir;
> > +
> > +             strbuf_addf(&buf, "%s/.git", ce->name);
> > +             git_dir = read_gitfile(buf.buf);
>
> Okey, so the "NULL" variant of read_gitfile_gently(), as we don't care
> about the sort of error we got, but...
>
> > +             if (!git_dir)
> > +                     git_dir = buf.buf;
> > +             if (!is_git_directory(git_dir)) {
>
> Isn't this something we could have asked read_gitfile_gently() about
> instead, i.e. the READ_GITFILE_ERR_NOT_A_REPO condition?
> > +                     if (is_directory(git_dir))
> > +                             die(_("'%s' not recognized as a git repository"), git_dir);
>
> It would be a detour from this, but perhaps setup.c can be tasked with
> knowing about this edge case and returning an error code, but even if we
> punt on that we can just do the is_directory() here, but get the
> !is_git_directory() for free it seems.

So there are two non-fatal errors in read_gitfile_gently() that return NULL rather than dieing: READ_GITFILE_ERR_STAT_FAILED READ_GITFILE_ERR_NOT_A_FILE

Then the edge case becomes: Not a git file (or stat failed) Not a git directory Is a directory

I'm not seeing how we would get !is_git_directory() for free.
Show 6 quoted lines
> > +             task->path = ce->name;
> > +             task->dirty_submodule = util->dirty_submodule;
> > +             task->ignore_untracked = util->ignore_untracked;
>
> Cn do we the same readability trick here that we did with "struct
> submodule_status_util tmp = {" earlier & the memcpy()? Looks like it...

Yes, if I make the readability change, then I should still be able to use xmalloc

Show 11 quoted lines
> > +
> > +     child_process_init(cp);
> > +     prepare_submodule_repo_env_in_gitdir(&cp->env);
> > +
> > +     strvec_init(&cp->args);
> > +     strvec_pushl(&cp->args, "status", "--porcelain=2", NULL);
> > +     if (task->ignore_untracked)
> > +             strvec_push(&cp->args, "-uno");
>
> Nit: Maybe worth spelling out --untracked-files=no (or maybe "-uno" is
> more idiomatic at this point...)

Same spelling as in is_submodule_modified(). Probably not worth changing /shrug.

Show 7 quoted lines
> > +int get_submodules_status(struct repository *r,
> > +                       struct string_list *submodules,
> > +                       int max_parallel_jobs);
>
> s/int/size_t/ because at this point you've already die()'d in the one
> caller if it's <0 from the config parser, so we know it's unsigned
> (actually >1, but unsigned will have to do :).

The caller is passing in an int anyways so I'm going to keep it as is for consistency.

Took the rest of your suggestions. Thanks!
Previous: Ævar Arnfjörð Bjarmason
Message 19 of 19 in “submodule: parallelize diff”
  1. 0/6 submodule: parallelize diffCalvin Wan, Oct 20, 2022
  2. 1/6 run-command: add pipe_output_fn to run_processes_parallel_optsCalvin Wan, Oct 20, 2022
  3. Ævar Arnfjörð BjarmasonOct 21, 2022
  4. Calvin WanOct 24, 2022
  5. Junio C HamanoOct 21, 2022
  6. Calvin WanOct 24, 2022
  7. Junio C HamanoOct 24, 2022
  8. Calvin WanOct 25, 2022
  9. 2/6 run-command: add hide_output to run_processes_parallel_optsCalvin Wan, Oct 20, 2022
  10. Ævar Arnfjörð BjarmasonOct 21, 2022
  11. Calvin WanOct 24, 2022
  12. Ævar Arnfjörð BjarmasonOct 25, 2022
  13. Calvin WanOct 25, 2022
  14. 3/6 submodule: strbuf variable renameCalvin Wan, Oct 20, 2022
  15. 4/6 submodule: move status parsing into functionCalvin Wan, Oct 20, 2022
  16. 5/6 diff-lib: refactor match_stat_with_submoduleCalvin Wan, Oct 20, 2022
  17. 6/6 diff-lib: parallelize run_diff_files for submodulesCalvin Wan, Oct 20, 2022
  18. Ævar Arnfjörð BjarmasonOct 21, 2022
  19. Calvin WanNov 3, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.