Re: "submodule foreach" much slower than removed "submodule--helper --list"

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Oct 17, 2022, 16:50 UTC
Message-ID: <221017.86h702jsiq.gmgdl@evledraar.gmail.com>
In-Reply-To: <87a65xrnwz.fsf@bernoul.li>

Replying to both this & the parent post:

On Sat, Oct 15 2022, Jonas Bernoulli wrote:

Show 22 quoted lines

> In v2.38.0 (31955475d1c283120d5d84247eb3fd55d9f5fdd9)
> "submodule--helper --list" was remove because
>
>> We're not getting anything useful from the "list | cut -f2"
>> invocation that we couldn't get from "foreach 'echo $sm_path'".
>
> But we get speed (this is with about one hundred modules):
>
> $ time git submodule foreach -q 'echo $sm_path' > /dev/null
>
> real    0m0.585s
> user    0m0.413s
> sys     0m0.182s
>
> $ time git submodule--helper list > /dev/null
>
> real    0m0.008s
> user    0m0.004s
> sys     0m0.004s
>
> Please consider restoring this subcommand or providing something
> equivalent that is just as fast.

Sorry about the slowdown, the removal of "list" was just an in-between step to migrating "submodule" to a full built-in.

I can't reproduce anything like the 8ms v.s. ~600ms difference you note here, but for me migrating it to a built-in is around 10% slower with "foreach" than the old "list". I wonder what results you get?

I sent in a topic to migrate it since you sent this report. I was going to do it in this development cycle, but this prompted me to do it earlier:

	https://lore.kernel.org/git/cover-00.10-00000000000-20221017T115544Z-avarab@gmail.com/

On Sat, Oct 15 2022, Jonas Bernoulli wrote:

Show 15 quoted lines

> I just noticed that "submodule--helper name" was also removed, which I
> also found useful in scripts.  Please tell me if I am missing something,
> but it seems I now have to do something like this instead:
>
>   git config -f .gitmodules --list |
>       sed -n "s|^submodule.\([^.]*\).path=$path\$|\1|p"
>
> The old way was nicer:
>
>   git submodule--helper name $path
>
> I realize submodule--helper is for internal use and using it anyway
> comes with the risk of such removals and other changes, but again,
> please consider restoring that or providing something similar in the
> public interface.

This however is another case, I removed "name" along with "list" and other leftover code we weren't using anymore for the internal-only "submodule--helper" (which is at turns out, was not as internal-only as we'd hoped).

For "list" it's clear how to use "foreach" instead, but for "name" then AFAICT the "best" replacement is to do a:

	git submodule foreach 'echo $displaypath $name'

And pipe that into grep/sed. If that's fast enough would it satisfy your use-case, or would a "name" equivalent be handy?

I think the best way to prove that would be e.g.:

	git submodule foreach-format '%{name}' -- <pathspec>

Which, due to the "foreach" taking N number of arguments isn't easy to add to "foreach" without the interface becoming somewhat tortured (we could add a [---pathspec=<pathspec>]...), but "-- <pathspec>" with a different subcommand name seems better.