threads / discuss / 40314

--progress option for git submodule update?

Subject: --progress option for git submodule update?

## tl;dr

5 messages between Sep 9, 2015 and Sep 11, 2015.

replies: 4people: 2as markdown or json

Vitali Lovich· Sep 9, 2015, 22:52 UTC · lore

Hi,

Git submodule doesn’t have a --progress option like regular clone/fetch does. This means that it can hang a long time without output as it’s transferring data, particularly for large repositories. This is problematic in automation scenarios where there can be upper-bounds on how long a process may run without any output (to protect against processes hanging for long periods of time without forward progress).

I’m sure this has been asked for before but having this option would be really nice for automation system (like buildbot) to take advantage of. The only alternative is a hacky solution to clone locally first with the —progress option & then somehow set up the submodule to use the local clone as a reference.

Thanks, Vitali

Stefan Beller· Sep 9, 2015, 23:40 UTC · re: Vitali Lovich · lore

Re: --progress option for git submodule update?

So submodules...

I am currently working on improving submodules (some basic performance improvements have been done, soon to be merged upstream, I currently try to get parallelism working for git fetch --recurse-submodules and for git submodule update eventually. I sent some early working patches for that, but I am doing a whole new redesign without threads now).

On Wed, Sep 9, 2015 at 3:52 PM, Vitali Lovich <vlovich@gmail.com> wrote:

> Hi,
>
> Git submodule doesn’t have a --progress option like regular clone/fetch does.  This means that it can hang a long time without output as it’s transferring data, particularly for large repositories.

For repositories with nested submodules it is impossible to estimate the progress because you don't know how many there are. Say you have a layout like:

A
 -> B
-> C
-> D
    -> E
    -> F

whereas each letter is a repository and B,C,D are submodules of A and E,F are submodules of D. So if D is not cloned yet, it looks like A has only 3 submodules, but in fact we need to update 5 submodules.

> This is problematic in automation scenarios where there can be upper-bounds on how long a process may run without any output (to protect against processes hanging for long periods of time without forward progress).

Maybe a better error-out-if-hanging would be better IMHO ? Another option would be to enumerate the submodules and give the currently estimated upper bound ?

Doh! I see what you're missing now after rereading the email closely. You can add a --quiet option, but --verbose or --progress just errors out, but you want that as a possible argument for git clone inside the git submodule update code.

Thanks, Stefan

Show 9 quoted lines

>
> I’m sure this has been asked for before but having this option would be really nice for automation system (like buildbot) to take advantage of.  The only alternative is a hacky solution to clone locally first with the —progress option
> & then somehow set up the submodule to use the local clone as a reference.
>
> Thanks,
> Vitali--
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Vitali Lovich· Sep 10, 2015, 03:06 UTC · re: Stefan Beller · lore

Re: --progress option for git submodule update?

Show 9 quoted lines

> On Sep 9, 2015, at 4:40 PM, Stefan Beller <sbeller@google.com> wrote:
> 
> So submodules...
> 
> I am currently working on improving submodules (some basic performance
> improvements have been done, soon to be merged upstream, I currently
> try to get parallelism working for git fetch --recurse-submodules and for
> git submodule update eventually. I sent some early working patches for that,
> but I am doing a whole new redesign without threads now).

That sounds exciting.  Can’t wait.

Show 21 quoted lines

> On Wed, Sep 9, 2015 at 3:52 PM, Vitali Lovich <vlovich@gmail.com> wrote:
>> Hi,
>> 
>> Git submodule doesn’t have a --progress option like regular clone/fetch does.  This means that it can hang a long time without output as it’s transferring data, particularly for large repositories.
> 
> For repositories with nested submodules it is impossible to estimate
> the progress because you don't know how many there are.
> Say you have a layout like:
> 
> A
> -> B
> -> C
> -> D
>    -> E
>    -> F
> 
> whereas each letter is a repository and B,C,D are submodules of A and
> E,F are submodules of D.
> So if D is not cloned yet, it looks like A has only 3 submodules, but
> in fact we need to update 5
> submodules.

I don’t think I’m asking for an overall --progress. As you point out that is very difficult/an intractable problem. I was thinking it would just report the progress for each submodule it encounters as it fetches/clones. The added benefit to that is that if there’s a lot of submodules, an overall progress might get stuck at a long time at a given percentage whereas it’s less likely cloning just a single module would depending on the size of repositories.

Show 5 quoted lines

>> This is problematic in automation scenarios where there can be upper-bounds on how long a process may run without any output (to protect against processes hanging for long periods of time without forward progress).
> 
> Maybe a better error-out-if-hanging would be better IMHO ?
> Another option would be to enumerate the submodules and give the
> currently estimated upper bound ?

That’s much more difficult & I’m still left with the original problem where I have to set a very conservative upper-bound which wastes valuable machine time & causes extra contention for automation resources.

Show 5 quoted lines

> Doh! I see what you're missing now after rereading the email closely.
> You can add a --quiet option,
> but --verbose or --progress just errors out, but you want that as a
> possible argument for git clone
> inside the git submodule update code.

Yes exactly.

Show 13 quoted lines

> 
> Thanks,
> Stefan
> 
>> 
>> I’m sure this has been asked for before but having this option would be really nice for automation system (like buildbot) to take advantage of.  The only alternative is a hacky solution to clone locally first with the —progress option
>> & then somehow set up the submodule to use the local clone as a reference.
>> 
>> Thanks,
>> Vitali--
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Beller· Sep 11, 2015, 23:05 UTC · re: Vitali Lovich · lore

Re: --progress option for git submodule update?

On Wed, Sep 9, 2015 at 8:06 PM, Vitali Lovich <vlovich@gmail.com> wrote:

Show 6 quoted lines

>> Doh! I see what you're missing now after rereading the email closely.
>> You can add a --quiet option,
>> but --verbose or --progress just errors out, but you want that as a
>> possible argument for git clone
>> inside the git submodule update code.
> Yes exactly.

Instead of cloning with submodules, you could also clone only the superproject and then do a git fetch --recuse-submodules=yes -v instead soonish.

Vitali Lovich· Sep 11, 2015, 23:34 UTC · re: Stefan Beller · lore

Re: --progress option for git submodule update?

& then do a git submodule init?

Show 13 quoted lines

> On Sep 11, 2015, at 4:05 PM, Stefan Beller <sbeller@google.com> wrote:
> 
> On Wed, Sep 9, 2015 at 8:06 PM, Vitali Lovich <vlovich@gmail.com> wrote:
>>> Doh! I see what you're missing now after rereading the email closely.
>>> You can add a --quiet option,
>>> but --verbose or --progress just errors out, but you want that as a
>>> possible argument for git clone
>>> inside the git submodule update code.
>> Yes exactly.
> 
> Instead of cloning with submodules, you could also clone only the
> superproject and then do a git fetch --recuse-submodules=yes -v
> instead soonish.