Re: failed submodule update re-run results in no checked out files?

From: Jacob Keller <jacob.keller@gmail.com>
Date: Feb 18, 2016, 00:26 UTC
Message-ID: <CA+P7+xqj2i7EFpOhsecRPniHL68sEcmnaCB+m6VuO4U3dfnwog@mail.gmail.com>
In-Reply-To: <CAGZ79kaM5JREvfCmNMn8FQm1mutmDB3e8DYmtxMEzmUO5mqccw@mail.gmail.com>

On Wed, Feb 17, 2016 at 4:15 PM, Stefan Beller <sbeller@google.com> wrote:

Show 18 quoted lines

> On Wed, Feb 17, 2016 at 3:54 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
>> Hi,
>>
>> I am having an issue currently when using Git with a remote server
>> which has a limited number of ssh connections.
>>
>> The ssh server sometimes closes connections due to too many concurrent
>> connections. I will get the following output from git in this case
>> when performing a submodule update of a submodule which is not yet
>> currently cloned/checked out.
>>
>> stdout: Cloning into 'src/SHARED'...
>
> Which version of Git are you using?
> (does it include origin/sb/submodule-parallel-update, which rewrites
> lots of the relevant code? Also it introduces parallelism, which may not be
> anticipated by the server admin)
>

I am not 100% sure, but multiple versions. I have multiple virtual machines that run automated tests, and I have not kept all the VMs up to date.

Show 6 quoted lines

>>
>> stderr: Total 10288 (delta 7577), reused 10190 (delta 7577)
>
> Seeing both stdout and stderr, I assume you're on master or even behind that,
> which doesn't include the rewrite.
>

I think I am using 2.4.x right now.

Show 13 quoted lines

>> Received disconnect from 10.96.8.71: 7: Too many concurrent connections
>> fatal: Could not read from remote repository.
>>
>> Please make sure you have the correct access rights
>> and the repository exists.
>> Unable to fetch in submodule path 'src/SHARED'
>>
>> The submodule is not cloned successfully, and this occurs somewhere in
>> the middle of the process.
>
> I wonder if the client should retry each submodule at least once, in case of
> transient errors such as this ssh config having too many connections.
>

Well, in my case it wouldn't be "transient" and my process already does an automatic re-run when it fails. (Technically I am using the Java wrapper around CLI Git for Jenkins, which does this re-wrapping). It is possible something is wrong in the java wrapper, but I determined that it boils down to this sequence. I have yet to be able to reproduce a failure locally though without the Java wrapper (due to being difficult to generate enough simultaneous connections to cause the failure).

Show 11 quoted lines

>>
>> If I run the command a 2nd time,
>>
>> git submodule update --remote src/SHARED,
>>
>> I get a successful run, but the files are not actually checked out.
>
> The submodules are cloned via clone --no-checkout.
> This is because you may have a custom update strategy configured,
> (or a preset such as "none" which would clone but do nothing further)
>

I see.

Show 9 quoted lines

>> I
>> believe this is because the clone that failed did succeed in getting
>> the repository into a state where all the files are "removed" so a
>> further submodule update will do nothing since it's "already" checked
>> out at the correct commit.
>
> you can pass --force into "git submodule update" which passes that
> flag along to the checkout.
>

Right, this is what worked as a workaround for me.

Show 22 quoted lines

>>
>> Am I right in my understanding? Is this a bug? I believe I can fix
>> this using --force.
>>
>> Note that i don't yet currently have a reliable reproduction of this
>> for various reasons, not least of which is that simulating network
>> error is difficult.
>
> I have a similar problem which I am debugging currently with the
> new code. (~1000 submodules which may fail randomly; This is
> why I wonder about either automated retries or ignoring the errors)
>
>>
>> Any thoughts on this? Should I just have my script that runs my
>> continuous integration builds add a check to ensure files are checked
>> out? Is "--force" enough to get the submodule to be re-checked out
>> even if it's already checked out at the location?
>
> I believe so.
>
> Stefan
>

Ok, sounds like for now I need to extend my setup to use --force instead, which will resolve the issue for me. I like the idea of having git itself perform some retries incase of transient failures though.

Show 7 quoted lines

>>
>> Thanks,
>> Jake
>> --
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html