Re: git push --mirror hung indefinitely

From: brian m. carlson <sandals@crustytoothpaste.net>
Date: Mar 29, 2025, 16:07 UTC
Message-ID: <Z-gaxw8ctTcFUUoV@tapette.crustytoothpaste.net>
In-Reply-To: <SA1PR19MB70131FCBD314F579B15F8C69C0A32@SA1PR19MB7013.namprd19.prod.outlook.com>

On 2025-03-29 at 14:07:17, Akash S wrote:

Show 36 quoted lines

> We compared the GIT_TRACE_PACKET=1 output of successful and
> unsuccessful pushes and noticed that the "git flush message 0000" is
> not being received from the Azure DevOps Server during unsuccessful
> push. 
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> Successful git push:
> 11:29:09.032050 pkt-line.c:85           packet:          git< ok refs/tags/v3.9.1
> 11:29:09.034601 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
> 11:29:09.034630 pkt-line.c:85           packet:     sideband< \10000
> 11:29:09.034643 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
> 11:29:09.034664 pkt-line.c:85           packet:     sideband< 0000
> 11:29:09.034664 pkt-line.c:85           packet:          git< 0000
> 11:29:09.034699 pkt-line.c:85           packet:          git> 0000
> 11:29:09.035667 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
> 11:29:09.035701 http.c:878              == Info: Connection #0 to host dev.azure.com left intact
> To https://dev.azure.com/ORGNAME/PRJNAME/_git/REPONAME
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> Unsuccessful git push:
> 11:33:58.759005 pkt-line.c:85           packet:     sideband< \10018ok refs/tags/v3.1.7
> 11:33:58.759035 pkt-line.c:85           packet:          git< ok refs/tags/v3.1.7
> 11:33:58.760165 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
> 11:33:58.760197 http.c:878              == Info: Connection #0 to host dev.azure.com left intact
> (process hung here)
> 
> -------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> We noticed that "git push mirror" ultimately spawns "git send-pack"
> with options "--stdin" and "--stateless-rpc" and from the git
> send-pack documentation we confirmed that the line must end with a
> flush packet which is ultimately not happening in the hung cases.
> pstack of send-pack process also shows it is waiting on reading
> packets as you mentioned earlier.

`--stateless-rpc` is used for HTTPS, so that's expected.

> Can you please confirm if the hung issue is due to the missing message or packet from the Azure DevOps Server?

We don't have a full trace output here, so it's hard to say completely for certain, but from what I know of the protocol off the top of my head, a flush packet is always required at the end of the section of message. So I would say that it's very likely that's the cause here.

Note that it could be that you're _also_ missing one or more `ok` responses from the ref updates you're doing here, and that it's hanging _before_ it would normally send the flush packet. That might point to a different cause on the server side, but the effect would be the same.

If you're talking to the Azure DevOps people, I'd provide them a full trace output of both working and non-working pushes so that they can inspect it themselves, which will help them figure out where it might be going wrong.

-- 
brian m. carlson (they/them)
Toronto, Ontario, CA