Re: Git clone fails with "bad pack header", how to get remote log

> I have a problem with my build system.
> 
> I have a remote server with a relatively large repository (around 12
> GB, each branch having a size of 3 GB). 
> 
> I have also 2 build servers (Mac, Windows) that are cloning the repo
> from the remote.
> 
> Sometimes (very often when several git clone are sent at the same
> time), I have the following error:
>         
>     remote: internal server error
>     fatal: protocol error: bad pack header
> 
> I know that it happens when the remote is compressing objects (thanks
> to `--progress -v` flags) because the last line of the log before the
> erro is: 
>     remote: Compressing objects:  93% (17959/19284)   [K
> 
>  * So I have 2 questions, does anybody what is the problem and what
> should I do?
>  * Is there a way to get a more precise log from the remote to debug
> this problem?

> On Mon, 29 Oct 2012 09:52:54 -0700 (PDT)
> Kevin Molcard <kev2041@gmail.com> wrote:
>
>> I have a problem with my build system.
>>
>> I have a remote server with a relatively large repository (around 12
>> GB, each branch having a size of 3 GB).
>>
>> I have also 2 build servers (Mac, Windows) that are cloning the repo
>> from the remote.
>>
>> Sometimes (very often when several git clone are sent at the same
>> time), I have the following error:
>>          
>>      remote: internal server error
>>      fatal: protocol error: bad pack header
>>
>> I know that it happens when the remote is compressing objects (thanks
>> to `--progress -v` flags) because the last line of the log before the
>> erro is:
>>      remote: Compressing objects:  93% (17959/19284)   [K
>>
>>   * So I have 2 questions, does anybody what is the problem and what
>> should I do?
>>   * Is there a way to get a more precise log from the remote to debug
>> this problem?
> This reminds me of a bug fixed in 1.7.12.1 [1]:
>
> * When "git push" triggered the automatic gc on the receiving end, a
>    message from "git prune" that said it was removing cruft leaked to
>    the standard output, breaking the communication protocol.
>
> In any case, bugs should be reported to the main Git list (which is
> git at vger.kernel.org), not here.
> I'm Cc'ing the main Git list so you'll get any responses from there, if
> any.
>
> Kevin, please answer to this message (keeping all the Ccs -- use "Reply
> to group" or "Reply to all" in your MUA) and describe exactly what Git
> versions on which platforms your have.
>
> 1. https://raw.github.com/git/git/master/Documentation/RelNotes/1.7.12.1.txt
>

> I have a problem with my build system.
>
> I have a remote server with a relatively large repository (around 12
> GB, each branch having a size of 3 GB).
>
> I have also 2 build servers (Mac, Windows) that are cloning the repo
> from the remote.
>
> Sometimes (very often when several git clone are sent at the same
> time), I have the following error:
>
>     remote: internal server error
>     fatal: protocol error: bad pack header
>
> I know that it happens when the remote is compressing objects (thanks
> to `--progress -v` flags) because the last line of the log before the
> erro is:
>     remote: Compressing objects:  93% (17959/19284)   [K
>
>  * So I have 2 questions, does anybody what is the problem and what
> should I do?
>  * Is there a way to get a more precise log from the remote to debug
> this problem?

> On Mon, 29 Oct 2012 09:52:54 -0700 (PDT)
> Kevin Molcard <kev2041-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> I have a problem with my build system.
>>
>> I have a remote server with a relatively large repository (around 12
>> GB, each branch having a size of 3 GB).
>>
>> I have also 2 build servers (Mac, Windows) that are cloning the repo
>> from the remote.
>>
>> Sometimes (very often when several git clone are sent at the same
>> time), I have the following error:
>>          
>>      remote: internal server error
>>      fatal: protocol error: bad pack header
>>
>> I know that it happens when the remote is compressing objects (thanks
>> to `--progress -v` flags) because the last line of the log before the
>> erro is:
>>      remote: Compressing objects:  93% (17959/19284)   [K
>>
>>   * So I have 2 questions, does anybody what is the problem and what
>> should I do?
>>   * Is there a way to get a more precise log from the remote to debug
>> this problem?
> This reminds me of a bug fixed in 1.7.12.1 [1]:
>
> * When "git push" triggered the automatic gc on the receiving end, a
>    message from "git prune" that said it was removing cruft leaked to
>    the standard output, breaking the communication protocol.
>
> In any case, bugs should be reported to the main Git list (which is
> git at vger.kernel.org), not here.
> I'm Cc'ing the main Git list so you'll get any responses from there, if
> any.
>
> Kevin, please answer to this message (keeping all the Ccs -- use "Reply
> to group" or "Reply to all" in your MUA) and describe exactly what Git
> versions on which platforms your have.
>
> 1. https://raw.github.com/git/git/master/Documentation/RelNotes/1.7.12.1.txt
>

> I tried to install git 1.8 on the remote server and get exactly the
> same problem :(.
> [...]
> >>Sometimes (very often when several git clone are sent at the same
> >>time), I have the following error:
> >>     remote: internal server error
> >>     fatal: protocol error: bad pack header

> On Tue, Oct 30, 2012 at 03:57:36PM +0100, kevin molcard wrote:
>
>> I tried to install git 1.8 on the remote server and get exactly the
>> same problem :(.
>> [...]
>>>> Sometimes (very often when several git clone are sent at the same
>>>> time), I have the following error:
>>>>      remote: internal server error
>>>>      fatal: protocol error: bad pack header
> I'm very confused about who is printing "internal server error". The
> "remote:" indicates that it came to the git client via the sideband,
> which means it probably came from the stderr of a child process (e.g.,
> pack-objects). But git does not and has never generated the phrase
> "internal server error".
>
> So what program is producing that? Is there some kind of custom layer
> that might be run when upload-pack runs "git pack-objects ..."? Can you
> try running strace on the server?
>
> -Peff
>

> On Tue, Oct 30, 2012 at 03:57:36PM +0100, kevin molcard wrote:
>
>> I tried to install git 1.8 on the remote server and get exactly the
>> same problem :(.
>> [...]
>>>> Sometimes (very often when several git clone are sent at the same
>>>> time), I have the following error:
>>>>      remote: internal server error
>>>>      fatal: protocol error: bad pack header
> I'm very confused about who is printing "internal server error". The
> "remote:" indicates that it came to the git client via the sideband,
> which means it probably came from the stderr of a child process (e.g.,
> pack-objects). But git does not and has never generated the phrase
> "internal server error".
>
> So what program is producing that? Is there some kind of custom layer
> that might be run when upload-pack runs "git pack-objects ..."? Can you
> try running strace on the server?
>
> -Peff
>

> On Wed, Oct 31, 2012 at 03:34:22PM +0100, kevin molcard wrote:
>
>> I forgot to mention that I am using scm manager:
>> https://bitbucket.org/sdorra/scm-manager/wiki/Home
>>
>> So that maybe the " custom layer you are talking about.
>
> Yeah, that is very important. If I am reading the scm-manager code right
> (and I might not be, as I only just looked at it), it is built entirely
> around JGit, and is not calling git-core programs at all. So it is
> either an issue in scm-manager, or in JGit.
>
> Your best bet is probably to report the issue to the scm-manager folks,
> who can probably help you dig into the problem further (it may even have
> more detailed logs of what happened, but I don't know).
>
> -Peff