GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

Hello,

We are seeing issue with GIT 2.14 version. When we try to clone the repos, it is taking HUGE amount of time on HPUX, whereas on the linux machine with same network configuration, it's getting cloned in less than mins. So we want to know has anyone reported this issue? What is the fix for this? Has the fix been released for this? Whom should we contact for this? 

Thanks & Regards, Ibrahim Vanak

Below is the HPUX system where we are seeing issue, it is taking 1 hr 45 mins and later it failed:

root@sstl002.in.rdlabs.hpecorp.net# uname -a HP-UX sstl002 B.11.31 U ia64 0158936019 unlimited-user license root@sstl002.in.rdlabs.hpecorp.net# time git clone git@github.hpe.com:HPUX/SysFaultMgmt.git Cloning into 'SysFaultMgmt'... remote: Enumerating objects: 63627, done. remote: Total 63627 (delta 0), reused 0 (delta 0), pack-reused 63627 Receiving objects: 100% (63627/63627), 681.90 MiB | 111.00 KiB/s, done. fatal: pack is corrupted (SHA1 mismatch) fatal: index-pack failed

real    104m3.373s user    0m36.220s sys     0m6.090s You have mail in /var/mail/root root@sstl002.in.rdlabs.hpecorp.net#

Below is the system where it has passed in less than 3 mins:

[root@chandana ~]# uname -a Linux chandana.in.rdlabs.hpecorp.net 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [root@chandana ~]# time git clone git@github.hpe.com:HPUX/SysFaultMgmt.git Cloning into 'SysFaultMgmt'... remote: Enumerating objects: 63627, done. remote: Total 63627 (delta 0), reused 0 (delta 0), pack-reused 63627 Receiving objects: 100% (63627/63627), 681.90 MiB | 5.59 MiB/s, done. Resolving deltas: 100% (35495/35495), done. Checking connectivity... done. Checking out files: 100% (10634/10634), done.

real    2m46.517s user    0m36.601s sys     0m10.503s

On Tue, May 28, 2019 at 09:10:12AM +0000, Vanak, Ibrahim wrote:

I don't know about the slowness, but...

Git v2.14 uses the sha1collision-detection implementation of sha1 by default. That has a bug that was fixed recently with:

  commit 4125f78222749cb8fc91115abec3ac83e5dfb194
  Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
  Date:   Tue May 14 00:17:01 2019 +0200
  
      sha1dc: update from upstream
      
      Update sha1dc from the latest version by the upstream
      maintainer[1]. See 07a20f569b ("Makefile: fix unaligned loads in
      sha1dc with UBSan", 2019-03-12) for the last update.
      
      This fixes an issue where HP-UX IA64 was wrongly detected as a
      Little-endian instead of a Big-endian system, see [2] and [3].
      
      1. https://github.com/cr-marcstevens/sha1collisiondetection/commit/855827c583bc30645ba427885caa40c5b81764d2
      2. https://public-inbox.org/git/603989bd-f86d-c61d-c6f5-fb6748a65ba9@siemens.com/
      3. https://github.com/cr-marcstevens/sha1collisiondetection/pull/50

which looks like it would impact your system. You can either:

  1. Try v2.22.0-rc1, which will be the first release with that fix.

  2. Try cherry-picking the various fixes on top of v2.14.0:

       git checkout v2.14.0
       git cherry-pick 23e37f8e9d5961c0c8d52ac481693d3fca5309ce
       git cherry-pick 07a20f569b4b1690e717eaac0954007a8edfbfc2
       git cherry-pick 4125f78222749cb8fc91115abec3ac83e5dfb194

  3. Compile with another sha1 implementation. E.g.:

       # if you have openssl available; otherwise,
       # try BLK_SHA1
       echo 'OPENSSL_SHA1 = Yes' >config.mak
       make

     Note that you won't be protected from collision attacks, but those
     are still impractically expensive to mount at this point. It may be
     a good tradeoff until you can upgrade to a more recent Git.

-Peff

On Tue, May 28 2019, Jeff King wrote:

As a follow-up perhaps we should hash_object_file_literally() early in main() (or maybe just clone & init) to detect this issue & exit with some "zomg broken!".

Vanak: Also, this issue suggests whoever compiled the package you're
using (you?) installed it with a failing "make test", so a lot of other
things may be broken...

Thanks everyone for the inputs !!! upgrade of GIT to 2.21 worked.

BUT still I have significant slowness(50 times slower than clone on linux machine) while cloning. HPUX box is having very good H/W configuration and network is also stable.

Do you have any inputs on this.?

-Ibrahim

-----Original Message-----
From: Jeff King [mailto:peff@peff.net] 
Sent: Tuesday, May 28, 2019 3:07 PM
To: Vanak, Ibrahim <ibrahim.vanak@hpe.com
Cc: git@vger.kernel.org
Subject: Re: GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

On Tue, May 28, 2019 at 09:10:12AM +0000, Vanak, Ibrahim wrote:

I don't know about the slowness, but...

Git v2.14 uses the sha1collision-detection implementation of sha1 by default. That has a bug that was fixed recently with:

  commit 4125f78222749cb8fc91115abec3ac83e5dfb194
  Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
  Date:   Tue May 14 00:17:01 2019 +0200
  
      sha1dc: update from upstream
      
      Update sha1dc from the latest version by the upstream
      maintainer[1]. See 07a20f569b ("Makefile: fix unaligned loads in
      sha1dc with UBSan", 2019-03-12) for the last update.
      
      This fixes an issue where HP-UX IA64 was wrongly detected as a
      Little-endian instead of a Big-endian system, see [2] and [3].
      
      1. https://github.com/cr-marcstevens/sha1collisiondetection/commit/855827c583bc30645ba427885caa40c5b81764d2
      2. https://public-inbox.org/git/603989bd-f86d-c61d-c6f5-fb6748a65ba9@siemens.com/
      3. https://github.com/cr-marcstevens/sha1collisiondetection/pull/50

which looks like it would impact your system. You can either:

  1. Try v2.22.0-rc1, which will be the first release with that fix.

  2. Try cherry-picking the various fixes on top of v2.14.0:

       git checkout v2.14.0
       git cherry-pick 23e37f8e9d5961c0c8d52ac481693d3fca5309ce
       git cherry-pick 07a20f569b4b1690e717eaac0954007a8edfbfc2
       git cherry-pick 4125f78222749cb8fc91115abec3ac83e5dfb194

  3. Compile with another sha1 implementation. E.g.:

       # if you have openssl available; otherwise,
       # try BLK_SHA1
       echo 'OPENSSL_SHA1 = Yes' >config.mak
       make

     Note that you won't be protected from collision attacks, but those
     are still impractically expensive to mount at this point. It may be
     a good tradeoff until you can upgrade to a more recent Git.

-Peff

On Tue, May 28, 2019 at 06:45:18PM +0000, Vanak, Ibrahim wrote:

From your output:

The main things going on there are:

  1. we're getting the bytes over the network

  2. we're computing a sha1 on the stream we're getting

  3. we're zlib inflating any non-delta objects we find and computing
     their sha1

  4. we're checking the filesystem to see if we have other copies of any
     of those objects

For (2) and (3) it could be that the sha1 implementation is not quite as fast. But 50x is much larger than I'd expect. If you've built Git from source, you could try running "t/helper/test-sha1 <big-file" and timing the result. That would isolate sha1 performance. If it's slow, try building with "make BLK_SHA1=Yes" or "make OPENSSL_SHA1=Yes" and see if that's faster.

For (4), we'll have to stat() in .git/objects to see if we have a loose version of the object. That can be slow if you have a really terrible NFS setup, for example. But I doubt that would still be slow in v2.21. There we should be using the new loose-cache which will only readdir() the object directories once.

If neither of those pan out, it seems like the network is slow? Or maybe ssh? Or pipes passing data from clone to index-pack?

-Peff

Hello Jeff,

Thanks again for the response. 

I am not building GIT and also don't know either how to build. I am just pulling up the binaries from http://hpux.connect.org.uk/. 

Both HPUX and linux machine are in same network/subnet so network shouldn't be a problem here. ssh to HPUX box is super quick and box is having very high configuration. There is no NFS involved here.

I am surprised on slowness.... If you get a chance to look more into this, please.

Thanks & Regards, Ibrahim Vanak

-----Original Message-----
From: Jeff King [mailto:peff@peff.net] 
Sent: Wednesday, May 29, 2019 3:00 AM
To: Vanak, Ibrahim <ibrahim.vanak@hpe.com>
Cc: git@vger.kernel.org
Subject: Re: GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

On Tue, May 28, 2019 at 06:45:18PM +0000, Vanak, Ibrahim wrote:

From your output:

The main things going on there are:

  1. we're getting the bytes over the network

  2. we're computing a sha1 on the stream we're getting

  3. we're zlib inflating any non-delta objects we find and computing
     their sha1

  4. we're checking the filesystem to see if we have other copies of any
     of those objects

For (2) and (3) it could be that the sha1 implementation is not quite as fast. But 50x is much larger than I'd expect. If you've built Git from source, you could try running "t/helper/test-sha1 <big-file" and timing the result. That would isolate sha1 performance.  If it's slow, try building with "make BLK_SHA1=Yes" or "make OPENSSL_SHA1=Yes" and see if that's faster.

For (4), we'll have to stat() in .git/objects to see if we have a loose version of the object. That can be slow if you have a really terrible NFS setup, for example. But I doubt that would still be slow in v2.21. There we should be using the new loose-cache which will only readdir() the object directories once.

If neither of those pan out, it seems like the network is slow? Or maybe ssh? Or pipes passing data from clone to index-pack?

-Peff

Hello Jeff,

Just sharing more information with you on this issue:

After cloning when I tried to checkout a branch on HPUX and Linux, I still significant time difference there as well even though network is not involve here. Do you suspect anything with HPUX OS? Do you have any mechanism to find out who is the causing gigantic delay ?

========================================
Checkout of a branch on HPUX as below
========================================
root@fiend6.in.rdlabs.hpecorp.net# time git checkout remotes/origin/11.23/sfm_1123_main
Checking out files: 100% (12949/12949), done.
Note: checking out 'remotes/origin/11.23/sfm_1123_main'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 39b6110f Commiting to isvn for WEB1303_1123 SFM release

real    8m17.234s user    0m7.070s sys     0m1.960s

==============================================
Checkout of the same branch on Linux as below:
==============================================
[root@chandana SysFaultMgmt]# time git checkout remotes/origin/11.23/sfm_1123_main
Checking out files: 100% (12949/12949), done.
Note: checking out 'remotes/origin/11.23/sfm_1123_main'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 39b6110... Commiting to isvn for WEB1303_1123 SFM release

real    0m4.709s user    0m3.539s sys     0m1.098s [root@chandana SysFaultMgmt]#

-----Original Message-----
From: Vanak, Ibrahim 
Sent: Wednesday, May 29, 2019 10:59 AM
To: Jeff King <peff@peff.net>
Cc: git@vger.kernel.org
Subject: RE: GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

Hello Jeff,

Thanks again for the response. 

I am not building GIT and also don't know either how to build. I am just pulling up the binaries from http://hpux.connect.org.uk/. 

Both HPUX and linux machine are in same network/subnet so network shouldn't be a problem here. ssh to HPUX box is super quick and box is having very high configuration. There is no NFS involved here.

I am surprised on slowness.... If you get a chance to look more into this, please.

Thanks & Regards, Ibrahim Vanak

-----Original Message-----
From: Jeff King [mailto:peff@peff.net]
Sent: Wednesday, May 29, 2019 3:00 AM
To: Vanak, Ibrahim <ibrahim.vanak@hpe.com>
Cc: git@vger.kernel.org
Subject: Re: GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

On Tue, May 28, 2019 at 06:45:18PM +0000, Vanak, Ibrahim wrote:

From your output:

The main things going on there are:

  1. we're getting the bytes over the network

  2. we're computing a sha1 on the stream we're getting

  3. we're zlib inflating any non-delta objects we find and computing
     their sha1

  4. we're checking the filesystem to see if we have other copies of any
     of those objects

For (2) and (3) it could be that the sha1 implementation is not quite as fast. But 50x is much larger than I'd expect. If you've built Git from source, you could try running "t/helper/test-sha1 <big-file" and timing the result. That would isolate sha1 performance.  If it's slow, try building with "make BLK_SHA1=Yes" or "make OPENSSL_SHA1=Yes" and see if that's faster.

For (4), we'll have to stat() in .git/objects to see if we have a loose version of the object. That can be slow if you have a really terrible NFS setup, for example. But I doubt that would still be slow in v2.21. There we should be using the new loose-cache which will only readdir() the object directories once.

If neither of those pan out, it seems like the network is slow? Or maybe ssh? Or pipes passing data from clone to index-pack?

-Peff

On Wed, May 29, 2019 at 09:06:18AM +0000, Vanak, Ibrahim wrote:

If it's happening for just a local checkout operation, I'd suspect very poor filesystem performance. On Linux, I'd run it under "perf" to see where the time is going. Or possibly "strace -T" to time system calls. I don't know what tools are available under HP-UX for that kind of profiling.

-Peff

Thank Peff. Unfortunately I am living with this issue, no one able to find out what is causing this significant delay. Re-initiating this thread if someone can help on this.

Thanks & Regards, Ibrahim Vanak

-----Original Message-----
From: Jeff King [mailto:peff@peff.net] 
Sent: Thursday, May 30, 2019 5:28 PM
To: Vanak, Ibrahim <ibrahim.vanak@hpe.com>
Cc: git@vger.kernel.org
Subject: Re: GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

On Wed, May 29, 2019 at 09:06:18AM +0000, Vanak, Ibrahim wrote:

If it's happening for just a local checkout operation, I'd suspect very poor filesystem performance. On Linux, I'd run it under "perf" to see where the time is going. Or possibly "strace -T" to time system calls. I don't know what tools are available under HP-UX for that kind of profiling.

-Peff

Hello ALL again,

Has anyone tested the performance of GIT on HP-UX platform? Can someone please look into the issue we are seeing.

Thanks, Ibrahim

-----Original Message-----
From: Vanak, Ibrahim 
Sent: Tuesday, June 11, 2019 10:09 PM
To: Jeff King <peff@peff.net>
Cc: git@vger.kernel.org
Subject: RE: GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

Thank Peff. Unfortunately I am living with this issue, no one able to find out what is causing this significant delay. Re-initiating this thread if someone can help on this.

Thanks & Regards, Ibrahim Vanak

-----Original Message-----
From: Jeff King [mailto:peff@peff.net]
Sent: Thursday, May 30, 2019 5:28 PM
To: Vanak, Ibrahim <ibrahim.vanak@hpe.com>
Cc: git@vger.kernel.org
Subject: Re: GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

On Wed, May 29, 2019 at 09:06:18AM +0000, Vanak, Ibrahim wrote:

If it's happening for just a local checkout operation, I'd suspect very poor filesystem performance. On Linux, I'd run it under "perf" to see where the time is going. Or possibly "strace -T" to time system calls. I don't know what tools are available under HP-UX for that kind of profiling.

-Peff

On 6/18/2019 10:31 AM, Vanak, Ibrahim wrote:

You might try setting some of the GIT_TRACE* environment variables listed in [1] on both your HP-UX and Linux machines and see if you can isolate any performance differences between them.

[1] https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables

Hello ALL,

I did try GIT_TRACE* environment variables but couldn't isolate the problem. 

But HPUX Dev team have seen one significance difference, when they were triaging this issue, the ways GIT operations handled by 2 OSs: On linux, around 40 processes are spawned for handling a GIT operation whereas on HPUX only one process manages the operation. Is there something special being written for linux on GIT code ?

Can you please have a look into this.

Thanks & Regards, Ibrahim Vanak

-----Original Message-----
From: Jeff Hostetler [mailto:git@jeffhostetler.com] 
Sent: Tuesday, June 18, 2019 10:45 PM
To: Vanak, Ibrahim <ibrahim.vanak@hpe.com>; Jeff King <peff@peff.net>
Cc: git@vger.kernel.org
Subject: Re: GIT issue while cloning (fatal: pack is corrupted (SHA1 mismatch)) !!!

On 6/18/2019 10:31 AM, Vanak, Ibrahim wrote:

You might try setting some of the GIT_TRACE* environment variables listed in [1] on both your HP-UX and Linux machines and see if you can isolate any performance differences between them.

[1] https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables 

On Fri, Jun 28, 2019 at 04:18:37PM +0000, Vanak, Ibrahim wrote:

On Linux (and any platform that supports pthreads), Git will do the lstat() operations to refresh the index in parallel. See the core.preloadIndex option (which I think defaults to true).

If you have a very high-latency lstat() and your build does not have pthread support, that could be a significant difference.

That might be worth exploring.

-Peff

Hi,

On Tue, 28 May 2019, Vanak, Ibrahim wrote:

You definitely want to upgrade ASAP. Not only is the issue that you reported fixed, but two distinct vulnerabilities have been fixed since v2.14.0. Your version is still vulnerable.

Ciao, Johannes