threads / bug / 46566

Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

Subject: Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

## tl;dr

4 messages between Aug 11, 2017 and Aug 14, 2017.

replies: 3people: 4as markdown or json

Dr.-Ing. Christoph Cullmann· Aug 11, 2017, 07:04 UTC · lore

Hi,

on Windows 64-bit, for a repository having a .pack file > 4GB I get during cloning:

$ git clone file:///repositories/test.git test Cloning into 'test'... remote: Counting objects: 210294, done. remote: error: bad object header remote: error: bad object header remote: fatal: packed object cae50116ebe36de8bded4811bd262d90670bde2f (stored in ./objects/pack/pack-30ba8b19b029cc6554853ae07729aa807995ebb8.pack) is corrupt error: git upload-pack: git-pack-objects died with error. fatal: git upload-pack: aborting due to possible repository remote: aborting due to possible repository corruption on the remote side. corruption on the remote side. fatal: early EOF fatal: index-pack failed

We use git 2.14.1, the 64-bit version.

On systems which have (long int) == 64-bit datatype, all works well, e.g. Linux 64-bit. (same goes for fscks/...., they work on this git on Linux 64-bit, not on Windows 64-bit)

It seems the Windows port did take care of using the right 64-bit offsets and all in the OS specific parts, but things like pack-objects.c in the generic parts use "unsigned long" like it behaves like size_t (at least on the first glance).

As I think the issue is in the generic part, I report this here, not on the Windows list, thought it seems to be an IL32P64 related issue.

Greetings Christoph

-- 
----------------------------- Dr.-Ing. Christoph Cullmann ---------
AbsInt Angewandte Informatik GmbH      Email: cullmann@AbsInt.com
Science Park 1                         Tel:   +49-681-38360-22
66123 Saarbrücken                      Fax:   +49-681-38360-20
GERMANY                                WWW:   http://www.AbsInt.com
--------------------------------------------------------------------
Geschäftsführung: Dr.-Ing. Christian Ferdinand
Eingetragen im Handelsregister des Amtsgerichts Saarbrücken, HRB 11234

Johannes Schindelin· Aug 14, 2017, 16:05 UTC · re: Dr.-Ing. Christoph Cullmann · lore

Re: Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

Hi Christoph,

On Fri, 11 Aug 2017, Dr.-Ing. Christoph Cullmann wrote:

> on Windows 64-bit, for a repository having a .pack file > 4GB I get
> during cloning:

The reason is Git's source code that over-uses the `unsigned long` datatype.

In a nearby-thread, an underappreciated effort by Martin Koegler is underway to get the ball rolling in getting it fixed. Maybe you can help making Martin a lot more welcome on the Git mailing list, and maybe even help getting his patches reviewed and integrated?

Ciao, Johannes

Stefan Beller· Aug 14, 2017, 17:07 UTC · re: Johannes Schindelin · lore

Re: Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

On Mon, Aug 14, 2017 at 9:05 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 14 quoted lines

> Hi Christoph,
>
> On Fri, 11 Aug 2017, Dr.-Ing. Christoph Cullmann wrote:
>
>> on Windows 64-bit, for a repository having a .pack file > 4GB I get
>> during cloning:
>
> The reason is Git's source code that over-uses the `unsigned long`
> datatype.
>
> In a nearby-thread, an underappreciated effort by Martin Koegler is
> underway to get the ball rolling in getting it fixed. Maybe you can help
> making Martin a lot more welcome on the Git mailing list, and maybe even
> help getting his patches reviewed and integrated?

'nearby' as in [1] ;-)

[1] https://public-inbox.org/git/1502527643-21944-1-git-send-email-martin@mail.zuhause/

I had the impression the review is going well there?

>
> Ciao,
> Johannes

Junio C Hamano· Aug 14, 2017, 18:57 UTC · re: Stefan Beller · lore

Re: Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

Stefan Beller <sbeller@google.com> writes:

Show 22 quoted lines

> On Mon, Aug 14, 2017 at 9:05 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> Hi Christoph,
>>
>> On Fri, 11 Aug 2017, Dr.-Ing. Christoph Cullmann wrote:
>>
>>> on Windows 64-bit, for a repository having a .pack file > 4GB I get
>>> during cloning:
>>
>> The reason is Git's source code that over-uses the `unsigned long`
>> datatype.
>>
>> In a nearby-thread, an underappreciated effort by Martin Koegler is
>> underway to get the ball rolling in getting it fixed. Maybe you can help
>> making Martin a lot more welcome on the Git mailing list, and maybe even
>> help getting his patches reviewed and integrated?
>
> 'nearby' as in [1] ;-)
>
> [1] https://public-inbox.org/git/1502527643-21944-1-git-send-email-martin@mail.zuhause/
>
> I had the impression the review is going well there?

I do not know if it is "going well", but I do not agree with the "underappreciated" bit at all. I find such a blanket statement toxic and detrimental to the community.

It is true that many topics in flight are broken with a tree-wide change that is presented as a single ball of wax. Unlike the ongoing "struct object_id" effort, which also is tree-wide but tries to find a step-wise refinement to reduce its impact on other topics, it is harder to integrate such a change. But that does not have any relation to how much the effort is appreciated.