{"thread":{"id":"50651","subject":"Large object issue (Windows)","startedAt":"2019-03-05T00:04:48Z","lastAt":"2019-03-07T17:29:57Z","messageCount":3,"participants":["Patrick Hogg","brian m. carlson","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"370648","messageId":"CAFOcBz=+XZtcxPxeE3j6dpHXA79R_BOyDCtNbwSu+nbBpPzGrw@mail.gmail.com","threadId":"50651","inReplyTo":null,"subject":"Large object issue (Windows)","fromName":"Patrick Hogg","fromEmail":"phogg@novamoon.net","sentAt":"2019-03-05T00:04:02Z","receivedAt":"2019-03-05T00:04:48Z","isPatch":false,"sender":{"key":"phogg@novamoon.net","avatar":"https://avatars.githubusercontent.com/u/7890416?v=4"},"body":"Hi all,\n\nWhile investigating the last issue I reported (and fixed) I was trying\nto come up with a good test case for repos with large objects. In the\nprocess I found an issue on Windows with objects at least 4g large:\n\ngit init test\ncd test\necho \"*.exe binary\" > .gitattributes\ntruncate -s 4g nullbytes.exe\ngit stage .\ngit commit -m \"Test\"\n# This will break, complaining that the object is corrupt.\ngit fsck --full\n# This will also break, complaining that the object is corrupt.\n#git gc\n\nI did some investigation and I think that this is a porting issue.\nunpack_object_header_buffer in packfile.c uses an unsigned long for the\nsize. On Linux this will be 64 bits (at least on the Linux systems I've\ntried) but on Windows it's 32 bits. The code then decides that the\nobject header is bad and bombs. However, if I move the repo to a Linux\nmachine it can handle the data just fine. (And ironically git generated\nthe object header when storing the object!)\n\nIs there any reason not to switch the unsigned longs in\nunpack_object_header_buffer (and its callers, wherever that may lead)\nto uint64_t? (Or any potential pitfalls in doing so that I would need\nto look out for?)\n\nThanks,\n-Patrick\n"},{"id":"370654","messageId":"20190305033526.GH601925@genre.crustytoothpaste.net","threadId":"50651","inReplyTo":"CAFOcBz=+XZtcxPxeE3j6dpHXA79R_BOyDCtNbwSu+nbBpPzGrw@mail.gmail.com","subject":"Re: Large object issue (Windows)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2019-03-05T03:35:26Z","receivedAt":"2019-03-05T03:35:36Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Mar 04, 2019 at 07:04:02PM -0500, Patrick Hogg wrote:\n> Hi all,\n> \n> While investigating the last issue I reported (and fixed) I was trying\n> to come up with a good test case for repos with large objects. In the\n> process I found an issue on Windows with objects at least 4g large:\n> \n> git init test\n> cd test\n> echo \"*.exe binary\" > .gitattributes\n> truncate -s 4g nullbytes.exe\n> git stage .\n> git commit -m \"Test\"\n> # This will break, complaining that the object is corrupt.\n> git fsck --full\n> # This will also break, complaining that the object is corrupt.\n> #git gc\n> \n> I did some investigation and I think that this is a porting issue.\n> unpack_object_header_buffer in packfile.c uses an unsigned long for the\n> size. On Linux this will be 64 bits (at least on the Linux systems I've\n> tried) but on Windows it's 32 bits. The code then decides that the\n> object header is bad and bombs. However, if I move the repo to a Linux\n> machine it can handle the data just fine. (And ironically git generated\n> the object header when storing the object!)\n> \n> Is there any reason not to switch the unsigned longs in\n> unpack_object_header_buffer (and its callers, wherever that may lead)\n> to uint64_t? (Or any potential pitfalls in doing so that I would need\n> to look out for?)\n\nIt's known that there are several problems with this, affecting various\nparts of the code. Patches to fix this are of course welcome.\n\nI think we've chosen to specify size_t for values which are stored\nentirely in memory, since a buffer can't be larger than this size, and\noff_t for sizes which refer to files or object sizes. The latter will be\n64-bit on 32-bit systems when compiled with _FILE_OFFSET_BITS set to 64,\nwhile the former will be 32-bit.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"370916","messageId":"44f5c2ae-149f-2b7c-77ac-90476ed7bd9d@iee.org","threadId":"50651","inReplyTo":"20190305033526.GH601925@genre.crustytoothpaste.net","subject":"Re: Large object issue (Windows)","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2019-03-07T17:29:52Z","receivedAt":"2019-03-07T17:29:57Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 05/03/2019 03:35, brian m. carlson wrote:\n> On Mon, Mar 04, 2019 at 07:04:02PM -0500, Patrick Hogg wrote:\n>> Hi all,\n>>\n>> While investigating the last issue I reported (and fixed) I was trying\n>> to come up with a good test case for repos with large objects. In the\n>> process I found an issue on Windows with objects at least 4g large:\n>>\n>> git init test\n>> cd test\n>> echo \"*.exe binary\" > .gitattributes\n>> truncate -s 4g nullbytes.exe\n>> git stage .\n>> git commit -m \"Test\"\n>> # This will break, complaining that the object is corrupt.\n>> git fsck --full\n>> # This will also break, complaining that the object is corrupt.\n>> #git gc\n>>\n>> I did some investigation and I think that this is a porting issue.\n>> unpack_object_header_buffer in packfile.c uses an unsigned long for the\n>> size. On Linux this will be 64 bits (at least on the Linux systems I've\n>> tried) but on Windows it's 32 bits. The code then decides that the\n>> object header is bad and bombs. However, if I move the repo to a Linux\n>> machine it can handle the data just fine. (And ironically git generated\n>> the object header when storing the object!)\n>>\n>> Is there any reason not to switch the unsigned longs in\n>> unpack_object_header_buffer (and its callers, wherever that may lead)\n>> to uint64_t? (Or any potential pitfalls in doing so that I would need\n>> to look out for?)\n> It's known that there are several problems with this, affecting various\n> parts of the code. Patches to fix this are of course welcome.\n>\n> I think we've chosen to specify size_t for values which are stored\n> entirely in memory, since a buffer can't be larger than this size, and\n> off_t for sizes which refer to files or object sizes. The latter will be\n> 64-bit on 32-bit systems when compiled with _FILE_OFFSET_BITS set to 64,\n> while the former will be 32-bit.\n\nHi Patrick,\n\nThere is also a thread on the Git-for-Windows list at \nhttps://github.com/git-for-windows/git/issues/1063 and also here at \nhttps://public-inbox.org/git/994568940.109648.1548957557643@ox.hosteurope.de/ \n\n\nPart of the issues is that zlib on windows 'sort of' fails to do >4Gb - \nsee their FAQ32 - in that the length value is only 'long' which is only \n32 bit, while in fact the zlib copes fine but returns a length modulo \nthat limit.\n\nTrying to get all the places that should be upcast to size_t (ptr) or \nptdiff_t rather than coerced down to windows 32bit long is part of the \nstruggle.\n\nPhilip\n\n"}]}