{"thread":{"id":"62811","subject":"large repository clone failure in git for windows","startedAt":"2025-01-15T08:23:25Z","lastAt":"2025-01-16T01:53:19Z","messageCount":2,"participants":["Ginger Luo 罗江","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"510559","messageId":"NTZPR01MB11138218E5A5F2EB6AD848518419A@NTZPR01MB1113.CHNPR01.prod.partner.outlook.cn","threadId":"62811","inReplyTo":null,"subject":"large repository clone failure in git for windows","fromName":"Ginger Luo 罗江","fromEmail":"jiang.luo@vastaitech.com","sentAt":"2025-01-15T08:07:30Z","receivedAt":"2025-01-15T08:23:25Z","isPatch":false,"sender":{"key":"jiang.luo@vastaitech.com","avatar":null},"body":"Hello, GIT gurus\n\nI have a large repository which is more than 30GB, when I clone it with git-for-windows (tried 2.47.1 and some old versions), it prompted me with \"fatal: pack has bad object at offset xxxxx: inflate returned 1\", usually died at \"receiving objects\" stage at about 4GiB.\nLooks like it's same issue in https://github.com/git-for-windows/git/pull/2179 ;\nSeems like it's a \"long\" versus \"size_t\" problem and should be fixed long ago, but why it's still there? I was using 64bit git and 64bit windows server;\n\nThanks!!\nG\n"},{"id":"510631","messageId":"Z4hmhjOkuhxYy3IZ@tapette.crustytoothpaste.net","threadId":"62811","inReplyTo":"NTZPR01MB11138218E5A5F2EB6AD848518419A@NTZPR01MB1113.CHNPR01.prod.partner.outlook.cn","subject":"Re: large repository clone failure in git for windows","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-01-16T01:53:10Z","receivedAt":"2025-01-16T01:53:19Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-01-15 at 08:07:30, Ginger Luo 罗江 wrote:\n> Hello, GIT gurus\n> \n> I have a large repository which is more than 30GB, when I clone it with git-for-windows (tried 2.47.1 and some old versions), it prompted me with \"fatal: pack has bad object at offset xxxxx: inflate returned 1\", usually died at \"receiving objects\" stage at about 4GiB.\n> Looks like it's same issue in https://github.com/git-for-windows/git/pull/2179 ;\n\nI think it's actually a different issue.  That's the limit for\nindividual large blobs, and pack size would be a different code path.\n\nI feel like, though, that lots of people have large repositories and we\nwould have heard before if Git for Windows simply could not handle pack\nfiles over 4 GiB.  I've CC'd Dscho, the Git for Windows maintainer, to\nverify that there's no known problems cloning large repositories.  He's\nvery capable and has seen a lot, so hopefully he can provide some good\ninsight as to what might be going wrong.\n\nDo you have a single file that is larger than 4 GiB?  That is known to\nhave had some problems on Git for Windows in the past, and could\ntheoretically be related to this.  However, typically files are\ncompressed in pack files, so I wouldn't expect a 4 GiB blob to cause a\nfailure at 4 GiB in the pack file unless it was uncompressible (such as\nrandom data).\n\nDo you have any sort of public repository or test case that can\nreproduce this?  That would really help us fix the problem or pin down\nwhat might be going on, since we could inspect it ourselves.  For\ninstance, on my system, a fresh bare clone of\nhttps://github.com/torvalds/linux.git is larger than 4 GiB, so it would\nbe good to know if that fails for you as well, or if it works correctly.\n\nAre you using any sort of antivirus or firewall other than the default,\nor any sort of proxy, including any TLS man-in-the-middle device or\ncorporate proxy?  This sounds a lot like some piece of software or\nhardware trying to buffer all the data for inspection or tampering with\nthe data.\n\nIf so, can you please try to completely uninstall the software and\nreboot, or use a different network that doesn't have such a device on\nit?  Note that simply disabling the software often does not fix the\nproblem.  If that works, then you should report a bug to the vendor of\nthat software or device.\n\n> Seems like it's a \"long\" versus \"size_t\" problem and should be fixed long ago, but why it's still there? I was using 64bit git and 64bit windows server;\n\nGit was originally written for Unix systems, which, when 64-bit, are\nLP64.  That means that `long`, `long long`, and pointers are 64-bit, and\nthe appropriate way to write a word-sized integer is `long`.  This has\nbeen the case since the DEC Alpha, which was one of the first 64-bit\nmachines.\n\nHowever, Windows decided to use an LLP64 approach, where `long` is\n32-bit and `long long` and pointers are 64-bit.  They did have reasons\nfor doing so, such as compatibility with existing software that\nspecified `long` as a 32-bit type, but it is incompatible with the rest\nof the world, and they knew that and did it anyway.  This is a common\nsource of portability problems when porting between OSes in both\ndirections.  Fortunately, newer languages like Rust and Go have avoided\nthat problem.\n\nBecause Windows was not originally a target for Git, there's a lot of\ncode which still uses `long` in places, and nobody has cared enough to\nsend patches fixing this.  The required changes to the code can be\npretty invasive, and so patches have to be carefully crafted to be\nreasonable in size and avoid conflicts with other series in flight.  Of\ncourse, you (and anyone else) are welcome to contribute and everyone\nwould be happy if you improved things in this regard, even if only a\nlittle bit.\n\nNote that `size_t` does not have to be 64-bit on 64-bit systems, only\nlarge enough to hold the largest possible allocation, and POSIX only\nrequires that it be 16 bits in size, so it is not always suitable to\nuse it when you want a word-sized unsigned integer.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"}]}