git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[bug] Reliably Reproducible Bad Packing of Objects

From
CMChristoph Michelbach <michelbach94@gmail.com>
Date
Jun 24, 2016, 22:38 UTC
Message-ID
<1466807902.28869.8.camel@gmail.com>
Hi,

when run on a 32 bit system (Linux system, don't know about other systems),

mkdir test && cd test && git init && touch someFile && git add someFile && git commit -m "Initial commit." && dd if=/dev/urandom of=bigBinaryFile bs=1MB count=4294 && git add bigBinaryFile && git commit -m "Introduced big biary file."

reliably produces this error message: "error: bad packed object CRC for"

Since git counts sizes in byte and uses ints for it, it can't handle files bigger than (2^32 - 1) byte. That's 4'294.967296 MB. If you give git a file bigger than it can handle, it usually just says "fatal: Cannot handle files this big" without corrupting the repository. Btw.: It'd be nice if the error message stated that this only occurs on 32 bit system and only with files 4 GiB in size or bigger.

To provoke the bug, the commands above creates a file which cannot be compressed slightly less than (2^32 - 1) byte big, probably resulting in a commit more than (2^32 - 1) byte big.

I was able to reproduce the bug on a Raspberry Pi Model 3 (ARMv7 CPU) and a virtual machine running Ubuntu 16.04 32 bit (which was specifically set up to test this, so it was a clean installation) on a host running Ubuntu 16.04 64 bit on an ARM 64 bit x86 CPU (i7-4720HQ).

Output on raspi: https://gist.github.com/m1cm1c/d874f03be5b12cbd8b86ced 79fa456d1 Output on virt. machine: https://gist.github.com/m1cm1c/d0dd47828386bb0 f1e001b9f750416e0

Note that on the raspi I concatenated `git gc` to the end to show that the object exists but is corrupt but you can already see that something went wrong before `git gc` is executed.

If you look at the output on the virtual machine, however, you will not see that something wrong until you read the part where I typed in `git gc` which I find particularly worrying.

When checking whether you get the same result, please make sure to use git 1.7.6 or newer. If you use an older version of git (older versions are still being distributed, for example for Ubuntu 14.04), git will try to memory-map a too big file resulting in a different error.

-- 
With kind regards
Christoph Michelbach
Next: Duy Nguyen
Message 1 of 39 in “[bug] Reliably Reproducible Bad Packing of Objects”
  1. Christoph MichelbachJun 24, 2016
  2. Duy NguyenJul 2, 2016
  3. Duy NguyenJul 2, 2016
  4. 0/5 Number truncation with 4+ GB files on 32-bit systemsNguyễn Thái Ngọc Duy, Jul 5, 2016
  5. 1/5 pack-objects: pass length to check_pack_crc() without truncationNguyễn Thái Ngọc Duy, Jul 5, 2016
  6. Junio C HamanoJul 12, 2016
  7. 5/5 index-pack: correct "offset" type in unpack_entry_data()Nguyễn Thái Ngọc Duy, Jul 5, 2016
  8. 4/5 index-pack: report correct bad object offsets even if they are largeNguyễn Thái Ngọc Duy, Jul 5, 2016
  9. Junio C HamanoJul 12, 2016
  10. Junio C HamanoJul 12, 2016
  11. 2/5 sha1_file.c: use type off_t* for object_info->disk_sizepNguyễn Thái Ngọc Duy, Jul 5, 2016
  12. Junio C HamanoJul 12, 2016
  13. Junio C HamanoJul 12, 2016
  14. 3/5 index-pack: correct "len" type in unpack_data()Nguyễn Thái Ngọc Duy, Jul 5, 2016
  15. Johannes SixtJul 5, 2016
  16. Duy NguyenJul 6, 2016
  17. Junio C HamanoJul 6, 2016
  18. Junio C HamanoJul 6, 2016
  19. Christoph MichelbachJul 5, 2016
  20. Duy NguyenJul 10, 2016
  21. 6/5 pack-objects: do not truncate result in-pack object size on 32-bit systemsNguyễn Thái Ngọc Duy, Jul 10, 2016
  22. 7/5 fsck: use streaming interface for large blobs in packNguyễn Thái Ngọc Duy, Jul 10, 2016
  23. Junio C HamanoJul 12, 2016
  24. Junio C HamanoJul 12, 2016
  25. Junio C HamanoJul 12, 2016
  26. Junio C HamanoJul 12, 2016
  27. Junio C HamanoJul 12, 2016
  28. Duy NguyenJul 13, 2016
  29. 0/7 Number truncation with 4+ GB files on 32-bit systemsNguyễn Thái Ngọc Duy, Jul 13, 2016
  30. 4/7 index-pack: report correct bad object offsets even if they are largeNguyễn Thái Ngọc Duy, Jul 13, 2016
  31. 3/7 index-pack: correct "len" type in unpack_data()Nguyễn Thái Ngọc Duy, Jul 13, 2016
  32. 2/7 sha1_file.c: use type off_t* for object_info->disk_sizepNguyễn Thái Ngọc Duy, Jul 13, 2016
  33. 5/7 index-pack: correct "offset" type in unpack_entry_data()Nguyễn Thái Ngọc Duy, Jul 13, 2016
  34. 7/7 fsck: use streaming interface for large blobs in packNguyễn Thái Ngọc Duy, Jul 13, 2016
  35. 6/7 pack-objects: do not truncate result in-pack object size on 32-bit systemsNguyễn Thái Ngọc Duy, Jul 13, 2016
  36. 1/7 pack-objects: pass length to check_pack_crc() without truncationNguyễn Thái Ngọc Duy, Jul 13, 2016
  37. Junio C HamanoJul 13, 2016
  38. Christoph MichelbachJul 6, 2016
  39. Duy NguyenJul 6, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.