git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Problem with large files on different OSes

From
CJChristopher Jefferson <caj@cs.st-andrews.ac.uk>
Date
May 27, 2009, 10:52 UTC
Message-ID
<submission.1M9Gk0-0000N8-MQ@mail.cs.st-andrews.ac.uk>

I recently came across a very annoying problem, characterised by the following example:

On a recent ubuntu install:

dd if=/dev/zero of=file bs=1300k count=1k git commit file -m "Add huge file"

The repository can be pulled and pushed successfully to other ubuntu installs, but on Mac OS X, 10.5.7 machine with 4GB ram git pull produces:

remote: Counting objects: 6, done. remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) failed (error code=12) remote: *** error: can't allocate region remote: *** set a breakpoint in malloc_error_break to debug remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) failed (error code=12) remote: *** error: can't allocate region remote: *** set a breakpoint in malloc_error_break to debug remote: fatal: Out of memory, malloc failed error: git upload-pack: git-pack-objects died with error. fatal: git upload-pack: aborting due to possible repository corruption on the remote side. remote: aborting due to possible repository corruption on the remote side. fatal: protocol error: bad pack header

The problem appears to be the different maximum mmap sizes available on different OSes. Whic I don't really mind the maximum file size restriction git imposes, this restriction varying from OS to OS is very annoying, fixing this required rewriting history to remove the commit, which caused problems as the commit had already been pulled, and built on, by a number of developers.

If the requirement that all files can be mmapped cannot be easily removed, would be it perhaps be acceptable to impose a (soft?) 1GB(ish) file size limit? I suggest 1GB as all the OSes I can get hold of easily (freeBSD, windows, Mac OS X, linux) support a mmap of size > 1GB.

Next: Andreas Ericsson
Message 1 of 31 in “Problem with large files on different OSes”
  1. Christopher JeffersonMay 27, 2009
  2. Andreas EricssonMay 27, 2009
  3. Christopher JeffersonMay 27, 2009
  4. John TapsellMay 27, 2009
  5. Christopher JeffersonMay 27, 2009
  6. John TapsellMay 27, 2009
  7. Tomas CarneckyMay 27, 2009
  8. Christopher JeffersonMay 27, 2009
  9. Andreas EricssonMay 27, 2009
  10. Jakub NarebskiMay 27, 2009
  11. Linus TorvaldsMay 27, 2009
  12. Linus TorvaldsMay 27, 2009
  13. Christopher JeffersonMay 27, 2009
  14. Jakub NarebskiMay 27, 2009
  15. Nicolas PitreMay 27, 2009
  16. Jeff KingMay 27, 2009
  17. Linus TorvaldsMay 27, 2009
  18. Alan Manuel GloriaMay 27, 2009
  19. Linus TorvaldsMay 28, 2009
  20. Nicolas PitreMay 28, 2009
  21. Eric RaibleMay 28, 2009
  22. Shawn O. PearceMay 28, 2009
  23. Eric RaibleMay 28, 2009
  24. Andreas EricssonMay 28, 2009
  25. Nicolas PitreMay 28, 2009
  26. Jeff KingMay 28, 2009
  27. Linus TorvaldsMay 28, 2009
  28. Nicolas PitreMay 27, 2009
  29. Jeff KingMay 28, 2009
  30. Nicolas PitreMay 28, 2009
  31. Jeff KingMay 28, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.