git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: serious performance issues with images, audio files, and other "non-code" data

From
Jakub Narebski <jnareb@gmail.com>
Date
May 12, 2010, 19:15 UTC
Message-ID
<m339xxdj71.fsf@localhost.localdomain>
In-Reply-To
<4BEAF941.6040609@puckerupgames.com>
John <john@puckerupgames.com> writes:
Show 20 quoted lines
> We're seeing serious performance issues with repos that store media
> files, even relatively small files. For example, a web site with less
> than 100 MB of images can take minutes to commit, push, or pull when
> images have changed.
> 
> Our first guess was that git is repeatedly attempting to
> compress/decompress data that had already been compressed. We tried
> these configuration settings (shooting in the dark) to no avail:
> 
>     core.compression 0   ## Docs say this disables compression. Didn't seem to work.
>     pack.depth 1     ## Unclear what this does.
>     pack.window 0    ## No idea what this does.
>     gc.auto 0        ## We hope this disables automatic packing.
> 
> Our guess that re-compression is to blame may not even be valid since
> we can manually re-compress these files in seconds, not minutes.
> 
> Is there a trick to getting git to simply "copy files as is"?  In
> other words, don't attempt to compress them, don't attempt to "diff"
> them, just store/copy/transfer the files as-is?

Search for `delta` attribute, which should be unset for files that you don't want for git to attempt (binary) delta against, in gitattributes manpage.

P.S. There is also git-bigfiles project that migth be of interest to you.

-- 
Jakub Narebski
Poland
ShadeHawk on #git
Previous: JohnNext: Jeff King
Message 2 of 28 in “serious performance issues with images, audio files, and other "non-code" data”
  1. JohnMay 12, 2010
  2. Jakub NarebskiMay 12, 2010
  3. Jeff KingMay 14, 2010
  4. JohnMay 14, 2010
  5. Dirk SüsserottMay 14, 2010
  6. Jeff KingMay 17, 2010
  7. Sverre RabbelierMay 17, 2010
  8. Jeff KingMay 18, 2010
  9. Sverre RabbelierMay 18, 2010
  10. Jeff KingMay 18, 2010
  11. Nicolas PitreMay 18, 2010
  12. JohnMay 18, 2010
  13. Sverre RabbelierMay 18, 2010
  14. Jeff KingMay 18, 2010
  15. Nicolas PitreMay 18, 2010
  16. Jeff KingMay 18, 2010
  17. Nicolas PitreMay 18, 2010
  18. JohnMay 24, 2010
  19. Junio C HamanoMay 24, 2010
  20. JohnMay 24, 2010
  21. Jeff KingMay 25, 2010
  22. Michael J GruberMay 25, 2010
  23. JohnMay 25, 2010
  24. Nicolas PitreMay 25, 2010
  25. JohnMay 25, 2010
  26. Jeff KingMay 24, 2010
  27. JohnMay 24, 2010
  28. Jeff KingMay 24, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.