git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: git-fetching from a big repository is slow

From
Shawn Pearce <spearce@spearce.org>
Date
Dec 14, 2006, 19:46 UTC
Message-ID
<20061214194636.GO1747@spearce.org>
In-Reply-To
<C287764F-6755-4291-A87A-3E8816E90B49@adacore.com>
Geert Bosch <bosch@adacore.com> wrote:
Show 20 quoted lines
> Such special magic based on filenames is always a bad idea. Tomorrow  
> somebody
> comes with .zip files (oh, and of course .ZIP), then it's .jpg's other
> compressed content. In the end git will be doing lots of magic and  
> still perform
> badly on unknown compressed content.
> 
> There is a very simple way of detecting compressed files: just look  
> at the
> size of the compressed blob and compare against the size of the  
> expanded blob.
> If the compressed blob has a non-trivial size which is close to the  
> expanded
> size, assume the file is not interesting as source or target for deltas.
> 
> Example:
>    if (compressed_size > expanded_size / 4 * 3 + 1024) {
>      /* don't try to deltify if blob doesn't compress well */
>      return ...;
>    }

And yet I get good delta compression on a number of ZIP formatted files which don't get good additional zlib compression (<3%). Doing the above would cause those packfiles to explode to about 10x their current size.

Next: Geert Bosch
Message 1 of 11 in “Re: git-fetching from a big repository is slow”
  1. Shawn PearceDec 14, 2006
  2. Geert BoschDec 14, 2006
  3. Johannes SchindelinDec 14, 2006
  4. Shawn PearceDec 14, 2006
  5. Johannes SchindelinDec 15, 2006
  6. Shawn PearceDec 15, 2006
  7. Nicolas PitreDec 15, 2006
  8. Horst H. von BrandDec 14, 2006
  9. Shawn PearceDec 14, 2006
  10. PazuDec 15, 2006
  11. Robin RosenbergDec 16, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.