git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Problem with large files on different OSes

From
Nicolas Pitre <nico@cam.org>
Date
May 28, 2009, 03:26 UTC
Message-ID
<alpine.LFD.2.00.0905272312370.3906@xanadu.home>
In-Reply-To
<alpine.LFD.2.01.0905271825520.3435@localhost.localdomain>
On Wed, 27 May 2009, Linus Torvalds wrote:
Show 22 quoted lines
> Something like the following may also work, as a more generic "just don't 
> even bother trying to delta huge files".
> 
> Totally untested. Maybe it works. Maybe it doesn't.
> 
> 		Linus
> 
> ---
>  Documentation/config.txt |    7 +++++++
>  builtin-pack-objects.c   |    9 +++++++++
>  2 files changed, 16 insertions(+), 0 deletions(-)
> 
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 2c03162..8c21027 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -1238,6 +1238,13 @@ older version of git. If the `{asterisk}.pack` file is smaller than 2 GB, howeve
>  you can use linkgit:git-index-pack[1] on the *.pack file to regenerate
>  the `{asterisk}.idx` file.
>  
> +pack.packDeltaLimit::
> +	The default maximum size of objects that we try to delta.

The option name feels a bit wrong here, like if it meant the max number of deltas in a pack. Nothing better comes to my mind at the moment though.

Show 19 quoted lines
> diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
> index 9742b45..9a0072b 100644
> --- a/builtin-pack-objects.c
> +++ b/builtin-pack-objects.c
> @@ -85,6 +85,7 @@ static struct progress *progress_state;
>  static int pack_compression_level = Z_DEFAULT_COMPRESSION;
>  static int pack_compression_seen;
>  
> +static unsigned long pack_delta_limit = 64*1024*1024;
>  static unsigned long delta_cache_size = 0;
>  static unsigned long max_delta_cache_size = 0;
>  static unsigned long cache_max_small_delta_size = 1000;
> @@ -1270,6 +1271,10 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
>  	if (trg_entry->type != src_entry->type)
>  		return -1;
>  
> +	/* If we limit delta generation, don't even bother for larger blobs */
> +	if (pack_delta_limit && trg_entry->size >= pack_delta_limit)
> +		return -1;

I'd suggest filtering delta candidates out of delta_list up front in prepare_pack() instead.

Nicolas
Previous: Linus TorvaldsNext: Eric Raible
Message 20 of 31 in “Problem with large files on different OSes”
  1. Christopher JeffersonMay 27, 2009
  2. Andreas EricssonMay 27, 2009
  3. Christopher JeffersonMay 27, 2009
  4. John TapsellMay 27, 2009
  5. Christopher JeffersonMay 27, 2009
  6. John TapsellMay 27, 2009
  7. Tomas CarneckyMay 27, 2009
  8. Christopher JeffersonMay 27, 2009
  9. Andreas EricssonMay 27, 2009
  10. Jakub NarebskiMay 27, 2009
  11. Linus TorvaldsMay 27, 2009
  12. Linus TorvaldsMay 27, 2009
  13. Christopher JeffersonMay 27, 2009
  14. Jakub NarebskiMay 27, 2009
  15. Nicolas PitreMay 27, 2009
  16. Jeff KingMay 27, 2009
  17. Linus TorvaldsMay 27, 2009
  18. Alan Manuel GloriaMay 27, 2009
  19. Linus TorvaldsMay 28, 2009
  20. Nicolas PitreMay 28, 2009
  21. Eric RaibleMay 28, 2009
  22. Shawn O. PearceMay 28, 2009
  23. Eric RaibleMay 28, 2009
  24. Andreas EricssonMay 28, 2009
  25. Nicolas PitreMay 28, 2009
  26. Jeff KingMay 28, 2009
  27. Linus TorvaldsMay 28, 2009
  28. Nicolas PitreMay 27, 2009
  29. Jeff KingMay 28, 2009
  30. Nicolas PitreMay 28, 2009
  31. Jeff KingMay 28, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.