Re: [PATCH] git-pack-objects: cache small deltas between big objects

From: Dana How <danahow@gmail.com>
Date: May 21, 2007, 04:35 UTC
Message-ID: <56b7f5510705202135s8c9cd9qf4489b2b5bb2e264@mail.gmail.com>
In-Reply-To: <11796954641778-git-send-email-mkoegler@auto.tuwien.ac.at>

On 5/20/07, Martin Koegler <mkoegler@auto.tuwien.ac.at> wrote:

> Creating deltas between big blobs is a CPU and memory intensive task.
> In the writing phase, all (not reused) deltas are redone.

Actually, just the ones selected, which is approx 1/window. Do you have any numbers describing the effects on runtime and memory size for a known repo like linux-2.6?

Show 5 quoted lines

> This patch adds support for caching deltas from the deltifing phase, so
> that that the writing phase is faster.
>
> The caching is limited to small deltas to avoid increasing memory usage very much.
> The implemented limit is (memory needed to create the delta)/1024.

Your limit is applied per-object, and there is no overall limit on the amount of memory not freed in the delta phase. I suspect this caching would be disastrous for the large repo with "megablobs" I'm trying to wrestle with at the moment.

Show 12 quoted lines

> @@ -1294,10 +1302,17 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
>         trg_entry->delta = src_entry;
>         trg_entry->delta_size = delta_size;
>         trg_entry->depth = src_entry->depth + 1;
> -       free(delta_buf);
> +       /* cache delta, if objects are large enough compared to delta size */
> +       if ((src_size >> 20) + (trg_size >> 21) > (delta_size >> 10))
> +               trg_entry->delta_data = delta_buf;
> +       else
> +               free(delta_buf);
>         return 1;
>  }

-- 
Dana L. How  danahow@gmail.com  +1 650 804 5991 cell