[PATCH v5 0/7] [PATCH v4 0/7] Add delta islands support

From: Christian Couder <christian.couder@gmail.com>
Date: Aug 16, 2018, 06:13 UTC
Message-ID: <20180816061313.19298-1-chriscool@tuxfamily.org>

This patch series is upstreaming work made by GitHub and available in:

https://github.com/peff/git/commits/jk/delta-islands

The above work has been already described in the following article:

https://githubengineering.com/counting-objects/

The above branch contains only one patch. In this patch series the patch has been split into 4 patches (1/7, 3/7, 4/7 and 5/7) with their own commit message, and on top of that 3 new patches (2/7, 6/7 and 7/7) have been added. The new patches implement things that have been requested on the mailing list.

I kept Peff as the author of the original 4 patches and took the liberty to add his Signed-off-by to them.

As explained in details in the "Counting Objects" article referenced above, the goal of the delta island mechanism is for a hosting provider to make it possible to have the "forks" of a repository share as much storage as possible while preventing object packs to contain deltas between different forks.

If deltas between different forks are not prevented, when users clone or fetch a fork, preparing the pack that should be sent to them can be very costly CPU wise, as objects from a different fork should not be sent, which means that a lot of deltas might need to be computed again (instead of reusing existing deltas).

The following changes have been made since the previous iteration:

* suggested by Ramsay: fix typo in "deltified" in commit message in
  patch 1/7

* suggested by Ramsay and Peff: use FLEX_ARRAY when defining 'struct
  island_bitmap' in delta-islands.c in patch 1/7

The diff against v4 is:

=======

diff --git a/delta-islands.c b/delta-islands.c
index 2ced34d99c..8e5018e406 100644
--- a/delta-islands.c
+++ b/delta-islands.c
@@ -35,7 +35,7 @@ struct remote_island {
 
 struct island_bitmap {
        uint32_t refcount;
-       uint32_t bits[];
+       uint32_t bits[FLEX_ARRAY];
 };
 
 static uint32_t island_bitmap_size;
=======

This patch series is also available on GitHub in:

https://github.com/chriscool/git/commits/delta-islands

The previous versions are available there:

V4: https://github.com/chriscool/git/commits/delta-islands32
V3: https://github.com/chriscool/git/commits/delta-islands29
V2: https://github.com/chriscool/git/commits/delta-islands19
V1: https://github.com/chriscool/git/commits/delta-islands6

V4: https://public-inbox.org/git/20180812051151.6425-1-chriscool@tuxfamily.org/
V3: https://public-inbox.org/git/20180809155532.26151-1-chriscool@tuxfamily.org/
V2: https://public-inbox.org/git/20180805172525.15278-1-chriscool@tuxfamily.org/
V1: https://public-inbox.org/git/20180722054836.28935-1-chriscool@tuxfamily.org/


Christian Couder (3):
  pack-objects: refactor code into compute_layer_order()
  pack-objects: move tree_depth into 'struct packing_data'
  pack-objects: move 'layer' into 'struct packing_data'

Jeff King (4):
  Add delta-islands.{c,h}
  pack-objects: add delta-islands support
  repack: add delta-islands support
  t: add t5319-delta-islands.sh

 Documentation/config.txt           |  19 ++
 Documentation/git-pack-objects.txt |  97 ++++++
 Documentation/git-repack.txt       |   5 +
 Makefile                           |   1 +
 builtin/pack-objects.c             | 137 +++++---
 builtin/repack.c                   |   9 +
 delta-islands.c                    | 502 +++++++++++++++++++++++++++++
 delta-islands.h                    |  11 +
 pack-objects.c                     |  12 +
 pack-objects.h                     |  39 +++
 t/t5319-delta-islands.sh           | 143 ++++++++
 11 files changed, 932 insertions(+), 43 deletions(-)
 create mode 100644 delta-islands.c
 create mode 100644 delta-islands.h
 create mode 100755 t/t5319-delta-islands.sh

-- 
2.18.0.673.gcd86e60100