From: ZheNing Hu via GitGitGadget <gitgitgadget@gmail.com>
Date: Sat, 22 Oct 2022 15:46:04 GMT
Subject: [PATCH] pack-objects: introduce --exclude-delta=<pattern> option
Message-ID: <pull.1392.git.1666453564661.gitgitgadget@gmail.com>

From: ZheNing Hu <adlternative@gmail.com>

The server uses delta compression during git clone to reduce
the amount of data transferred over the network, but delta
compression for large binary blobs often does not reduce
storage size significantly and wastes a lot of CPU. Git now
disables delta compression for objects that meet these conditions:

1. files that have -delta set in .gitattributes
2. files that its size exceed the big_file_threshold

However, in 1, .gitattributes needs to be set manually by the user,
and in most cases the user does not actively set it, and it is not
something that can be actively adjusted on the server aside. In 2,
the big_file_threshold now defaults to 512MB, and many binary files
smaller than that will be uselessly delta-compressed, and this is
made worse if the server actively increases the big_file_threshold.

Therefore, we need a way to be able to actively skip the delta
compression of some files on the server. Introduces the
`-exclude-delta=<pattern>` option, which can be used to disable delta
compression for objects that satisfy the pattern.

Signed-off-by: ZheNing Hu <adlternative@gmail.com>
---
    pack-objects: introduce --exclude-delta= option
    
    While analyzing some repositories using git filter-repo -analyze, I
    noticed that many huge binaries in the repositories were
    delta-compressed without much reduction in size.
    
    $ cat .git/filter-repo/analysis/path-all-sizes.txt | more === All paths
    by reverse accumulated size === Format: unpacked size, packed size, date
    deleted, path name 23816778 23765921 2022-08-22
    managed/src/universal/ybc/ybc-1.0.0-b1-linux-x86_64.tar.gz 22504398
    22445676 2022-08-22
    managed/src/universal/ybc/ybc-1.0.0-b1-el8-aarch64.tar.gz 11726471
    6424233 2022-08-09 managed/yba-installer/yba-installer_linux_amd64
    294644800 5794201 src/yb/master/catalog_manager.cc 2912780 2872186
    docs/static/images/yp/tables-view-ycql.png 2992192 2634232
    docs/static/images/yb-cloud/cloud-clusters-backups.png 2757095 2501915
    docs/static/images/deploy/aws/aws-cf-configure-options.png ...
    
    The current solution to avoid delta compression is not very suitable for
    git servers. First, files that exceed the big_file_threshold are not
    delta compressed, but the above analysis indicates that many big binary
    files do not exceed the the big_file_threshold (default to 512MB).
    Second, there is not .gitattrbutes to disable delta compression for
    them, we also don't really can let repo administrators add it manually.
    
    But we can also see that the large files in these repositories often
    have some common characteristics: they end in ".tar.gz"or “.png". So
    perhaps we can take advantage of this feature and disable delta
    compression on the server for some common type binary files.
    
    This is currently implemented by command line parameters
    --exclude-delta=<pattern>. But maybe we can also try passing it through
    git config.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1392%2Fadlternative%2Fadl%2Fpack-object-no-try-delta-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1392/adlternative/adl/pack-object-no-try-delta-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1392

 Documentation/git-pack-objects.txt |  6 +++++-
 builtin/pack-objects.c             | 28 +++++++++++++++++++++++++++-
 2 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt
index a9995a932ca..92cfee83df5 100644
--- a/Documentation/git-pack-objects.txt
+++ b/Documentation/git-pack-objects.txt
@@ -13,7 +13,7 @@ SYNOPSIS
 	[--no-reuse-delta] [--delta-base-offset] [--non-empty]
 	[--local] [--incremental] [--window=<n>] [--depth=<n>]
 	[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
-	[--cruft] [--cruft-expiration=<time>]
+	[--cruft] [--cruft-expiration=<time>] [--exclude-delta=<file>]
 	[--stdout [--filter=<filter-spec>] | <base-name>]
 	[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>
 
@@ -221,6 +221,10 @@ depth is 4095.
 	This flag tells the command not to reuse existing deltas
 	but compute them from scratch.
 
+--exclude-delta=<pattern>::
+	Delta compression will not be attempted for blobs for paths
+	matching pattern. See linkgit:gitignore[5] for pattern details.
+
 --no-reuse-object::
 	This flag tells the command not to reuse existing object data at all,
 	including non deltified object, forcing recompression of everything.
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 3658c05cafc..ab9cff98e3a 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -272,6 +272,8 @@ static struct commit **indexed_commits;
 static unsigned int indexed_commits_nr;
 static unsigned int indexed_commits_alloc;
 
+static struct pattern_list *exclude_delta_patterns;
+
 static void index_commit_for_bitmap(struct commit *commit)
 {
 	if (indexed_commits_nr >= indexed_commits_alloc) {
@@ -1315,13 +1317,20 @@ static void write_pack_file(void)
 static int no_try_delta(const char *path)
 {
 	static struct attr_check *check;
+	int dtype;
 
 	if (!check)
 		check = attr_check_initl("delta", NULL);
 	git_check_attr(the_repository->index, path, check);
 	if (ATTR_FALSE(check->items[0].value))
 		return 1;
-	return 0;
+
+	return exclude_delta_patterns &&
+		path_matches_pattern_list(path,
+					  strlen(path),
+					  path, &dtype,
+					  exclude_delta_patterns,
+					  the_repository->index) == MATCHED;
 }
 
 /*
@@ -4149,6 +4158,19 @@ static int option_parse_cruft_expiration(const struct option *opt,
 	return 0;
 }
 
+static int option_parse_exclude_delta(const struct option *opt,
+					 const char *arg, int unset)
+{
+	BUG_ON_OPT_NEG(unset);
+
+	if (!exclude_delta_patterns)
+		exclude_delta_patterns = xcalloc(1, sizeof(*exclude_delta_patterns));
+
+	if (arg)
+		add_pattern(arg, "", 0, exclude_delta_patterns, 0);
+	return 0;
+}
+
 struct po_filter_data {
 	unsigned have_revs:1;
 	struct rev_info revs;
@@ -4242,6 +4264,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		OPT_CALLBACK_F(0, "cruft-expiration", NULL, N_("time"),
 		  N_("expire cruft objects older than <time>"),
 		  PARSE_OPT_OPTARG, option_parse_cruft_expiration),
+		OPT_CALLBACK_F(0, "exclude-delta", NULL, N_("pattern"),
+		  N_("disable delta compression for files matching pattern"),
+		  PARSE_OPT_NONEG, option_parse_exclude_delta),
 		OPT_BOOL(0, "sparse", &sparse,
 			 N_("use the sparse reachability algorithm")),
 		OPT_BOOL(0, "thin", &thin,
@@ -4514,6 +4539,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 
 cleanup:
 	strvec_clear(&rp);
+	FREE_AND_NULL(exclude_delta_patterns);
 
 	return 0;
 }

base-commit: 1fc3c0ad407008c2f71dd9ae1241d8b75f8ef886
-- 
gitgitgadget