{"thread":{"id":"8058","subject":"[PATCH v3] Custom compression levels for objects and packs","startedAt":"2007-05-09T20:56:50Z","lastAt":"2007-05-09T20:56:50Z","messageCount":1,"participants":["Dana How"],"isPatch":true,"patchVersion":3,"patchTotal":null},"messages":[{"id":"41664","messageId":"46423592.1050506@gmail.com","threadId":"8058","inReplyTo":null,"subject":"[PATCH v3] Custom compression levels for objects and packs","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-09T20:56:50Z","receivedAt":"2007-05-09T20:56:50Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"\nAdd config variables pack.compression and core.loosecompression ,\nand switch --compression=level to pack-objects.\n\nLoose objects will be compressed using core.loosecompression if set,\nelse core.compression if set, else Z_BEST_SPEED.\nPacked objects will be compressed using --compression=level if seen,\nelse pack.compression if set, else core.compression if set,\nelse Z_DEFAULT_COMPRESSION.  This is the \"pack compression level\".\n\nLoose objects added to a pack undeltified will be recompressed\nto the pack compression level if it is unequal to the current\nloose compression level by the preceding rules,  or if the loose\nobject was written while core.legacyheaders = true.  Newly\ndeltified loose objects are always compressed to the current\npack compression level.\n\nPreviously packed objects added to a pack are recompressed\nto the current pack compression level exactly when their\ndeltification status changes,  since the previous pack data\ncannot be reused.\n\nIn either case,  the --no-reuse-object switch from the first\npatch below will always force recompression to the current pack\ncompression level,  instead of assuming the pack compression level\nhasn't changed and pack data can be reused when possible.\n\nThis applies on top of the following patches from Nicolas Pitre:\n[PATCH] allow for undeltified objects not to be reused\n[PATCH] make \"repack -f\" imply \"pack-objects --no-reuse-object\"\n\nSigned-off-by: Dana L. How <danahow@gmail.com>\n---\n Documentation/config.txt           |   17 +++++++++++++++--\n Documentation/git-pack-objects.txt |   14 +++++++++++++-\n builtin-pack-objects.c             |   33 +++++++++++++++++++++++++++++++--\n cache.h                            |    2 ++\n config.c                           |   18 +++++++++++++++++-\n csum-file.c                        |    4 ++--\n csum-file.h                        |    2 +-\n environment.c                      |    4 +++-\n 8 files changed, 84 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ea434af..382a31b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -204,10 +204,16 @@ core.warnAmbiguousRefs::\n \tand might match multiple refs in the .git/refs/ tree. True by default.\n \n core.compression::\n+\tAn integer -1..9, indicating a default compression level.\n+\t-1 is the zlib default. 0 means no compression,\n+\tand 1..9 are various speed/size tradeoffs, 9 being slowest.\n+\n+core.loosecompression::\n \tAn integer -1..9, indicating the compression level for objects that\n-\tare not in a pack file. -1 is the zlib and git default. 0 means no\n+\tare not in a pack file. -1 is the zlib default. 0 means no\n \tcompression, and 1..9 are various speed/size tradeoffs, 9 being\n-\tslowest.\n+\tslowest.  If not set,  defaults to core.compression.  If that is\n+\tnot set,  defaults to 0 (best speed).\n \n core.legacyheaders::\n \tA boolean which\n@@ -550,6 +556,13 @@ pack.depth::\n \tThe maximum delta depth used by gitlink:git-pack-objects[1] when no\n \tmaximum depth is given on the command line. Defaults to 50.\n \n+pack.compression::\n+\tAn integer -1..9, indicating the compression level for objects\n+\tin a pack file. -1 is the zlib default. 0 means no\n+\tcompression, and 1..9 are various speed/size tradeoffs, 9 being\n+\tslowest.  If not set,  defaults to core.compression.  If that is\n+\tnot set,  defaults to -1.\n+\n pull.octopus::\n \tThe default merge strategy to use when pulling multiple branches\n \tat once.\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex ce89214..2531238 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -130,10 +130,22 @@ base-name::\n --no-reuse-object::\n \tThis flag tells the command not to reuse existing object data at all,\n \tincluding non deltified object, forcing recompression of everything.\n-\tThis implies --no-reuse-delta. Useful only in the obscur case where\n+\tThis implies --no-reuse-delta. Useful only in the obscure case where\n \twholesale enforcement of a different compression level on the\n \tpacked data is desired.\n \n+--compression=[N]::\n+\tSpecifies compression level for newly-compressed data in the\n+\tgenerated pack.  If not specified,  pack compression level is\n+\tdetermined first by pack.compression,  then by core.compression,\n+\tand defaults to -1,  the zlib default,  if neither is set.\n+\tData copied from loose objects will be recompressed\n+\tif core.legacyheaders was true when they were created or if\n+\tthe loose compression level (see core.loosecompression and\n+\tcore.compression) is now a different value than the pack\n+\tcompression level.  Add --no-reuse-object if you want to force\n+\ta uniform compression level on all data no matter the source.\n+\n --delta-base-offset::\n \tA packed archive can express base object of a delta as\n \teither 20-byte object name or as an offset in the\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex d94c79a..5c468aa 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -68,6 +68,8 @@ static int depth = 50;\n static int pack_to_stdout;\n static int num_preferred_base;\n static struct progress progress_state;\n+static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n+static int pack_compression_seen;\n \n /*\n  * The object names in objects array are hashed with this hashtable,\n@@ -427,7 +429,10 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t\t * and we do not need to deltify it.\n \t\t\t\t */\n \n-\tif (!no_reuse_object && !entry->in_pack && !entry->delta) {\n+\t/* differing core & pack compression when loose object -> must recompress */\n+\tif (!entry->in_pack && pack_compression_level != zlib_compression_level)\n+\t\tto_reuse = 0;\n+\telse if (!no_reuse_object && !entry->in_pack && !entry->delta) {\n \t\tunsigned char *map;\n \t\tunsigned long mapsize;\n \t\tmap = map_sha1_file(entry->sha1, &mapsize);\n@@ -487,7 +492,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\tsha1write(f, entry->delta->sha1, 20);\n \t\t\thdrlen += 20;\n \t\t}\n-\t\tdatalen = sha1write_compressed(f, buf, size);\n+\t\tdatalen = sha1write_compressed(f, buf, size, pack_compression_level);\n \t\tfree(buf);\n \t}\n \telse {\n@@ -1496,6 +1501,16 @@ static int git_pack_config(const char *k, const char *v)\n \t\tdepth = git_config_int(k, v);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.compression\")) {\n+\t\tint level = git_config_int(k, v);\n+\t\tif (level == -1)\n+\t\t\tlevel = Z_DEFAULT_COMPRESSION;\n+\t\telse if (level < 0 || level > Z_BEST_COMPRESSION)\n+\t\t\tdie(\"bad pack compression level %d\", level);\n+\t\tpack_compression_level = level;\n+\t\tpack_compression_seen = 1;\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(k, v);\n }\n \n@@ -1607,6 +1622,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \trp_ac = 2;\n \n \tgit_config(git_pack_config);\n+\tif (!pack_compression_seen && core_compression_seen)\n+\t\tpack_compression_level = core_compression_level;\n \n \tprogress = isatty(2);\n \tfor (i = 1; i < argc; i++) {\n@@ -1627,6 +1644,18 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\tincremental = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--compression=\")) {\n+\t\t\tchar *end;\n+\t\t\tint level = strtoul(arg+14, &end, 0);\n+\t\t\tif (!arg[14] || *end)\n+\t\t\t\tusage(pack_usage);\n+\t\t\tif (level == -1)\n+\t\t\t\tlevel = Z_DEFAULT_COMPRESSION;\n+\t\t\telse if (level < 0 || level > Z_BEST_COMPRESSION)\n+\t\t\t\tdie(\"bad pack compression level %d\", level);\n+\t\t\tpack_compression_level = level;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!prefixcmp(arg, \"--window=\")) {\n \t\t\tchar *end;\n \t\t\twindow = strtoul(arg+9, &end, 0);\ndiff --git a/cache.h b/cache.h\nindex 8e76152..2b3f359 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -283,6 +283,8 @@ extern int warn_ambiguous_refs;\n extern int shared_repository;\n extern const char *apply_default_whitespace;\n extern int zlib_compression_level;\n+extern int core_compression_level;\n+extern int core_compression_seen;\n extern size_t packed_git_window_size;\n extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\ndiff --git a/config.c b/config.c\nindex 70d1055..5627ed6 100644\n--- a/config.c\n+++ b/config.c\n@@ -12,6 +12,8 @@\n static FILE *config_file;\n static const char *config_file_name;\n static int config_linenr;\n+static int zlib_compression_seen;\n+\n static int get_next_char(void)\n {\n \tint c;\n@@ -304,13 +306,27 @@ int git_default_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n-\tif (!strcmp(var, \"core.compression\")) {\n+\tif (!strcmp(var, \"core.loosecompression\")) {\n \t\tint level = git_config_int(var, value);\n \t\tif (level == -1)\n \t\t\tlevel = Z_DEFAULT_COMPRESSION;\n \t\telse if (level < 0 || level > Z_BEST_COMPRESSION)\n \t\t\tdie(\"bad zlib compression level %d\", level);\n \t\tzlib_compression_level = level;\n+\t\tzlib_compression_seen = 1;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(var, \"core.compression\")) {\n+\t\tint level = git_config_int(var, value);\n+\t\tif (level == -1)\n+\t\t\tlevel = Z_DEFAULT_COMPRESSION;\n+\t\telse if (level < 0 || level > Z_BEST_COMPRESSION)\n+\t\t\tdie(\"bad zlib compression level %d\", level);\n+\t\tcore_compression_level = level;\n+\t\tcore_compression_seen = 1;\n+\t\tif (!zlib_compression_seen)\n+\t\t\tzlib_compression_level = level;\n \t\treturn 0;\n \t}\n \ndiff --git a/csum-file.c b/csum-file.c\nindex 7c806ad..7088f6e 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -119,14 +119,14 @@ struct sha1file *sha1fd(int fd, const char *name)\n \treturn f;\n }\n \n-int sha1write_compressed(struct sha1file *f, void *in, unsigned int size)\n+int sha1write_compressed(struct sha1file *f, void *in, unsigned int size, int level)\n {\n \tz_stream stream;\n \tunsigned long maxsize;\n \tvoid *out;\n \n \tmemset(&stream, 0, sizeof(stream));\n-\tdeflateInit(&stream, zlib_compression_level);\n+\tdeflateInit(&stream, level);\n \tmaxsize = deflateBound(&stream, size);\n \tout = xmalloc(maxsize);\n \ndiff --git a/csum-file.h b/csum-file.h\nindex 7e13391..4e8b83e 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -16,7 +16,7 @@ extern struct sha1file *sha1fd(int fd, const char *name);\n extern struct sha1file *sha1create(const char *fmt, ...) __attribute__((format (printf, 1, 2)));\n extern int sha1close(struct sha1file *, unsigned char *, int);\n extern int sha1write(struct sha1file *, void *, unsigned int);\n-extern int sha1write_compressed(struct sha1file *, void *, unsigned int);\n+extern int sha1write_compressed(struct sha1file *, void *, unsigned int, int);\n extern void crc32_begin(struct sha1file *);\n extern uint32_t crc32_end(struct sha1file *);\n \ndiff --git a/environment.c b/environment.c\nindex 2231659..b7aeb1a 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -24,7 +24,9 @@ const char *git_commit_encoding;\n const char *git_log_output_encoding;\n int shared_repository = PERM_UMASK;\n const char *apply_default_whitespace;\n-int zlib_compression_level = Z_DEFAULT_COMPRESSION;\n+int zlib_compression_level = Z_BEST_SPEED;\n+int core_compression_level;\n+int core_compression_seen;\n size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;\n size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;\n size_t delta_base_cache_limit = 16 * 1024 * 1024;\n-- \n1.5.2.rc0.787.g0014\n"}]}