git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCHv3 4/9] pack-objects: Teach new option --max-object-count, similar to --max-pack-size

From
Johan Herland <johan@herland.net>
Date
May 15, 2011, 21:37 UTC
Message-ID
<1305495440-30836-5-git-send-email-johan@herland.net>
In-Reply-To
<1305495440-30836-1-git-send-email-johan@herland.net>

The new --max-object-count option behaves similarly to --max-pack-size, except that the decision to split packs is determined by the number of objects in the pack, and not by the size of the pack.

The new option also has a corresponding configuration variable, named pack.objectCountLimit, which works similarly to pack.packSizeLimit, subject to the difference mentioned above.

As with --max-pack-size, you can use --max-object-count together with --stdout to put a limit on the number of objects in the pack written to stdout. If the pack would exceed this limit, pack-objects will abort with an error message.

Finally, for completeness, the new option is also added to git-repack, which simply forwards it to pack-objects

Documentation and tests are included.
Signed-off-by: Johan Herland <johan@herland.net>
---
 Documentation/config.txt           |    8 ++++++
 Documentation/git-pack-objects.txt |    9 +++++++
 Documentation/git-repack.txt       |    6 +++++
 builtin/pack-objects.c             |   20 ++++++++++++++++
 git-repack.sh                      |   27 +++++++++++----------
 t/t5300-pack-object.sh             |   44 ++++++++++++++++++++++++++++++++++++
 6 files changed, 101 insertions(+), 13 deletions(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 285c7f7..056e01f 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1523,6 +1523,14 @@ pack.packSizeLimit::
 	Common unit suffixes of 'k', 'm', or 'g' are
 	supported.
 
+pack.objectCountLimit::
+	The maximum number of objects in a pack.  This setting only
+	affects packing to a file when repacking, i.e. the git://
+	protocol is unaffected.  It can be overridden by the
+	`\--max-object-count` option of linkgit:git-repack[1].
+	The default is unlimited. Common unit suffixes of 'k', 'm',
+	or 'g' are supported.
+
 pager.<cmd>::
 	If the value is boolean, turns on or off pagination of the
 	output of a particular git subcommand when writing to a tty.
diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt
index ca97463..5ab1fe9 100644
--- a/Documentation/git-pack-objects.txt
+++ b/Documentation/git-pack-objects.txt
@@ -116,6 +116,15 @@ base-name::
 When used together with --stdout, the command will fail with an error
 message if the pack output exceeds the given limit.
 
+--max-object-count=<n>::
+	Maximum number of objects in each output pack file. The number
+	can be suffixed with "k", "m", or "g". If specified, multiple
+	packfiles may be created. The default is unlimited, unless
+	the config variable `pack.objectCountLimit` is set.
++
+When used together with --stdout, the command will fail with an error
+message if the pack output exceeds the given limit.
+
 --honor-pack-keep::
 	This flag causes an object already in a local pack that
 	has a .keep file to be ignored, even if it would have
diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt
index 0decee2..f9a44b8 100644
--- a/Documentation/git-repack.txt
+++ b/Documentation/git-repack.txt
@@ -109,6 +109,12 @@ other objects in that pack they already have locally.
 	The default is unlimited, unless the config variable
 	`pack.packSizeLimit` is set.
 
+--max-object-count=<n>::
+	Maximum number of objects in each output pack file. The number
+	can be suffixed with "k", "m", or "g". If specified, multiple
+	packfiles may be created. The default is unlimited, unless
+	the config variable `pack.objectCountLimit` is set.
+
 
 Configuration
 -------------
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 69f1c51..c4fbc54 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -74,6 +74,7 @@ static const char *base_name;
 static int progress = 1;
 static int window = 10;
 static unsigned long pack_size_limit, pack_size_limit_cfg;
+static unsigned long object_count_limit, object_count_limit_cfg;
 static int depth = 50;
 static int delta_search_threads;
 static int pack_to_stdout;
@@ -227,6 +228,10 @@ static unsigned long write_object(struct sha1file *f,
 	else
 		limit = pack_size_limit - write_offset;
 
+	/* Trigger new pack when we reach object count limit */
+	if (object_count_limit && nr_written >= object_count_limit)
+		return 0;
+
 	if (!entry->delta)
 		usable_delta = 0;	/* no delta */
 	else if (!pack_size_limit || pack_to_stdout)
@@ -1897,6 +1902,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)
 		pack_size_limit_cfg = git_config_ulong(k, v);
 		return 0;
 	}
+	if (!strcmp(k, "pack.objectcountlimit")) {
+		object_count_limit_cfg = git_config_ulong(k, v);
+		return 0;
+	}
 	return git_default_config(k, v, cb);
 }
 
@@ -2182,6 +2191,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 				usage(pack_usage);
 			continue;
 		}
+		if (!prefixcmp(arg, "--max-object-count=")) {
+			object_count_limit_cfg = 0;
+			if (!git_parse_ulong(arg+19, &object_count_limit))
+				usage(pack_usage);
+			continue;
+		}
 		if (!prefixcmp(arg, "--window=")) {
 			char *end;
 			window = strtoul(arg+9, &end, 0);
@@ -2322,6 +2337,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		warning("minimum pack size limit is 1 MiB");
 		pack_size_limit = 1024*1024;
 	}
+	if (!pack_to_stdout && !object_count_limit)
+		object_count_limit = object_count_limit_cfg;
 
 	if (!pack_to_stdout && thin)
 		die("--thin cannot be used to build an indexable pack.");
@@ -2349,6 +2366,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 
 	if (non_empty && !nr_result)
 		return 0;
+	if (pack_to_stdout && object_count_limit && object_count_limit < nr_result)
+		die("unable to make pack within the object count limit"
+			" (%lu objects)", object_count_limit);
 	if (nr_result)
 		prepare_pack(window, depth);
 	write_pack_file();
diff --git a/git-repack.sh b/git-repack.sh
index 624feec..e8693a6 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -7,19 +7,20 @@ OPTIONS_KEEPDASHDASH=
 OPTIONS_SPEC="\
 git repack [options]
 --
-a               pack everything in a single pack
-A               same as -a, and turn unreachable objects loose
-d               remove redundant packs, and run git-prune-packed
-f               pass --no-reuse-delta to git-pack-objects
-F               pass --no-reuse-object to git-pack-objects
-n               do not run git-update-server-info
-q,quiet         be quiet
-l               pass --local to git-pack-objects
+a                  pack everything in a single pack
+A                  same as -a, and turn unreachable objects loose
+d                  remove redundant packs, and run git-prune-packed
+f                  pass --no-reuse-delta to git-pack-objects
+F                  pass --no-reuse-object to git-pack-objects
+n                  do not run git-update-server-info
+q,quiet            be quiet
+l                  pass --local to git-pack-objects
  Packing constraints
-window=         size of the window used for delta compression
-window-memory=  same as the above, but limit memory size instead of entries count
-depth=          limits the maximum delta depth
-max-pack-size=  maximum size of each packfile
+window=            size of the window used for delta compression
+window-memory=     same as the above, but limit memory size instead of entries count
+depth=             limits the maximum delta depth
+max-pack-size=     maximum size of each packfile
+max-object-count=  maximum number of objects in each packfile
 "
 SUBDIRECTORY_OK='Yes'
 . git-sh-setup
@@ -38,7 +39,7 @@ do
 	-f)	no_reuse=--no-reuse-delta ;;
 	-F)	no_reuse=--no-reuse-object ;;
 	-l)	local=--local ;;
-	--max-pack-size|--window|--window-memory|--depth)
+	--max-pack-size|--max-object-count|--window|--window-memory|--depth)
 		extra="$extra $1=$2"; shift ;;
 	--) shift; break;;
 	*)	usage ;;
diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh
index 00f1bd8..d133c28 100755
--- a/t/t5300-pack-object.sh
+++ b/t/t5300-pack-object.sh
@@ -439,6 +439,50 @@ test_expect_success '--stdout fails when pack exceeds --max-pack-size' '
 	grep -q "pack size limit" errs
 '
 
+test_expect_success 'honor pack.objectCountLimit' '
+	git config pack.objectCountLimit 5 &&
+	packname_17=$(git pack-objects test-17 <obj-list) &&
+	test 2 = $(ls test-17-*.pack | wc -l)
+'
+
+test_expect_success 'verify resulting packs' '
+	git verify-pack test-17-*.pack
+'
+
+test_expect_success '--stdout ignores pack.objectCountLimit' '
+	git pack-objects --stdout <obj-list >test-18.pack &&
+	git index-pack --strict test-18.pack
+'
+
+test_expect_success 'verify resulting pack' '
+	git verify-pack test-18.pack
+'
+
+test_expect_success 'honor --max-object-count' '
+	git config --unset pack.objectCountLimit &&
+	packname_19=$(git pack-objects --max-object-count=5 test-19 <obj-list) &&
+	test 2 = $(ls test-19-*.pack | wc -l)
+'
+
+test_expect_success 'verify resulting packs' '
+	git verify-pack test-19-*.pack
+'
+
+test_expect_success '--stdout works with large enough --max-object-count' '
+	git pack-objects --stdout --max-object-count=8 <obj-list >test-20.pack &&
+	git index-pack --strict test-20.pack
+'
+
+test_expect_success 'verify resulting pack' '
+	git verify-pack test-20.pack
+'
+
+test_expect_success '--stdout fails when pack exceeds --max-object-count' '
+	test_must_fail git pack-objects --stdout --max-object-count=1 <obj-list >test-21.pack 2>errs &&
+	test_must_fail git index-pack --strict test-21.pack &&
+	grep -q "object count limit" errs
+'
+
 #
 # WARNING!
 #

-- 
1.7.5.rc1.3.g4d7b
Previous: Johan HerlandNext: Shawn Pearce
Message 31 of 53 in “receive-pack: Add receive.denyObjectLimit to refuse push with too many objects”
  1. 2/2 receive-pack: Add receive.denyObjectLimit to refuse push with too many objectsJohan Herland, May 13, 2011
  2. Junio C HamanoMay 13, 2011
  3. Johan HerlandMay 14, 2011
  4. 2/2 receive-pack: Add receive.objectCountLimit to refuse push with too many objectsJohan Herland, May 14, 2011
  5. Shawn PearceMay 14, 2011
  6. Johan HerlandMay 14, 2011
  7. Shawn PearceMay 14, 2011
  8. Johan HerlandMay 15, 2011
  9. 0/9 Push limitsJohan Herland, May 15, 2011
  10. 1/9 Update technical docs to reflect side-band-64k capability in receive-packJohan Herland, May 15, 2011
  11. 2/9 send-pack: Attempt to retrieve remote status even if pack-objects failsJohan Herland, May 15, 2011
  12. Jeff KingMay 16, 2011
  13. Jeff KingMay 16, 2011
  14. Jeff KingMay 16, 2011
  15. 1/3 connect: treat generic proxy processes like ssh processesJeff King, May 16, 2011
  16. Johannes SixtMay 16, 2011
  17. Junio C HamanoMay 16, 2011
  18. Jeff KingMay 17, 2011
  19. Johannes SixtMay 17, 2011
  20. Jeff KingMay 18, 2011
  21. 2/3 connect: let callers know if connection is a socketJeff King, May 16, 2011
  22. 3/3 send-pack: avoid deadlock on git:// push with failed pack-objectsJeff King, May 16, 2011
  23. Johannes SixtMay 16, 2011
  24. Jeff KingMay 17, 2011
  25. Windows: add a wrapper for the shutdown() system callJohannes Sixt, May 18, 2011
  26. 3/9 pack-objects: Allow --max-pack-size to be used together with --stdoutJohan Herland, May 15, 2011
  27. Shawn PearceMay 15, 2011
  28. Johan HerlandMay 16, 2011
  29. Junio C HamanoMay 16, 2011
  30. Johan HerlandMay 16, 2011
  31. 4/9 pack-objects: Teach new option --max-object-count, similar to --max-pack-sizeJohan Herland, May 15, 2011
  32. Shawn PearceMay 15, 2011
  33. Johan HerlandMay 15, 2011
  34. Shawn PearceMay 15, 2011
  35. Junio C HamanoMay 16, 2011
  36. Johan HerlandMay 16, 2011
  37. 5/9 pack-objects: Teach new option --max-commit-count, limiting #commits in packJohan Herland, May 15, 2011
  38. 6/9 receive-pack: Prepare for addition of the new 'limit-*' family of capabilitiesJohan Herland, May 15, 2011
  39. Junio C HamanoMay 16, 2011
  40. Johan HerlandMay 16, 2011
  41. Sverre RabbelierMay 16, 2011
  42. Junio C HamanoMay 16, 2011
  43. Sverre RabbelierMay 16, 2011
  44. Junio C HamanoMay 16, 2011
  45. Sverre RabbelierMay 16, 2011
  46. 7/9 send-pack/receive-pack: Allow server to refuse pushes with too many objectsJohan Herland, May 15, 2011
  47. 8/9 send-pack/receive-pack: Allow server to refuse pushing too large packsJohan Herland, May 15, 2011
  48. 9/9 send-pack/receive-pack: Allow server to refuse pushes with too many commitsJohan Herland, May 15, 2011
  49. Ævar Arnfjörð BjarmasonMay 15, 2011
  50. Junio C HamanoMay 14, 2011
  51. Shawn PearceMay 14, 2011
  52. Johannes SixtMay 13, 2011
  53. Johan HerlandMay 14, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.