git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 3/5] clone: add --bundle-uri option

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
Aug 9, 2022, 13:11 UTC
Message-ID
<00debaf6e77852efe1dcad4bfda5ebd5bf590ac4.1660050704.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1300.v3.git.1660050703.gitgitgadget@gmail.com>
From: Derrick Stolee <derrickstolee@github.com>

Cloning a remote repository is one of the most expensive operations in Git. The server can spend a lot of CPU time generating a pack-file for the client's request. The amount of data can clog the network for a long time, and the Git protocol is not resumable. For users with poor network connections or are located far away from the origin server, this can be especially painful.

Add a new '--bundle-uri' option to 'git clone' to bootstrap a clone from a bundle. If the user is aware of a bundle server, then they can tell Git to bootstrap the new repository with these bundles before fetching the remaining objects from the origin server.

Reviewed-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 Documentation/git-clone.txt |  6 ++++++
 builtin/clone.c             | 15 +++++++++++++++
 t/t5558-clone-bundle-uri.sh | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+)
 create mode 100755 t/t5558-clone-bundle-uri.sh
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 632bd1348ea..60fedf7eb5e 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -323,6 +323,12 @@ or `--mirror` is given)
 	for `host.xz:foo/.git`).  Cloning into an existing directory
 	is only allowed if the directory is empty.
 
+--bundle-uri=<uri>::
+	Before fetching from the remote, fetch a bundle from the given
+	`<uri>` and unbundle the data into the local repository. The refs
+	in the bundle will be stored under the hidden `refs/bundle/*`
+	namespace.
+
 :git-clone: 1
 include::urls.txt[]
 
diff --git a/builtin/clone.c b/builtin/clone.c
index c4ff4643ecd..4224d562758 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -34,6 +34,7 @@
 #include "list-objects-filter-options.h"
 #include "hook.h"
 #include "bundle.h"
+#include "bundle-uri.h"
 
 /*
  * Overall FIXMEs:
@@ -77,6 +78,7 @@ static int option_filter_submodules = -1;    /* unspecified */
 static int config_filter_submodules = -1;    /* unspecified */
 static struct string_list server_options = STRING_LIST_INIT_NODUP;
 static int option_remote_submodules;
+static const char *bundle_uri;
 
 static int recurse_submodules_cb(const struct option *opt,
 				 const char *arg, int unset)
@@ -160,6 +162,8 @@ static struct option builtin_clone_options[] = {
 		    N_("any cloned submodules will use their remote-tracking branch")),
 	OPT_BOOL(0, "sparse", &option_sparse_checkout,
 		    N_("initialize sparse-checkout file to include only files at root")),
+	OPT_STRING(0, "bundle-uri", &bundle_uri,
+		   N_("uri"), N_("a URI for downloading bundles before fetching from origin remote")),
 	OPT_END()
 };
 
@@ -1232,6 +1236,17 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	if (transport->smart_options && !deepen && !filter_options.choice)
 		transport->smart_options->check_self_contained_and_connected = 1;
 
+	/*
+	 * Before fetching from the remote, download and install bundle
+	 * data from the --bundle-uri option.
+	 */
+	if (bundle_uri) {
+		/* At this point, we need the_repository to match the cloned repo. */
+		repo_init(the_repository, git_dir, work_tree);
+		if (fetch_bundle_uri(the_repository, bundle_uri))
+			warning(_("failed to fetch objects from bundle URI '%s'"),
+				bundle_uri);
+	}
 
 	strvec_push(&transport_ls_refs_options.ref_prefixes, "HEAD");
 	refspec_ref_prefixes(&remote->fetch,
diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh
new file mode 100755
index 00000000000..f709bcb729c
--- /dev/null
+++ b/t/t5558-clone-bundle-uri.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description='test fetching bundles with --bundle-uri'
+
+. ./test-lib.sh
+
+test_expect_success 'fail to clone from non-existent file' '
+	test_when_finished rm -rf test &&
+	git clone --bundle-uri="$(pwd)/does-not-exist" . test 2>err &&
+	grep "failed to download bundle from URI" err
+'
+
+test_expect_success 'fail to clone from non-bundle file' '
+	test_when_finished rm -rf test &&
+	echo bogus >bogus &&
+	git clone --bundle-uri="$(pwd)/bogus" . test 2>err &&
+	grep "is not a bundle" err
+'
+
+test_expect_success 'create bundle' '
+	git init clone-from &&
+	git -C clone-from checkout -b topic &&
+	test_commit -C clone-from A &&
+	test_commit -C clone-from B &&
+	git -C clone-from bundle create B.bundle topic
+'
+
+test_expect_success 'clone with path bundle' '
+	git clone --bundle-uri="clone-from/B.bundle" \
+		clone-from clone-path &&
+	git -C clone-path rev-parse refs/bundles/topic >actual &&
+	git -C clone-from rev-parse topic >expect &&
+	test_cmp expect actual
+'
+
+test_done
-- 
gitgitgadget
Previous: Derrick Stolee via GitGitGadgetNext: Junio C Hamano
Message 29 of 36 in “Bundle URIs II: git clone --bundle-uri”
  1. 0/5 Bundle URIs II: git clone --bundle-uriDerrick Stolee via GitGitGadget, Jul 25, 2022
  2. 1/5 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Jul 25, 2022
  3. Josh SteadmonJul 27, 2022
  4. Ævar Arnfjörð BjarmasonJul 27, 2022
  5. Derrick StoleeAug 1, 2022
  6. Ævar Arnfjörð BjarmasonAug 1, 2022
  7. 2/5 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Jul 25, 2022
  8. Josh SteadmonJul 27, 2022
  9. Derrick StoleeAug 1, 2022
  10. 3/5 clone: add --bundle-uri optionDerrick Stolee via GitGitGadget, Jul 25, 2022
  11. 5/5 clone: --bundle-uri cannot be combined with --depthDerrick Stolee via GitGitGadget, Jul 25, 2022
  12. 4/5 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Jul 25, 2022
  13. Josh SteadmonJul 27, 2022
  14. Derrick StoleeAug 1, 2022
  15. Josh SteadmonJul 27, 2022
  16. Derrick StoleeAug 1, 2022
  17. 0/5 Bundle URIs II: git clone --bundle-uriDerrick Stolee via GitGitGadget, Aug 2, 2022
  18. 1/5 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Aug 2, 2022
  19. 2/5 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Aug 2, 2022
  20. 3/5 clone: add --bundle-uri optionDerrick Stolee via GitGitGadget, Aug 2, 2022
  21. 4/5 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Aug 2, 2022
  22. Junio C HamanoAug 2, 2022
  23. Derrick StoleeAug 4, 2022
  24. Junio C HamanoAug 4, 2022
  25. 5/5 clone: --bundle-uri cannot be combined with --depthDerrick Stolee via GitGitGadget, Aug 2, 2022
  26. 0/5 Bundle URIs II: git clone --bundle-uriDerrick Stolee via GitGitGadget, Aug 9, 2022
  27. 1/5 remote-curl: add 'get' capabilityDerrick Stolee via GitGitGadget, Aug 9, 2022
  28. 2/5 bundle-uri: create basic file-copy logicDerrick Stolee via GitGitGadget, Aug 9, 2022
  29. 3/5 clone: add --bundle-uri optionDerrick Stolee via GitGitGadget, Aug 9, 2022
  30. Junio C HamanoAug 22, 2022
  31. Derrick StoleeAug 23, 2022
  32. Junio C HamanoAug 24, 2022
  33. 4/5 bundle-uri: add support for http(s):// and file://Derrick Stolee via GitGitGadget, Aug 9, 2022
  34. 4/5 bundle-uri: add support for http(s):// and file://Teng Long, Aug 29, 2022
  35. Derrick StoleeAug 30, 2022
  36. 5/5 clone: --bundle-uri cannot be combined with --depthDerrick Stolee via GitGitGadget, Aug 9, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.