git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v7 04/16] upload-pack: add object filtering for partial clone

From
JHJeff Hostetler <git@jeffhostetler.com>
Date
Dec 8, 2017, 15:58 UTC
Message-ID
<20171208155851.855-5-git@jeffhostetler.com>
In-Reply-To
<20171208155851.855-1-git@jeffhostetler.com>
From: Jeff Hostetler <jeffhost@microsoft.com>

Teach upload-pack to negotiate object filtering over the protocol and to send filter parameters to pack-objects. This is intended for partial clone and fetch.

The idea to make upload-pack configurable using uploadpack.allowFilter comes from Jonathan Tan's work in [1].

[1] https://public-inbox.org/git/f211093280b422c32cc1b7034130072f35c5ed51.1506714999.git.jonathantanmy@google.com/
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
---
 Documentation/config.txt                          |  4 ++++
 Documentation/technical/pack-protocol.txt         |  8 +++++++
 Documentation/technical/protocol-capabilities.txt |  8 +++++++
 upload-pack.c                                     | 26 ++++++++++++++++++++++-
 4 files changed, 45 insertions(+), 1 deletion(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 1ac0ae6..e528210 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -3268,6 +3268,10 @@ uploadpack.packObjectsHook::
 	was run. I.e., `upload-pack` will feed input intended for
 	`pack-objects` to the hook, and expects a completed packfile on
 	stdout.
+
+uploadpack.allowFilter::
+	If this option is set, `upload-pack` will advertise partial
+	clone and partial fetch object filtering.
 +
 Note that this configuration variable is ignored if it is seen in the
 repository-level config (this is a safety measure against fetching from
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index ed1eae8..a43a113 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.
   upload-request    =  want-list
 		       *shallow-line
 		       *1depth-request
+		       [filter-request]
 		       flush-pkt
 
   want-list         =  first-want
@@ -227,6 +228,8 @@ out of what the server said it could do with the first 'want' line.
   additional-want   =  PKT-LINE("want" SP obj-id)
 
   depth             =  1*DIGIT
+
+  filter-request    =  PKT-LINE("filter" SP filter-spec)
 ----
 
 Clients MUST send all the obj-ids it wants from the reference
@@ -249,6 +252,11 @@ complete those commits. Commits whose parents are not received as a
 result are defined as shallow and marked as such in the server. This
 information is sent back to the client in the next step.
 
+The client can optionally request that pack-objects omit various
+objects from the packfile using one of several filtering techniques.
+These are intended for use with partial clone and partial fetch
+operations.  See `rev-list` for possible "filter-spec" values.
+
 Once all the 'want's and 'shallow's (and optional 'deepen') are
 transferred, clients MUST send a flush-pkt, to tell the server side
 that it is done sending the list.
diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt
index 26dcc6f..332d209 100644
--- a/Documentation/technical/protocol-capabilities.txt
+++ b/Documentation/technical/protocol-capabilities.txt
@@ -309,3 +309,11 @@ to accept a signed push certificate, and asks the <nonce> to be
 included in the push certificate.  A send-pack client MUST NOT
 send a push-cert packet unless the receive-pack server advertises
 this capability.
+
+filter
+------
+
+If the upload-pack server advertises the 'filter' capability,
+fetch-pack may send "filter" commands to request a partial clone
+or partial fetch and request that the server omit various objects
+from the packfile.
diff --git a/upload-pack.c b/upload-pack.c
index e25f725..e6d38b9 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -10,6 +10,8 @@
 #include "diff.h"
 #include "revision.h"
 #include "list-objects.h"
+#include "list-objects-filter.h"
+#include "list-objects-filter-options.h"
 #include "run-command.h"
 #include "connect.h"
 #include "sigchain.h"
@@ -18,6 +20,7 @@
 #include "parse-options.h"
 #include "argv-array.h"
 #include "prio-queue.h"
+#include "quote.h"
 
 static const char * const upload_pack_usage[] = {
 	N_("git upload-pack [<options>] <dir>"),
@@ -64,6 +67,10 @@ static int advertise_refs;
 static int stateless_rpc;
 static const char *pack_objects_hook;
 
+static int filter_capability_requested;
+static int filter_advertise;
+static struct list_objects_filter_options filter_options;
+
 static void reset_timeout(void)
 {
 	alarm(timeout);
@@ -131,6 +138,12 @@ static void create_pack_file(void)
 		argv_array_push(&pack_objects.args, "--delta-base-offset");
 	if (use_include_tag)
 		argv_array_push(&pack_objects.args, "--include-tag");
+	if (filter_options.filter_spec) {
+		struct strbuf buf = STRBUF_INIT;
+		sq_quote_buf(&buf, filter_options.filter_spec);
+		argv_array_pushf(&pack_objects.args, "--filter=%s", buf.buf);
+		strbuf_release(&buf);
+	}
 
 	pack_objects.in = -1;
 	pack_objects.out = -1;
@@ -794,6 +807,12 @@ static void receive_needs(void)
 			deepen_rev_list = 1;
 			continue;
 		}
+		if (skip_prefix(line, "filter ", &arg)) {
+			if (!filter_capability_requested)
+				die("git upload-pack: filtering capability not negotiated");
+			parse_list_objects_filter(&filter_options, arg);
+			continue;
+		}
 		if (!skip_prefix(line, "want ", &arg) ||
 		    get_oid_hex(arg, &oid_buf))
 			die("git upload-pack: protocol error, "
@@ -821,6 +840,8 @@ static void receive_needs(void)
 			no_progress = 1;
 		if (parse_feature_request(features, "include-tag"))
 			use_include_tag = 1;
+		if (parse_feature_request(features, "filter"))
+			filter_capability_requested = 1;
 
 		o = parse_object(&oid_buf);
 		if (!o) {
@@ -940,7 +961,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 		struct strbuf symref_info = STRBUF_INIT;
 
 		format_symref_info(&symref_info, cb_data);
-		packet_write_fmt(1, "%s %s%c%s%s%s%s%s agent=%s\n",
+		packet_write_fmt(1, "%s %s%c%s%s%s%s%s%s agent=%s\n",
 			     oid_to_hex(oid), refname_nons,
 			     0, capabilities,
 			     (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?
@@ -949,6 +970,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 				     " allow-reachable-sha1-in-want" : "",
 			     stateless_rpc ? " no-done" : "",
 			     symref_info.buf,
+			     filter_advertise ? " filter" : "",
 			     git_user_agent_sanitized());
 		strbuf_release(&symref_info);
 	} else {
@@ -1027,6 +1049,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused)
 	} else if (current_config_scope() != CONFIG_SCOPE_REPO) {
 		if (!strcmp("uploadpack.packobjectshook", var))
 			return git_config_string(&pack_objects_hook, var, value);
+	} else if (!strcmp("uploadpack.allowfilter", var)) {
+		filter_advertise = git_config_bool(var, value);
 	}
 	return parse_hide_refs_config(var, value, "uploadpack");
 }

-- 
2.9.3
Previous: Jeff HostetlerNext: Junio C Hamano
Message 19 of 24 in “Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests”
  1. 00/16 Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and testsJeff Hostetler, Dec 8, 2017
  2. 03/16 gc: do not repack promisor packfilesJeff Hostetler, Dec 8, 2017
  3. 09/16 fetch: support filtersJeff Hostetler, Dec 8, 2017
  4. Duy NguyenAug 19, 2018
  5. Jeff HostetlerAug 20, 2018
  6. 13/16 fetch-pack: restore save_commit_buffer after useJeff Hostetler, Dec 8, 2017
  7. 12/16 unpack-trees: batch fetching of missing blobsJeff Hostetler, Dec 8, 2017
  8. 11/16 clone: partial cloneJeff Hostetler, Dec 8, 2017
  9. 15/16 fetch: inherit filter-spec from partial cloneJeff Hostetler, Dec 8, 2017
  10. 16/16 t5616: test bulk prefetch after partial fetchJeff Hostetler, Dec 8, 2017
  11. 10/16 partial-clone: define partial clone settings in configJeff Hostetler, Dec 8, 2017
  12. 14/16 t5616: end-to-end tests for partial cloneJeff Hostetler, Dec 8, 2017
  13. 02/16 rev-list: support termination at promisor objectsJeff Hostetler, Dec 8, 2017
  14. 08/16 fetch: refactor calculation of remote listJeff Hostetler, Dec 8, 2017
  15. 07/16 fetch-pack: test support excluding large blobsJeff Hostetler, Dec 8, 2017
  16. 01/16 sha1_file: support lazily fetching missing objectsJeff Hostetler, Dec 8, 2017
  17. 05/16 fetch-pack, index-pack, transport: partial cloneJeff Hostetler, Dec 8, 2017
  18. 06/16 fetch-pack: add --no-filterJeff Hostetler, Dec 8, 2017
  19. 04/16 upload-pack: add object filtering for partial cloneJeff Hostetler, Dec 8, 2017
  20. Junio C HamanoDec 8, 2017
  21. Jeff HostetlerDec 8, 2017
  22. Junio C HamanoDec 8, 2017
  23. Brandon WilliamsDec 8, 2017
  24. Jonathan TanDec 11, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.