git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2] Add unix domain socket support to HTTP transport

From
LGLeslie Cheng via GitGitGadget <gitgitgadget@gmail.com>
Date
Feb 23, 2024, 01:58 UTC
Message-ID
<pull.1681.v2.git.git.1708653536115.gitgitgadget@gmail.com>
In-Reply-To
<pull.1681.git.git.1708506863243.gitgitgadget@gmail.com>
From: Leslie Cheng <leslie.cheng5@gmail.com>

This changeset introduces an `http.unixSocket` option so that users can proxy their git over HTTP remotes to a unix domain socket. In terms of why, since UDS are local and git already has a local protocol: some corporate environments use a UDS to proxy requests to internal resources (ie. source control), so this change would support those use-cases. This proxy can occasionally be necessary to attach MFA tokens or client certificates for CLI tools.

The implementation leverages `--unix-socket` option [0] via the `CURLOPT_UNIX_SOCKET_PATH` flag available with libcurl [1].

`GIT_CURL_HAVE_CURLOPT_UNIX_SOCKET_PATH` and `NO_UNIX_SOCKETS` were kept separate so that we can spit out better error messages for users if git was compiled with `NO_UNIX_SOCKETS`.

[0] https://curl.se/docs/manpage.html#--unix-socket [1] https://curl.se/libcurl/c/CURLOPT_UNIX_SOCKET_PATH.html

Signed-off-by: Leslie Cheng <leslie@lc.fyi>
---
    Add unix domain socket support to HTTP transport
    
    Changes since v1:
    
     * Updated test to use Perl instead of nc to proxy between UDS and TCP
       socket; I chose not to split this out into a library since its use is
       hyper-specific and has a dependency on lib-httpd.sh
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1681%2Flcfyi%2Flcfyi%2Fadd-unix-socket-support-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1681/lcfyi/lcfyi/add-unix-socket-support-v2
Pull-Request: https://github.com/git/git/pull/1681
Range-diff vs v1:
 1:  3e531632329 ! 1:  2af5cc8089b Add unix domain socket support to HTTP transport.
     @@ Metadata
      Author: Leslie Cheng <leslie.cheng5@gmail.com>
      
       ## Commit message ##
     -    Add unix domain socket support to HTTP transport.
     +    Add unix domain socket support to HTTP transport
      
          This changeset introduces an `http.unixSocket` option so that users can
          proxy their git over HTTP remotes to a unix domain socket. In terms of
     @@ t/t5565-http-unix-domain-socket.sh (new)
      +	test_done
      +}
      +
     -+UDS_TO_TCP_FIFO=uds_to_tcp
     -+TCP_TO_UDS_FIFO=tcp_to_uds
     -+UDS_PID=
     -+TCP_PID=
     ++if ! test_have_prereq PERL
     ++then
     ++	skip_all='skipping http-unix-socket tests; perl not available'
     ++	test_done
     ++fi
     ++
     ++SOCKET_PROXY_PIDFILE="$(pwd)/proxy.pid"
      +UDS_SOCKET="$(pwd)/uds.sock"
     -+UNRESOLVABLE_ENDPOINT=http://localhost:4242
     ++UNRESOLVABLE_ENDPOINT=http://unresolved
      +
      +start_proxy_unix_to_tcp() {
     -+    local socket_path="$UDS_SOCKET"
     -+    local host=127.0.0.1
     -+    local port=$LIB_HTTPD_PORT
     -+
     -+    rm -f "$UDS_TO_TCP_FIFO"
     -+    rm -f "$TCP_TO_UDS_FIFO"
     -+    rm -f "$socket_path"
     -+    mkfifo "$UDS_TO_TCP_FIFO"
     -+    mkfifo "$TCP_TO_UDS_FIFO"
     -+    nc -klU "$socket_path" <tcp_to_uds >uds_to_tcp &
     -+    UDS_PID=$!
     -+
     -+    nc "$host" "$port" >tcp_to_uds <uds_to_tcp &
     -+    TCP_PID=$!
     -+
     -+    test_atexit 'stop_proxy_unix_to_tcp'
     ++	test_atexit 'stop_proxy_unix_to_tcp'
     ++
     ++	perl -Mstrict -MIO::Select -MIO::Socket::INET -MIO::Socket::UNIX -e '
     ++		my $uds_path = $ARGV[0];
     ++		my $host = $ARGV[1];
     ++		my $port = $ARGV[2];
     ++		my $pidfile = $ARGV[3];
     ++
     ++		open(my $fh, ">", $pidfile) or die "failed to create pidfile";
     ++		print $fh "$$";
     ++		close($fh);
     ++
     ++		my $uds = IO::Socket::UNIX->new(
     ++			Local => $uds_path,
     ++			Type => SOCK_STREAM,
     ++			Listen => 5,
     ++		) or die "failed to create unix domain socket";
     ++
     ++		while (my $conn = $uds->accept()) {
     ++			my $tcp_client = IO::Socket::INET->new(
     ++				PeerAddr => $host,
     ++				PeerPort => $port,
     ++				Proto => "tcp",
     ++			) or die "failed to create TCP socket";
     ++
     ++			my $sel = IO::Select->new($conn, $tcp_client);
     ++
     ++			while (my @ready = $sel->can_read(10)) {
     ++				foreach my $socket (@ready) {
     ++					my $other = ($socket == $conn) ? $tcp_client : $conn;
     ++					my $data;
     ++					my $bytes = $socket->sysread($data, 4096);
     ++
     ++					if ($bytes) {
     ++						$other->syswrite($data, $bytes);
     ++					} else {
     ++						$socket->close();
     ++					}
     ++				}
     ++			}
     ++		}
     ++	' "$UDS_SOCKET" "127.0.0.1" "$LIB_HTTPD_PORT" "$SOCKET_PROXY_PIDFILE" &
     ++	SOCKET_PROXY_PID=$!
      +}
      +
      +stop_proxy_unix_to_tcp() {
     -+    kill "$UDS_PID"
     -+    kill "$TCP_PID"
     -+    rm -f "$UDS_TO_TCP_FIFO"
     -+    rm -f "$TCP_TO_UDS_FIFO"
     ++	kill -9 "$(cat "$SOCKET_PROXY_PIDFILE")"
     ++	rm -f "$SOCKET_PROXY_PIDFILE"
     ++	rm -f "$UDS_SOCKET"
      +}
      +
      +start_httpd
     @@ t/t5565-http-unix-domain-socket.sh (new)
      +
      +# sanity check that we can't clone normally
      +test_expect_success 'cloning without UDS fails' '
     -+    test_must_fail git clone "$UNRESOLVABLE_ENDPOINT/smart/repo.git" clone
     ++	test_must_fail git clone "$UNRESOLVABLE_ENDPOINT/smart/repo.git" clone
      +'
      +
      +test_expect_success 'cloning with UDS succeeds' '
     -+    test_when_finished "rm -rf clone" &&
     ++	test_when_finished "rm -rf clone" &&
      +	test_config_global http.unixsocket "$UDS_SOCKET" &&
      +	git clone "$UNRESOLVABLE_ENDPOINT/smart/repo.git" clone
      +'
      +
      +test_expect_success 'cloning with a non-existent http proxy fails' '
     -+    git clone $HTTPD_URL/smart/repo.git clone &&
     -+    rm -rf clone &&
     -+    test_config_global http.proxy 127.0.0.1:0 &&
     -+    test_must_fail git clone $HTTPD_URL/smart/repo.git clone
     ++	git clone $HTTPD_URL/smart/repo.git clone &&
     ++	rm -rf clone &&
     ++	test_config_global http.proxy 127.0.0.1:0 &&
     ++	test_must_fail git clone $HTTPD_URL/smart/repo.git clone
      +'
      +
      +test_expect_success 'UDS socket takes precedence over http proxy' '
     -+    test_when_finished "rm -rf clone" &&
     -+    test_config_global http.proxy 127.0.0.1:0 &&
     -+    test_config_global http.unixsocket "$UDS_SOCKET" &&
     -+    git clone $HTTPD_URL/smart/repo.git clone
     ++	test_when_finished "rm -rf clone" &&
     ++	test_config_global http.proxy 127.0.0.1:0 &&
     ++	test_config_global http.unixsocket "$UDS_SOCKET" &&
     ++	git clone $HTTPD_URL/smart/repo.git clone
      +'
      +
      +test_done
 Documentation/config/http.txt      |   5 ++
 git-curl-compat.h                  |   7 ++
 http.c                             |  23 ++++++
 t/t5565-http-unix-domain-socket.sh | 109 +++++++++++++++++++++++++++++
 4 files changed, 144 insertions(+)
 create mode 100755 t/t5565-http-unix-domain-socket.sh
diff --git a/Documentation/config/http.txt b/Documentation/config/http.txt
index 2d4e0c9b869..bf48cbd599a 100644
--- a/Documentation/config/http.txt
+++ b/Documentation/config/http.txt
@@ -277,6 +277,11 @@ http.followRedirects::
 	the base for the follow-up requests, this is generally
 	sufficient. The default is `initial`.
 
+http.unixSocket::
+	Connect through this Unix domain socket via HTTP, instead of using the
+	network. If set, this config takes precendence over `http.proxy` and
+	is incompatible with the proxy options (see `curl(1)`).
+
 http.<url>.*::
 	Any of the http.* options above can be applied selectively to some URLs.
 	For a config key to match a URL, each element of the config key is
diff --git a/git-curl-compat.h b/git-curl-compat.h
index fd96b3cdffd..f0f3bec0e17 100644
--- a/git-curl-compat.h
+++ b/git-curl-compat.h
@@ -74,6 +74,13 @@
 #define GIT_CURL_HAVE_CURLE_SSL_PINNEDPUBKEYNOTMATCH 1
 #endif
 
+/**
+ * CURLOPT_UNIX_SOCKET_PATH was added in 7.40.0, released in January 2015.
+ */
+#if LIBCURL_VERSION_NUM >= 0x074000
+#define GIT_CURL_HAVE_CURLOPT_UNIX_SOCKET_PATH 1
+#endif
+
 /**
  * CURL_HTTP_VERSION_2 was added in 7.43.0, released in June 2015.
  *
diff --git a/http.c b/http.c
index e73b136e589..8cfdcaeac82 100644
--- a/http.c
+++ b/http.c
@@ -79,6 +79,9 @@ static const char *http_proxy_ssl_ca_info;
 static struct credential proxy_cert_auth = CREDENTIAL_INIT;
 static int proxy_ssl_cert_password_required;
 
+#if defined(GIT_CURL_HAVE_CURLOPT_UNIX_SOCKET_PATH) && !defined(NO_UNIX_SOCKETS)
+static const char *curl_unix_socket_path;
+#endif
 static struct {
 	const char *name;
 	long curlauth_param;
@@ -455,6 +458,20 @@ static int http_options(const char *var, const char *value,
 		return 0;
 	}
 
+	if (!strcmp("http.unixsocket", var)) {
+#ifdef GIT_CURL_HAVE_CURLOPT_UNIX_SOCKET_PATH
+#ifndef NO_UNIX_SOCKETS
+		return git_config_string(&curl_unix_socket_path, var, value);
+#else
+		warning(_("Unix socket support unavailable in this build of Git"));
+		return 0;
+#endif
+#else
+		warning(_("Unix socket support is not supported with cURL < 7.40.0"));
+		return 0;
+#endif
+	}
+
 	if (!strcmp("http.cookiefile", var))
 		return git_config_pathname(&curl_cookie_file, var, value);
 	if (!strcmp("http.savecookies", var)) {
@@ -1203,6 +1220,12 @@ static CURL *get_curl_handle(void)
 	}
 	init_curl_proxy_auth(result);
 
+#if defined(GIT_CURL_HAVE_CURLOPT_UNIX_SOCKET_PATH) && !defined(NO_UNIX_SOCKETS)
+	if (curl_unix_socket_path) {
+		curl_easy_setopt(result, CURLOPT_UNIX_SOCKET_PATH, curl_unix_socket_path);
+	}
+#endif
+
 	set_curl_keepalive(result);
 
 	return result;
diff --git a/t/t5565-http-unix-domain-socket.sh b/t/t5565-http-unix-domain-socket.sh
new file mode 100755
index 00000000000..2f9c53ab14f
--- /dev/null
+++ b/t/t5565-http-unix-domain-socket.sh
@@ -0,0 +1,109 @@
+#!/bin/sh
+
+test_description="test fetching through http via unix domain socket"
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-httpd.sh
+
+test -z "$NO_UNIX_SOCKETS" || {
+	skip_all='skipping http-unix-socket tests, unix sockets not available'
+	test_done
+}
+
+if ! test_have_prereq PERL
+then
+	skip_all='skipping http-unix-socket tests; perl not available'
+	test_done
+fi
+
+SOCKET_PROXY_PIDFILE="$(pwd)/proxy.pid"
+UDS_SOCKET="$(pwd)/uds.sock"
+UNRESOLVABLE_ENDPOINT=http://unresolved
+
+start_proxy_unix_to_tcp() {
+	test_atexit 'stop_proxy_unix_to_tcp'
+
+	perl -Mstrict -MIO::Select -MIO::Socket::INET -MIO::Socket::UNIX -e '
+		my $uds_path = $ARGV[0];
+		my $host = $ARGV[1];
+		my $port = $ARGV[2];
+		my $pidfile = $ARGV[3];
+
+		open(my $fh, ">", $pidfile) or die "failed to create pidfile";
+		print $fh "$$";
+		close($fh);
+
+		my $uds = IO::Socket::UNIX->new(
+			Local => $uds_path,
+			Type => SOCK_STREAM,
+			Listen => 5,
+		) or die "failed to create unix domain socket";
+
+		while (my $conn = $uds->accept()) {
+			my $tcp_client = IO::Socket::INET->new(
+				PeerAddr => $host,
+				PeerPort => $port,
+				Proto => "tcp",
+			) or die "failed to create TCP socket";
+
+			my $sel = IO::Select->new($conn, $tcp_client);
+
+			while (my @ready = $sel->can_read(10)) {
+				foreach my $socket (@ready) {
+					my $other = ($socket == $conn) ? $tcp_client : $conn;
+					my $data;
+					my $bytes = $socket->sysread($data, 4096);
+
+					if ($bytes) {
+						$other->syswrite($data, $bytes);
+					} else {
+						$socket->close();
+					}
+				}
+			}
+		}
+	' "$UDS_SOCKET" "127.0.0.1" "$LIB_HTTPD_PORT" "$SOCKET_PROXY_PIDFILE" &
+	SOCKET_PROXY_PID=$!
+}
+
+stop_proxy_unix_to_tcp() {
+	kill -9 "$(cat "$SOCKET_PROXY_PIDFILE")"
+	rm -f "$SOCKET_PROXY_PIDFILE"
+	rm -f "$UDS_SOCKET"
+}
+
+start_httpd
+start_proxy_unix_to_tcp
+
+test_expect_success 'setup repository' '
+	test_commit foo &&
+	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+	git push --mirror "$HTTPD_DOCUMENT_ROOT_PATH/repo.git"
+'
+
+# sanity check that we can't clone normally
+test_expect_success 'cloning without UDS fails' '
+	test_must_fail git clone "$UNRESOLVABLE_ENDPOINT/smart/repo.git" clone
+'
+
+test_expect_success 'cloning with UDS succeeds' '
+	test_when_finished "rm -rf clone" &&
+	test_config_global http.unixsocket "$UDS_SOCKET" &&
+	git clone "$UNRESOLVABLE_ENDPOINT/smart/repo.git" clone
+'
+
+test_expect_success 'cloning with a non-existent http proxy fails' '
+	git clone $HTTPD_URL/smart/repo.git clone &&
+	rm -rf clone &&
+	test_config_global http.proxy 127.0.0.1:0 &&
+	test_must_fail git clone $HTTPD_URL/smart/repo.git clone
+'
+
+test_expect_success 'UDS socket takes precedence over http proxy' '
+	test_when_finished "rm -rf clone" &&
+	test_config_global http.proxy 127.0.0.1:0 &&
+	test_config_global http.unixsocket "$UDS_SOCKET" &&
+	git clone $HTTPD_URL/smart/repo.git clone
+'
+
+test_done

base-commit: 3e0d3cd5c7def4808247caf168e17f2bbf47892b

-- 
gitgitgadget
Previous: Leslie ChengNext: Junio C Hamano
Message 4 of 7 in “Add unix domain socket support to HTTP transport.”
  1. Add unix domain socket support to HTTP transport.Leslie Cheng via GitGitGadget, Feb 21, 2024
  2. Eric WongFeb 21, 2024
  3. Leslie ChengFeb 22, 2024
  4. Add unix domain socket support to HTTP transportLeslie Cheng via GitGitGadget, Feb 23, 2024
  5. Junio C HamanoFeb 23, 2024
  6. Junio C HamanoFeb 23, 2024
  7. Leslie ChengFeb 23, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.