Re: [WIP RFC 2/5] Documentation: add Packfile URIs design doc

From: brian m. carlson <sandals@crustytoothpaste.net>
Date: Dec 4, 2018, 01:54 UTC
Message-ID: <20181204015446.GX890086@genre.crustytoothpaste.net>
In-Reply-To: <0461b362569362c6d0e73951469c547a03a1b59d.1543879256.git.jonathantanmy@google.com>

On Mon, Dec 03, 2018 at 03:37:35PM -0800, Jonathan Tan wrote:

Show 43 quoted lines

> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
> ---
>  Documentation/technical/packfile-uri.txt | 83 ++++++++++++++++++++++++
>  Documentation/technical/protocol-v2.txt  |  6 +-
>  2 files changed, 88 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/technical/packfile-uri.txt
> 
> diff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt
> new file mode 100644
> index 0000000000..6535801486
> --- /dev/null
> +++ b/Documentation/technical/packfile-uri.txt
> @@ -0,0 +1,83 @@
> +Packfile URIs
> +=============
> +
> +This feature allows servers to serve part of their packfile response as URIs.
> +This allows server designs that improve scalability in bandwidth and CPU usage
> +(for example, by serving some data through a CDN), and (in the future) provides
> +some measure of resumability to clients.
> +
> +This feature is available only in protocol version 2.
> +
> +Protocol
> +--------
> +
> +The server advertises `packfile-uris`.
> +
> +If the client replies with the following arguments:
> +
> + * packfile-uris
> + * thin-pack
> + * ofs-delta
> +
> +when the server sends the packfile, it MAY send a `packfile-uris` section
> +directly before the `packfile` section (right after `wanted-refs` if it is
> +sent) containing HTTP(S) URIs. See protocol-v2.txt for the documentation of
> +this section.
> +
> +Clients then should understand that the returned packfile could be incomplete,
> +and that it needs to download all the given URIs before the fetch or clone is
> +complete. Each URI should point to a Git packfile (which may be a thin pack and
> +which may contain offset deltas).

Some thoughts here:

First, I'd like to see a section (and a bit in the implementation) requiring HTTPS if the original protocol is secure (SSH or HTTPS). Allowing the server to downgrade to HTTP, even by accident, would be a security problem.

Second, this feature likely should be opt-in for SSH. One issue I've seen repeatedly is that people don't want to use HTTPS to fetch things when they're using SSH for Git. Many people in corporate environments have proxies that break HTTP for non-browser use cases[0], and using SSH is the only way that they can make a functional Git connection.

Third, I think the server needs to be required to both support Range headers and never change the content of a URI, so that we can have resumable clone implicit in this design. There are some places in the world where connections are poor and fetching even the initial packfile at once might be a problem. (I've seen such questions on Stack Overflow, for example.)

Having said that, I think overall this is a good idea and I'm glad to see a proposal for it.

[0] For example, a naughty-word filter may corrupt or block certain byte sequences that occur incidentally in the pack stream.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204