git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Proposed approaches to supporting HTTP remotes in "git archive"

From
Josh Steadmon <steadmon@google.com>
Date
Jul 27, 2018, 21:47 UTC
Message-ID
<CANq=j3tK7QeBJOC7VNWkh4+WBNibMJJp5YUkd9te5NaYwukAow@mail.gmail.com>
# Supporting HTTP remotes in "git archive"

We would like to allow remote archiving from HTTP servers. There are a few possible implementations to be discussed:

## Shallow clone to temporary repo

This approach builds on existing endpoints. Clients will connect to the remote server's git-upload-pack service and fetch a shallow clone of the requested commit into a temporary local repo. The write_archive() function is then called on the local clone to write out the requested archive.

### Benefits
* This can be implemented entirely in builtin/archive.c. No new service
  endpoints or server code are required.
* The archive is generated and compressed on the client side. This
  reduces CPU load on the server (for compressed archives) which would
   otherwise be a potential DoS vector.
* This provides a git-native way to archive any HTTP servers that
  support the git-upload-pack service; some providers (including GitHub)
  do not currently allow the git-upload-archive service.
### Drawbacks
* Archives generated remotely may not be bit-for-bit identical compared
  to those generated locally, if the versions of git used on the client
  and on the server differ.
* This requires higher bandwidth compared to transferring a compressed
  archive generated on the server.
## Use git-upload-archive

This approach requires adding support for the git-upload-archive endpoint to the HTTP backend. Clients will connect to the remote server's git-upload-archive service and the server will generate the archive which is then delivered to the client.

### Benefits
* Matches existing "git archive" behavior for other remotes.
* Requires less bandwidth to send a compressed archive than a shallow
  clone.
* Resulting archive does not depend in any way on the client
  implementation.
### Drawbacks
* Implementation is more complicated; it will require changes to (at
  least) builtin/archive.c, http-backend.c, and
  builtin/upload-archive.c.
* Generates more CPU load on the server when compressing archives. This
  is potentially a DoS vector.
* Does not allow archiving from servers that don't support the
  git-upload-archive service.
## Add a new protocol v2 "archive" command

I am still a bit hazy on the exact details of this approach, please forgive any inaccuracies (I'm a new contributor and haven't examined custom v2 commands in much detail yet).

This approach builds off the existing v2 upload-pack endpoint. The client will issue an archive command (with options to select particular paths or a tree-ish). The server will generate the archive and deliver it to the client.

### Benefits
* Requires less bandwidth to send a compressed archive than a shallow
  clone.
* Resulting archive does not depend in any way on the client
  implementation.
### Drawbacks
* Generates more CPU load on the server when compressing archives. This
  is potentially a DoS vector.
* Servers must support the v2 protocol (although the client could
  potentially fallback to some other supported remote archive
   functionality).
### Unknowns
* I am not clear on the relative complexity of this approach compared to
  the others, and would appreciate any guidance offered.
## Summary

Personally, I lean towards the first approach. It could give us an opportunity to remove server-side complexity; there is no reason that the shallow-clone approach must be restricted to the HTTP transport, and we could re-implement other transports using this method. Additionally, it would allow clients to pull archives from remotes that would not otherwise support it.

That said, I am happy to work on whichever approach the community deems most worthwhile.

Next: Jonathan Nieder
Message 1 of 6 in “Proposed approaches to supporting HTTP remotes in "git archive"”
  1. Josh SteadmonJul 27, 2018
  2. Jonathan NiederJul 27, 2018
  3. Jonathan NiederJul 27, 2018
  4. Junio C HamanoJul 27, 2018
  5. René ScharfeJul 29, 2018
  6. brian m. carlsonJul 28, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.