{"thread":{"id":"48966","subject":"Proposed approaches to supporting HTTP remotes in \"git archive\"","startedAt":"2018-07-27T21:47:14Z","lastAt":"2018-07-29T11:54:48Z","messageCount":6,"participants":["Josh Steadmon","Jonathan Nieder","Junio C Hamano","brian m. carlson","René Scharfe"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"353760","messageId":"CANq=j3tK7QeBJOC7VNWkh4+WBNibMJJp5YUkd9te5NaYwukAow@mail.gmail.com","threadId":"48966","inReplyTo":null,"subject":"Proposed approaches to supporting HTTP remotes in \"git archive\"","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2018-07-27T21:47:00Z","receivedAt":"2018-07-27T21:47:14Z","isPatch":false,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"# Supporting HTTP remotes in \"git archive\"\n\nWe would like to allow remote archiving from HTTP servers. There are a\nfew possible implementations to be discussed:\n\n## Shallow clone to temporary repo\n\nThis approach builds on existing endpoints. Clients will connect to the\nremote server's git-upload-pack service and fetch a shallow clone of the\nrequested commit into a temporary local repo. The write_archive()\nfunction is then called on the local clone to write out the requested\narchive.\n\n### Benefits\n\n* This can be implemented entirely in builtin/archive.c. No new service\n  endpoints or server code are required.\n\n* The archive is generated and compressed on the client side. This\n  reduces CPU load on the server (for compressed archives) which would\n   otherwise be a potential DoS vector.\n\n* This provides a git-native way to archive any HTTP servers that\n  support the git-upload-pack service; some providers (including GitHub)\n  do not currently allow the git-upload-archive service.\n\n### Drawbacks\n\n* Archives generated remotely may not be bit-for-bit identical compared\n  to those generated locally, if the versions of git used on the client\n  and on the server differ.\n\n* This requires higher bandwidth compared to transferring a compressed\n  archive generated on the server.\n\n\n## Use git-upload-archive\n\nThis approach requires adding support for the git-upload-archive\nendpoint to the HTTP backend. Clients will connect to the remote\nserver's git-upload-archive service and the server will generate the\narchive which is then delivered to the client.\n\n### Benefits\n\n* Matches existing \"git archive\" behavior for other remotes.\n\n* Requires less bandwidth to send a compressed archive than a shallow\n  clone.\n\n* Resulting archive does not depend in any way on the client\n  implementation.\n\n### Drawbacks\n\n* Implementation is more complicated; it will require changes to (at\n  least) builtin/archive.c, http-backend.c, and\n  builtin/upload-archive.c.\n\n* Generates more CPU load on the server when compressing archives. This\n  is potentially a DoS vector.\n\n* Does not allow archiving from servers that don't support the\n  git-upload-archive service.\n\n\n## Add a new protocol v2 \"archive\" command\n\nI am still a bit hazy on the exact details of this approach, please\nforgive any inaccuracies (I'm a new contributor and haven't examined\ncustom v2 commands in much detail yet).\n\nThis approach builds off the existing v2 upload-pack endpoint. The\nclient will issue an archive command (with options to select particular\npaths or a tree-ish). The server will generate the archive and deliver\nit to the client.\n\n### Benefits\n\n* Requires less bandwidth to send a compressed archive than a shallow\n  clone.\n\n* Resulting archive does not depend in any way on the client\n  implementation.\n\n### Drawbacks\n\n* Generates more CPU load on the server when compressing archives. This\n  is potentially a DoS vector.\n\n* Servers must support the v2 protocol (although the client could\n  potentially fallback to some other supported remote archive\n   functionality).\n\n### Unknowns\n\n* I am not clear on the relative complexity of this approach compared to\n  the others, and would appreciate any guidance offered.\n\n\n## Summary\n\nPersonally, I lean towards the first approach. It could give us an\nopportunity to remove server-side complexity; there is no reason that\nthe shallow-clone approach must be restricted to the HTTP transport, and\nwe could re-implement other transports using this method.  Additionally,\nit would allow clients to pull archives from remotes that would not\notherwise support it.\n\nThat said, I am happy to work on whichever approach the community deems\nmost worthwhile.\n"},{"id":"353762","messageId":"20180727215644.GA223387@aiede.svl.corp.google.com","threadId":"48966","inReplyTo":"CANq=j3tK7QeBJOC7VNWkh4+WBNibMJJp5YUkd9te5NaYwukAow@mail.gmail.com","subject":"Re: Proposed approaches to supporting HTTP remotes in \"git archive\"","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-07-27T21:56:44Z","receivedAt":"2018-07-27T21:57:31Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(just cc-ing René Scharfe, archive expert; Peff; Dscho; Franck Bui-Huu\nto see how his creation is evolving)\nJosh Steadmon wrote:\n\n> # Supporting HTTP remotes in \"git archive\"\n>\n> We would like to allow remote archiving from HTTP servers. There are a\n> few possible implementations to be discussed:\n>\n> ## Shallow clone to temporary repo\n>\n> This approach builds on existing endpoints. Clients will connect to the\n> remote server's git-upload-pack service and fetch a shallow clone of the\n> requested commit into a temporary local repo. The write_archive()\n> function is then called on the local clone to write out the requested\n> archive.\n>\n> ### Benefits\n>\n> * This can be implemented entirely in builtin/archive.c. No new service\n>   endpoints or server code are required.\n>\n> * The archive is generated and compressed on the client side. This\n>   reduces CPU load on the server (for compressed archives) which would\n>    otherwise be a potential DoS vector.\n>\n> * This provides a git-native way to archive any HTTP servers that\n>   support the git-upload-pack service; some providers (including GitHub)\n>   do not currently allow the git-upload-archive service.\n>\n> ### Drawbacks\n>\n> * Archives generated remotely may not be bit-for-bit identical compared\n>   to those generated locally, if the versions of git used on the client\n>   and on the server differ.\n>\n> * This requires higher bandwidth compared to transferring a compressed\n>   archive generated on the server.\n>\n>\n> ## Use git-upload-archive\n>\n> This approach requires adding support for the git-upload-archive\n> endpoint to the HTTP backend. Clients will connect to the remote\n> server's git-upload-archive service and the server will generate the\n> archive which is then delivered to the client.\n>\n> ### Benefits\n>\n> * Matches existing \"git archive\" behavior for other remotes.\n>\n> * Requires less bandwidth to send a compressed archive than a shallow\n>   clone.\n>\n> * Resulting archive does not depend in any way on the client\n>   implementation.\n>\n> ### Drawbacks\n>\n> * Implementation is more complicated; it will require changes to (at\n>   least) builtin/archive.c, http-backend.c, and\n>   builtin/upload-archive.c.\n>\n> * Generates more CPU load on the server when compressing archives. This\n>   is potentially a DoS vector.\n>\n> * Does not allow archiving from servers that don't support the\n>   git-upload-archive service.\n>\n>\n> ## Add a new protocol v2 \"archive\" command\n>\n> I am still a bit hazy on the exact details of this approach, please\n> forgive any inaccuracies (I'm a new contributor and haven't examined\n> custom v2 commands in much detail yet).\n>\n> This approach builds off the existing v2 upload-pack endpoint. The\n> client will issue an archive command (with options to select particular\n> paths or a tree-ish). The server will generate the archive and deliver\n> it to the client.\n>\n> ### Benefits\n>\n> * Requires less bandwidth to send a compressed archive than a shallow\n>   clone.\n>\n> * Resulting archive does not depend in any way on the client\n>   implementation.\n>\n> ### Drawbacks\n>\n> * Generates more CPU load on the server when compressing archives. This\n>   is potentially a DoS vector.\n>\n> * Servers must support the v2 protocol (although the client could\n>   potentially fallback to some other supported remote archive\n>    functionality).\n>\n> ### Unknowns\n>\n> * I am not clear on the relative complexity of this approach compared to\n>   the others, and would appreciate any guidance offered.\n>\n>\n> ## Summary\n>\n> Personally, I lean towards the first approach. It could give us an\n> opportunity to remove server-side complexity; there is no reason that\n> the shallow-clone approach must be restricted to the HTTP transport, and\n> we could re-implement other transports using this method.  Additionally,\n> it would allow clients to pull archives from remotes that would not\n> otherwise support it.\n>\n> That said, I am happy to work on whichever approach the community deems\n> most worthwhile.\n"},{"id":"353763","messageId":"20180727220021.GB223387@aiede.svl.corp.google.com","threadId":"48966","inReplyTo":"CANq=j3tK7QeBJOC7VNWkh4+WBNibMJJp5YUkd9te5NaYwukAow@mail.gmail.com","subject":"Re: Proposed approaches to supporting HTTP remotes in \"git archive\"","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-07-27T22:00:21Z","receivedAt":"2018-07-27T22:00:26Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(just cc-ing René Scharfe, archive expert; Peff; Dscho; Franck Bui-Huu\nto see how his creation is evolving.\n\nUsing the correct address for René this time. Sorry for the noise.)\n\nJosh Steadmon wrote:\n\n> # Supporting HTTP remotes in \"git archive\"\n>\n> We would like to allow remote archiving from HTTP servers. There are a\n> few possible implementations to be discussed:\n>\n> ## Shallow clone to temporary repo\n>\n> This approach builds on existing endpoints. Clients will connect to the\n> remote server's git-upload-pack service and fetch a shallow clone of the\n> requested commit into a temporary local repo. The write_archive()\n> function is then called on the local clone to write out the requested\n> archive.\n>\n> ### Benefits\n>\n> * This can be implemented entirely in builtin/archive.c. No new service\n>   endpoints or server code are required.\n>\n> * The archive is generated and compressed on the client side. This\n>   reduces CPU load on the server (for compressed archives) which would\n>    otherwise be a potential DoS vector.\n>\n> * This provides a git-native way to archive any HTTP servers that\n>   support the git-upload-pack service; some providers (including GitHub)\n>   do not currently allow the git-upload-archive service.\n>\n> ### Drawbacks\n>\n> * Archives generated remotely may not be bit-for-bit identical compared\n>   to those generated locally, if the versions of git used on the client\n>   and on the server differ.\n>\n> * This requires higher bandwidth compared to transferring a compressed\n>   archive generated on the server.\n>\n>\n> ## Use git-upload-archive\n>\n> This approach requires adding support for the git-upload-archive\n> endpoint to the HTTP backend. Clients will connect to the remote\n> server's git-upload-archive service and the server will generate the\n> archive which is then delivered to the client.\n>\n> ### Benefits\n>\n> * Matches existing \"git archive\" behavior for other remotes.\n>\n> * Requires less bandwidth to send a compressed archive than a shallow\n>   clone.\n>\n> * Resulting archive does not depend in any way on the client\n>   implementation.\n>\n> ### Drawbacks\n>\n> * Implementation is more complicated; it will require changes to (at\n>   least) builtin/archive.c, http-backend.c, and\n>   builtin/upload-archive.c.\n>\n> * Generates more CPU load on the server when compressing archives. This\n>   is potentially a DoS vector.\n>\n> * Does not allow archiving from servers that don't support the\n>   git-upload-archive service.\n>\n>\n> ## Add a new protocol v2 \"archive\" command\n>\n> I am still a bit hazy on the exact details of this approach, please\n> forgive any inaccuracies (I'm a new contributor and haven't examined\n> custom v2 commands in much detail yet).\n>\n> This approach builds off the existing v2 upload-pack endpoint. The\n> client will issue an archive command (with options to select particular\n> paths or a tree-ish). The server will generate the archive and deliver\n> it to the client.\n>\n> ### Benefits\n>\n> * Requires less bandwidth to send a compressed archive than a shallow\n>   clone.\n>\n> * Resulting archive does not depend in any way on the client\n>   implementation.\n>\n> ### Drawbacks\n>\n> * Generates more CPU load on the server when compressing archives. This\n>   is potentially a DoS vector.\n>\n> * Servers must support the v2 protocol (although the client could\n>   potentially fallback to some other supported remote archive\n>    functionality).\n>\n> ### Unknowns\n>\n> * I am not clear on the relative complexity of this approach compared to\n>   the others, and would appreciate any guidance offered.\n>\n>\n> ## Summary\n>\n> Personally, I lean towards the first approach. It could give us an\n> opportunity to remove server-side complexity; there is no reason that\n> the shallow-clone approach must be restricted to the HTTP transport, and\n> we could re-implement other transports using this method.  Additionally,\n> it would allow clients to pull archives from remotes that would not\n> otherwise support it.\n>\n> That said, I am happy to work on whichever approach the community deems\n> most worthwhile.\n"},{"id":"353767","messageId":"xmqqtvokjonm.fsf@gitster-ct.c.googlers.com","threadId":"48966","inReplyTo":"CANq=j3tK7QeBJOC7VNWkh4+WBNibMJJp5YUkd9te5NaYwukAow@mail.gmail.com","subject":"Re: Proposed approaches to supporting HTTP remotes in \"git archive\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-07-27T22:32:13Z","receivedAt":"2018-07-27T22:32:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon <steadmon@google.com> writes:\n\n> # Supporting HTTP remotes in \"git archive\"\n>\n> We would like to allow remote archiving from HTTP servers. There are a\n> few possible implementations to be discussed:\n>\n> ## Shallow clone to temporary repo\n>\n> This approach builds on existing endpoints. Clients will connect to the\n> remote server's git-upload-pack service and fetch a shallow clone of the\n> requested commit into a temporary local repo. The write_archive()\n> function is then called on the local clone to write out the requested\n> archive.\n>\n> ...\n>\n> ## Summary\n>\n> Personally, I lean towards the first approach. It could give us an\n> opportunity to remove server-side complexity; there is no reason that\n> the shallow-clone approach must be restricted to the HTTP transport, and\n> we could re-implement other transports using this method.  Additionally,\n> it would allow clients to pull archives from remotes that would not\n> otherwise support it.\n\nI consider the first one (i.e. make a shallow clone and tar it up\nlocally) a hack that does *not* belong to \"git archive --remote\"\ncommand, especially when it is only done to \"http remotes\".  The\nonly reason HTTP remotes are special is because there is no ready\n\"http-backend\" equivalent that passes the \"git archive\" traffic over\nsmart-http transport, unlike the one that exists for \"git\nupload-pack\".\n\nIt however still _is_ attractive to drive such a hack from \"git\narchive\" at the UI level, as the end users do not care how ugly the\nhack is ;-)  As you mentioned, the approach would work for any\ntransport that allows one-commit shallow clone, so it might become\nmore palatable if it is designed as a different _mode_ of operation\nof \"git archive\" that is orthogonal to the underlying transport,\ni.e.\n\n  $ git archive --remote=<repo> --shallow-clone-then-local-archive-hack master\n\nor\n\n  $ git config archive.<repo>.useShallowCloneThenLocalArchiveHack true\n  $ git archive --remote=<repo> master\n\nIt might turn out that it may work better than the native \"git\narchive\" access against servers that offer both shallow clone\nand native archive access.  I doubt a single-commit shallow clone\nwould benefit from reusing of premade deltas and compressed bases\nstreamed straight out of packfiles from the server side that much,\nbut you'd never know until you measure ;-)\n\n\n"},{"id":"353802","messageId":"20180728185251.GB945730@genre.crustytoothpaste.net","threadId":"48966","inReplyTo":"CANq=j3tK7QeBJOC7VNWkh4+WBNibMJJp5YUkd9te5NaYwukAow@mail.gmail.com","subject":"Re: Proposed approaches to supporting HTTP remotes in \"git archive\"","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-07-28T18:52:51Z","receivedAt":"2018-07-28T18:52:59Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Jul 27, 2018 at 02:47:00PM -0700, Josh Steadmon wrote:\n> ## Use git-upload-archive\n> \n> This approach requires adding support for the git-upload-archive\n> endpoint to the HTTP backend. Clients will connect to the remote\n> server's git-upload-archive service and the server will generate the\n> archive which is then delivered to the client.\n> \n> ### Benefits\n> \n> * Matches existing \"git archive\" behavior for other remotes.\n> \n> * Requires less bandwidth to send a compressed archive than a shallow\n>   clone.\n> \n> * Resulting archive does not depend in any way on the client\n>   implementation.\n> \n> ### Drawbacks\n> \n> * Implementation is more complicated; it will require changes to (at\n>   least) builtin/archive.c, http-backend.c, and\n>   builtin/upload-archive.c.\n> \n> * Generates more CPU load on the server when compressing archives. This\n>   is potentially a DoS vector.\n> \n> * Does not allow archiving from servers that don't support the\n>   git-upload-archive service.\n\nI happen to like this option because it has the potential to be driven\nby a non-git client (e.g. a curl invocation).  That would be enormously\nvaluable, especially in cases where authentication isn't desired or an\nSSH key isn't a good form of authentication.\n\nI'm not really worried about the DoS vector because an implementation is\nalmost certainly going to support both SSH and HTTPS or neither, and the\nDoS potential is the same either way.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"353820","messageId":"e769cd31-49cb-c49b-7653-5692e065f097@web.de","threadId":"48966","inReplyTo":"xmqqtvokjonm.fsf@gitster-ct.c.googlers.com","subject":"Re: Proposed approaches to supporting HTTP remotes in \"git archive\"","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-07-29T11:54:37Z","receivedAt":"2018-07-29T11:54:48Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 28.07.2018 um 00:32 schrieb Junio C Hamano:\n> Josh Steadmon <steadmon@google.com> writes:\n> \n>> # Supporting HTTP remotes in \"git archive\"\n>>\n>> We would like to allow remote archiving from HTTP servers. There are a\n>> few possible implementations to be discussed:\n>>\n>> ## Shallow clone to temporary repo\n>>\n>> This approach builds on existing endpoints. Clients will connect to the\n>> remote server's git-upload-pack service and fetch a shallow clone of the\n>> requested commit into a temporary local repo. The write_archive()\n>> function is then called on the local clone to write out the requested\n>> archive.\n\nA prototype would require just a few lines of shell script, I guess..\n\nA downside that was only stated implicitly: This method needs temporary\ndisk space for the clone, while the existing archive modes only ever\nwrite out the resulting file.  I guess the required space is in the same\norder as the compressed archive.  This shouldn't be a problem if we\nassume the user would eventually want to extract its contents, right?\n\n>> ## Summary\n>>\n>> Personally, I lean towards the first approach. It could give us an\n>> opportunity to remove server-side complexity; there is no reason that\n>> the shallow-clone approach must be restricted to the HTTP transport, and\n>> we could re-implement other transports using this method.  Additionally,\n>> it would allow clients to pull archives from remotes that would not\n>> otherwise support it.\n> \n> I consider the first one (i.e. make a shallow clone and tar it up\n> locally) a hack that does *not* belong to \"git archive --remote\"\n> command, especially when it is only done to \"http remotes\".  The\n> only reason HTTP remotes are special is because there is no ready\n> \"http-backend\" equivalent that passes the \"git archive\" traffic over\n> smart-http transport, unlike the one that exists for \"git\n> upload-pack\".\n> \n> It however still _is_ attractive to drive such a hack from \"git\n> archive\" at the UI level, as the end users do not care how ugly the\n> hack is ;-)  As you mentioned, the approach would work for any\n> transport that allows one-commit shallow clone, so it might become\n> more palatable if it is designed as a different _mode_ of operation\n> of \"git archive\" that is orthogonal to the underlying transport,\n> i.e.\n> \n>    $ git archive --remote=<repo> --shallow-clone-then-local-archive-hack master\n> \n> or\n> \n>    $ git config archive.<repo>.useShallowCloneThenLocalArchiveHack true\n>    $ git archive --remote=<repo> master\n\nArchive-via-clone would also work with full clones (if shallow ones are\nnot available), but that would be wasteful and a bit cruel, of course.\n\nAnyway, I think we should find a better (shorter) name for that option;\nthat could turn out to be the hardest part. :)\n\n> It might turn out that it may work better than the native \"git\n> archive\" access against servers that offer both shallow clone\n> and native archive access.  I doubt a single-commit shallow clone\n> would benefit from reusing of premade deltas and compressed bases\n> streamed straight out of packfiles from the server side that much,\n> but you'd never know until you measure ;-)\n\nIt could benefit from GIT_ALTERNATE_OBJECT_DIRECTORIES, but I guess\ntypical users of git archive --remote won't have any good ones lying\naround.\n\nRené\n"}]}