{"thread":{"id":"45377","subject":"Proposal for \"fetch-any-blob Git protocol\" and server design","startedAt":"2017-03-14T22:58:05Z","lastAt":"2017-04-26T22:51:42Z","messageCount":10,"participants":["Jonathan Tan","Junio C Hamano","Stefan Beller","Kevin David"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"314102","messageId":"ffd92ad9-39fe-c76b-178d-6e3d6a425037@google.com","threadId":"45377","inReplyTo":null,"subject":"Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-03-14T22:57:57Z","receivedAt":"2017-03-14T22:58:05Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"As described in \"Background\" below, there have been at least 2 patch \nsets to support \"partial clones\" and on-demand blob fetches, where the \nserver part that supports on-demand blob fetches was treated at least in \noutline. Here is a proposal treating that server part in detail.\n\n== Background\n\nThe desire for Git to support (i) missing blobs and (ii) fetching them \nas needed from a remote repository has surfaced on the mailing list a \nfew times, most recently in the form of RFC patch sets [1] [2].\n\nA local repository that supports (i) will be created by a \"partial \nclone\", that is, a clone with some special parameters (exact parameters \nare still being discussed) that does not download all blobs normally \ndownloaded. Such a repository should support (ii), which is what this \nproposal describes.\n\n== Design\n\nA new endpoint \"server\" is created. The client will send a message in \nthe following format:\n\n----\nfbp-request = PKT-LINE(\"fetch-blob-pack\")\n               1*want\n               flush-pkt\nwant = PKT-LINE(\"want\" SP obj-id)\n----\n\nThe client may send one or more SHA-1s for which it wants blobs, then a \nflush-pkt.\n\nThe server will then reply:\n\n----\nserver-reply = flush-pkt | PKT-LINE(\"ERR\" SP message)\n----\n\nIf there was no error, the server will then send them in a packfile, \nformatted like described in \"Packfile Data\" in pack-protocol.txt with \n\"side-band-64k\" enabled.\n\nAny server that supports \"partial clone\" will also support this, and the \nclient will automatically assume this. (How a client discovers \"partial \nclone\" is not covered by this proposal.)\n\nThe server will perform reachability checks on requested blobs through \nthe equivalent of \"git rev-list --use-bitmap-index\" (like \"git \nupload-pack\" when using the allowreachablesha1inwant option), unless \nconfigured to suppress reachability checks through a config option. The \nserver administrator is highly recommended to regularly regenerate the \nbitmap (or suppress reachability checks).\n\n=== Endpoint support for forward compatibility\n\nThis \"server\" endpoint requires that the first line be understood, but \nwill ignore any other lines starting with words that it does not \nunderstand. This allows new \"commands\" to be added (distinguished by \ntheir first lines) and existing commands to be \"upgraded\" with backwards \ncompatibility.\n\n=== Related improvements possible with new endpoint\n\nPrevious protocol upgrade suggestions have had to face the difficulty of \nallowing updated clients to discover the server support while not \nslowing down (for example, through extra network round-trips) any \nclient, whether non-updated or updated. The introduction of \"partial \nclone\" allows clients to rely on the guarantee that any server that \nsupports \"partial clone\" supports \"fetch-blob-pack\", and we can extend \nthe guarantee to other protocol upgrades that such repos would want.\n\nOne such upgrade is \"ref-in-want\" [3]. The full details can be obtained \nfrom that email thread, but to summarize, the patch set eliminates the \nneed for the initial ref advertisement and allows communication in ref \nname globs, making it much easier for multiple load-balanced servers to \nserve large repos to clients - this is something that would greatly \nbenefit the Android project, for example, and possibly many others.\n\nBundling support for \"ref-in-want\" with \"fetch-blob-pack\" simplifies \nmatters for the client in that a client needs to only handle one \n\"version\" of server (a server that supports both). If \"ref-in-want\" were \nadded later, instead of now, clients would need to be able to handle two \n\"versions\" (one with only \"fetch-blob-pack\" and one with both \n\"fetch-blob-pack\" and \"ref-in-want\").\n\nAs for its implementation, that email thread already contains a patch \nset that makes it work with the existing \"upload-pack\" endpoint; I can \nupdate that patch set to use the proposed \"server\" endpoint (with a \n\"fetch-commit-pack\" message) if need be.\n\n== Client behavior\n\nThis proposal is concerned with server behavior only, but it is useful \nto envision how the client would use this to ensure that the server \nbehavior is useful.\n\n=== Indication to use the proposed endpoint\n\nThe client will probably already record that at least one of its remotes \n(the one that it successfully performed a \"partial clone\" from) supports \nthis new endpoint (if not, it can’t determine whether a missing blob was \ncaused by repo corruption or by the \"partial clone\"). This knowledge can \nbe used both to know that the server supports \"fetch-blob-pack\" and \n\"fetch-commit-pack\" (for the latter, the client can fall back to \n\"fetch-pack\"/\"upload-pack\" when fetching from other servers).\n\n=== Multiple remotes\n\nFetches of missing blobs should (at least by default?) go to the remote \nthat sent the tree that points to them. This means that if there are \nmultiple remotes, the client needs to remember which remote it learned \nabout a given missing blob from.\n\n== Alternatives considered\n\nThe \"fetch-blob-pack\" and \"fetch-commit-pack\" messages could be split \ninto their own endpoints. It seemed more reasonable to combine them \ntogether since they serve similar use cases (large repos), and (for \nexample) reduces the number of binaries in PATH, but I do not feel \nstrongly about this.\n\nThe client could supply commit information about the blobs it wants (or \nother information that could help the reachability analysis). However, \nthese lines wouldn’t be used by the proposed server design. And if we do \ndiscover that these lines are useful, the protocol could be extended \nwith new lines that contain this information (since old servers will \nignore all lines that they do not understand).\n\nWe could extend \"upload-pack\" to allow blobs in \"want\" lines instead of \nhaving a new endpoint. Due to a quirk in the Git implementation (but \npossibly not other implementations like JGit), this is already supported \n[4]. However, each invocation would require the server to generate an \nunnecessary ref list, and would require both the server and the client \nto undergo more network traffic.\n\nAlso, the new \"server\" endpoint might be made to be discovered through \nanother mechanism (for example, a capability advertisement on another \nendpoint). It is probably simpler to tie it to the \"partial clone\" \nfeature, though, since they are so likely to be used together.\n\n[1] <20170304191901.9622-1-markbt@efaref.net>\n[2] <1488999039-37631-1-git-send-email-git@jeffhostetler.com>\n[3] <cover.1485381677.git.jonathantanmy@google.com>\n[4] <20170309003547.6930-1-jonathantanmy@google.com>\n"},{"id":"314151","messageId":"xmqqinnafml4.fsf@gitster.mtv.corp.google.com","threadId":"45377","inReplyTo":"ffd92ad9-39fe-c76b-178d-6e3d6a425037@google.com","subject":"Re: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-15T17:59:03Z","receivedAt":"2017-03-15T17:59:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> == Design\n>\n> A new endpoint \"server\" is created. The client will send a message in\n> the following format:\n>\n> ----\n> fbp-request = PKT-LINE(\"fetch-blob-pack\")\n>               1*want\n>               flush-pkt\n> want = PKT-LINE(\"want\" SP obj-id)\n> ----\n>\n> The client may send one or more SHA-1s for which it wants blobs, then\n> a flush-pkt.\n\nBy \"SHA-1s for which it wants blobs\", you mean that \"want\" only\nallows one exact blob object name?  I think it is necessary to\nsupport that mode of operation as a base case, and it is a good\nstarting point.\n\nWhen you know\n\n - you have a \"partial\" clone that initially asked to contain only\n   blobs that are smaller than 10MB, and\n\n - you are now trying to do a \"git checkout v1.0 -- this/directory\"\n   so that the directory is fully populated\n\ninstead of enumerating all the missing blobs from the output of\n\"ls-tree -r v1.0 this/directory\" on separate \"want\" requests, you\nmay want to say \"I want all the blobs that are not smaller than 10MB\nin this tree object $(git rev-parse v1.0:this/directory)\".\n\nI am not saying that you should add something like this right away,\nbut I am wondering how you would extend the proposed system to do\nso.  Would you add \"fetch-size-limited-blob-in-tree-pack\" that runs\nparallel to \"fetch-blob-pack\" request?  Would you add a new type of\nrequest packet \"want-blob-with-expression\" for fbp-request, which is\nprotected by some \"protocol capability\" exchange?\n\nIf the former, how does a client discover if a particular server\nalready supports the new \"fetch-size-limited-blob-in-tree-pack\"\nrequest, so that it does not have to send a bunch of \"want\" request\nby enumerating the blobs itself?  If the latter, how does a client\ndiscover if a particular server's \"fetch-blob-pack\" already supports\nthe new \"want-blob-with-expression\" request packet?\n\n> === Endpoint support for forward compatibility\n>\n> This \"server\" endpoint requires that the first line be understood, but\n> will ignore any other lines starting with words that it does not\n> understand. This allows new \"commands\" to be added (distinguished by\n> their first lines) and existing commands to be \"upgraded\" with\n> backwards compatibility.\n"},{"id":"314233","messageId":"90381e66-d91f-6412-6294-701f5f780645@google.com","threadId":"45377","inReplyTo":"xmqqinnafml4.fsf@gitster.mtv.corp.google.com","subject":"Re: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-03-16T17:31:09Z","receivedAt":"2017-03-16T17:31:31Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 03/15/2017 10:59 AM, Junio C Hamano wrote:\n> By \"SHA-1s for which it wants blobs\", you mean that \"want\" only\n> allows one exact blob object name?  I think it is necessary to\n> support that mode of operation as a base case, and it is a good\n> starting point.\n>\n> When you know\n>\n>  - you have a \"partial\" clone that initially asked to contain only\n>    blobs that are smaller than 10MB, and\n>\n>  - you are now trying to do a \"git checkout v1.0 -- this/directory\"\n>    so that the directory is fully populated\n>\n> instead of enumerating all the missing blobs from the output of\n> \"ls-tree -r v1.0 this/directory\" on separate \"want\" requests, you\n> may want to say \"I want all the blobs that are not smaller than 10MB\n> in this tree object $(git rev-parse v1.0:this/directory)\".\n>\n> I am not saying that you should add something like this right away,\n> but I am wondering how you would extend the proposed system to do\n> so.  Would you add \"fetch-size-limited-blob-in-tree-pack\" that runs\n> parallel to \"fetch-blob-pack\" request?  Would you add a new type of\n> request packet \"want-blob-with-expression\" for fbp-request, which is\n> protected by some \"protocol capability\" exchange?\n>\n> If the former, how does a client discover if a particular server\n> already supports the new \"fetch-size-limited-blob-in-tree-pack\"\n> request, so that it does not have to send a bunch of \"want\" request\n> by enumerating the blobs itself?  If the latter, how does a client\n> discover if a particular server's \"fetch-blob-pack\" already supports\n> the new \"want-blob-with-expression\" request packet?\n\nI'm not sure if that use case is something we need to worry about (if \nyou're downloading x * 10MB, uploading x * 50B shouldn't be a problem, I \nthink), but if we want to handle that use case in the future, I agree \nthat extending this system would be difficult.\n\nThe best way I can think of right now is for the client to send a \nfetch-blob-pack request with no \"want\" lines and at least one \n\"want-tree\" line, and then if there is an error (which will happen if \nthe server is old, and therefore sees that there is not at least \"want\" \nline), to retry with the \"want\" lines. This allows us to add alternative \nways of specifying blobs later (if we want to), but also means that \nupgrading a client without upgrading the corresponding server incurs a \nround-trip penalty.\n\nAlternatively we could add rudimentary support for trees now and add \nfilter-by-size later (so that such requests made to old servers will \ndownload extra blobs, but at least it works), but it still doesn't solve \nthe general problem of specifying blobs by some other rule than its own \nSHA-1 or its tree's SHA-1.\n\n"},{"id":"314258","messageId":"xmqq7f3oc45v.fsf@gitster.mtv.corp.google.com","threadId":"45377","inReplyTo":"90381e66-d91f-6412-6294-701f5f780645@google.com","subject":"Re: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-16T21:17:32Z","receivedAt":"2017-03-16T21:17:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> On 03/15/2017 10:59 AM, Junio C Hamano wrote:\n>> ...\n>> but I am wondering how you would extend the proposed system to do\n>> so.  Would you add \"fetch-size-limited-blob-in-tree-pack\" that runs\n>> parallel to \"fetch-blob-pack\" request?  Would you add a new type of\n>> request packet \"want-blob-with-expression\" for fbp-request, which is\n>> protected by some \"protocol capability\" exchange?\n>>\n>> If the former, how does a client discover if a particular server\n>> already supports the new \"fetch-size-limited-blob-in-tree-pack\"\n>> request, so that it does not have to send a bunch of \"want\" request\n>> by enumerating the blobs itself?  If the latter, how does a client\n>> discover if a particular server's \"fetch-blob-pack\" already supports\n>> the new \"want-blob-with-expression\" request packet?\n>\n> I'm not sure if that use case is something we need to worry about (if\n> you're downloading x * 10MB, uploading x * 50B shouldn't be a problem,\n> I think), but if we want to handle that use case in the future, I\n> agree that extending this system would be difficult.\n\nYeah, the example was solely to see how the system was to be\nextended, as one of the selling point of the proposal was:\n\n    > === Endpoint support for forward compatibility\n    >\n    > This \"server\" endpoint requires that the first line be understood, but\n    > will ignore any other lines starting with words that it does not\n    > understand. This allows new \"commands\" to be added (distinguished by\n    > their first lines) and existing commands to be \"upgraded\" with\n    > backwards compatibility.\n\n> The best way I can think of right now is for the client to send a\n> fetch-blob-pack request with no \"want\" lines and at least one\n> \"want-tree\" line, ...\n\nSo it is not by adding new type of \"request\" that sits next to\n\"fetch-blob-pack\" request, but by adding a new way to drive that\nexisting \"fetch-blob-pack\" request.  \n\n> and then if there is an error (which will happen if\n> the server is old, and therefore sees that there is not at least\n> \"want\" line), to retry with the \"want\" lines. This allows us to add\n> alternative ways of specifying blobs later (if we want to), but also\n> means that upgrading a client without upgrading the corresponding\n> server incurs a round-trip penalty.\n\nAnd the lack of \"capability negotiation\" is substituted by \"assume\nthe better server, fallback to lower common denominator by detecting\nerrors\"?\n\n> Alternatively we could add rudimentary support for trees now and add\n> filter-by-size later ...\n\nI am not particularly interested in \"blobs in this tree\" request.\nIt was merely an example to make it easier to discuss the main\npoint, which is the bigger picture that the proposal was painting\naround \"forward compatibility\".\n\nThanks.\n"},{"id":"314287","messageId":"3c89fee5-2226-6760-f2fd-222432230612@google.com","threadId":"45377","inReplyTo":"xmqq7f3oc45v.fsf@gitster.mtv.corp.google.com","subject":"Re: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-03-16T22:48:19Z","receivedAt":"2017-03-16T22:58:28Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 03/16/2017 02:17 PM, Junio C Hamano wrote:\n> Yeah, the example was solely to see how the system was to be\n> extended, as one of the selling point of the proposal was:\n>\n>     > === Endpoint support for forward compatibility\n>     >\n>     > This \"server\" endpoint requires that the first line be understood, but\n>     > will ignore any other lines starting with words that it does not\n>     > understand. This allows new \"commands\" to be added (distinguished by\n>     > their first lines) and existing commands to be \"upgraded\" with\n>     > backwards compatibility.\n\n<snip>\n\n>\n> And the lack of \"capability negotiation\" is substituted by \"assume\n> the better server, fallback to lower common denominator by detecting\n> errors\"?\n\nYes. I probably should have mentioned that this \"forward compatibility\" \nis limited - it does not include any potential new feature intending to \nreduce the size of the request. I was thinking more of this being able \nto be extended to, for example, add \"hint\" lines that certain blobs come \nfrom certain commits, or add \"have\" lines to present blobs that might be \ngood delta bases (and even if the server doesn't understand these lines, \nits output is still correct).\n"},{"id":"315632","messageId":"CAGZ79ka-YaF5dBXJmhXsfwwrwSBy0tkun0yDysG581E0mpqDVw@mail.gmail.com","threadId":"45377","inReplyTo":"ffd92ad9-39fe-c76b-178d-6e3d6a425037@google.com","subject":"Re: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-03-28T23:19:33Z","receivedAt":"2017-03-28T23:19:41Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"+cc Ben Peart, who sent\n\"[RFC] Add support for downloading blobs on demand\" to the list recently.\nThis proposal here seems like it has the same goal, so maybe your review\ncould go a long way here?\n\nThanks,\nStefan\n\nOn Tue, Mar 14, 2017 at 3:57 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> As described in \"Background\" below, there have been at least 2 patch sets to\n> support \"partial clones\" and on-demand blob fetches, where the server part\n> that supports on-demand blob fetches was treated at least in outline. Here\n> is a proposal treating that server part in detail.\n>\n> == Background\n>\n> The desire for Git to support (i) missing blobs and (ii) fetching them as\n> needed from a remote repository has surfaced on the mailing list a few\n> times, most recently in the form of RFC patch sets [1] [2].\n>\n> A local repository that supports (i) will be created by a \"partial clone\",\n> that is, a clone with some special parameters (exact parameters are still\n> being discussed) that does not download all blobs normally downloaded. Such\n> a repository should support (ii), which is what this proposal describes.\n>\n> == Design\n>\n> A new endpoint \"server\" is created. The client will send a message in the\n> following format:\n>\n> ----\n> fbp-request = PKT-LINE(\"fetch-blob-pack\")\n>               1*want\n>               flush-pkt\n> want = PKT-LINE(\"want\" SP obj-id)\n> ----\n>\n> The client may send one or more SHA-1s for which it wants blobs, then a\n> flush-pkt.\n>\n> The server will then reply:\n>\n> ----\n> server-reply = flush-pkt | PKT-LINE(\"ERR\" SP message)\n> ----\n>\n> If there was no error, the server will then send them in a packfile,\n> formatted like described in \"Packfile Data\" in pack-protocol.txt with\n> \"side-band-64k\" enabled.\n>\n> Any server that supports \"partial clone\" will also support this, and the\n> client will automatically assume this. (How a client discovers \"partial\n> clone\" is not covered by this proposal.)\n>\n> The server will perform reachability checks on requested blobs through the\n> equivalent of \"git rev-list --use-bitmap-index\" (like \"git upload-pack\" when\n> using the allowreachablesha1inwant option), unless configured to suppress\n> reachability checks through a config option. The server administrator is\n> highly recommended to regularly regenerate the bitmap (or suppress\n> reachability checks).\n>\n> === Endpoint support for forward compatibility\n>\n> This \"server\" endpoint requires that the first line be understood, but will\n> ignore any other lines starting with words that it does not understand. This\n> allows new \"commands\" to be added (distinguished by their first lines) and\n> existing commands to be \"upgraded\" with backwards compatibility.\n>\n> === Related improvements possible with new endpoint\n>\n> Previous protocol upgrade suggestions have had to face the difficulty of\n> allowing updated clients to discover the server support while not slowing\n> down (for example, through extra network round-trips) any client, whether\n> non-updated or updated. The introduction of \"partial clone\" allows clients\n> to rely on the guarantee that any server that supports \"partial clone\"\n> supports \"fetch-blob-pack\", and we can extend the guarantee to other\n> protocol upgrades that such repos would want.\n>\n> One such upgrade is \"ref-in-want\" [3]. The full details can be obtained from\n> that email thread, but to summarize, the patch set eliminates the need for\n> the initial ref advertisement and allows communication in ref name globs,\n> making it much easier for multiple load-balanced servers to serve large\n> repos to clients - this is something that would greatly benefit the Android\n> project, for example, and possibly many others.\n>\n> Bundling support for \"ref-in-want\" with \"fetch-blob-pack\" simplifies matters\n> for the client in that a client needs to only handle one \"version\" of server\n> (a server that supports both). If \"ref-in-want\" were added later, instead of\n> now, clients would need to be able to handle two \"versions\" (one with only\n> \"fetch-blob-pack\" and one with both \"fetch-blob-pack\" and \"ref-in-want\").\n>\n> As for its implementation, that email thread already contains a patch set\n> that makes it work with the existing \"upload-pack\" endpoint; I can update\n> that patch set to use the proposed \"server\" endpoint (with a\n> \"fetch-commit-pack\" message) if need be.\n>\n> == Client behavior\n>\n> This proposal is concerned with server behavior only, but it is useful to\n> envision how the client would use this to ensure that the server behavior is\n> useful.\n>\n> === Indication to use the proposed endpoint\n>\n> The client will probably already record that at least one of its remotes\n> (the one that it successfully performed a \"partial clone\" from) supports\n> this new endpoint (if not, it can’t determine whether a missing blob was\n> caused by repo corruption or by the \"partial clone\"). This knowledge can be\n> used both to know that the server supports \"fetch-blob-pack\" and\n> \"fetch-commit-pack\" (for the latter, the client can fall back to\n> \"fetch-pack\"/\"upload-pack\" when fetching from other servers).\n>\n> === Multiple remotes\n>\n> Fetches of missing blobs should (at least by default?) go to the remote that\n> sent the tree that points to them. This means that if there are multiple\n> remotes, the client needs to remember which remote it learned about a given\n> missing blob from.\n>\n> == Alternatives considered\n>\n> The \"fetch-blob-pack\" and \"fetch-commit-pack\" messages could be split into\n> their own endpoints. It seemed more reasonable to combine them together\n> since they serve similar use cases (large repos), and (for example) reduces\n> the number of binaries in PATH, but I do not feel strongly about this.\n>\n> The client could supply commit information about the blobs it wants (or\n> other information that could help the reachability analysis). However, these\n> lines wouldn’t be used by the proposed server design. And if we do discover\n> that these lines are useful, the protocol could be extended with new lines\n> that contain this information (since old servers will ignore all lines that\n> they do not understand).\n>\n> We could extend \"upload-pack\" to allow blobs in \"want\" lines instead of\n> having a new endpoint. Due to a quirk in the Git implementation (but\n> possibly not other implementations like JGit), this is already supported\n> [4]. However, each invocation would require the server to generate an\n> unnecessary ref list, and would require both the server and the client to\n> undergo more network traffic.\n>\n> Also, the new \"server\" endpoint might be made to be discovered through\n> another mechanism (for example, a capability advertisement on another\n> endpoint). It is probably simpler to tie it to the \"partial clone\" feature,\n> though, since they are so likely to be used together.\n>\n> [1] <20170304191901.9622-1-markbt@efaref.net>\n> [2] <1488999039-37631-1-git-send-email-git@jeffhostetler.com>\n> [3] <cover.1485381677.git.jonathantanmy@google.com>\n> [4] <20170309003547.6930-1-jonathantanmy@google.com>\n"},{"id":"316703","messageId":"BY2PR21MB0052067C2CDC6F727CCB6969EE030@BY2PR21MB0052.namprd21.prod.outlook.com","threadId":"45377","inReplyTo":"00bf01d2aed7$b13492a0$139db7e0$@gmail.com","subject":"RE: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Kevin David","fromEmail":"kevin.david@microsoft.com","sentAt":"2017-04-12T22:02:31Z","receivedAt":"2017-04-12T22:02:40Z","isPatch":false,"sender":{"key":"kevin.david@microsoft.com","avatar":null},"body":"Hi Jonathan, \n\nI work on the network protocols for the GVFS project at Microsoft.\nI shared a couple thoughts and questions below.\n\nThanks,\n\nKevin\n\n-----Original Message-----\nFrom: Stefan Beller [mailto:sbeller@google.com]\nSent: Tuesday, March 28, 2017 7:20 PM\nTo: Jonathan Tan <jonathantanmy@google.com>; Ben Peart <peartben@gmail.com>\nCc: git@vger.kernel.org; Mark Thomas <markbt@efaref.net>; Jeff Hostetler <git@jeffhostetler.com>\nSubject: Re: Proposal for \"fetch-any-blob Git protocol\" and server design\n\n+cc Ben Peart, who sent\n\"[RFC] Add support for downloading blobs on demand\" to the list recently.\nThis proposal here seems like it has the same goal, so maybe your review could go a long way here?\n\nThanks,\nStefan\n\nOn Tue, Mar 14, 2017 at 3:57 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> As described in \"Background\" below, there have been at least 2 patch \n> sets to support \"partial clones\" and on-demand blob fetches, where the \n> server part that supports on-demand blob fetches was treated at least \n> in outline. Here is a proposal treating that server part in detail.\n>\n> == Background\n>\n> The desire for Git to support (i) missing blobs and (ii) fetching them \n> as needed from a remote repository has surfaced on the mailing list a \n> few times, most recently in the form of RFC patch sets [1] [2].\n>\n> A local repository that supports (i) will be created by a \"partial \n> clone\", that is, a clone with some special parameters (exact \n> parameters are still being discussed) that does not download all blobs \n> normally downloaded. Such a repository should support (ii), which is what this proposal describes.\n>\n> == Design\n>\n> A new endpoint \"server\" is created. The client will send a message in \n> the following format:\n>\n> ----\n> fbp-request = PKT-LINE(\"fetch-blob-pack\")\n>               1*want\n>               flush-pkt\n> want = PKT-LINE(\"want\" SP obj-id)\n> ----\n>\n> The client may send one or more SHA-1s for which it wants blobs, then \n> a flush-pkt.\n\nI know we're considering server behavior here, but how large do you generally\nexpect these blob-want requests to be? I ask because we took an initial approach\nvery similar to this, however, we had a hard time being clever about figuring out\nwhat set of blobs to request for those clients that didn't want the entire set, and\nended up falling back to single-blob requests. \n\nObviously, this could be due to thenature of our filesystem-virtualization-based client, \nbut I also suspect that the teams attacking this problem are more often than not dealing\nwith very large blob objects, so the cost of a round-trip becomes lower relative to sending\nthe object content itself.\n\n>\n> The server will then reply:\n>\n> ----\n> server-reply = flush-pkt | PKT-LINE(\"ERR\" SP message)\n> ----\n>\n> If there was no error, the server will then send them in a packfile, \n> formatted like described in \"Packfile Data\" in pack-protocol.txt with \n> \"side-band-64k\" enabled.\n> Any server that supports \"partial clone\" will also support this, and \n> the client will automatically assume this. (How a client discovers \n> \"partial clone\" is not covered by this proposal.)\n\nAlong the same lines as above, this is where we started and it worked well for\nlow-volume requests. However, when we started ramping up the load,\n`pack-objects` operating on a very large packed repository (~150 GiB) became\nvery computationally expensive, even with `--compression=1 --depth=0 --window=0`.\n\nBeing a bit more clever about packing objects (e.g. splitting blobs out from commits\nand trees) improved this a bit, but we still hit a bottlenecks from what appeared to\nbe a large number of memory-mapping operations on a ~140GiB packfile of blobs.\n\nEach `pack-objects` process would consume approximately one CPU core for the\nduration of the request. It's possible that further splitting of these large blob packs\nwould have improved performance in some scenarios, but that would increase the\namount of pack-index lookups necessary to find a single object.\n\n>\n> The server will perform reachability checks on requested blobs through \n> the equivalent of \"git rev-list --use-bitmap-index\" (like \"git \n> upload-pack\" when using the allowreachablesha1inwant option), unless \n> configured to suppress reachability checks through a config option.\n> The server administrator is highly recommended to regularly regenerate \n> the bitmap (or suppress reachability checks).\n>\n> === Endpoint support for forward compatibility\n>\n> This \"server\" endpoint requires that the first line be understood, but \n> will ignore any other lines starting with words that it does not \n> understand. This allows new \"commands\" to be added (distinguished by \n> their first lines) and existing commands to be \"upgraded\" with backwards compatibility.\n\nThis seems like a clever way to avoid the canonical `/info/refs?service=git-upload-pack` \ncapability negotiation on every call. However, using error handling to fallback seems\nslightly wonky to me. Hopefully users are incentivized to upgrade their clients.\n\n> === Related improvements possible with new endpoint\n>\n> Previous protocol upgrade suggestions have had to face the difficulty \n> of allowing updated clients to discover the server support while not \n> slowing down (for example, through extra network round-trips) any \n> client, whether non-updated or updated. The introduction of \"partial \n> clone\" allows clients to rely on the guarantee that any server that supports \"partial clone\"\n> supports \"fetch-blob-pack\", and we can extend the guarantee to other \n> protocol upgrades that such repos would want.\n>\n> One such upgrade is \"ref-in-want\" [3]. The full details can be \n> obtained from that email thread, but to summarize, the patch set \n> eliminates the need for the initial ref advertisement and allows \n> communication in ref name globs, making it much easier for multiple \n> load-balanced servers to serve large repos to clients - this is \n> something that would greatly benefit the Android project, for example, and possibly many others.\n>\n> Bundling support for \"ref-in-want\" with \"fetch-blob-pack\" simplifies \n> matters for the client in that a client needs to only handle one \n> \"version\" of server (a server that supports both). If \"ref-in-want\"\n> were added later, instead of now, clients would need to be able to \n> handle two \"versions\" (one with only \"fetch-blob-pack\" and one with both \"fetch-blob-pack\" and \"ref-in-want\").\n>\n> As for its implementation, that email thread already contains a patch \n> set that makes it work with the existing \"upload-pack\" endpoint; I can \n> update that patch set to use the proposed \"server\" endpoint (with a \n> \"fetch-commit-pack\" message) if need be.\n>\n> == Client behavior\n>\n> This proposal is concerned with server behavior only, but it is useful \n> to envision how the client would use this to ensure that the server \n> behavior is useful.\n>\n> === Indication to use the proposed endpoint\n>\n> The client will probably already record that at least one of its \n> remotes (the one that it successfully performed a \"partial clone\"\n> from) supports this new endpoint (if not, it can’t determine whether a \n> missing blob was caused by repo corruption or by the \"partial clone\").\n> This knowledge can be used both to know that the server supports \n> \"fetch-blob-pack\" and \"fetch-commit-pack\" (for the latter, the client \n> can fall back to \"fetch-pack\"/\"upload-pack\" when fetching from other servers).\n\nThis makes a lot of sense to me. When we built our caching proxy, we had to be careful\nwhen designing how we'd handle clients requesting objects missing from the proxy. \n\nFor example, a client requests a single blob and the proxy doesn't have it - we can't simply\ndownload that object from the \"authoritative\" remote and stick it in the `.git\\objects\\xx\\yyy...` \ndirectory, because the repository would be made corrupt. \n\nHaving a way to specify that the repo is a \"partial clone\" and allowing \"holes\" would help a lot,\nI believe. I know there have been varying opinions on how these missing objects should be\n\"marked\" and I'm not ready to propose anything there - just agreeing the problem is important.\n\n> === Multiple remotes\n>\n> Fetches of missing blobs should (at least by default?) go to the \n> remote that sent the tree that points to them. This means that if \n> there are multiple remotes, the client needs to remember which remote \n> it learned about a given missing blob from.\n\nDefinitely handy for someone writing a proxy, but some level of error fallback might be good.\n\n> == Alternatives considered\n>\n> The \"fetch-blob-pack\" and \"fetch-commit-pack\" messages could be split \n> into their own endpoints. It seemed more reasonable to combine them \n> together since they serve similar use cases (large repos), and (for\n> example) reduces the number of binaries in PATH, but I do not feel strongly about this.\n\nNot do derail us too far off blobs, but I wonder if we need a `fetch-commit-pack` endpoint,\nor could get away with introducing a new capability (e.g. `no-blobs`) to `upload-pack` instead.\nAs a casual observer, this seems like it would be a much smaller change since the rest of the \nnegotiation/reachability calculation would look the same, right? Or would this `fetch-commit-pack` \nnot return trees either? \n\nI only ask because, in our observations, when git wants to read commits it's \nusually followed by a lot of \"related\" trees - again caveated with the fact that \nwe're intercepting many things at the filesystem layer.\n\n> The client could supply commit information about the blobs it wants \n> (or other information that could help the reachability analysis).\n> However, these lines wouldn’t be used by the proposed server design. \n> And if we do discover that these lines are useful, the protocol could \n> be extended with new lines that contain this information (since old \n> servers will ignore all lines that they do not understand).\n>\n> We could extend \"upload-pack\" to allow blobs in \"want\" lines instead \n> of having a new endpoint. Due to a quirk in the Git implementation \n> (but possibly not other implementations like JGit), this is already \n> supported [4]. However, each invocation would require the server to \n> generate an unnecessary ref list, and would require both the server \n> and the client to undergo more network traffic.\n>\n> Also, the new \"server\" endpoint might be made to be discovered through \n> another mechanism (for example, a capability advertisement on another \n> endpoint). It is probably simpler to tie it to the \"partial clone\"\n> feature, though, since they are so likely to be used together.\n\nJust to keep the discussion interesting, I'll throw an alternative out there that's\nworked well for us. As I understand it, the HTTP-based dumb transfer protocol\nsupports returning objects in loose object format, but only if they already exist\nin loose format. \n\nExtending this to have the remote provide these objects via a \"dumb\" protocol\nwhen they are packed as well - i.e. the server would \"loosens\" them upon request -\nis basically what we do and it works quite well for low-latency clients. To further improve\nperformance at the cost of complexity, we've added caching at the memory and disk layer\nfor these loose objects in the same format we send to the client. \n\nThere's a clear tradeoff here - the servers must have adequate disk and/or memory to store\nthese loose objects in optimal format. In addition, the higher the latency is to the remote,\nthe worse this solution will perform. Fortunately, in our case, network topology allows us to\nput these caching proxies close enough to clients for it not to matter.\n\n> [1] <20170304191901.9622-1-markbt@efaref.net>\n> [2] <1488999039-37631-1-git-send-email-git@jeffhostetler.com>\n> [3] <cover.1485381677.git.jonathantanmy@google.com>\n> [4] <20170309003547.6930-1-jonathantanmy@google.com>\n\n"},{"id":"316779","messageId":"18ebcd04-4765-bdc7-3880-b0e8cb90d35c@google.com","threadId":"45377","inReplyTo":"BY2PR21MB0052067C2CDC6F727CCB6969EE030@BY2PR21MB0052.namprd21.prod.outlook.com","subject":"Re: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-04-13T20:12:54Z","receivedAt":"2017-04-13T20:13:03Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 04/12/2017 03:02 PM, Kevin David wrote:\n> Hi Jonathan,\n>\n> I work on the network protocols for the GVFS project at Microsoft.\n> I shared a couple thoughts and questions below.\n\nThanks for your reply!\n\n> I know we're considering server behavior here, but how large do you generally\n> expect these blob-want requests to be? I ask because we took an initial approach\n> very similar to this, however, we had a hard time being clever about figuring out\n> what set of blobs to request for those clients that didn't want the entire set, and\n> ended up falling back to single-blob requests.\n>\n> Obviously, this could be due to thenature of our filesystem-virtualization-based client,\n> but I also suspect that the teams attacking this problem are more often than not dealing\n> with very large blob objects, so the cost of a round-trip becomes lower relative to sending\n> the object content itself.\n\nI am envisioning (1a) as described in Jeff Hostetler's e-mail [1] (\"a \npre-command or hook to identify needed blobs and pre-fetch them before \nallowing the actual command to start\"), so a Git command would typically \nmake a single request that contains all the blobs required, but my \nproposal can also handle (1c) ('\"fault\" them in as necessary in \nread_object() while the command is running and without any pre-fetch \n(either synchronously or asynchronously and with/without a helper \nprocess)').\n\nEven if we decided to go with single-blob requests and responses, it is \nstill important to send them as packfiles, so that the server can serve \nthem directly from its compressed storage without first having to \nuncompress them.\n\n[1] \nhttps://public-inbox.org/git/1488999039-37631-1-git-send-email-git@jeffhostetler.com/\n\n> Along the same lines as above, this is where we started and it worked well for\n> low-volume requests. However, when we started ramping up the load,\n> `pack-objects` operating on a very large packed repository (~150 GiB) became\n> very computationally expensive, even with `--compression=1 --depth=0 --window=0`.\n>\n> Being a bit more clever about packing objects (e.g. splitting blobs out from commits\n> and trees) improved this a bit, but we still hit a bottlenecks from what appeared to\n> be a large number of memory-mapping operations on a ~140GiB packfile of blobs.\n>\n> Each `pack-objects` process would consume approximately one CPU core for the\n> duration of the request. It's possible that further splitting of these large blob packs\n> would have improved performance in some scenarios, but that would increase the\n> amount of pack-index lookups necessary to find a single object.\n\nI'm not very experienced with mmap, but I thought that memory-mapping a \nlarge file in itself does not incur much of a performance penalty (if \nany) - it is the accesses that count. I experimented with 15,000 and \n150,000 MiB files and mmap and they seem to be handled quite well. Also, \nhow many objects are \"pack-objects\" packing here?\n\n>> === Endpoint support for forward compatibility\n>>\n>> This \"server\" endpoint requires that the first line be understood, but\n>> will ignore any other lines starting with words that it does not\n>> understand. This allows new \"commands\" to be added (distinguished by\n>> their first lines) and existing commands to be \"upgraded\" with backwards compatibility.\n>\n> This seems like a clever way to avoid the canonical `/info/refs?service=git-upload-pack`\n> capability negotiation on every call. However, using error handling to fallback seems\n> slightly wonky to me. Hopefully users are incentivized to upgrade their clients.\n\nBy \"error handling to fallback\", do you mean in my proposal or in a \npossible future one (assuming my proposal is implemented)? I don't think \nmy proposal requires any error handling to fallback (since only new \nclients can clone partially - old clients will just clone totally and \nobliviously), but I acknowledge that this proposal does not mean that \nany future proposal can be done without requiring error handling to \nfallback.\n\n>> === Indication to use the proposed endpoint\n>>\n>> The client will probably already record that at least one of its\n>> remotes (the one that it successfully performed a \"partial clone\"\n>> from) supports this new endpoint (if not, it can’t determine whether a\n>> missing blob was caused by repo corruption or by the \"partial clone\").\n>> This knowledge can be used both to know that the server supports\n>> \"fetch-blob-pack\" and \"fetch-commit-pack\" (for the latter, the client\n>> can fall back to \"fetch-pack\"/\"upload-pack\" when fetching from other servers).\n>\n> This makes a lot of sense to me. When we built our caching proxy, we had to be careful\n> when designing how we'd handle clients requesting objects missing from the proxy.\n>\n> For example, a client requests a single blob and the proxy doesn't have it - we can't simply\n> download that object from the \"authoritative\" remote and stick it in the `.git\\objects\\xx\\yyy...`\n> directory, because the repository would be made corrupt.\n\nBy proxy, do you mean a Git repository? Sorry, I don't really understand \nthis part.\n\n> Having a way to specify that the repo is a \"partial clone\" and allowing \"holes\" would help a lot,\n> I believe. I know there have been varying opinions on how these missing objects should be\n> \"marked\" and I'm not ready to propose anything there - just agreeing the problem is important.\n\nAgreed.\n\n> Not do derail us too far off blobs, but I wonder if we need a `fetch-commit-pack` endpoint,\n> or could get away with introducing a new capability (e.g. `no-blobs`) to `upload-pack` instead.\n> As a casual observer, this seems like it would be a much smaller change since the rest of the\n> negotiation/reachability calculation would look the same, right? Or would this `fetch-commit-pack`\n> not return trees either?\n>\n> I only ask because, in our observations, when git wants to read commits it's\n> usually followed by a lot of \"related\" trees - again caveated with the fact that\n> we're intercepting many things at the filesystem layer.\n\nThe main reason for this extra command is not to exclude blobs (which, \nas you said, can be done with a new capability - I suspect that we will \nneed a capability or parameter of some sort anyway to indicate which \nsize of blobs to filter out) but to eliminate the mandatory ref \nadvertisement that is done whenever the client fetches. One of our use \ncases (internal Android) has large blobs and many (more than 700k) refs, \nso it would benefit greatly from blob filtering and elimination of the \nmandatory ref advertisement (tens of megabytes per fetch).\n\nAs for the size of the change, I have a work in progress that implements \nthis [2].\n\n[2] \nhttps://public-inbox.org/git/cover.1491851452.git.jonathantanmy@google.com/\n\n> Just to keep the discussion interesting, I'll throw an alternative out there that's\n> worked well for us. As I understand it, the HTTP-based dumb transfer protocol\n> supports returning objects in loose object format, but only if they already exist\n> in loose format.\n>\n> Extending this to have the remote provide these objects via a \"dumb\" protocol\n> when they are packed as well - i.e. the server would \"loosens\" them upon request -\n> is basically what we do and it works quite well for low-latency clients. To further improve\n> performance at the cost of complexity, we've added caching at the memory and disk layer\n> for these loose objects in the same format we send to the client.\n>\n> There's a clear tradeoff here - the servers must have adequate disk and/or memory to store\n> these loose objects in optimal format. In addition, the higher the latency is to the remote,\n> the worse this solution will perform. Fortunately, in our case, network topology allows us to\n> put these caching proxies close enough to clients for it not to matter.\n\nThis does make sense in the situation you describe, but (as you said) I \ndon't think we can guarantee this in the majority of situations. I think \nsome sort of batching (like the (1a) solution I talked about near the \nstart of this e-mail) and serving packed data from packed storage should \nform the baseline, and any situation-specific optimizations (e.g. \nserving unpacked data from topologically-close servers) can be \nadditional steps.\n"},{"id":"317493","messageId":"CY4PR21MB05047397A12B8692C923FC67EE1A0@CY4PR21MB0504.namprd21.prod.outlook.com","threadId":"45377","inReplyTo":"18ebcd04-4765-bdc7-3880-b0e8cb90d35c@google.com","subject":"RE: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Kevin David","fromEmail":"kevin.david@microsoft.com","sentAt":"2017-04-21T16:41:14Z","receivedAt":"2017-04-21T16:42:13Z","isPatch":false,"sender":{"key":"kevin.david@microsoft.com","avatar":null},"body":"Hi Jonathan,\n\nSorry for the delayed response - other work got in the way, unfortunately!\n\nKevin\n\n-----Original Message-----\nFrom: Jonathan Tan [mailto:jonathantanmy@google.com] \nSent: Thursday, April 13, 2017 4:13 PM\n\n>> I know we're considering server behavior here, but how large do you generally\n>> expect these blob-want requests to be? I ask because we took an initial approach\n>> very similar to this, however, we had a hard time being clever about figuring out\n>> what set of blobs to request for those clients that didn't want the entire set, and\n>> ended up falling back to single-blob requests.\n>>\n>> Obviously, this could be due to thenature of our filesystem-virtualization-based client,\n>> but I also suspect that the teams attacking this problem are more often than not dealing\n>> with very large blob objects, so the cost of a round-trip becomes lower relative to sending\n>> the object content itself.\n>\n> I am envisioning (1a) as described in Jeff Hostetler's e-mail [1] (\"a \n> pre-command or hook to identify needed blobs and pre-fetch them before \n> allowing the actual command to start\"), so a Git command would typically \n> make a single request that contains all the blobs required, but my \n> proposal can also handle (1c) ('\"fault\" them in as necessary in \n> read_object() while the command is running and without any pre-fetch \n> (either synchronously or asynchronously and with/without a helper \n> process)').\n> \n> Even if we decided to go with single-blob requests and responses, it is \n> still important to send them as packfiles, so that the server can serve \n> them directly from its compressed storage without first having to \n> uncompress them.\n> \n> [1] https://public-inbox.org/git/1488999039-37631-1-git-send-email-git@jeffhostetler.com/\n\nAh, I missed this. If we think we can build meaningfully-sized requests via (1a) and (1b), \nthen I agree - packs are optimal.\n\nHowever, if single-blob requests/responses dominate, I have a few concerns, mostly from \nexperience playing with something similar: \n* Regardless of the size of the object or number returned, the client will need to \n  `index-pack` the result and create a corresponding `.idx` file, which requires \n  decompression to construct (right?)\n\n* Unless the client's repository is repacked aggressively, we'll pollute the \n  `.git\\objects\\pack` directory with little indexes (somewhere around 4KiB minimum) \n  and packfiles rapidly, which would degrade performance. One rudimentary workaround \n  would be to loosen these packs on the client if they were under a certain \n  size/object count. I think fetch does this already?\n\nIn either case, shifting the pack decompression/loose object recompression problem \nto the client instead of the server is probably a good principle, but in our case it \nsimply wasn't fast enough to serve single blobs interactively (e.g. opening a source \nfile you don't have locally). I'm hopeful that the proposed partial clone solutions you \nreferenced would reduce the frequency of this being required.\n\n>> Being a bit more clever about packing objects (e.g. splitting blobs out from commits\n>> and trees) improved this a bit, but we still hit a bottlenecks from what appeared to\n>> be a large number of memory-mapping operations on a ~140GiB packfile of blobs.\n>> \n>> Each `pack-objects` process would consume approximately one CPU core for the\n>> duration of the request. It's possible that further splitting of these large blob packs\n>> would have improved performance in some scenarios, but that would increase the\n>> amount of pack-index lookups necessary to find a single object.\n>\n> I'm not very experienced with mmap, but I thought that memory-mapping a \n> large file in itself does not incur much of a performance penalty (if \n> any) - it is the accesses that count. I experimented with 15,000 and \n> 150,000 MiB files and mmap and they seem to be handled quite well. Also, \n> how many objects are \"pack-objects\" packing here?\n\nBack when we took this approach, it was ~4000 blob objects at a time. \nPerhaps we were being bitten by the Windows implementation of git_mmap[3]?. \nWhen I profiled the 4k blob scenario, the majority of CPU and wall time was \nspent in MapViewOfFileEx, which looks like it could mean accesses as well.\n\n[3] https://github.com/git/git/blob/master/compat/win32mmap.c\n\n>> This seems like a clever way to avoid the canonical `/info/refs?service=git-upload-pack`\n>> capability negotiation on every call. However, using error handling to fallback seems\n>> slightly wonky to me. Hopefully users are incentivized to upgrade their clients.\n>\n> By \"error handling to fallback\", do you mean in my proposal or in a \n> possible future one (assuming my proposal is implemented)? I don't think \n> my proposal requires any error handling to fallback (since only new \n> clients can clone partially - old clients will just clone totally and \n> obliviously), but I acknowledge that this proposal does not mean that \n> any future proposal can be done without requiring error handling to \n> fallback.\n\nRight, I was talking about the possible future one - more around the \nconcept of back-compat in the event of any protocol changes. I don't want \nto spend too much time focusing on what we might want in the future, but a \nthought I just had: what about versioning as a part of the URL? For example, \n`/server-endpoint?version=1.0`. This could also enable breaking changes for \nexisting commands.\n\n>> This makes a lot of sense to me. When we built our caching proxy, we had to be careful\n>> when designing how we'd handle clients requesting objects missing from the proxy.\n>>\n>> For example, a client requests a single blob and the proxy doesn't have it - we can't simply\n>> download that object from the \"authoritative\" remote and stick it in the `.git\\objects\\xx\\yyy...`\n>> directory, because the repository would be made corrupt.\n>\n> By proxy, do you mean a Git repository? Sorry, I don't really understand \n> this part.\n\nYes, apologies for not being clear. The proxy we created maintains a \nnearly-up-to-date mirror of the remote as a bare repo and exposes a few \ncustom endpoints I've alluded to. What I'm trying to say is that updating \nthe bare repo on disk via methods other than a full `fetch` is dangerous \nwithout the partial-clone-marking capability.\n\n>> Not do derail us too far off blobs, but I wonder if we need a `fetch-commit-pack` endpoint,\n>> or could get away with introducing a new capability (e.g. `no-blobs`) to `upload-pack` instead.\n>> As a casual observer, this seems like it would be a much smaller change since the rest of the\n>> negotiation/reachability calculation would look the same, right? Or would this `fetch-commit-pack`\n>> not return trees either?\n>>\n>> I only ask because, in our observations, when git wants to read commits it's\n>> usually followed by a lot of \"related\" trees - again caveated with the fact that\n>> we're intercepting many things at the filesystem layer.\n>\n> The main reason for this extra command is not to exclude blobs (which, \n> as you said, can be done with a new capability - I suspect that we will \n> need a capability or parameter of some sort anyway to indicate which \n> size of blobs to filter out) but to eliminate the mandatory ref \n> advertisement that is done whenever the client fetches. One of our use \n> cases (internal Android) has large blobs and many (more than 700k) refs, \n> so it would benefit greatly from blob filtering and elimination of the \n> mandatory ref advertisement (tens of megabytes per fetch).\n> \n> As for the size of the change, I have a work in progress that implements \n> this [2].\n> \n> [2] https://public-inbox.org/git/cover.1491851452.git.jonathantanmy@google.com/\n\nAh, interesting. We solved the many-refs problem using a different approach - \nbasically, limiting what the server returns based on who the user is and preferences \nthey set via our web interface or an API. I've been promised by a few colleagues that \nwe'll have more to share here soon... With your proposal, how does the client choose \nwhich refs they want to fetch?  \n\nI took a look at look at your patch series. I'm nowhere near qualified to give feedback \ngiven my lack of experience with the core git implementation and C in general, but it \nlooks reasonable. Hoping we can come up with a better name than \"server-endpoint\" though :)\n\n>> Just to keep the discussion interesting, I'll throw an alternative out there that's\n>> worked well for us. As I understand it, the HTTP-based dumb transfer protocol\n>> supports returning objects in loose object format, but only if they already exist\n>> in loose format.\n>>\n>> Extending this to have the remote provide these objects via a \"dumb\" protocol\n>> when they are packed as well - i.e. the server would \"loosens\" them upon request -\n>> is basically what we do and it works quite well for low-latency clients. To further improve\n>> performance at the cost of complexity, we've added caching at the memory and disk layer\n>> for these loose objects in the same format we send to the client.\n>>\n>> There's a clear tradeoff here - the servers must have adequate disk and/or memory to store\n>> these loose objects in optimal format. In addition, the higher the latency is to the remote,\n>> the worse this solution will perform. Fortunately, in our case, network topology allows us to\n>> put these caching proxies close enough to clients for it not to matter.\n>\n>This does make sense in the situation you describe, but (as you said) I \n>don't think we can guarantee this in the majority of situations. I think \n>some sort of batching (like the (1a) solution I talked about near the \n>start of this e-mail) and serving packed data from packed storage should \n>form the baseline, and any situation-specific optimizations (e.g. \n>serving unpacked data from topologically-close servers) can be \n>additional steps.\n\nAgreed - our choice of solution was largely driven by our lack of ability to batch, \nperformance issues with `pack-objects` at scale, and ability to deploy the \npreviously-mentioned proxies very close to clients. If we can solve the first \nproblem (batching), optimizing for the hopefully-less-common single blob at a time \nscenario becomes less important. If we assume that happens, I think your proposal \nlooks sound overall.\n"},{"id":"318006","messageId":"e9f18a53-c4cc-9927-7579-03d88a9169cd@google.com","threadId":"45377","inReplyTo":"CY4PR21MB05047397A12B8692C923FC67EE1A0@CY4PR21MB0504.namprd21.prod.outlook.com","subject":"Re: Proposal for \"fetch-any-blob Git protocol\" and server design","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-04-26T22:51:31Z","receivedAt":"2017-04-26T22:51:42Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 04/21/2017 09:41 AM, Kevin David wrote:\n> Hi Jonathan,\n>\n> Sorry for the delayed response - other work got in the way, unfortunately!\n\nNo worries!\n\n>> I am envisioning (1a) as described in Jeff Hostetler's e-mail [1] (\"a\n>> pre-command or hook to identify needed blobs and pre-fetch them before\n>> allowing the actual command to start\"), so a Git command would typically\n>> make a single request that contains all the blobs required, but my\n>> proposal can also handle (1c) ('\"fault\" them in as necessary in\n>> read_object() while the command is running and without any pre-fetch\n>> (either synchronously or asynchronously and with/without a helper\n>> process)').\n>>\n>> Even if we decided to go with single-blob requests and responses, it is\n>> still important to send them as packfiles, so that the server can serve\n>> them directly from its compressed storage without first having to\n>> uncompress them.\n>>\n>> [1] https://public-inbox.org/git/1488999039-37631-1-git-send-email-git@jeffhostetler.com/\n>\n> Ah, I missed this. If we think we can build meaningfully-sized requests via (1a) and (1b),\n> then I agree - packs are optimal.\n>\n> However, if single-blob requests/responses dominate, I have a few concerns, mostly from\n> experience playing with something similar:\n> * Regardless of the size of the object or number returned, the client will need to\n>   `index-pack` the result and create a corresponding `.idx` file, which requires\n>   decompression to construct (right?)\n\nYes, although I think that loose objects would need to be decompressed \ntoo to verify the SHA-1.\n\n> * Unless the client's repository is repacked aggressively, we'll pollute the\n>   `.git\\objects\\pack` directory with little indexes (somewhere around 4KiB minimum)\n>   and packfiles rapidly, which would degrade performance. One rudimentary workaround\n>   would be to loosen these packs on the client if they were under a certain\n>   size/object count. I think fetch does this already?\n>\n> In either case, shifting the pack decompression/loose object recompression problem\n> to the client instead of the server is probably a good principle, but in our case it\n> simply wasn't fast enough to serve single blobs interactively (e.g. opening a source\n> file you don't have locally). I'm hopeful that the proposed partial clone solutions you\n> referenced would reduce the frequency of this being required.\n\nI wrote up a proposal for how Git should handle missing blobs, including \na hook that, when invoked, is allowed to write packed or loose objects \n[1]. I was thinking along similar lines too about the many-packfile \nissue, and that was part of the reason why [1] was written the way it is.\n\nI think that, by default, the wire protocol should transmit packs \nbecause it causes less load on the server (as you describe) and it \nsupports batch requests. Also, this protocol with single-blob fetching \nmight be usable as-is for projects that only exclude large blobs (so \nthere are relatively few of them), and will be more and more usable as \nwe migrate the more commonly used Git commands to provide batch \nrequests. However, the ability to customize the hook allows users to use \nanother download scheme more suited to their use cases (for example, in \nyours, where you can provision servers that are topologically close to \nyour clients).\n\n[1] <20170426221346.25337-1-jonathantanmy@google.com>\n\n>>> Being a bit more clever about packing objects (e.g. splitting blobs out from commits\n>>> and trees) improved this a bit, but we still hit a bottlenecks from what appeared to\n>>> be a large number of memory-mapping operations on a ~140GiB packfile of blobs.\n>>>\n>>> Each `pack-objects` process would consume approximately one CPU core for the\n>>> duration of the request. It's possible that further splitting of these large blob packs\n>>> would have improved performance in some scenarios, but that would increase the\n>>> amount of pack-index lookups necessary to find a single object.\n>>\n>> I'm not very experienced with mmap, but I thought that memory-mapping a\n>> large file in itself does not incur much of a performance penalty (if\n>> any) - it is the accesses that count. I experimented with 15,000 and\n>> 150,000 MiB files and mmap and they seem to be handled quite well. Also,\n>> how many objects are \"pack-objects\" packing here?\n>\n> Back when we took this approach, it was ~4000 blob objects at a time.\n> Perhaps we were being bitten by the Windows implementation of git_mmap[3]?.\n> When I profiled the 4k blob scenario, the majority of CPU and wall time was\n> spent in MapViewOfFileEx, which looks like it could mean accesses as well.\n>\n> [3] https://github.com/git/git/blob/master/compat/win32mmap.c\n\nHmm...I don't have much experience with this, but this may be something \nto keep in mind.\n\n>>> This seems like a clever way to avoid the canonical `/info/refs?service=git-upload-pack`\n>>> capability negotiation on every call. However, using error handling to fallback seems\n>>> slightly wonky to me. Hopefully users are incentivized to upgrade their clients.\n>>\n>> By \"error handling to fallback\", do you mean in my proposal or in a\n>> possible future one (assuming my proposal is implemented)? I don't think\n>> my proposal requires any error handling to fallback (since only new\n>> clients can clone partially - old clients will just clone totally and\n>> obliviously), but I acknowledge that this proposal does not mean that\n>> any future proposal can be done without requiring error handling to\n>> fallback.\n>\n> Right, I was talking about the possible future one - more around the\n> concept of back-compat in the event of any protocol changes. I don't want\n> to spend too much time focusing on what we might want in the future, but a\n> thought I just had: what about versioning as a part of the URL? For example,\n> `/server-endpoint?version=1.0`. This could also enable breaking changes for\n> existing commands.\n\nThat could work, although we could put a version number in the command \nname (e.g. \"fetch-any-blob-2\") too.\n\n> Ah, interesting. We solved the many-refs problem using a different approach -\n> basically, limiting what the server returns based on who the user is and preferences\n> they set via our web interface or an API. I've been promised by a few colleagues that\n> we'll have more to share here soon... With your proposal, how does the client choose\n> which refs they want to fetch?\n\nThe \"want\" lines can now take names (including glob characters). A list \nof refs (names and hashes) that correspond to those \"want\"s will be \nreturned by the server right before the packfile.\n\n(So the client doesn't \"choose\" in the sense that it has a list of \noptions and it picks a subset of them - it chooses by sending one or \nmore strings that the server will match.)\n\n> I took a look at look at your patch series. I'm nowhere near qualified to give feedback\n> given my lack of experience with the core git implementation and C in general, but it\n> looks reasonable. Hoping we can come up with a better name than \"server-endpoint\" though :)\n\nThanks, and I'm open to suggestions for the name. :-)\n\n>>> Just to keep the discussion interesting, I'll throw an alternative out there that's\n>>> worked well for us. As I understand it, the HTTP-based dumb transfer protocol\n>>> supports returning objects in loose object format, but only if they already exist\n>>> in loose format.\n>>>\n>>> Extending this to have the remote provide these objects via a \"dumb\" protocol\n>>> when they are packed as well - i.e. the server would \"loosens\" them upon request -\n>>> is basically what we do and it works quite well for low-latency clients. To further improve\n>>> performance at the cost of complexity, we've added caching at the memory and disk layer\n>>> for these loose objects in the same format we send to the client.\n>>>\n>>> There's a clear tradeoff here - the servers must have adequate disk and/or memory to store\n>>> these loose objects in optimal format. In addition, the higher the latency is to the remote,\n>>> the worse this solution will perform. Fortunately, in our case, network topology allows us to\n>>> put these caching proxies close enough to clients for it not to matter.\n>>\n>> This does make sense in the situation you describe, but (as you said) I\n>> don't think we can guarantee this in the majority of situations. I think\n>> some sort of batching (like the (1a) solution I talked about near the\n>> start of this e-mail) and serving packed data from packed storage should\n>> form the baseline, and any situation-specific optimizations (e.g.\n>> serving unpacked data from topologically-close servers) can be\n>> additional steps.\n>\n> Agreed - our choice of solution was largely driven by our lack of ability to batch,\n> performance issues with `pack-objects` at scale, and ability to deploy the\n> previously-mentioned proxies very close to clients. If we can solve the first\n> problem (batching), optimizing for the hopefully-less-common single blob at a time\n> scenario becomes less important. If we assume that happens, I think your proposal\n> looks sound overall.\n\nThanks.\n"}]}