git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [WIP 0/7] CDN offloading of fetch response

From
Christian Couder <christian.couder@gmail.com>
Date
Feb 26, 2019, 08:30 UTC
Message-ID
<CAP8UFD1tKNFO9wU8CbgNnSnQyvHYPsZMk1Bit2y1jxH4vk62qQ@mail.gmail.com>
In-Reply-To
<20190225234528.GD16965@google.com>
Hi,
On Tue, Feb 26, 2019 at 12:45 AM Jonathan Nieder <jrnieder@gmail.com> wrote:
Show 22 quoted lines
>
> Christian Couder wrote:
> > On Sun, Feb 24, 2019 at 12:39 AM Jonathan Tan <jonathantanmy@google.com> wrote:
>
> > Especially I'd like to know what should the client do if they find out
> > that for example a repo that contains a lot of large files is
> > configured so that the large files should be fetched from a CDN that
> > the client cannot use? Is the client forced to find or setup another
> > repo configured differently if the client still wants to use CDN
> > offloading?
>
> The example from that message:
>
>   For example I think the Great Firewall of China lets people in China
>   use GitHub.com but not Google.com. So if people start configuring
>   their repos on GitHub so that they send packs that contain Google.com
>   CDN URLs (or actually anything that the Firewall blocks), it might
>   create many problems for users in China if they don't have a way to
>   opt out of receiving packs with those kind of URLs.
>
> But the same thing can happen with redirects, with embedded assets in
> web pages, and so on.

I don't think it's the same situation, because the CDN offloading is likely to be used for large objects that some hosting sites like GitHub, GitLab and BitBucket might not be ok to have them store for free on their machines. (I think the current limitations are around 10GB or 20GB, everything included, for each user.)

So it's likely that users will want a way to host on such sites incomplete repos using CDN offloading to a CDN on another site. And then if the CDN is not accessible for some reason, things will completely break when users will clone.

You could say that it's the same issue as when a video is not available on a web page, but the web browser can still render the page when a video is not available. So I don't think it's the same kind of issue.

And by the way that's a reason why I think it's important to think about this in relation to promisor/partial clone remotes. Because with them it's less of a big deal if the CDN is unavailable, temporarily or not, for some reason.

> I think in this situation the user would likely
> (and rightly) blame the host (github.com) for requiring access to a
> separate inaccessible site, and the problem could be resolved with
> them.

The host will say that it's repo admins' responsibility to use a CDN that works for the repo users (or to pay for more space on the host). Then repo admins will say that they use this CDN because it's simpler for them or the only thing they can afford or deal with. (For example I don't think it would be easy for westerners to use a Chinese CDN.) Then users will likely blame Git for not supporting a way to use a different CDN than the one configured in each repo.

> The beauty of this is that it's transparent to the client: the fact
> that packfile transfer was offloaded to a CDN is an implementation
> detail, and the server takes full responsibility for it.

Who is "the server" in real life? Are you sure they would be ok to take full responsibility?

And yes, I agree that transparency for the client is nice. And if it's really nice, then why not have it for promisor/partial clone remotes too? But then do we really need duplicated functionality between promisor remotes and CDN offloading?

And also I just think that in real life there needs to be an easy way to override this transparency, and we already have that with promisor remotes.

Show 5 quoted lines
> This doesn't stop a hosting provider from using e.g. server options to
> allow the client more control over how their response is served, just
> like can be done for other features of how the transfer works (how
> often to send progress updates, whether to prioritize latency or
> throughput, etc).
Could you give a more concrete example of what could be done?
> What the client *can* do is turn off support for packfile URLs in a
> request completely.  This is required for backward compatibility and
> allows working around a host that has configured the feature
> incorrectly.

If the full content of a repo is really large, the size of a full pack file sent by an initial clone could be really big and many client machines could not have enough memory to deal with that. And this suppose that repo hosting providers would be ok to host very large repos in the first place.

With promisor remotes, it's less of a problem if for example:
- a repo hosting provider is not ok with very large repos,
- a CDN is unavailable,
- a repo admin has not configured some repos very well.

Thanks for your answer, Christian.

Previous: Jonathan NiederNext: Ævar Arnfjörð Bjarmason
Message 17 of 45 in “[WIP 0/7] CDN offloading of fetch response”
  1. Jonathan TanFeb 23, 2019
  2. 1/7 http: use --stdin and --keep when downloading packJonathan Tan, Feb 23, 2019
  3. 2/7 http: improve documentation of http_pack_requestJonathan Tan, Feb 23, 2019
  4. 3/7 http-fetch: support fetching packfiles by URLJonathan Tan, Feb 23, 2019
  5. 4/7 Documentation: order protocol v2 sectionsJonathan Tan, Feb 23, 2019
  6. 5/7 Documentation: add Packfile URIs design docJonathan Tan, Feb 23, 2019
  7. 6/7 upload-pack: refactor reading of pack-objects outJonathan Tan, Feb 23, 2019
  8. 7/7 upload-pack: send part of packfile response as uriJonathan Tan, Feb 23, 2019
  9. Junio C HamanoFeb 24, 2019
  10. Christian CouderFeb 25, 2019
  11. Jonathan NiederFeb 26, 2019
  12. Christian CouderFeb 26, 2019
  13. Josh SteadmonMar 1, 2019
  14. Jonathan TanMar 1, 2019
  15. Christian CouderFeb 25, 2019
  16. Jonathan NiederFeb 25, 2019
  17. Christian CouderFeb 26, 2019
  18. Ævar Arnfjörð BjarmasonFeb 26, 2019
  19. Christian CouderMar 4, 2019
  20. Jonathan NiederFeb 28, 2019
  21. Christian CouderMar 4, 2019
  22. 0/8 CDN offloading of fetch responseJonathan Tan, Mar 8, 2019
  23. 1/8 http: use --stdin when getting dumb HTTP packJonathan Tan, Mar 8, 2019
  24. 2/8 http: improve documentation of http_pack_requestJonathan Tan, Mar 8, 2019
  25. 3/8 http-fetch: support fetching packfiles by URLJonathan Tan, Mar 8, 2019
  26. 4/8 Documentation: order protocol v2 sectionsJonathan Tan, Mar 8, 2019
  27. 5/8 Documentation: add Packfile URIs design docJonathan Tan, Mar 8, 2019
  28. Jeff KingApr 23, 2019
  29. Jonathan TanApr 23, 2019
  30. Ævar Arnfjörð BjarmasonApr 23, 2019
  31. Jonathan NiederApr 23, 2019
  32. Ævar Arnfjörð BjarmasonApr 23, 2019
  33. Jonathan NiederApr 23, 2019
  34. Jonathan NiederApr 23, 2019
  35. Ævar Arnfjörð BjarmasonApr 23, 2019
  36. Jonathan NiederApr 23, 2019
  37. Ævar Arnfjörð BjarmasonApr 24, 2019
  38. Junio C HamanoApr 24, 2019
  39. 6/8 upload-pack: refactor reading of pack-objects outJonathan Tan, Mar 8, 2019
  40. 7/8 fetch-pack: support more than one pack lockfileJonathan Tan, Mar 8, 2019
  41. 8/8 upload-pack: send part of packfile response as uriJonathan Tan, Mar 8, 2019
  42. Josh SteadmonMar 19, 2019
  43. Jeff KingApr 23, 2019
  44. Jonathan TanApr 23, 2019
  45. Ævar Arnfjörð BjarmasonApr 24, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.