{"thread":{"id":"45018","subject":"git-scm.com status report","startedAt":"2017-02-02T02:33:59Z","lastAt":"2017-05-18T15:42:35Z","messageCount":15,"participants":["Jeff King","Eric Wong","Samuel Lijin","Pranit Bauva","brian m. carlson","Lars Schneider"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"310697","messageId":"20170202023349.7fopb3a6pc6dkcmd@sigill.intra.peff.net","threadId":"45018","inReplyTo":null,"subject":"git-scm.com status report","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-02T02:33:51Z","receivedAt":"2017-02-02T02:33:59Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"We (the Git project) got control of the git-scm.com domain this year. We\nhave never really had an \"official\" website, but I think a lot of people\nconsider this to be one.\n\nThis is an overview of the current state, as well as some possible\nissues and future work.\n\n## What's on the site\n\nWe have the domains git-scm.com and git-scm.org (the latter we've had\nfor a while). They both point to the same website, which has general\ninformation about Git, including:\n\n  - a general overview of Git\n\n  - links to the latest releases (both source and some binary\n    installers)\n\n  - HTML-rendered copies of the manpages (both for the current version\n    and historical versions)\n\n  - an HTML rendering of the contents of the Pro Git book, along with\n    translations. The book content is licensed cc-by-nc-sa and developed\n    openly.\n\n  - various external links to books, tutorials, GUI tools, etc\n\n## How is it developed and hosted\n\nThe site is a Ruby on Rails app. The git repository is\nhttps://github.com/git/git-scm.com. Modifications are generally done by\npull requests there. I have admin access on the repository.\n\nThe deployed site is hosted on Heroku. It's part of GitHub's\nmeta-account, and they pay the bills. I have access to it, and am the\nonly person who deploys updates. Other technical staff at GitHub have\naccess, too, because of the account setup, but don't generally\nparticipate in maintenance.\n\nIt uses three 1GB Heroku dynos for scaling, which is $150/mo. It also\nuses some Heroku addons which add up to another $80/mo.\n\n## Who's the maintainer\n\nThese days, it's pretty much me, with a lot of help from Jean-Noël Avila\non issues with the Pro Git import and formatting code.\n\nLong ago, the site content and code was done by Scott Chacon, with\ngraphic design help from Jason Long.  Scott maintained the site with\nhelp from Bryan Turner for many years. But over time, they both seemed\nto get less active, and I haven't seen a peep from either on the site's\nGitHub repo in the past year. I've started trying to respond to issues\nand pull requests to keep things healthy.\n\nThe site is mostly in maintenance mode, but things do need addressing.\nPeople show up with new additions, fixes for typos, broken links and\nother formatting problems, etc. There are a lot of long-standing\nAsciidoc formatting problems both for the manpages and the imported Pro\nGit content.\n\n## What next\n\nWe can probably continue in maintenance mode like this for a while.\nWe've fixed a lot of of the long-standing formatting issues over the\npast year, so maintaining seems to have subsided in the past few months\nto mostly just merging or rejecting the occasional PR.\n\nStill, if anybody is interested in helping with this work, I'd love to\nhave more eyes on it. I can give people access to the GitHub repo.\nUnfortunately, I can't do so for the Heroku deploy, and part of the\nmaintenance burden is that the site is finicky and often needs manual\nintervention (e.g., a fix to formatting requires rebuilding the\nmanpages, which needs a job run manually on Heroku).\n\nIt's possible that the content or visual design of the site could be\nimproved in various ways. I don't have any strong desires myself, but\nmaybe others do. If people start doing larger work, though, we have a\nreal lack of reviewers, and I have very little expertise with Rails or\nwith visual design. So anybody who wants to do this should be prepared to\ntake maintenance ownership.\n\nAt some point, GitHub may boot us off of the shared Heroku account,\nbecause my impression is that it's somewhat of an administrative\nheadache. I don't think the Git project could afford the $230/mo hosting\nfees; that's basically all the money we make. On the other hand, we\nhaven't actively solicited funds to any great degree, and it's possible\nwe could get GitHub or some other entity to just sponsor us with site\nfees (I've heard zero complaints from GitHub about the money; it's\nmostly just that the site is an oddball among their other assets).\n\nWith the caveat that I know very little about web hosting, $230/mo\nsounds like an awful lot for what is essentially a static web site.\nThe site does see a lot of hits, but most of the content is a few basic\nweb pages, and copies of other static content that is updated\nonly occasionally (manpage content, lists of downloads, etc).  The biggest\ndynamic component is the site search, I think.\n\nI do wonder if there's room for improvement either:\n\n  - by measuring and optimizing the Heroku deploy. I have no idea about\n    scaling Rails or Heroku apps. Do we really need three expensive\n    dynos, or a $50/mo database plan? I'm not even sure what to measure,\n    or how. There are some analytics on the site, but I don't have\n    access to them (I could probably dig around for access if there was\n    somebody who actually knew how to do something productive with\n    them).\n\n  - by moving to a simpler model. I wonder if we could build the site\n    once and then deploy a more static variant of it to a cheaper\n    hosting platform. I'm not really sure what our options would be, how\n    much work it would take to do the conversion, and if we'd lose any\n    functionality.\n\nIf anybody is interested in tackling a project like this, let me know,\nand I can try to provide access to whatever parts are needed.\n\n-Peff\n"},{"id":"310700","messageId":"20170202043610.GA12738@starla","threadId":"45018","inReplyTo":"20170202023349.7fopb3a6pc6dkcmd@sigill.intra.peff.net","subject":"Re: git-scm.com status report","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-02-02T04:36:10Z","receivedAt":"2017-02-02T04:36:17Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jeff King <peff@peff.net> wrote:\n> With the caveat that I know very little about web hosting, $230/mo\n> sounds like an awful lot for what is essentially a static web site.\n\nYes, that's a lot.\n\nFwiw, that covers a year of low-end VPS hosting for the main\npublic-inbox.org/git machine + mail host\n(~1GB git objects + ~3GB Xapian index).\n\n> The site does see a lot of hits, but most of the content is a few basic\n> web pages, and copies of other static content that is updated\n> only occasionally (manpage content, lists of downloads, etc).  The biggest\n> dynamic component is the site search, I think.\n\nMaybe optimize search if that's slowest, first.  public-inbox\nuses per-host Xapian indexes so there's no extra network latency\nand it seems to work well.  But maybe you don't get FS write\naccess without full VPS access on Heroku...\n\nnginx handles static content easily, and since it looks like you\nguys use unicorn[*] for running the Ruby part.  I really hope\nnginx is in front of unicorn, since (AFAIK) Heroku doesn't put\nnginx in front of it by default.\n\n\n[*] I wrote and maintain unicorn; and have not yet recommended\n    any reverse proxy besides nginx to buffer for it.\n    However, having varnish or any other caching layer in\n    between nginx and unicorn is great, too.  I dunno how Heroku\n    (or any proprietary deployment systems) handle it, though.\n\n> I do wonder if there's room for improvement either:\n> \n>   - by measuring and optimizing the Heroku deploy. I have no idea about\n>     scaling Rails or Heroku apps. Do we really need three expensive\n>     dynos, or a $50/mo database plan? I'm not even sure what to measure,\n>     or how. There are some analytics on the site, but I don't have\n>     access to them (I could probably dig around for access if there was\n>     somebody who actually knew how to do something productive with\n>     them).\n\nI track down the most expensive requests in per-request timing\nlogs and work on profiling + optimizations from there...\nNothing fancy and no relying on proprietary tools like NewRelic.\n\nI also watch for queueing in listen socket backlog (with\nraindrops <https://raindrops-demo.bogomips.org/> or ss(8) to\nnotice overloading.  Again, I don't know how much visibility\nyou have with Heroku.\n\n>   - by moving to a simpler model. I wonder if we could build the site\n>     once and then deploy a more static variant of it to a cheaper\n>     hosting platform. I'm not really sure what our options would be, how\n>     much work it would take to do the conversion, and if we'd lose any\n>     functionality.\n\n*shrug*  That'd be more work, at least.  I'd figure out what's\nslow, first.\n\nFwiw, Varnish definitely helps public-inbox when slammed by\nHN/Reddit traffic.  It's great as long as you don't have\nper-user data to invalidate, which seems to be the case for\ngit-scm.\n\n> If anybody is interested in tackling a project like this, let me know,\n> and I can try to provide access to whatever parts are needed.\n\nWhile I'm not up-to-date with modern Rails or deployment stuff,\nI'm available via email if you have any lower-level\nRuby/unicorn/nginx-related questions.  I'm sure GitHub/GitLab\nalso has folks familiar with nginx+unicorn deployment on\nbare metal or VPS who could also help.\n"},{"id":"310701","messageId":"CAJZjrdUGsp5-HsA0Hxk4Qo+2s6crLbu-LuX=ECuC2QpM6HCWgg@mail.gmail.com","threadId":"45018","inReplyTo":"20170202043610.GA12738@starla","subject":"Re: git-scm.com status report","fromName":"Samuel Lijin","fromEmail":"sxlijin@gmail.com","sentAt":"2017-02-02T06:54:53Z","receivedAt":"2017-02-02T06:55:42Z","isPatch":false,"sender":{"key":"sxlijin@gmail.com","avatar":"https://gravatar.com/avatar/01777bf1eae64e2b4dca97dcac182a6abbcf6fd8cb4d5b8fa33edf9f8cc21746?d=mp&s=160"},"body":"In theory, you could also dump the build artifacts to a GH Pages repo\nand host it from there, although I don't know if you would run up\nagainst any of the usage limits[0]. The immediate problem I see with\nthat approach, though, is that I have no idea how any of the dynamic\nstuff (e.g. search) would be replaced.\n\nA question: there's a DB schema in there. Does the site still use a DB?\n\n[0] https://help.github.com/articles/what-is-github-pages/#usage-limits\n\nOn Wed, Feb 1, 2017 at 10:36 PM, Eric Wong <e@80x24.org> wrote:\n> Jeff King <peff@peff.net> wrote:\n>> With the caveat that I know very little about web hosting, $230/mo\n>> sounds like an awful lot for what is essentially a static web site.\n>\n> Yes, that's a lot.\n>\n> Fwiw, that covers a year of low-end VPS hosting for the main\n> public-inbox.org/git machine + mail host\n> (~1GB git objects + ~3GB Xapian index).\n>\n>> The site does see a lot of hits, but most of the content is a few basic\n>> web pages, and copies of other static content that is updated\n>> only occasionally (manpage content, lists of downloads, etc).  The biggest\n>> dynamic component is the site search, I think.\n>\n> Maybe optimize search if that's slowest, first.  public-inbox\n> uses per-host Xapian indexes so there's no extra network latency\n> and it seems to work well.  But maybe you don't get FS write\n> access without full VPS access on Heroku...\n>\n> nginx handles static content easily, and since it looks like you\n> guys use unicorn[*] for running the Ruby part.  I really hope\n> nginx is in front of unicorn, since (AFAIK) Heroku doesn't put\n> nginx in front of it by default.\n>\n>\n> [*] I wrote and maintain unicorn; and have not yet recommended\n>     any reverse proxy besides nginx to buffer for it.\n>     However, having varnish or any other caching layer in\n>     between nginx and unicorn is great, too.  I dunno how Heroku\n>     (or any proprietary deployment systems) handle it, though.\n>\n>> I do wonder if there's room for improvement either:\n>>\n>>   - by measuring and optimizing the Heroku deploy. I have no idea about\n>>     scaling Rails or Heroku apps. Do we really need three expensive\n>>     dynos, or a $50/mo database plan? I'm not even sure what to measure,\n>>     or how. There are some analytics on the site, but I don't have\n>>     access to them (I could probably dig around for access if there was\n>>     somebody who actually knew how to do something productive with\n>>     them).\n>\n> I track down the most expensive requests in per-request timing\n> logs and work on profiling + optimizations from there...\n> Nothing fancy and no relying on proprietary tools like NewRelic.\n>\n> I also watch for queueing in listen socket backlog (with\n> raindrops <https://raindrops-demo.bogomips.org/> or ss(8) to\n> notice overloading.  Again, I don't know how much visibility\n> you have with Heroku.\n>\n>>   - by moving to a simpler model. I wonder if we could build the site\n>>     once and then deploy a more static variant of it to a cheaper\n>>     hosting platform. I'm not really sure what our options would be, how\n>>     much work it would take to do the conversion, and if we'd lose any\n>>     functionality.\n>\n> *shrug*  That'd be more work, at least.  I'd figure out what's\n> slow, first.\n>\n> Fwiw, Varnish definitely helps public-inbox when slammed by\n> HN/Reddit traffic.  It's great as long as you don't have\n> per-user data to invalidate, which seems to be the case for\n> git-scm.\n>\n>> If anybody is interested in tackling a project like this, let me know,\n>> and I can try to provide access to whatever parts are needed.\n>\n> While I'm not up-to-date with modern Rails or deployment stuff,\n> I'm available via email if you have any lower-level\n> Ruby/unicorn/nginx-related questions.  I'm sure GitHub/GitLab\n> also has folks familiar with nginx+unicorn deployment on\n> bare metal or VPS who could also help.\n"},{"id":"310803","messageId":"20170203115814.vxn33ivuhrnxmkzf@sigill.intra.peff.net","threadId":"45018","inReplyTo":"CAJZjrdUGsp5-HsA0Hxk4Qo+2s6crLbu-LuX=ECuC2QpM6HCWgg@mail.gmail.com","subject":"Re: git-scm.com status report","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-03T11:58:15Z","receivedAt":"2017-02-03T11:58:23Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 02, 2017 at 12:54:53AM -0600, Samuel Lijin wrote:\n\n> In theory, you could also dump the build artifacts to a GH Pages repo\n> and host it from there, although I don't know if you would run up\n> against any of the usage limits[0]. The immediate problem I see with\n> that approach, though, is that I have no idea how any of the dynamic\n> stuff (e.g. search) would be replaced.\n\nI've talked with Pages people and they say it shouldn't be a big deal to\nhost. The main issue is that it's not _just_ a static site. It's a site\nthat's static once built, but a lot of the content is auto-generated\nfrom other sources (git manpages, Pro Git and its translations, etc).\n\nSo there's work involved in moving that generation step to whatever the\nnew process is (it's fine if it's running \"make\" locally after a Git\nrelease and pushing up the result).\n\n> A question: there's a DB schema in there. Does the site still use a DB?\n\nIt does use the database to hold all of the bits that aren't checked\ninto Git. So renderings of the manpages, the latest release git version,\netc. AFAIK, it's all things that I would be comfortable committing into\na git repository.\n\n-Peff\n"},{"id":"310819","messageId":"CAJZjrdVOZrCAa9s72zYrQv8wY+DezrENzA0AgT_Bgr-oMgsFTA@mail.gmail.com","threadId":"45018","inReplyTo":"20170203115814.vxn33ivuhrnxmkzf@sigill.intra.peff.net","subject":"Re: git-scm.com status report","fromName":"Samuel Lijin","fromEmail":"sxlijin@gmail.com","sentAt":"2017-02-03T20:56:59Z","receivedAt":"2017-02-03T20:57:46Z","isPatch":false,"sender":{"key":"sxlijin@gmail.com","avatar":"https://gravatar.com/avatar/01777bf1eae64e2b4dca97dcac182a6abbcf6fd8cb4d5b8fa33edf9f8cc21746?d=mp&s=160"},"body":"On Fri, Feb 3, 2017 at 5:58 AM, Jeff King <peff@peff.net> wrote:\n> On Thu, Feb 02, 2017 at 12:54:53AM -0600, Samuel Lijin wrote:\n>\n>> In theory, you could also dump the build artifacts to a GH Pages repo\n>> and host it from there, although I don't know if you would run up\n>> against any of the usage limits[0]. The immediate problem I see with\n>> that approach, though, is that I have no idea how any of the dynamic\n>> stuff (e.g. search) would be replaced.\n>\n> I've talked with Pages people and they say it shouldn't be a big deal to\n> host. The main issue is that it's not _just_ a static site. It's a site\n> that's static once built, but a lot of the content is auto-generated\n> from other sources (git manpages, Pro Git and its translations, etc).\n>\n> So there's work involved in moving that generation step to whatever the\n> new process is (it's fine if it's running \"make\" locally after a Git\n> release and pushing up the result).\n\nYep, noticed that when I cloned the repo the other day. Still\nwrangling with my own setup so that I can build everything locally. I\nimagine it would also be possible to set up some sort of CI/CD\npipeline to handle generating build artifacts automatically; so to be\nhonest, I don't think any of the static assets would pose a\nsignificant problem.\n\nThe bigger issue, in my opinion, is that there seems to be a fair\namount of non-trivial back-end stuff\n(https://github.com/git/git-scm.com/blob/master/spec/controllers/site_controller_spec.rb,\nhttps://github.com/git/git-scm.com/blob/master/app/controllers/site_controller.rb)\nincluding an Elasticsearch layer. (The redirects would be mildly\ninconvenient to handle with Pages, but like the static asset\ngeneration, should be more than doable.)\n\n>> A question: there's a DB schema in there. Does the site still use a DB?\n>\n> It does use the database to hold all of the bits that aren't checked\n> into Git. So renderings of the manpages, the latest release git version,\n> etc. AFAIK, it's all things that I would be comfortable committing into\n> a git repository.\n>\n> -Peff\n\nIn the meantime, I've also pinged a friend at Digital Ocean about\ntheir hosting options and they've expressed interest. At the very\nleast, they seem to offer a lot more than Heroku for 230$/mo[0], and I\nimagine it wouldn't be impossible to reduce the hosting costs by an\norder of magnitude. Think it's worth looking into?\n\n[0] https://www.digitalocean.com/pricing/#droplet\n"},{"id":"310878","messageId":"CAFZEwPPX4LngKrOHxgEp4aMGKOs0w4LHUBKumtmeRJSZ2_iV_Q@mail.gmail.com","threadId":"45018","inReplyTo":"20170202023349.7fopb3a6pc6dkcmd@sigill.intra.peff.net","subject":"Re: git-scm.com status report","fromName":"Pranit Bauva","fromEmail":"pranit.bauva@gmail.com","sentAt":"2017-02-05T20:11:04Z","receivedAt":"2017-02-05T20:11:18Z","isPatch":false,"sender":{"key":"pranit.bauva@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2959938?v=4"},"body":"Hey Peff,\n\nOn Thu, Feb 2, 2017 at 8:03 AM, Jeff King <peff@peff.net> wrote:\n> ## What's on the site\n>\n> We have the domains git-scm.com and git-scm.org (the latter we've had\n> for a while). They both point to the same website, which has general\n> information about Git, including:\n\nSince we have an \"official\" control over the website, shouldn't we be\nusing the .org domain more because we are more of an organization?\nWhat I mean is that in many places, we have referred to git-scm.com,\nwhich was perfectly fine because it was done by github which is a\ncompany but now I think it would be more appropriate to use\ngit-scm.org domain. We can forward all .com requests to .org and try\nto move all reference we know about, to .org. What do you all think?\n\nRegards,\nPranit Bauva\n"},{"id":"310919","messageId":"20170206162436.wiydnq7sv43wqg3d@sigill.intra.peff.net","threadId":"45018","inReplyTo":"CAFZEwPPX4LngKrOHxgEp4aMGKOs0w4LHUBKumtmeRJSZ2_iV_Q@mail.gmail.com","subject":"Re: git-scm.com status report","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-06T16:24:36Z","receivedAt":"2017-02-06T16:24:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 06, 2017 at 01:41:04AM +0530, Pranit Bauva wrote:\n\n> On Thu, Feb 2, 2017 at 8:03 AM, Jeff King <peff@peff.net> wrote:\n> > ## What's on the site\n> >\n> > We have the domains git-scm.com and git-scm.org (the latter we've had\n> > for a while). They both point to the same website, which has general\n> > information about Git, including:\n> \n> Since we have an \"official\" control over the website, shouldn't we be\n> using the .org domain more because we are more of an organization?\n> What I mean is that in many places, we have referred to git-scm.com,\n> which was perfectly fine because it was done by github which is a\n> company but now I think it would be more appropriate to use\n> git-scm.org domain. We can forward all .com requests to .org and try\n> to move all reference we know about, to .org. What do you all think?\n\nI don't have a preference myself. I know a lot of non-commercial groups\n(which I think the Git project is) try to prefer \".org\" to signal that.\n\nSwitching it around would require some DNS changes. I think \".org\" goes\nto a server the DNS provider (Gandi) runs which issues an HTTP 301 to\n\".com\". So we'd want to reverse that, or possibly just treat them both\nas equals. That shouldn't be too hard, and will have to be done via\nConservancy.\n\nI don't know what it would mean in terms of search-engine optimization.\nI know Google tries to detect duplicate names for sites and treat one as\ncanonical. And that's going to be \".com\" now, based on the existing\nredirect and on the fact that most people will have linked to .com.\n\nI'm not sure what disadvantages there are to switching now, or if there\nare things we should be doing to tell search engines (I seem to recall\nGoogle's Webmaster tools have options to say \"this is the canonical\nname\"). This is pretty far outside my area of expertise, so it may not\neven be something to care about at all. Just things to consider (and\nhopefully more clueful people than I can comment on it).\n\n-Peff\n"},{"id":"310930","messageId":"20170206182754.qxgzl7546i7u5wnw@sigill.intra.peff.net","threadId":"45018","inReplyTo":"20170202023349.7fopb3a6pc6dkcmd@sigill.intra.peff.net","subject":"Re: git-scm.com status report","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-06T18:27:54Z","receivedAt":"2017-02-06T18:28:04Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 02, 2017 at 03:33:50AM +0100, Jeff King wrote:\n\n> We (the Git project) got control of the git-scm.com domain this year. We\n> have never really had an \"official\" website, but I think a lot of people\n> consider this to be one.\n> \n> This is an overview of the current state, as well as some possible\n> issues and future work.\n\nThanks everybody, for your responses here and off-list. After my mail\ngot posted to HN, I got quite a lot of private responses, including\noffers to sponsor hosting, work on the site, etc. I'm still working my\nway through them, but I wanted to try to respond in aggregate here.\n\nFirst, a few clarifications:\n\n  - The money for the site wasn't mentioned to me by GitHub at all.  I'm\n    quite sure they would continue to sponsor the site financially if\n    need be. The only reason I didn't promise that is because I hadn't\n    arranged it specifically, and \"step 0\" seemed like first making sure\n    our costs were reasonable.\n\n  - Spinning the site out of GitHub's Heroku account isn't an urgent or\n    impending change. It came out of a conversation I had with people\n    auditing the GitHub account, where it is clearly a funny historical\n    anomaly. So I suspect we could just stay there indefinitely if need\n    be. But it seems to me like the right thing is to move it out for\n    two reasons:\n\n      1. The site was always intended to serve the Git community, not\n         GitHub, and it has increasingly become a community asset (e.g.,\n\t with the transfer of the domain name). The hosting assets\n\t should be held by the community, too, to help with things like\n\t continuity. If I get hit by a bus, the rest of the Git PLC\n\t should have access to the site without having to figure out who\n\t owns what.\n\n      2. Right now I can't add any other co-admins to handle operational\n         issues. So the bus factor and load of that part of operating\n\t the site can't be spread.\n\nThe responses I've gotten fall into a few buckets, I think:\n\n  - Yes, the current hosting cost really is unnecessarily high. Most of\n    this is due to scaling wrong. The main costs are:\n\n      1. Using 2x dynos; these have 1GB of RAM versus 512MB. The site\n         does seem to use about 750MB. I have no idea why that is the\n\t case. There's probably some low-hanging fruit in reducing the\n\t memory use to keep it below 512MB, but I don't think anybody\n\t has dug in there.\n\n      2. The site is scaled by using 3 dynos. It would be simpler and\n         cheaper to stick a CDN in front of it, since the pages change\n\t very rarely. That's something I haven't looked into setting up\n\t yet.\n\n\t The prerequisite to using a CDN is actually making sure the\n\t content is deterministic and cacheable. There was a nice PR\n\t opened at https://github.com/git/git-scm.com/pull/941 towards\n\t that end.\n\n  - It's mostly silly for this to be a Rails app at all. It's a static\n    site which occasionally sucks in and formats new content (like the\n    latest git version, new manpages, etc). The intent here was to make\n    something that would \"just run\" forever and pick up new versions\n    without human intervention. And that _does_ work, but it also makes\n    things more expensive and complicated than they need to be.\n\n    So a viable alternative is to use some kind of static site\n    generator and have someone (or something) responsible for pulling in\n    the new git versions occasionally.\n\n    A few people have expressed interesting this. There's some\n    preliminary work here:\n\n      https://github.com/git/git-scm.com/pull/941\n\n    and at least GitLab has expressed some interest. So I'll let people\n    coordinate in that PR or a new one what the result should look like.\n    Working patches trump discussion. :)\n\n    I have also talked with the GitHub Pages people, and they think\n    hosting it as a Jekyll page wouldn't be a big deal performance-wise\n    (with the caveat that we'd need to pre-render the asciidoctor bits\n    ourselves, as Jekyll doesn't do asciidoc). So that's a viable option\n    for hosting it for effectively free (though I think we _would_ still\n    want to put a CDN in front of it). But if somebody has an\n    alternative option, that's fine, too.\n\n  - Some people offered to help with running the site, or making major\n    transitions (like converting to a static site). The most important\n    thing to me there is that we have a solid maintenance plan. So I\n    would want some evidence that anybody doing a major work would stick\n    around in the community afterwards, or that it be done in a way that\n    the handoff back to community members is easy. So I'd probably look\n    for somebody already involved in the community, or somebody who\n    wants to join it building up that trust by taking on site\n    responsibilities over time.\n\n  - Lots of people asked about small tasks to do. Mostly reviewing and\n    responding to issues and PR is the simplest thing. You can do it in\n    a drive-by way, and that helps take the load off of me. As the same\n    reviewers show up more and more, I think we can build a community\n    and I'd eventually hand out greater access to the site to match.\n\n    I notice I've got over 100 GitHub notifications from people sifting\n    through back-issues, so that will take some time to go through. I'm\n    hoping a lot of them are \"already fixed, click closed\". :)\n\n  - Several people offered money out of pocket to pay for hosting, and\n    several hosters contacted me to work out hosting deals ranging from\n    cheap to free. I'd prefer to explore the technical bits for now and\n    see what the final shape and cost actually is (if we move to a\n    non-Rails site, then Rails hosting is less appealing, obviously).\n\nSo that's where we're at. I think the next step is either sticking a CDN\nin front of Heroku and dialing down the scaling there, or moving to a\nstatic site. I'll probably stall for a bit and see if patches for the\nlatter materialize, and if not pursue the CDN thing (where most of the\nwork will be administrative in getting it set up, not technical. I think\nthat makes it more or less my thing to do, but if anybody is interested\nin setting it up and handing off an account to the project, that\ncertainly makes things easy).\n\nThanks again for everybody who has offered to help, and everybody who\ncontinues to do so.\n\n-Peff\n"},{"id":"311136","messageId":"20170209021208.rvy7uww3qsktbf4a@genre.crustytoothpaste.net","threadId":"45018","inReplyTo":"20170206182754.qxgzl7546i7u5wnw@sigill.intra.peff.net","subject":"Re: git-scm.com status report","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2017-02-09T02:12:09Z","receivedAt":"2017-02-09T02:13:33Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Feb 06, 2017 at 07:27:54PM +0100, Jeff King wrote:\n>   - It's mostly silly for this to be a Rails app at all. It's a static\n>     site which occasionally sucks in and formats new content (like the\n>     latest git version, new manpages, etc). The intent here was to make\n>     something that would \"just run\" forever and pick up new versions\n>     without human intervention. And that _does_ work, but it also makes\n>     things more expensive and complicated than they need to be.\n> \n>     So a viable alternative is to use some kind of static site\n>     generator and have someone (or something) responsible for pulling in\n>     the new git versions occasionally.\n> \n>     A few people have expressed interesting this. There's some\n>     preliminary work here:\n> \n>       https://github.com/git/git-scm.com/pull/941\n> \n>     and at least GitLab has expressed some interest. So I'll let people\n>     coordinate in that PR or a new one what the result should look like.\n>     Working patches trump discussion. :)\n> \n>     I have also talked with the GitHub Pages people, and they think\n>     hosting it as a Jekyll page wouldn't be a big deal performance-wise\n>     (with the caveat that we'd need to pre-render the asciidoctor bits\n>     ourselves, as Jekyll doesn't do asciidoc). So that's a viable option\n>     for hosting it for effectively free (though I think we _would_ still\n>     want to put a CDN in front of it). But if somebody has an\n>     alternative option, that's fine, too.\n\nMy only concern with using GitHub Pages is that I don't believe it\ncurrently supports TLS on custom domains.  Since we currently have TLS\nenabled, along with HTTP Strict Transport Security (as we should), such\na configuration literally wouldn't work[0].  I think it's important that\nwe continue to serve HTTPS only, anyway.\n\nI agree that a static site is the way to go from a maintenance\nperspective, though.  Jekyll does support Asciidoctor with a plugin,\njust not on GitHub Pages, so it would theoretically be possible to build\nthe site as one big unit if we did it that way.  I've played around with\nthat plugin, so I'm happy to provide guidance if we want to do that.\n\n[0] HSTS would prevent anyone who had visited the page from downgrading\nto an insecure connection for the next year.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"311138","messageId":"20170209025030.52frbjekatebjoii@sigill.intra.peff.net","threadId":"45018","inReplyTo":"20170209021208.rvy7uww3qsktbf4a@genre.crustytoothpaste.net","subject":"Re: git-scm.com status report","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-09T02:50:30Z","receivedAt":"2017-02-09T02:57:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 09, 2017 at 02:12:09AM +0000, brian m. carlson wrote:\n\n> My only concern with using GitHub Pages is that I don't believe it\n> currently supports TLS on custom domains.  Since we currently have TLS\n> enabled, along with HTTP Strict Transport Security (as we should), such\n> a configuration literally wouldn't work[0].  I think it's important that\n> we continue to serve HTTPS only, anyway.\n\nI agree we should continue to serve HTTPS. The usual solution for our\nuse case is to stick a CDN like Cloudflare in front of GitHub Pages (and\nI think we'd want to do that anyway for performance).\n\nI haven't done it, but there are various guides. Here's the one from\nCloudflare:\n\n  https://blog.cloudflare.com/secure-and-fast-github-pages-with-cloudflare/\n\n> I agree that a static site is the way to go from a maintenance\n> perspective, though.  Jekyll does support Asciidoctor with a plugin,\n> just not on GitHub Pages, so it would theoretically be possible to build\n> the site as one big unit if we did it that way.  I've played around with\n> that plugin, so I'm happy to provide guidance if we want to do that.\n\nWe already massage the data coming from Git (and from the Pro Git books)\na bit before and after feeding it to asciidoctor. So I always assumed\nthat any static site would involve some import steps for those things,\nand we'd commit the intermediate product into the repository.\n\n-Peff\n"},{"id":"311142","messageId":"20170209043034.GA30409@dcvr.yhbt.net","threadId":"45018","inReplyTo":"20170209025030.52frbjekatebjoii@sigill.intra.peff.net","subject":"Re: git-scm.com status report","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-02-09T04:30:34Z","receivedAt":"2017-02-09T04:30:52Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jeff King <peff@peff.net> wrote:\n> I agree we should continue to serve HTTPS. The usual solution for our\n> use case is to stick a CDN like Cloudflare in front of GitHub Pages (and\n> I think we'd want to do that anyway for performance).\n> \n> I haven't done it, but there are various guides. Here's the one from\n> Cloudflare:\n> \n>   https://blog.cloudflare.com/secure-and-fast-github-pages-with-cloudflare/\n\nAFAIK, there's a way to keep CloudFlare stuff accessible to Tor\nusers.  If there is, please do so.  As a Tor user, it's been\ndisappointing to see so much of the web walled off by CAPTCHAs.\n\nThank you.\n\nHeck, maybe a .onion mirror would be nice :)\nI wouldn't mind hosting one myself if it's static.\n"},{"id":"319983","messageId":"CAJZjrdXCYwbH4wtY44UiEQii+r47a-coTwLd0qYzhV3VZwyd1A@mail.gmail.com","threadId":"45018","inReplyTo":"20170206182754.qxgzl7546i7u5wnw@sigill.intra.peff.net","subject":"Re: git-scm.com status report","fromName":"Samuel Lijin","fromEmail":"sxlijin@gmail.com","sentAt":"2017-05-17T01:56:37Z","receivedAt":"2017-05-17T01:57:24Z","isPatch":false,"sender":{"key":"sxlijin@gmail.com","avatar":"https://gravatar.com/avatar/01777bf1eae64e2b4dca97dcac182a6abbcf6fd8cb4d5b8fa33edf9f8cc21746?d=mp&s=160"},"body":"On Mon, Feb 6, 2017 at 1:27 PM, Jeff King <peff@peff.net> wrote:\n> On Thu, Feb 02, 2017 at 03:33:50AM +0100, Jeff King wrote:\n>\n>> We (the Git project) got control of the git-scm.com domain this year. We\n>> have never really had an \"official\" website, but I think a lot of people\n>> consider this to be one.\n>>\n>> This is an overview of the current state, as well as some possible\n>> issues and future work.\n>\n> Thanks everybody, for your responses here and off-list. After my mail\n> got posted to HN, I got quite a lot of private responses, including\n> offers to sponsor hosting, work on the site, etc. I'm still working my\n> way through them, but I wanted to try to respond in aggregate here.\n>\n> First, a few clarifications:\n>\n>   - The money for the site wasn't mentioned to me by GitHub at all.  I'm\n>     quite sure they would continue to sponsor the site financially if\n>     need be. The only reason I didn't promise that is because I hadn't\n>     arranged it specifically, and \"step 0\" seemed like first making sure\n>     our costs were reasonable.\n>\n>   - Spinning the site out of GitHub's Heroku account isn't an urgent or\n>     impending change. It came out of a conversation I had with people\n>     auditing the GitHub account, where it is clearly a funny historical\n>     anomaly. So I suspect we could just stay there indefinitely if need\n>     be. But it seems to me like the right thing is to move it out for\n>     two reasons:\n>\n>       1. The site was always intended to serve the Git community, not\n>          GitHub, and it has increasingly become a community asset (e.g.,\n>          with the transfer of the domain name). The hosting assets\n>          should be held by the community, too, to help with things like\n>          continuity. If I get hit by a bus, the rest of the Git PLC\n>          should have access to the site without having to figure out who\n>          owns what.\n>\n>       2. Right now I can't add any other co-admins to handle operational\n>          issues. So the bus factor and load of that part of operating\n>          the site can't be spread.\n>\n> The responses I've gotten fall into a few buckets, I think:\n>\n>   - Yes, the current hosting cost really is unnecessarily high. Most of\n>     this is due to scaling wrong. The main costs are:\n>\n>       1. Using 2x dynos; these have 1GB of RAM versus 512MB. The site\n>          does seem to use about 750MB. I have no idea why that is the\n>          case. There's probably some low-hanging fruit in reducing the\n>          memory use to keep it below 512MB, but I don't think anybody\n>          has dug in there.\n>\n>       2. The site is scaled by using 3 dynos. It would be simpler and\n>          cheaper to stick a CDN in front of it, since the pages change\n>          very rarely. That's something I haven't looked into setting up\n>          yet.\n>\n>          The prerequisite to using a CDN is actually making sure the\n>          content is deterministic and cacheable. There was a nice PR\n>          opened at https://github.com/git/git-scm.com/pull/941 towards\n>          that end.\n>\n>   - It's mostly silly for this to be a Rails app at all. It's a static\n>     site which occasionally sucks in and formats new content (like the\n>     latest git version, new manpages, etc). The intent here was to make\n>     something that would \"just run\" forever and pick up new versions\n>     without human intervention. And that _does_ work, but it also makes\n>     things more expensive and complicated than they need to be.\n>\n>     So a viable alternative is to use some kind of static site\n>     generator and have someone (or something) responsible for pulling in\n>     the new git versions occasionally.\n>\n>     A few people have expressed interesting this. There's some\n>     preliminary work here:\n>\n>       https://github.com/git/git-scm.com/pull/941\n>\n>     and at least GitLab has expressed some interest. So I'll let people\n>     coordinate in that PR or a new one what the result should look like.\n>     Working patches trump discussion. :)\n>\n>     I have also talked with the GitHub Pages people, and they think\n>     hosting it as a Jekyll page wouldn't be a big deal performance-wise\n>     (with the caveat that we'd need to pre-render the asciidoctor bits\n>     ourselves, as Jekyll doesn't do asciidoc). So that's a viable option\n>     for hosting it for effectively free (though I think we _would_ still\n>     want to put a CDN in front of it). But if somebody has an\n>     alternative option, that's fine, too.\n>\n>   - Some people offered to help with running the site, or making major\n>     transitions (like converting to a static site). The most important\n>     thing to me there is that we have a solid maintenance plan. So I\n>     would want some evidence that anybody doing a major work would stick\n>     around in the community afterwards, or that it be done in a way that\n>     the handoff back to community members is easy. So I'd probably look\n>     for somebody already involved in the community, or somebody who\n>     wants to join it building up that trust by taking on site\n>     responsibilities over time.\n>\n>   - Lots of people asked about small tasks to do. Mostly reviewing and\n>     responding to issues and PR is the simplest thing. You can do it in\n>     a drive-by way, and that helps take the load off of me. As the same\n>     reviewers show up more and more, I think we can build a community\n>     and I'd eventually hand out greater access to the site to match.\n>\n>     I notice I've got over 100 GitHub notifications from people sifting\n>     through back-issues, so that will take some time to go through. I'm\n>     hoping a lot of them are \"already fixed, click closed\". :)\n>\n>   - Several people offered money out of pocket to pay for hosting, and\n>     several hosters contacted me to work out hosting deals ranging from\n>     cheap to free. I'd prefer to explore the technical bits for now and\n>     see what the final shape and cost actually is (if we move to a\n>     non-Rails site, then Rails hosting is less appealing, obviously).\n>\n> So that's where we're at. I think the next step is either sticking a CDN\n> in front of Heroku and dialing down the scaling there, or moving to a\n> static site. I'll probably stall for a bit and see if patches for the\n> latter materialize, and if not pursue the CDN thing (where most of the\n> work will be administrative in getting it set up, not technical. I think\n> that makes it more or less my thing to do, but if anybody is interested\n> in setting it up and handing off an account to the project, that\n> certainly makes things easy).\n>\n> Thanks again for everybody who has offered to help, and everybody who\n> continues to do so.\n>\n> -Peff\n\nSo I've finally found the time to get everything set up (in the\nprocess discovering that remote_genbook2 consistently induces a\nsegfault in VirtualBox's networking driver, impressively enough) and\nam taking a look at how much work it would take to get the site\nrunning on AWS EC2/DO or some other hosting provider.\n\nSome things I'm wondering about:\n\n- You mentioned a lot of people reaching out off-list about hosting\noptions. Do any of them look particularly appealing at the moment?\n- How do I set up the ES service?\n\nSam\n"},{"id":"319984","messageId":"20170517020334.ncp3c6meak4mkp2j@sigill.intra.peff.net","threadId":"45018","inReplyTo":"CAJZjrdXCYwbH4wtY44UiEQii+r47a-coTwLd0qYzhV3VZwyd1A@mail.gmail.com","subject":"Re: git-scm.com status report","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-05-17T02:03:34Z","receivedAt":"2017-05-17T02:03:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 16, 2017 at 09:56:37PM -0400, Samuel Lijin wrote:\n\n> So I've finally found the time to get everything set up (in the\n> process discovering that remote_genbook2 consistently induces a\n> segfault in VirtualBox's networking driver, impressively enough) and\n> am taking a look at how much work it would take to get the site\n> running on AWS EC2/DO or some other hosting provider.\n> \n> Some things I'm wondering about:\n> \n> - You mentioned a lot of people reaching out off-list about hosting\n> options. Do any of them look particularly appealing at the moment?\n\nYes. I actually have stuff to announce there soon, but was holding off\nuntil the final pieces are in place. But basically, the architecture\nwould remain largely the same, but hosted on community-owned accounts\n(that I can share access to), with sponsorship from various hosting\nservices.\n\n> - How do I set up the ES service?\n\nI haven't ever tried to do this in the local development environment.\nThe production site currently just use a cloud-hosted ES (Bonsai). They\nhave free \"Sandbox\" plans for testing, so you could probably use that as\na test resource after setting up the relevant environment variables. Or\nalternatively, I think ElasticSearch folks produce binary builds you can\ntry, and you could host locally.\n\nThere's a rake job that inserts documents into the search index (see\nlib/tasks/search.rake).\n\n-Peff\n"},{"id":"320156","messageId":"469A925D-E5E4-4B9D-8AE2-A23DBA3595AE@gmail.com","threadId":"45018","inReplyTo":"20170517020334.ncp3c6meak4mkp2j@sigill.intra.peff.net","subject":"Re: git-scm.com status report","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-18T12:06:16Z","receivedAt":"2017-05-18T12:06:28Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 17 May 2017, at 04:03, Jeff King <peff@peff.net> wrote:\n> \n> On Tue, May 16, 2017 at 09:56:37PM -0400, Samuel Lijin wrote:\n> \n>> So I've finally found the time to get everything set up (in the\n>> process discovering that remote_genbook2 consistently induces a\n>> segfault in VirtualBox's networking driver, impressively enough) and\n>> am taking a look at how much work it would take to get the site\n>> running on AWS EC2/DO or some other hosting provider.\n>> \n>> Some things I'm wondering about:\n>> \n>> - You mentioned a lot of people reaching out off-list about hosting\n>> options. Do any of them look particularly appealing at the moment?\n> \n> Yes. I actually have stuff to announce there soon, but was holding off\n> until the final pieces are in place. But basically, the architecture\n> would remain largely the same, but hosted on community-owned accounts\n> (that I can share access to), with sponsorship from various hosting\n> services.\n> \n>> - How do I set up the ES service?\n> \n> I haven't ever tried to do this in the local development environment.\n> The production site currently just use a cloud-hosted ES (Bonsai). They\n> have free \"Sandbox\" plans for testing, so you could probably use that as\n> a test resource after setting up the relevant environment variables. Or\n> alternatively, I think ElasticSearch folks produce binary builds you can\n> try, and you could host locally.\n> \n> There's a rake job that inserts documents into the search index (see\n> lib/tasks/search.rake).\n\nDisclaimer: I am jumping in here without much knowledge. Feel free\nto ignore me :-)\n\nIn our TravisCI builds we create the AsciiDoc/Doctor documentation\nalready. Couldn't we push that result to some static hosting service?\nWould that help in any way with git-scm.com?\n\n- Lars\n"},{"id":"320166","messageId":"20170518154159.s2ysocfzlf2ffgqg@sigill.intra.peff.net","threadId":"45018","inReplyTo":"469A925D-E5E4-4B9D-8AE2-A23DBA3595AE@gmail.com","subject":"Re: git-scm.com status report","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-05-18T15:42:00Z","receivedAt":"2017-05-18T15:42:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 18, 2017 at 02:06:16PM +0200, Lars Schneider wrote:\n\n> > I haven't ever tried to do this in the local development environment.\n> > The production site currently just use a cloud-hosted ES (Bonsai). They\n> > have free \"Sandbox\" plans for testing, so you could probably use that as\n> > a test resource after setting up the relevant environment variables. Or\n> > alternatively, I think ElasticSearch folks produce binary builds you can\n> > try, and you could host locally.\n> > \n> > There's a rake job that inserts documents into the search index (see\n> > lib/tasks/search.rake).\n> \n> Disclaimer: I am jumping in here without much knowledge. Feel free\n> to ignore me :-)\n> \n> In our TravisCI builds we create the AsciiDoc/Doctor documentation\n> already. Couldn't we push that result to some static hosting service?\n> Would that help in any way with git-scm.com?\n\nNot really. The site builds the asciidoctor documentation already via an\nautomated job. This question was just about putting it into the search\nindex (which also happens in production with an automated job; this is\njust about setting up the search database).  So I don't think there's\nany real problem to be solved with respect to generating pages.\n\n-Peff\n"}]}