{"thread":{"id":"33215","subject":"random server hacks on top of git","startedAt":"2013-03-18T12:12:43Z","lastAt":"2013-03-18T18:24:52Z","messageCount":2,"participants":["Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"211559","messageId":"20130318121243.GC14789@sigill.intra.peff.net","threadId":"33215","inReplyTo":null,"subject":"random server hacks on top of git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-03-18T12:12:43Z","receivedAt":"2013-03-18T12:12:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"[Re-titled, as we are off-topic from the original patch series]\n\nOn Sun, Mar 17, 2013 at 05:38:59PM +0100, René Scharfe wrote:\n\n> Am 17.03.2013 06:40, schrieb Jeff King:\n> >We do have the capability to roll out to one or a few of our servers\n> >(the granularity is not 0.2%, but it is still small). I'm going to try\n> >to keep us more in sync with upstream git, but I don't know if I will\n> >get to the point of ever deploying \"master\" or \"next\", even for a small\n> >portion of the population. We are accumulating more hacks[1] on top of\n> >git, so it is not just \"run master for an hour on this server\"; I have\n> >to actually merge our fork.\n> \n> Did you perhaps intend to list these hacks in a footnote or link to a\n> repository containing them?  (I can't find the counterpart of that\n> [1].)\n\nI was actually just going to say \"some of which are gross hacks that\nwill never see the light of day, some of which have already gone\nupstream, and some of which I am planning on submitting upstream\".\n\nBut since I happened to be cataloguing them recently, here is the list\nof things that have not yet gone upstream.  If anybody is interested in\na particular topic, I'm happy to discuss and/or prioritize moving it\nforward.\n\n  - blame-tree; re-rolled from my submission last year to build on top\n    of the revision machinery, handle merges sanely, etc. Mostly this\n    needs documentation and a clean-up of the output format (which is\n    very utilitarian, but probably should share output with git-blame).\n\n  - diff --max-depth; this is a requirement to do blame-tree efficiently\n    if you want to do GitHub-style listings (you must recurse to find\n    the history of some/subdir, but you do not want to recurse past that\n    for efficiency reasons). This is hung up on two things:\n\n      1. It does not integrate with the pathspec max-depth code, because\n         we do not use struct pathspec in the tree diff (but I think\n         Duy's patches are changing that).\n\n      2. My definition of --max-depth is subtly different from that of\n         \"git grep\".  But I think mine is more useful, and I haven't\n         decided how to reconcile it.\n\n  - share ref selection code between \"git branch\", \"git tag\", and \"git\n    for-each-ref\". This includes cleaning up the \"tag --contains\" code\n    to be safer for general use (so that \"branch --contains\" can benefit\n    from the speedup), and then getting the same options for all three\n    commands (tag doesn't know about --merged, and for-each-ref\n    doesn't know about --contains or --merged).\n\n   - receive.maxsize; index-pack will happily spool data to disk\n     forever, and you never even get a chance to make a policy decision\n     like \"hey, this is too big\". This patch lets index-pack cut off the\n     client after a certain number of bytes. It's not elegant because\n     the cutoff transfer is not resumable, but we use it is as a\n     last-ditch for DoS protection (the client can reconnect and send\n     more, of course, but at that point we have the opportunity to make\n     external policy decisions like locking their account). Not sure if\n     other sites would want this or not.\n\n   - receive.advertisealternates; basically turn off \".have\"\n     advertisement. Some of our alternates networks are so huge that\n     the cost of collecting all of the alternate refs is very high (even\n     though it can save some transfer bandwidth). Not sure if other\n     sites want this or not (and I think it would be more elegant to\n     have a small static set of common refs that people build off of,\n     and advertise those. e.g., if you fork rails/rails, then we should\n     advertise rails/rails/refs/heads/master as a \".have\", but not\n     anybody else's fork).\n\n    - receive.hiderefs; this is going to become redundant with Junio's\n      implementation\n\n    - an audit reflog; we keep a reflog for all refs at the root of the\n      repository. It differs from a regular reflog in that:\n\n        1. It never expires.\n\n        2. It is not part of reachability analysis.\n\n        3. It includes the refname for each entry, so you can see\n           deletions.\n\n      It's mostly useful for forensics when somebody has screwed up\n      their repository (or we're chasing down a git bug; it helped me\n      find the pack-refs race recently). Probably too GitHub-specific\n      for other people to want it (especially because it grows without\n      bound).\n\n    - statistics instrumentation; we keep counters for various things in\n      code (e.g., which phase of protocol upload-pack is in, how many\n      bytes sent, etc) and expose them in a few ways. One is over a\n      socket to run a \"top\"-like interface. Another is to tweak the argv\n      array of the process so that \"ps\" shows the process state. I think\n      it would be useful to other people running git servers, but the\n      code is currently quite nasty and invasive. I have a\n      work-in-progress to clean it up, but it's got a ways to go.\n\n    - hacks to set niceness and io-priority; this should be done by a\n      wrapper, but in our case it was simpler to catch all processes by\n      just building it into git. Too gross to go upstream.\n\n    - ignore some fsck warnings under transfer.fsckobjects; some of them\n      are annoyingly common when people pull old history from an\n      existing project and try to push it back up. It's not indicative\n      of a new bug in an implementation, but we have to live with the\n      broken history forever (e.g., zero-padded modes in trees).\n\n-Peff\n"},{"id":"211588","messageId":"7v7gl4o9aj.fsf@alter.siamese.dyndns.org","threadId":"33215","inReplyTo":"20130318121243.GC14789@sigill.intra.peff.net","subject":"Re: random server hacks on top of git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-18T18:24:52Z","receivedAt":"2013-03-18T18:24:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>   - blame-tree; re-rolled from my submission last year to build on top\n>\n>   - diff --max-depth; this is a requirement to do blame-tree efficiently\n\nBoth look mildly interesting, especially after the magic pathspec\nsettles the latter may be a good addition.\n\n>   - share ref selection code between \"git branch\", \"git tag\", and \"git\n>     for-each-ref\".\n\nNice.\n\n>    - receive.maxsize; index-pack will happily spool data to disk\n\nAgain nice.\n\n>    - receive.advertisealternates; basically turn off \".have\"\n>      advertisement.\n\nI think this is a sane thing to do, especially with some hints on\nthe most common reference tree everybody is expected to know about.\n\n>     - receive.hiderefs; this is going to become redundant with Junio's\n>       implementation\n\nYup.\n\n>     - an audit reflog; we keep a reflog for all refs at the root of the\n>       repository. It differs from a regular reflog in that:\n>\n>         1. It never expires.\n>\n>         2. It is not part of reachability analysis.\n>\n>         3. It includes the refname for each entry, so you can see\n>            deletions.\n\nInteresting.\n\n>     - ignore some fsck warnings under transfer.fsckobjects; some of them\n>       are annoyingly common when people pull old history from an\n>       existing project and try to push it back up.\n\nDepending on the implementation, this may be very much valuable.\n"}]}