{"thread":{"id":"27825","subject":"Google Code supports Git","startedAt":"2011-07-16T10:24:50Z","lastAt":"2011-07-17T15:45:36Z","messageCount":6,"participants":["Nguyen Thai Ngoc Duy","Sverre Rabbelier","A Large Angry SCM","Shawn Pearce","Sebastian Schuberth","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"171458","messageId":"CACsJy8BMK4UqVVNnhs6riVSS-OKcFKT9=XScrAw5nK2FH_u92A@mail.gmail.com","threadId":"27825","inReplyTo":null,"subject":"Google Code supports Git","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-07-16T10:24:50Z","receivedAt":"2011-07-16T10:24:50Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Just out of curiousity and because I happen to know we have Googlers\nhere. If it's not confidential, are there any changes in git to make\nit work with Google Code? I am particularly interested in whether\nGoogle modifies git to use bigtable (or cassandra, I remember Shawn\nhad a prototype).\n-- \nDuy\n"},{"id":"171489","messageId":"CAGdFq_h4E75LDWkQjkp_DvOAWUC_H+2KEvWMeGviPuFSJWGu+w@mail.gmail.com","threadId":"27825","inReplyTo":"CACsJy8BMK4UqVVNnhs6riVSS-OKcFKT9=XScrAw5nK2FH_u92A@mail.gmail.com","subject":"Re: Google Code supports Git","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-07-16T20:44:34Z","receivedAt":"2011-07-16T20:44:34Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Sat, Jul 16, 2011 at 12:24, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n> Just out of curiousity and because I happen to know we have Googlers\n> here. If it's not confidential, are there any changes in git to make\n> it work with Google Code? I am particularly interested in whether\n> Google modifies git to use bigtable (or cassandra, I remember Shawn\n> had a prototype).\n\nIf nothing else, the \"hg on bigtable\" talk from I/O 2008 is probably relevant.\n\nhttp://www.youtube.com/watch?v=ri796Hx8las\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"171491","messageId":"4E220018.3080005@gmail.com","threadId":"27825","inReplyTo":"CAGdFq_h4E75LDWkQjkp_DvOAWUC_H+2KEvWMeGviPuFSJWGu+w@mail.gmail.com","subject":"Re: Google Code supports Git","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2011-07-16T21:18:16Z","receivedAt":"2011-07-16T21:18:16Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"On 07/16/2011 04:44 PM, Sverre Rabbelier wrote:\n> Heya,\n>\n> On Sat, Jul 16, 2011 at 12:24, Nguyen Thai Ngoc Duy<pclouds@gmail.com>  wrote:\n>> Just out of curiousity and because I happen to know we have Googlers\n>> here. If it's not confidential, are there any changes in git to make\n>> it work with Google Code? I am particularly interested in whether\n>> Google modifies git to use bigtable (or cassandra, I remember Shawn\n>> had a prototype).\n>\n> If nothing else, the \"hg on bigtable\" talk from I/O 2008 is probably relevant.\n>\n> http://www.youtube.com/watch?v=ri796Hx8las\n>\n\nI know I would would appreciate it and I believe many on this list would \nalso appreciate understanding how git hosting in Google code was \nimplemented.\n"},{"id":"171495","messageId":"CAJo=hJtqjgvEJzSqhJBUufatCpeLgpMVN0Eqps2W8WNMaQR7Vw@mail.gmail.com","threadId":"27825","inReplyTo":"CACsJy8BMK4UqVVNnhs6riVSS-OKcFKT9=XScrAw5nK2FH_u92A@mail.gmail.com","subject":"Re: Google Code supports Git","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-07-17T02:26:11Z","receivedAt":"2011-07-17T02:26:11Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sat, Jul 16, 2011 at 03:24, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n> Just out of curiousity and because I happen to know we have Googlers\n> here. If it's not confidential, are there any changes in git to make\n> it work with Google Code? I am particularly interested in whether\n> Google modifies git to use bigtable\n\nA major milestone in Git was adding smart HTTP. If you watch the talk\nSverre linked to, you will learn that Google is based heavily on HTTP.\nA fundamental issue at the time Hg on Google Code was added was Git\ndidn't really work well over HTTP. Adding smart HTTP in Git 1.6.6 made\nit more realistic for Google to support Git on Project Hosting. I\nadded smart HTTP support for kernel.org so their users behind\nfirewalls could still use an efficient Git protocol to fetch revisions\nfrom kernel.org for projects that are hosted there. Its a nice bonus\nthat this work made Git on Google Code more realistic for Google.\n\nWe are trying to get the engineer responsible for making Git on Google\nCode possible to give a recorded tech talk like the one Sverre linked\nto. I don't want to steal his thunder, but I can say the Git on Google\nCode work is not based on C Git or JGit.  :-)\n\n> (or cassandra, I remember Shawn\n> had a prototype).\n\nThis was an unrelated project, and is what I deem to be a failure...\nquite unlike Git on Google Code. :-)\n\nFor some background, at GitTogether in Oct. 2010 I showed a demo of\nJGit using the Apache Cassandra database as an object / reference\nstore. This prototype didn't really scale well; even though I demoed\nthe linux-2.6 repository being cloned through a JGit daemon using\nCassandra as the backing store, it was slow and used too much CPU and\nmemory resources to be useful in any context beyond a \"Look, I can do\nthis!\" demo. I managed to open source this work, it may still be\nlaying around somewhere, but I basically threw it out the window and\nsaid \"that isn't good, and I can't believe I put my name on it!\". (And\nfor the record I was not the first to try this, Scott Chacon at GitHub\ntried something similar first and demoed it at GitTogether in 2009.)\n\nIn late Jan/Feb 2011 I released a series of patches for JGit that\nadded what I called \"DHT\" (distributed hash table) support. These\npatches are now part of the JGit project. Its different from the\noriginal Cassandra prototype. With this work, JGit tries to treat the\nDHT as though it were a virtual memory system. Relatively standard\npack files are segmented into ~ 1 MiB chunks, then stored into the DHT\nwith row keys based on the SHA-1 hash of the content of the \"pack\nchunk\". The bet here is that the locality of data in a pack file is\nquite good, so loading a chunk of commits ~1 MiB in size should get us\na number of related commits, amortizing out the round-trip time to the\ndatabase. This was to resolve one of the latency problems I saw with\nthe Cassandra prototype, which stored 1 commit per row and had awful\nperformance during a major revision traversal like a clone has.\n\nThe JGit DHT work lead me to discover the pack locality is not as good\nas we think it is. Its good, but it can be better. I added some\npatches to JGit's PackWriter to reorder objects in an order that gave\nbetter data locality. After Junio and I started sharing an office, I\nbegan nagging him about this locality problem in Git pack files... and\nthat nagging lead to a series of patches Junio posted about a week\nback to improve pack-objects.c. The improvement is small on local\ndisk, it reduces some minor page faults, however there isn't much\ndifference in overall running time. Over higher latency filesystems\nhowever, like an NFS server in another city, it should help reading.\n\nJust recently I posted a message to the jgit-dev mailing list saying I\nalso now think JGit DHT isn't a viable solution, and am likely to\ndiscard it in the future. Its implementation is very complicated, and\nit just doesn't perform as well as I had hoped. FWIW, this work was\nnot for Google, but was for open source Git hosting sites like\nsource.android.com, eclipse.org, KDE, etc. where they need to manage a\nlarge number of Git repositories, and want to have hot-failover and\nload-balancing to reduce down time caused by hardware failures.\nUnfortunately it hasn't been panning out, because the performance loss\nis a lot compared to the small administrative improvements it might\nbring. Not to mention the additional complexity of running the\nclustered database vs. just a bunch of Git repositories in a\ndirectory.\n\n\nI can tell you that none of this is what Git for Google Code does.\n\nAs for how Git on Google Code is implemented... you'll just have to\nwait for the tech talk from the engineer responsible. I can say it\nwasn't me, and it wasn't Junio. I am too busy with JGit and Gerrit\nCode Review, and Junio is too busy being Git maintainer to work on a\nmajor new feature like this.\n\n:-)\n\n-- \nShawn.\n"},{"id":"171498","messageId":"4E22BD8E.1080207@gmail.com","threadId":"27825","inReplyTo":"CAJo=hJtqjgvEJzSqhJBUufatCpeLgpMVN0Eqps2W8WNMaQR7Vw@mail.gmail.com","subject":"Re: Google Code supports Git","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2011-07-17T10:46:38Z","receivedAt":"2011-07-17T10:46:38Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On 17.07.2011 04:26, Shawn Pearce wrote:\n\n> to. I don't want to steal his thunder, but I can say the Git on Google\n> Code work is not based on C Git or JGit.  :-)\n\nThen it's probably JavaScript ;-)\n\nhttps://github.com/danlucraft/git.js\n\n-- \nSebastian Schuberth\n"},{"id":"171504","messageId":"CACBZZX40M3-N1gkiHsKHZpK2dd4DeKEBA2FcvmcHzGXMgT+ihA@mail.gmail.com","threadId":"27825","inReplyTo":"CAJo=hJtqjgvEJzSqhJBUufatCpeLgpMVN0Eqps2W8WNMaQR7Vw@mail.gmail.com","subject":"Re: Google Code supports Git","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2011-07-17T15:45:36Z","receivedAt":"2011-07-17T15:45:36Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sun, Jul 17, 2011 at 04:26, Shawn Pearce <spearce@spearce.org> wrote:\n\n> We are trying to get the engineer responsible for making Git on Google\n> Code possible to give a recorded tech talk like the one Sverre linked\n> to. I don't want to steal his thunder, but I can say the Git on Google\n> Code work is not based on C Git or JGit.  :-)\n\nI don't think you have to worry about that, looks like he already\nstole his own thunder:\n\n http://hackerne.ws/item?id=2769816\n http://code.google.com/u/dborowitz@google.com/\n\nI.e. Google Code is using Dulwich:\n\n http://www.samba.org/~jelmer/dulwich/\n"}]}