{"thread":{"id":"11248","subject":"[(not so) random thoughts] using git as its own caching tool","startedAt":"2007-12-12T00:38:13Z","lastAt":"2007-12-12T16:27:42Z","messageCount":6,"participants":["Pierre Habouzit","Mike Hommey","Andreas Ericsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"62837","messageId":"20071212003813.GG29110@artemis.madism.org","threadId":"11248","inReplyTo":null,"subject":"[(not so) random thoughts] using git as its own caching tool","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2007-12-12T00:38:13Z","receivedAt":"2007-12-12T00:38:13Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":" That's an idea I have for quite some time, and I wonder why it's not\nused in git tools as a general rule.\n\n This idea is simple, git objects database has two (for this\ndiscussion) very interesting features: its delta compressed cached that\nis _very_ efficient, and the reflog.\n\n I wonder if that would be possible to write some git porcelains (and\nbuiltin API too) that would be more \"map\" oriented. I mean, we could use\na reference as a pointer to a given tree that would be the map (where\nkeys have a path form, which is nice). When I say that, I'm thinking\nabout git-svn, that even with the recent improvements of its .rev_db's\nstill eats a lot of space with the unhandled.log _and_ the indexes it\nstores for _each_ svn branch/tag. This way, instead of many:\n foo/index\n foo/.rev_map.6ef976f9-4de5-0310-a40d-91cae572ec18\n foo/unhandled.log\nwe would just have a special refs/db/git-svn/foo reference that would be\na tree with three blobs in it: index, rev_map.xxxx, unhandled.log. (or\nprobably index would even be a tree but that's another matter). This\nway, all the unhandled.log that share a lot of common content would be\nnicely compressed by the delta-compression algorithms, with a negligible\noverhead (git-svn is _very_ slow because of svn anyways, we don't really\ncare if it needs to get a blob contents instead opening a flat file).\n\n\n Another nifty usage we could have is memoization databases that don't\nrequire a specific tool to expire them, but use the reflog expiration\nfor that. I remember that we discussed quite some time ago, the idea of\nannotating objects. We could use such annotations to link some objects\nto memoized datas under different namespaces for each caching scheme\ninvolved, and with one reference per namespace that will have in its\nreflog each of the linked objects created over time for caching. Good\ncandidates to use that are the rr-cache, or git-annotate/blame caching.\nOf course that would need to write a tool that removes weak annotations\nthat point to objects that don't exist anymore. We could also cache the\nrename/copies/… detection results, and make those really really cheap to\nuse[0].\n\n\n I know that some will say something about hammers, problems and nails,\nthough it would allow to develop quite efficient tools with a generic\nand easy to use API, that could directly benefit from already existing\ninfrastructure in git. I mean it's silly to write yet-another cache\nexpirer when you have the reflog. Or to speak about git-svn again, it\ncould even version its state per branch the way I propose, it'll end up\nusing less disk that what it does now, with the immediate gain that it\nwould be fully clone-able[1] (which would be a _really_ nice feature).\n\n\n So am I having crazy thoughts and should I throw my crack-pipe away ?\nOr does parts of this mumbling makes any sense to someone ?\n\n\n\nPS: It's late, and I'm tired, hence my english is probably very clumsy,\n and I hope I'm understandable enough. I'd be glad to rephrase parts\n that needs it.\n\n [0] and if the copy/rename/… detection algorithm gets smarter, we just\n need to change its memoization namespace to throw the old cache\n away at once.\n\n [1] and the really nice part here is that even if you don't create one\n new step per svn revision you import but do macro-steps with\n hundreds of svn revisions at a time, the merge of two differnt\n git-svn states of two clones of the _same_ svn repository will\n have a trivial exact merge: the one that knows the biggest svn\n revision is the new state to use.\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"62861","messageId":"20071212065125.GA6063@glandium.org","threadId":"11248","inReplyTo":"20071212003813.GG29110@artemis.madism.org","subject":"Re: [(not so) random thoughts] using git as its own caching tool","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-12-12T06:51:25Z","receivedAt":"2007-12-12T06:51:25Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Wed, Dec 12, 2007 at 01:38:13AM +0100, Pierre Habouzit wrote:\n> So am I having crazy thoughts and should I throw my crack-pipe away ?\n> Or does parts of this mumbling makes any sense to someone ?\n\nI love this idea.\n\nMike\n"},{"id":"62914","messageId":"475FFFB7.4010102@op5.se","threadId":"11248","inReplyTo":"20071212003813.GG29110@artemis.madism.org","subject":"Re: [(not so) random thoughts] using git as its own caching tool","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2007-12-12T15:35:19Z","receivedAt":"2007-12-12T15:35:19Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Pierre Habouzit wrote:\n> That's an idea I have for quite some time, and I wonder why it's not\n> used in git tools as a general rule.\n> \n> This idea is simple, git objects database has two (for this\n> discussion) very interesting features: its delta compressed cached that\n> is _very_ efficient, and the reflog.\n> \n> I wonder if that would be possible to write some git porcelains (and\n> builtin API too) that would be more \"map\" oriented. I mean, we could use\n> a reference as a pointer to a given tree that would be the map (where\n> keys have a path form, which is nice). When I say that, I'm thinking\n> about git-svn, that even with the recent improvements of its .rev_db's\n> still eats a lot of space with the unhandled.log _and_ the indexes it\n> stores for _each_ svn branch/tag. This way, instead of many:\n> foo/index\n> foo/.rev_map.6ef976f9-4de5-0310-a40d-91cae572ec18\n> foo/unhandled.log\n> we would just have a special refs/db/git-svn/foo reference that would be\n> a tree with three blobs in it: index, rev_map.xxxx, unhandled.log. (or\n> probably index would even be a tree but that's another matter). This\n> way, all the unhandled.log that share a lot of common content would be\n> nicely compressed by the delta-compression algorithms, with a negligible\n> overhead (git-svn is _very_ slow because of svn anyways, we don't really\n> care if it needs to get a blob contents instead opening a flat file).\n> \n> \n> Another nifty usage we could have is memoization databases that don't\n> require a specific tool to expire them, but use the reflog expiration\n> for that. I remember that we discussed quite some time ago, the idea of\n> annotating objects. We could use such annotations to link some objects\n> to memoized datas under different namespaces for each caching scheme\n> involved, and with one reference per namespace that will have in its\n> reflog each of the linked objects created over time for caching. Good\n> candidates to use that are the rr-cache, or git-annotate/blame caching.\n> Of course that would need to write a tool that removes weak annotations\n> that point to objects that don't exist anymore. We could also cache the\n> rename/copies/… detection results, and make those really really cheap to\n> use[0].\n> \n> \n> I know that some will say something about hammers, problems and nails,\n> though it would allow to develop quite efficient tools with a generic\n> and easy to use API, that could directly benefit from already existing\n> infrastructure in git. I mean it's silly to write yet-another cache\n> expirer when you have the reflog. Or to speak about git-svn again, it\n> could even version its state per branch the way I propose, it'll end up\n> using less disk that what it does now, with the immediate gain that it\n> would be fully clone-able[1] (which would be a _really_ nice feature).\n> \n> \n> So am I having crazy thoughts and should I throw my crack-pipe away ?\n> Or does parts of this mumbling makes any sense to someone ?\n> \n\nA bit of both ;-)\n\nI like the idea to use the git object store, because that certainly\nhas an API that can't be done away with by user config. The reflog\nand its expiration mechanism is subject to human control though, and\neveryone doesn't even have them enabled. I don't for some repos where\nI know I'll create a thousand-and-one loose objects by rebasing,\n--amend'ing and otherwise fiddling with history rewrites.\n\nHaving a tool that works on some repos but not on others because it\nrelies on me living with an auto-gc after pretty much every operation\nwould be very tiresome indeed.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"62917","messageId":"20071212154848.GA19294@glandium.org","threadId":"11248","inReplyTo":"475FFFB7.4010102@op5.se","subject":"Re: [(not so) random thoughts] using git as its own caching tool","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-12-12T15:48:48Z","receivedAt":"2007-12-12T15:48:48Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Wed, Dec 12, 2007 at 04:35:19PM +0100, Andreas Ericsson wrote:\n> A bit of both ;-)\n> \n> I like the idea to use the git object store, because that certainly\n> has an API that can't be done away with by user config. The reflog\n> and its expiration mechanism is subject to human control though, and\n> everyone doesn't even have them enabled. I don't for some repos where\n> I know I'll create a thousand-and-one loose objects by rebasing,\n> --amend'ing and otherwise fiddling with history rewrites.\n> \n> Having a tool that works on some repos but not on others because it\n> relies on me living with an auto-gc after pretty much every operation\n> would be very tiresome indeed.\n\nThere is already a tool that relies on reflogs: stash.\n\nMike\n"},{"id":"62922","messageId":"47600634.3030207@op5.se","threadId":"11248","inReplyTo":"20071212154848.GA19294@glandium.org","subject":"Re: [(not so) random thoughts] using git as its own caching tool","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2007-12-12T16:03:00Z","receivedAt":"2007-12-12T16:03:00Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Mike Hommey wrote:\n> On Wed, Dec 12, 2007 at 04:35:19PM +0100, Andreas Ericsson wrote:\n>> A bit of both ;-)\n>>\n>> I like the idea to use the git object store, because that certainly\n>> has an API that can't be done away with by user config. The reflog\n>> and its expiration mechanism is subject to human control though, and\n>> everyone doesn't even have them enabled. I don't for some repos where\n>> I know I'll create a thousand-and-one loose objects by rebasing,\n>> --amend'ing and otherwise fiddling with history rewrites.\n>>\n>> Having a tool that works on some repos but not on others because it\n>> relies on me living with an auto-gc after pretty much every operation\n>> would be very tiresome indeed.\n> \n> There is already a tool that relies on reflogs: stash.\n> \n\nNo, \"git stash save\" works anyway. It's when you want to use multiple\nstashes that it becomes tricky, but even that works if you're willing\nto put some effort into it (although I don't use stash a lot, and not\nat all in the very rebase-heavy ones).\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"62932","messageId":"20071212162742.GE14538@artemis.madism.org","threadId":"11248","inReplyTo":"475FFFB7.4010102@op5.se","subject":"Re: [(not so) random thoughts] using git as its own caching tool","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2007-12-12T16:27:42Z","receivedAt":"2007-12-12T16:27:42Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Dec 12, 2007 at 03:35:19PM +0000, Andreas Ericsson wrote:\n> Pierre Habouzit wrote:\n> > So am I having crazy thoughts and should I throw my crack-pipe away ?\n> >Or does parts of this mumbling makes any sense to someone ?\n> \n> A bit of both ;-)\n> \n> I like the idea to use the git object store, because that certainly\n> has an API that can't be done away with by user config. The reflog\n> and its expiration mechanism is subject to human control though, and\n> everyone doesn't even have them enabled. I don't for some repos where\n> I know I'll create a thousand-and-one loose objects by rebasing,\n> --amend'ing and otherwise fiddling with history rewrites.\n> \n> Having a tool that works on some repos but not on others because it\n> relies on me living with an auto-gc after pretty much every operation\n> would be very tiresome indeed.\n\n Well if you disable the reflog on some repositories, those commands\nwill just be slow. But would still work.\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"}]}