{"thread":{"id":"24544","subject":"Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","startedAt":"2010-07-28T00:13:09Z","lastAt":"2010-08-13T19:19:54Z","messageCount":16,"participants":["Elijah Newren","Avery Pennarun","Nguyen Thai Ngoc Duy","Sverre Rabbelier","Enrico Weigelt","Jonathan Nieder"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"146556","messageId":"AANLkTikJhSVJw2hXkp0j6XA+k-J-AtSYzKWumjnqqsgz@mail.gmail.com","threadId":"24544","inReplyTo":null,"subject":"Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-07-28T00:13:09Z","receivedAt":"2010-07-28T00:13:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\n2010/7/27 Shawn O. Pearce <spearce@spearce.org>:\n> I would prefer doing something more like what we do with shallow\n> on the client side.  Record in a magic file the path(s) that we\n> did actually obtain.  During fsck, rev-list, or read-tree the\n> client skips over any paths that don't match that file's listing.\n> Then we can keep the same commit SHA-1s, but we won't complain that\n> there are objects missing.\n\nI recently decided to take a crack at implementing sparse clones, due\nto a crazy idea I had (which might not be as crazy as I thought since\nyou suggest something similar, though more limited).  I was going to\nwait until I actually got somewhere tangible with it to post an RFC,\nparticularly since it may take me a while, but since it's fresh on\neveryone's minds perhaps now is good anyway.\n\nDoes the following seem sane, or are there big gotchas that I'm just unaware of?\n\n0) Sparse clones have \"all\" commit objects, but not all trees/blobs.\n\nNote that \"all\" only means all that are reachable from the refs being\ndownloaded, of course.  I think this is widely agreed upon and has\nbeen suggested many times on this list.\n\n1) A user controls sparseness by passing rev-list arguments to clone.\n\nThis allows a user to control sparseness both in terms of span of\ncontent (files/directories) and depth of history.  It can also be used\nto limit to a subset of refs (cloning just one or two branches instead\nof all branches and tags).  For example,\n  $ git clone ssh://repo.git dst -- Documentation/\n  $ git clone ssh://repo.git dst master~6..master\n  $ git clone ssh://repo.git dst -3\n(Note that the destination argument becomes mandatory for those doing\na sparse clone in order to disambiguate it from rev-list options.)\n\nThis method also means users don't need much training to learn how to\nuse sparse clones -- they just use syntax they've already learned with\nlog, and clone will pass this info on to upload-pack.\n\nThere is a slight question as to whether users should have to specify\n\"--all HEAD\" with all sparse clones or whether it should be assumed\nwhen no other refs are listed.\n\n2) Sparse checkouts are automatically invoked with the path(s) from\n   the specified rev-list arguments.\n\nCan't checkout content that we don't have.  :-)\n\nThis has a slight downside -- it makes sparse checkouts and sparse\nclones slight misfits: the syntax (.gitignore style vs. rev-list\narguments) is a bit different, and sparse checkouts can exclude\ncertain paths whereas my sparse clones would only be able to *include*\npaths.  I don't see this as a deal-breaker, but even if others\ndisagree I think a more general path-exclusion mechanism for the\nrevision walking machinery would be really nice for reasons beyond\njust this one.  I've often wanted to do something like\n  git log -S'important code phrase' --EXCLUDE-PATH=big-data-dir\n\n3) The limiting rev-list arguments passed to clone are stored.\n\nHowever, relative arguments such as \"-3\" or \"master~6\" first need to\nbe translated into one or more exclude ranges written as \"^<sha1>\".\n\n4) All revision-walking operations automatically use these limiting args.\n\nThis should be a simple code change, and would enable rev-list, log,\netc. to avoid missing blobs/trees and thus enable them to work with\nsparse clones.  fsck would take a bit more work, since it doesn't use\nthe setup_revisions() and revision.h walking machinery, but shouldn't\nbe too bad (I hope).\n\nThere are also performance ramifications: There should be no\nmeasurable performance overhead for non-sparse clones (something that\nmight be a problem with a different implementation that did\ndoes-this-exist check each time it references a blob).  It should also\nbe a significant performance boost for those using it, as operations\nwill only need to deal with the subset of the repository they specify\n(faster downloads, stats, logs, etc.)\n\n5) \"Densifying\" a sparse clone can be done\n\nOne can fetch a new pack and replace the limiting rev-list args with\nthe new choice.  The sparse checkout information needs to be updated\ntoo.\n\n(So users probably would want to densify a sparse clone with \"pull\"\nrather than \"fetch\", as manually updating sparse checkouts may be a\nbit of a hassle.)\n\n6) Cloning-from/fetching-from/pushing-to sparse clones is supported.\n\nFuture fetches and pushes also make use of the limiting arguments.\nReceives do as well, but only to make sure the pack obtained is not\n\"more sparse\" than what the receiving repository already has.\n(uploads ignore the stored rev-list arguments, instead using the\nrev-list arguments passed to it -- it will die if asked for content\nnot locally available to it.)\n\n7) Operations that need unavailable data simply error out\n\nExamples: merge, cherry-pick, rebase (and upload-pack in a sparse\nclone).  However, hopefully the error messages state what extra\ninformation needs to be downloaded so the user can appropriately\n\"densify\" their repository.\n\n\nThanks,\nElijah\n"},{"id":"146559","messageId":"AANLkTikMLOFet-VMT7MntPgoSkvqGAXPd8Z1aaDpY1xs@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTikJhSVJw2hXkp0j6XA+k-J-AtSYzKWumjnqqsgz@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-28T01:05:10Z","receivedAt":"2010-07-28T01:05:10Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"2010/7/27 Elijah Newren <newren@gmail.com>:\n> 0) Sparse clones have \"all\" commit objects, but not all trees/blobs.\n>\n> Note that \"all\" only means all that are reachable from the refs being\n> downloaded, of course.  I think this is widely agreed upon and has\n> been suggested many times on this list.\n\nI think downloading all commit objects would require very low\nbandwidth and storage space, so it should be harmless.\n\nIn fact, I have a pretty strong impression that also downloading all\n*tree* objects would be fine too.  But I've never actually gone and\ncounted them to see what the stats are like.  Still, I'd assume that\nthe vast majority of repo space is blobs, not trees, and that trees\nare highly compatible with deltafication.\n\nNote that if you happen to want to implement it in a way that you'll\nalso get all the commit objects from your submodules too (which I\nhighly encourage :)) then downloading the trees is the easiest way.\nOtherwise you won't know which submodule commits you need.\n\n> 1) A user controls sparseness by passing rev-list arguments to clone.\n>\n> This allows a user to control sparseness both in terms of span of\n> content (files/directories) and depth of history.  It can also be used\n> to limit to a subset of refs (cloning just one or two branches instead\n> of all branches and tags).  For example,\n>  $ git clone ssh://repo.git dst -- Documentation/\n>  $ git clone ssh://repo.git dst master~6..master\n>  $ git clone ssh://repo.git dst -3\n> (Note that the destination argument becomes mandatory for those doing\n> a sparse clone in order to disambiguate it from rev-list options.)\n\nIt's really too bad that the dst argument took up that slot which, in\nevery other git command, is where the list of revs would go :(  Other\nthan that, I think the syntax looks nice.\n\n> There is a slight question as to whether users should have to specify\n> \"--all HEAD\" with all sparse clones or whether it should be assumed\n> when no other refs are listed.\n\nSince downloading commits is so cheap anyway, I'd suggest just\ndefaulting to downloading all the refs, as clone currently does.  If\npeople don't like it, they can do what they currently do:\n\n   git init\n   git remote add ...\n   git fetch\n\nNot that pretty, but then again, it's rarely needed.\n\n> 2) Sparse checkouts are automatically invoked with the path(s) from\n>   the specified rev-list arguments.\n>\n> Can't checkout content that we don't have.  :-)\n>\n> This has a slight downside -- it makes sparse checkouts and sparse\n> clones slight misfits: the syntax (.gitignore style vs. rev-list\n> arguments) is a bit different, and sparse checkouts can exclude\n> certain paths whereas my sparse clones would only be able to *include*\n> paths.  I don't see this as a deal-breaker, but even if others\n> disagree I think a more general path-exclusion mechanism for the\n> revision walking machinery would be really nice for reasons beyond\n> just this one.  I've often wanted to do something like\n>  git log -S'important code phrase' --EXCLUDE-PATH=big-data-dir\n\nI don't totally understand what you mean here.  But I do think that if\nyou can *mostly* trim down a tree, excluding every little thing is not\nthat important.  As was discussed on the other thread, it seems like\n*most* people are trimming down their trees (currently using\nsubmodules) just to make stuff faster, and getting rid of 90% of the\nunwanted cruft is probably fine; getting rid of 100% of it isn't that\nmuch more of a speed boost.\n\nI guess my point is, more complex exclusions could always be added\nlater but they aren't so important right away.\n\n> 3) The limiting rev-list arguments passed to clone are stored.\n>\n> However, relative arguments such as \"-3\" or \"master~6\" first need to\n> be translated into one or more exclude ranges written as \"^<sha1>\".\n\nJust run them through rev-parse, I think.\n\n> 4) All revision-walking operations automatically use these limiting args.\n>\n> This should be a simple code change, and would enable rev-list, log,\n> etc. to avoid missing blobs/trees and thus enable them to work with\n> sparse clones.  fsck would take a bit more work, since it doesn't use\n> the setup_revisions() and revision.h walking machinery, but shouldn't\n> be too bad (I hope).\n\nI don't know if this implementation detail would be better or worse\nthan just having the tools auto-trim their activities when they run\ninto a missing object.  But maybe.  It does sound sort of elegant:\nthis way they *won't* run into the missing objects.\n\nBeware, however, that\n\n   git log -- Documentation\n\noutputs a different set of commits than just\n\n   git log\n\nYou don't want to enable history simplification here; I think that\nmeans you want --full-history on by default for the \"stored\" path\nlimiting, but not for any command-line path limiting.  That could be\nslightly messy.\n\n> 5) \"Densifying\" a sparse clone can be done\n>\n> One can fetch a new pack and replace the limiting rev-list args with\n> the new choice.  The sparse checkout information needs to be updated\n> too.\n>\n> (So users probably would want to densify a sparse clone with \"pull\"\n> rather than \"fetch\", as manually updating sparse checkouts may be a\n> bit of a hassle.)\n\nI think this would work, but unless you want to re-download some\n(possibly lots of) objects you've already got, it would require some\nkind of extra support from the server, I think.  Maybe that's a rare\nenough case that few people will care and it could be fixed later.\n\nI don't think the pull vs. fetch distinction is valid; I would be very\nsurprised if pull un-sparsified my checkout, just as I would be\nsurprised if merge did.  And pull is just fetch+merge.\n\n> 6) Cloning-from/fetching-from/pushing-to sparse clones is supported.\n>\n> Future fetches and pushes also make use of the limiting arguments.\n> Receives do as well, but only to make sure the pack obtained is not\n> \"more sparse\" than what the receiving repository already has.\n> (uploads ignore the stored rev-list arguments, instead using the\n> rev-list arguments passed to it -- it will die if asked for content\n> not locally available to it.)\n\nThis scares me a little.  It's a reminder that it's all-too-easy to\nget your repository into a really messed up state by going in and\nscrewing with your sparseness parameters at the wrong time.\n\nIt would make me more comfortable if there was some kind of \"oh god,\njust fix it by downloading any objects you think are missing\" mode :)\nIn fact, git could benefit from that in general - every now and then\nsomeone on the list asks about a repository they managed to mangle by\ncorrupting a pack or something, and there's no really good answer to\nthat.\n\n> 7) Operations that need unavailable data simply error out\n>\n> Examples: merge, cherry-pick, rebase (and upload-pack in a sparse\n> clone).  However, hopefully the error messages state what extra\n> information needs to be downloaded so the user can appropriately\n> \"densify\" their repository.\n\nThat sounds good to me.\n\nHave fun,\n\nAvery\n"},{"id":"146567","messageId":"AANLkTin3OB6mw6rih208ybCL2+QQ8LpuCP5y=DywMuyw@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTikMLOFet-VMT7MntPgoSkvqGAXPd8Z1aaDpY1xs@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-28T03:06:57Z","receivedAt":"2010-07-28T03:06:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/7/28 Avery Pennarun <apenwarr@gmail.com>:\n> 2010/7/27 Elijah Newren <newren@gmail.com>:\n>> 0) Sparse clones have \"all\" commit objects, but not all trees/blobs.\n>>\n>> Note that \"all\" only means all that are reachable from the refs being\n>> downloaded, of course.  I think this is widely agreed upon and has\n>> been suggested many times on this list.\n>\n> I think downloading all commit objects would require very low\n> bandwidth and storage space, so it should be harmless.\n\nHere you go. A pack with only commits and trees of git.git#master is\n15M. With blobs, it is 28M. Git is not a typical repo with large blobs\nthough.\n-- \nDuy\n"},{"id":"146568","messageId":"AANLkTi=8u5VROYQygAXoCS4c+eAoEoP8V4t5rJ=wXL8q@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTikMLOFet-VMT7MntPgoSkvqGAXPd8Z1aaDpY1xs@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-07-28T03:31:55Z","receivedAt":"2010-07-28T03:31:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"2010/7/27 Avery Pennarun <apenwarr@gmail.com>:\n> Note that if you happen to want to implement it in a way that you'll\n> also get all the commit objects from your submodules too (which I\n> highly encourage :)) then downloading the trees is the easiest way.\n> Otherwise you won't know which submodule commits you need.\n\nMakes sense.  Seems like a good reason to include all the trees.\n\n> Since downloading commits is so cheap anyway, I'd suggest just\n> defaulting to downloading all the refs, as clone currently does.  If\n> people don't like it, they can do what they currently do:\n>\n>   git init\n>   git remote add ...\n>   git fetch\n>\n> Not that pretty, but then again, it's rarely needed.\n\nWould you suggest then parsing the limiting arguments passed to clone\nand disallowing refs?  Or just making it non-useful by always\nappending \"--all HEAD\"?\n\n>> 2) Sparse checkouts are automatically invoked with the path(s) from\n>>   the specified rev-list arguments.\n<snip>\n> I don't totally understand what you mean here.  But I do think that if\nBasically, I mean what you stated much more succinctly and eloquently\nright here:\n> I guess my point is, more complex exclusions could always be added\n> later but they aren't so important right away.\n\n>> 4) All revision-walking operations automatically use these limiting args.\n<snip>\n> It does sound sort of elegant: this way they *won't* run into the missing objects.\n> Beware, however, that\n>\n>   git log -- Documentation\n>\n> outputs a different set of commits than just\n>\n>   git log\n\nYes, exactly.  In a sparse clone, why wouldn't one want the behavior\nof the former automatically, without having to specify the paths on\nthe command line every time they ran log (or rev-list or fast-export\nor...etc., especially if they cloned N directories rather than just\n1)?\n\nActually, I can kind of see the desire to see the 'real' log since the\nusers do happen to have all commits locally, but it almost seems like\nit should be the case that requires a special option to be passed to\ngit log ('--ignore-sparse-limiting'?).  But trying to get that option\nto work in conjunction with other options (--stat, -S, -p, etc.) would\nbe really hard, if not impossible.\n\n>> 5) \"Densifying\" a sparse clone can be done\n<snip>\n> I think this would work, but unless you want to re-download some\n> (possibly lots of) objects you've already got, it would require some\n> kind of extra support from the server, I think.  Maybe that's a rare\n> enough case that few people will care and it could be fixed later.\n\nFor my first implementation, my plan was to simply re-download ALL\n(not just some or lots of) objects I've already got in such a case.  A\nbit wasteful to be sure, but I was hoping it was rare enough to\n\"densify\" a clone that it wouldn't be a big deal...and that support\nfor smarter downloads could be added later.\n\n> I don't think the pull vs. fetch distinction is valid; I would be very\n> surprised if pull un-sparsified my checkout, just as I would be\n> surprised if merge did.  And pull is just fetch+merge.\n\nRight, I don't think pull should un-sparsify either the checkout OR\nthe clone by default (it should have fetch pass the same limiting\narguments and only download an equivalently sparse set of updates).\nYour point about pull=fetch+merge (or fetch+rebase) makes sense, which\nI guess means that un-sparsifying a clone+checkout should be a\nseparate toplevel command (\"densify\"?) rather than a special option\nfor fetch/pull.\n\n>> 6) Cloning-from/fetching-from/pushing-to sparse clones is supported.\n>>\n>> Future fetches and pushes also make use of the limiting arguments.\n>> Receives do as well, but only to make sure the pack obtained is not\n>> \"more sparse\" than what the receiving repository already has.\n>> (uploads ignore the stored rev-list arguments, instead using the\n>> rev-list arguments passed to it -- it will die if asked for content\n>> not locally available to it.)\n>\n> This scares me a little.  It's a reminder that it's all-too-easy to\n> get your repository into a really messed up state by going in and\n> screwing with your sparseness parameters at the wrong time.\n\nI don't follow.  Why would people be \"screwing with sparseness parameters\"?\n\nMy basic idea was that there would be only three ways to change\nsparseness parameters for clones, with only the first two documented:\nthe initial clone command, the \"densify\" command (someone probably\nneeds to think of a better name), and reading the source code to\nfigure out what bits on your disk to change and changing them.\n\n\nHere's why I want the clone-able/fetch-able/pull-able sparse clone\nfunctionality:\n\nI like having translators (who only need maybe one file) or technical\nwriters (who only need the Documentation/ subdirectory) or other\nsimilar folks having the ability to collaborate on the subset of the\nrepository that they need to do their work.  Thus, it makes sense for\nthem to be able to clone from, pull from, and push to each other.  The\nonly two rules that I think are necessary to enable such behavior are:\n\n* No repository can provide information that it doesn't have (should\nbe pretty easy to enforce...)\n* No repository accepts less data than it expects in its repository\n(i.e. you can push to a sparse clone or a real clone, but need to\nprovide data that fulfills it's rev-list limiting arguments)\n\n> It would make me more comfortable if there was some kind of \"oh god,\n> just fix it by downloading any objects you think are missing\" mode :)\n> In fact, git could benefit from that in general - every now and then\n> someone on the list asks about a repository they managed to mangle by\n> corrupting a pack or something, and there's no really good answer to\n> that.\n\nFor sparse clones, Isn't that mode just running the \"densify\" command\nwith no limiting arguments?\n\n>> 7) Operations that need unavailable data simply error out\n>>\n>> Examples: merge, cherry-pick, rebase (and upload-pack in a sparse\n>> clone).  However, hopefully the error messages state what extra\n>> information needs to be downloaded so the user can appropriately\n>> \"densify\" their repository.\n>\n> That sounds good to me.\n\nThanks for the detailed feedback.  :-)\n"},{"id":"146569","messageId":"AANLkTikvhi-Bm-eV9ZfjoZdugRpiCwg8n-TQ4oOVvC=G@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTikJhSVJw2hXkp0j6XA+k-J-AtSYzKWumjnqqsgz@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-28T03:36:47Z","receivedAt":"2010-07-28T03:36:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2010/7/28 Elijah Newren <newren@gmail.com>:\n> 1) A user controls sparseness by passing rev-list arguments to clone.\n>\n> This allows a user to control sparseness both in terms of span of\n> content (files/directories) and depth of history.  It can also be used\n> to limit to a subset of refs (cloning just one or two branches instead\n> of all branches and tags).  For example,\n>  $ git clone ssh://repo.git dst -- Documentation/\n\nDoes pathspec is supported to in addition to prefix?\n\n>  $ git clone ssh://repo.git dst master~6..master\n>  $ git clone ssh://repo.git dst -3\n> (Note that the destination argument becomes mandatory for those doing\n> a sparse clone in order to disambiguate it from rev-list options.)\n>\n> This method also means users don't need much training to learn how to\n> use sparse clones -- they just use syntax they've already learned with\n> log, and clone will pass this info on to upload-pack.\n>\n> There is a slight question as to whether users should have to specify\n> \"--all HEAD\" with all sparse clones or whether it should be assumed\n> when no other refs are listed.\n\nSo you basically kill off shallow clone too, with \"master~6..master\".\nI wonder what happens if user does \"git clone ... master~6..master~3\"?\n\n> 4) All revision-walking operations automatically use these limiting args.\n>\n> This should be a simple code change, and would enable rev-list, log,\n> etc. to avoid missing blobs/trees and thus enable them to work with\n> sparse clones.  fsck would take a bit more work, since it doesn't use\n> the setup_revisions() and revision.h walking machinery, but shouldn't\n> be too bad (I hope).\n>\n> There are also performance ramifications: There should be no\n> measurable performance overhead for non-sparse clones (something that\n> might be a problem with a different implementation that did\n> does-this-exist check each time it references a blob).  It should also\n> be a significant performance boost for those using it, as operations\n> will only need to deal with the subset of the repository they specify\n> (faster downloads, stats, logs, etc.)\n\nRevision walking is not the only gate to access objects. Others like\ndiff machinery needs also be taught about rev-list limits.\n\n> 5) \"Densifying\" a sparse clone can be done\n>\n> One can fetch a new pack and replace the limiting rev-list args with\n> the new choice.  The sparse checkout information needs to be updated\n> too.\n>\n> (So users probably would want to densify a sparse clone with \"pull\"\n> rather than \"fetch\", as manually updating sparse checkouts may be a\n> bit of a hassle.)\n\nWhat information would you send to the server to request new pack in\nsparse clone? Currently we send all commit tips. rev-list has a notion\nto subtract commit trees. I don't know if it can \"add\" or \"subtract\"\ntree prefix though.\n-- \nDuy\n"},{"id":"146570","messageId":"AANLkTinpGO=TU3+1wZw68tUmPL8yQatPUEJ6uTGynbHq@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTin3OB6mw6rih208ybCL2+QQ8LpuCP5y=DywMuyw@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-28T03:38:34Z","receivedAt":"2010-07-28T03:38:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"(corrected reply context, sorry)\n\nOn Wed, Jul 28, 2010 at 1:06 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n> 2010/7/28 Avery Pennarun <apenwarr@gmail.com>:\n>> 2010/7/27 Elijah Newren <newren@gmail.com>:\n>>> 0) Sparse clones have \"all\" commit objects, but not all trees/blobs.\n>>>\n>>> Note that \"all\" only means all that are reachable from the refs being\n>>> downloaded, of course.  I think this is widely agreed upon and has\n>>> been suggested many times on this list.\n>>\n>> I think downloading all commit objects would require very low\n>> bandwidth and storage space, so it should be harmless.\n> >\n> > In fact, I have a pretty strong impression that also downloading\n> > all *tree* objects would be fine too.\n>\n> Here you go. A pack with only commits and trees of git.git#master is\n> 15M. With blobs, it is 28M. Git is not a typical repo with large blobs\n> though.\n> --\n> Duy\n>\n\n\n\n-- \nDuy\n"},{"id":"146571","messageId":"AANLkTi=WjBHYngSDgowD2+0FT3hG+oDYefp2aUostV3U@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTinpGO=TU3+1wZw68tUmPL8yQatPUEJ6uTGynbHq@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-28T03:58:18Z","receivedAt":"2010-07-28T03:58:18Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Wed, Jul 28, 2010 at 1:06 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n> 2010/7/28 Avery Pennarun <apenwarr@gmail.com>:\n>> 2010/7/27 Elijah Newren <newren@gmail.com>:\n>>> 0) Sparse clones have \"all\" commit objects, but not all trees/blobs.\n>>>\n>>> Note that \"all\" only means all that are reachable from the refs being\n>>> downloaded, of course.  I think this is widely agreed upon and has\n>>> been suggested many times on this list.\n>>\n>> I think downloading all commit objects would require very low\n>> bandwidth and storage space, so it should be harmless.\n> >\n> > In fact, I have a pretty strong impression that also downloading\n> > all *tree* objects would be fine too.\n>\n> Here you go. A pack with only commits and trees of git.git#master is\n> 15M. With blobs, it is 28M. Git is not a typical repo with large blobs\n> though.\n\nHmm, that's very interesting - more than half the repo is just tree\nand commit objects?  Maybe that's not so shocking after all, given the\ntendency in the git project to use long commit messages and relatively\nshort patches.\n\nWas your pack carefully ordered for best deltification?\n\nKnowing how much of that is commits vs. trees would also be very interesting.\n\nBut if so, only saving half the space is kind of disappointing.  If\nyou have a script around for generating this, it would be very\ninteresting to compare the results with, say, the Linux kernel repo\n(especially since it seems to be the #1 example of \"submodules people\ndon't want to check out because they're so bloody huge\").\n\nIn bup, I know the trees+commits are much smaller than the blobs, so\nmy intuition was telling me it would be the same in git.  It's\nentirely possible that I was wrong, though.  In retrospect, bup uses\nreally short computer-generated commit messages, and backs up large\nnumbers of files at once, most of which never change (and thus most of\nthe trees never change).  Commits+trees end up somewhere around 0.5%\nof the total repo size.\n\nHave fun,\n\nAvery\n"},{"id":"146572","messageId":"AANLkTikoOL0jYi4kD0L1uLU=UCmG9p0Rc3VLOU-djQnY@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTikvhi-Bm-eV9ZfjoZdugRpiCwg8n-TQ4oOVvC=G@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-07-28T03:59:25Z","receivedAt":"2010-07-28T03:59:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jul 27, 2010 at 9:36 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n> 2010/7/28 Elijah Newren <newren@gmail.com>:\n>> 1) A user controls sparseness by passing rev-list arguments to clone.\n<snip>\n>> For example,\n>>  $ git clone ssh://repo.git dst -- Documentation/\n>\n> Does pathspec is supported to in addition to prefix?\n\nBasically, whatever git log or git rev-list accepts.  I think I saw\nsome other discussion about making those adopt some of the\ncode/capability of git grep, which would automatically benefit sparse\nclones.  But until then, no, because I need to be able to take these\narguments and automatically pass them on to log, rev-list, etc.\n\n> So you basically kill off shallow clone too, with \"master~6..master\".\n\nYes, that was part of the plan...extend the capabilities of shallow\nclones in two ways: allowing the user to specify a cutoff via a\nrevision identifier as well as a number of commits, and allow people\nto clone (and fetch-from/push-to) other \"shallow\" clones.\n\n> I wonder what happens if user does \"git clone ... master~6..master~3\"?\n\nCurrently, that'd break -- just like it similarly does for fast-export\n(see t/t9350-fast-export.sh, 'no exact-ref revisions included').  I\nhad been thinking of trying to get that fixed for both cases by making\nit result in a \"master\" branch that is \"three commits behind\" what you\nclone/fast-export from.  You'd have to look for and disallow other\nspecial cases like \"git fast-export ... master^1 master^2\" or \"git\nclone ... :/searchstring\".\n\nI'm not sure how this interacts with Avery's suggestion to just ignore\nbranch/tag limiting.\n\n> Revision walking is not the only gate to access objects. Others like\n> diff machinery needs also be taught about rev-list limits.\n\nRight, good point.  Are there others than the diff machinery (and the\nfsck special case) that you know of?\n\n> What information would you send to the server to request new pack in\n> sparse clone? Currently we send all commit tips. rev-list has a notion\n> to subtract commit trees. I don't know if it can \"add\" or \"subtract\"\n> tree prefix though.\n\nWhen \"densifying\" a sparse clone, I was (initially at least) just\ngoing to treat it like an initial clone and re-download _everything_\n(even if sparsifying rather than densifying).  I assumed it'd be rare\nto want to do such an operation, but yeah, in the future someone might\nwant a smarter way to handle it.\n"},{"id":"146576","messageId":"AANLkTiktuQV+jEOy+Shm13-LVc0W-d8qsGS6XuxyTW44@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTi=WjBHYngSDgowD2+0FT3hG+oDYefp2aUostV3U@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-07-28T06:12:32Z","receivedAt":"2010-07-28T06:12:32Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Tue, Jul 27, 2010 at 22:58, Avery Pennarun <apenwarr@gmail.com> wrote:\n> Hmm, that's very interesting - more than half the repo is just tree\n> and commit objects?  Maybe that's not so shocking after all, given the\n> tendency in the git project to use long commit messages and relatively\n> short patches.\n\nNote that in the case of the ginormous-tree this holds as well. Many\nsmall files, but in insanely deeply nested directories with insane\nfan-out. If you need to download all the trees you don't save that\nmuch bandwith. OTOH, I'm not only concerned about bandwidth, just\nbeing able to run 'git status' without it taking half a minute would\nbe sweet.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"146580","messageId":"AANLkTikcV7BdJ_07yLei9_H_kR2GEOi4707J1gGhS-Mr@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTi=WjBHYngSDgowD2+0FT3hG+oDYefp2aUostV3U@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-28T07:11:40Z","receivedAt":"2010-07-28T07:11:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 28, 2010 at 1:58 PM, Avery Pennarun <apenwarr@gmail.com> wrote:\n> On Wed, Jul 28, 2010 at 1:06 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n>> 2010/7/28 Avery Pennarun <apenwarr@gmail.com>:\n>>> 2010/7/27 Elijah Newren <newren@gmail.com>:\n>>>> 0) Sparse clones have \"all\" commit objects, but not all trees/blobs.\n>>>>\n>>>> Note that \"all\" only means all that are reachable from the refs being\n>>>> downloaded, of course.  I think this is widely agreed upon and has\n>>>> been suggested many times on this list.\n>>>\n>>> I think downloading all commit objects would require very low\n>>> bandwidth and storage space, so it should be harmless.\n>> >\n>> > In fact, I have a pretty strong impression that also downloading\n>> > all *tree* objects would be fine too.\n>>\n>> Here you go. A pack with only commits and trees of git.git#master is\n>> 15M. With blobs, it is 28M. Git is not a typical repo with large blobs\n>> though.\n>\n> Hmm, that's very interesting - more than half the repo is just tree\n> and commit objects?  Maybe that's not so shocking after all, given the\n> tendency in the git project to use long commit messages and relatively\n> short patches.\n>\n> Was your pack carefully ordered for best deltification?\n\nI did not do any optimization. It's pack-objects' defaults. I only\nfiltered blobs out and that's what fetch-pack would receive.\n\n>\n> Knowing how much of that is commits vs. trees would also be very interesting.\n\nCommits only takes 7.8 MB. Well.. all those commit messages..\n\n> But if so, only saving half the space is kind of disappointing.  If\n> you have a script around for generating this, it would be very\n> interesting to compare the results with, say, the Linux kernel repo\n> (especially since it seems to be the #1 example of \"submodules people\n> don't want to check out because they're so bloody huge\").\n\nI modified pack-objects.c, show_object() to certain objects. Actually\nI started with git-fetch-pack, but you can do\n\ngit rev-parse master | git pack-objects --revs --delta-base-offset pack\n\nThen verify what's in the pack with\n\ngit verify-pack -v pack-*.idx\n-- \nDuy\n"},{"id":"146591","messageId":"AANLkTi=zgbT=uw=t18mBUHMxg050pMx_YD835Gbontph@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTiktuQV+jEOy+Shm13-LVc0W-d8qsGS6XuxyTW44@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-28T07:59:20Z","receivedAt":"2010-07-28T07:59:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 28, 2010 at 4:12 PM, Sverre Rabbelier <srabbelier@gmail.com> wrote:\n> OTOH, I'm not only concerned about bandwidth, just\n> being able to run 'git status' without it taking half a minute would\n> be sweet.\n\nDoesn't assume-unchanged bit or sparse checkout help?\n-- \nDuy\n"},{"id":"146630","messageId":"AANLkTi=e37AGx-Pehe_PPn-0swxrf9Gowekm4cN80G1C@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTi=zgbT=uw=t18mBUHMxg050pMx_YD835Gbontph@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-07-28T14:48:01Z","receivedAt":"2010-07-28T14:48:01Z","isPatch":true,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Jul 28, 2010 at 02:59, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n> Doesn't assume-unchanged bit or sparse checkout help?\n\nSee my earlier part, assume-unchanged doesn't help due to the\ngitignore files. Sparse checkout isn't an option since I really need\nthose files to be there, I just don't ever modify them. Really what I\nneed is read-only checkout ;). \"Git, please ignore the existence of\nthis directory and all it's files/subdirectories, ktnx\".\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"146706","messageId":"AANLkTi=-+9TbR7=McSsh55gdzfKEsdqcao67kf0yFxsr@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTikoOL0jYi4kD0L1uLU=UCmG9p0Rc3VLOU-djQnY@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-07-29T10:29:32Z","receivedAt":"2010-07-29T10:29:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 28, 2010 at 1:59 PM, Elijah Newren <newren@gmail.com> wrote:\n>> Revision walking is not the only gate to access objects. Others like\n>> diff machinery needs also be taught about rev-list limits.\n>\n> Right, good point.  Are there others than the diff machinery (and the\n> fsck special case) that you know of?\n\nA lot (just found out as I was pushing subtree clone as far as I\ncould). For merging, you can hardly limit sha1 access. When writing\ntree, git's particularly paranoid and check for sha1 existence\n(has_sha1_file and assert_sha1_type). You can find that has_sha1_file\nis used in many places, not just commit/write-tree.\n\nNarrow/Sparse/Subtree/Lazy/Whatever-it-is clone will have hard time..\nOh.. the lazy one does not.\n-- \nDuy\n"},{"id":"146878","messageId":"AANLkTin4XbVzwj7M86uTvXxJ_qEfzoyNrJgf3U8u=WdH@mail.gmail.com","threadId":"24544","inReplyTo":"AANLkTi=8u5VROYQygAXoCS4c+eAoEoP8V4t5rJ=wXL8q@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2010-07-31T22:36:10Z","receivedAt":"2010-07-31T22:36:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\n2010/7/27 Elijah Newren <newren@gmail.com>:\n> 2010/7/27 Avery Pennarun <apenwarr@gmail.com>:\n>> Note that if you happen to want to implement it in a way that you'll\n>> also get all the commit objects from your submodules too (which I\n>> highly encourage :)) then downloading the trees is the easiest way.\n>> Otherwise you won't know which submodule commits you need.\n>\n> Makes sense.  Seems like a good reason to include all the trees.\n\nActually, having thought about it more, I don't see the reason for\ngetting all the commit objects from submodules (unless those\nsubmodules are at paths specified for download).  If a user has\nspecified that they just want the Documentation subdirectory, why\nwould it matter if the submodule under src/widgets was downloaded?\nThey don't want to do anything with any of its contents, so I don't\nsee why they'd needs its trees or commits.  Am I missing something?\n\n\nAlso, I'm rethinking the download-all-commits aspect too.  This is\npartially due to Nguyễn's stats (and special usecases like\ntranslators), partially because of security issues (it has already\nbeen stated that only including stuff meant to be public is an\nimportant security concern for clone[1], and commit logs for changes\ncompletely outside specified paths might be considered non-public\ndata[2]), and partially because it reinforces my whole rev-list\nlimiting args idea (it makes it really clear that 'git log' should\nautomatically behave like 'git log -- Documentation/' in a sparse\nclone of just Documentation/).\n\n[1] e.g. http://article.gmane.org/gmane.comp.version-control.git/115835\n\n[2] This isn't just theoretical either.  I have a couple big important\n(to $dayjob and thus me) sparse-clone usecases in this situation and\nhave for a few years, but gave up on it thinking it wouldn't be\npossible with sparse clones.  I instead wrote a fast filtering\nmechanism using fast-export/fast-import that creates a new repository\nand keeps track of the mapping between sha1sums in unfiltered and\nfiltered repos, allowing changes to be grafted between the two.  Kind\nof a pain, and suboptimal for a few reasons.  It'd be really nice if I\ncould replace this stuff with sparse clones, but can't do that if\ncommit logs corresponding to changes completely outside the sparse\npaths are included.\n"},{"id":"148010","messageId":"20100813173147.GB27540@nibiru.local","threadId":"24544","inReplyTo":"AANLkTikJhSVJw2hXkp0j6XA+k-J-AtSYzKWumjnqqsgz@mail.gmail.com","subject":"Re: Sparse clones (Was: Re: [PATCH 1/2] upload-pack: support subtree packing)","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-08-13T17:31:48Z","receivedAt":"2010-08-13T17:31:48Z","isPatch":true,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Elijah Newren <newren@gmail.com> wrote:\n\nHi folks,\n\n\nas I'm doing many backups via git (eg. hourly sql dumps), I'd \nlike to cut off the history (eg. at the n'th past commit)\nand reclaim the space - both on local and remote side (even\ndifferently).\n\nSo let me propose another approach: fake-root's\n\nFake-roots are special refs that declare certain commit objects\nas root-commits). Each time git walks down the history, it checks\nwhether the current commit is an fake-root and so treats it as\nhaving no ancestor. That should be generic enough let everything\nelse (commit, push, gc, etc) work as usual.\n\nThe only tricky point is when to update remote fake-roots: the\nremote should not cut off my local repo (unless explicitly asked).\nSo remote fake-roots should only be imported if the local/receiving\nside has not the dropped commits anymore.\n\nhmm, maybe it's even more wise to get one step back in history\nand introduce fake-empty's (which also have no parents) instead\nof fake-root's ? A fake-empty is imported as soon as the original\nobject is missing.\n\n\nOf course, it's important that this feature has to be explicitly\nenabled (maybe even on per-remote basis) to prevent security flaws.\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone:  +49 36207 519931  email: weigelt@metux.de\n mobile: +49 151 27565287  icq:   210169427         skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"148015","messageId":"20100813191954.GA2003@burratino","threadId":"24544","inReplyTo":"20100813173147.GB27540@nibiru.local","subject":"Truncating history (Re: Sparse clones)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-13T19:19:54Z","receivedAt":"2010-08-13T19:19:54Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nEnrico Weigelt wrote:\n\n> Fake-roots are special refs that declare certain commit objects\n> as root-commits). Each time git walks down the history, it checks\n> whether the current commit is an fake-root and so treats it as\n> having no ancestor. That should be generic enough let everything\n> else (commit, push, gc, etc) work as usual.\n> \n> The only tricky point is when to update remote fake-roots: the\n> remote should not cut off my local repo (unless explicitly asked).\n> So remote fake-roots should only be imported if the local/receiving\n> side has not the dropped commits anymore.\n\nYou may be interested in grafts and replacement refs; see\ngit-filter-branch(1) and git-replace(1) for some hints.\n\nGood luck,\nJonathan\n"}]}