{"thread":{"id":"44427","subject":"Regarding \"git log\" on \"git series\" metadata","startedAt":"2016-11-04T17:57:18Z","lastAt":"2016-11-13T17:58:24Z","messageCount":35,"participants":["Junio C Hamano","Jacob Keller","Christian Couder","Jeff King","Josh Triplett","Duy Nguyen","Stefano Zacchiroli"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"305401","messageId":"xmqqa8dfdt6y.fsf@gitster.mtv.corp.google.com","threadId":"44427","inReplyTo":null,"subject":"Regarding \"git log\" on \"git series\" metadata","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-04T17:57:09Z","receivedAt":"2016-11-04T17:57:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"After your talk at LPC2016, I was thinking about your proposal to\ngive an option to hide certain parents from \"git log\" traversal.\n\nWhile I do not think we would terribly mind a new feature in the\ncore to support third-party additions like \"git series\" better, I\nthink this particular one is a big mistake that we shouldn't take.\n\nFor those listening from sidelines, here is a version of my\nunderstanding of \"git series\":\n\n * \"git series\" wants to represent a patch series evolution. It is\n a history of history, and each element of this evolution is\n represented by:\n\n - a commit object, that is used to describe what this reroll of\n the topic is about, and its parent links point at previous\n rerolls (it could be a merge of two independent incarnations of\n a series).\n\n - the tree contained in the commit object records the base commit\n where the topic forks from the main history, and the tip commit\n where the topic ends. These are pointers into the main history\n DAG.\n\n - the tree may have other metadata, an example of which is the\n cover letter contents to be used when the topic becomes ready\n for re-submission. There may be more metadata you would want\n to add in the future versions of \"git series\".\n\n Needless to say, the commits that represent the history of a\n series record a tree that is completely differently shaped. The\n only relation between the series history and main history is that\n the former has pointers into the latter.\n\n * You chose to represent the base and tip commit object as gitlinks\n in the tree of a series commit, simply because it was a way that\n was already implemented to record a commit object name in a tree.\n\n * However, because gitlink is designed to be used for \"external\"\n things (the prominent example is submodule), recording these as\n gitlinks would guarantee that they will get GCed as a series\n progresses, the main history rewound and rewritten thereby making\n the base and tip recorded in the older part of the series history\n unreachable from the main history. Because you want to make sure\n that base and tip objects will stay in the repository even after\n the topic branch in the main history gets rewound, this is not\n what you want.\n\n * In order to workaround that reachability issue, the hack you\n invented is to add the tip commit as a \"parent\" of a commit that\n represents one step in the series. This may guarantee the\n reachability---as long as a commit in a series history is\n reachable from a ref, the tip and base commits will be reachable\n from there even if they are rebased away from the main history.\n But of course, there are downsides.\n\n * Due to this hack, feeding \"gitk\" (or \"git log\") a commit in the\n series history will give you nonsense results. You are not\n interested in traversing or viewing the commits in the main\n history.\n\n * Because of the above, you propose another hack to tell the\n revision traversal machinery to optionally omit a parent commit\n that appear as a gitlink in the tree.\n\nI think this is backwards. The root cause of the issue you have\nwith \"gitk\" is because you added something that is *NOT* a parent to\nyour commit. We shouldn't have to add a mechanism to filter\nsomething that shouldn't have been added there in the first place.\n\nI am wondering if an alternative approach would work better.\n\nImagine we invent a new tree entry type, \"gitref\", that is similar\nto \"gitlink\" in that it can record a commit object name in a tree,\nbut unlike \"gitlink\" it does imply reachability. And you do not add\nphony parents to your commit object. A tree that has \"gitref\"s in\nit is about annotating the commits in the same repository (e.g. the\ntree references two commits, \"base\" and \"tip\", to point into a slice\nof the main history). And it is perfectly sensible for such a\npointer to imply reachability---after all it serves different\npurposes from \"gitlink\".\n\nAnother alternative that I am negative about (but is probably a\nbetter hack than how you abused the \"parent\" link) might be to add a\nnew commit object header field that behaves similarly to \"parent\"\nonly in that it implies reachability. But recording the extra\nparent in commit object was not something you wanted to do in the\nfirst place (i.e. your series processing is done solely on the\ncontents of the tree, and you do not read this extra parent). If you\nneed to add an in-tree reference to another commit in your future\nversions of \"git series\", with either this variant or your original\nimplementation, you would end up needing adding more \"parent\" (or\npseudo parent) only to preserve reachability. At that point, I\nthink it makes more sense to have entries in the tree to directly\nensure reachability, if you want these entries to always point at an\nin-tree object.\n\nI am afraid that I probably am two steps ahead of myself, because I\nam reasonably sure that it is quite possible that I have overlooked\nsomething trivially obvious that makes the \"gitref\" approach\nunworkable.\n\n"},{"id":"305402","messageId":"CA+P7+xq0LLFBJRNNvCMQ4QR7XBg9H7NSsifiqOYqr+PUBqYRGQ@mail.gmail.com","threadId":"44427","inReplyTo":"xmqqa8dfdt6y.fsf@gitster.mtv.corp.google.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-11-04T19:19:55Z","receivedAt":"2016-11-04T19:20:22Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Nov 4, 2016 at 10:57 AM, Junio C Hamano wrote:\n> I think this is backwards. The root cause of the issue you have\n> with \"gitk\" is because you added something that is *NOT* a parent to\n> your commit. We shouldn't have to add a mechanism to filter\n> something that shouldn't have been added there in the first place.\n>\n> I am wondering if an alternative approach would work better.\n>\n> Imagine we invent a new tree entry type, \"gitref\", that is similar\n> to \"gitlink\" in that it can record a commit object name in a tree,\n> but unlike \"gitlink\" it does imply reachability. And you do not add\n> phony parents to your commit object. A tree that has \"gitref\"s in\n> it is about annotating the commits in the same repository (e.g. the\n> tree references two commits, \"base\" and \"tip\", to point into a slice\n> of the main history). And it is perfectly sensible for such a\n> pointer to imply reachability---after all it serves different\n> purposes from \"gitlink\".\n>\n\nI agree with your assessment here. The main difficulty in implementing\ngitrefs is to ensure that they actually do get picked up by\nreachability checks to prevent dropping commits. I'm not sure how easy\nthis is, but I would much rather we go this route rather than\ncontinuing along with the hack. This seems like the ideal solution,\nsince it solves the entire problem and doesn't need more hacks bolted\non.\n\nIt would of course mean some work for people who previously used git\nseries as you would want to re-write the commits to drop the parent\nlinks and become gitrefs instead of gitlinks. However, this can\n(probably?) be solved by some sort of use of the filter-branch code.\n\nI don't think you've hit upon any trivially obvious unworkable things.\nIt is probably somewhat complex to make the reachability checks detect\nin-tree gitrefs but I don't think it would be impossible.\n\nThanks,\nJake\n"},{"id":"305409","messageId":"CAP8UFD2+A0MUKazAfSwCvv61TJRPuoOzH5EkqcrBOUi4TcuoDw@mail.gmail.com","threadId":"44427","inReplyTo":"xmqqa8dfdt6y.fsf@gitster.mtv.corp.google.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-11-04T20:47:41Z","receivedAt":"2016-11-04T20:47:48Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Nov 4, 2016 at 6:57 PM, Junio C Hamano wrote:\n>\n> Imagine we invent a new tree entry type, \"gitref\", that is similar\n> to \"gitlink\" in that it can record a commit object name in a tree,\n> but unlike \"gitlink\" it does imply reachability. And you do not add\n> phony parents to your commit object. A tree that has \"gitref\"s in\n> it is about annotating the commits in the same repository (e.g. the\n> tree references two commits, \"base\" and \"tip\", to point into a slice\n> of the main history). And it is perfectly sensible for such a\n> pointer to imply reachability---after all it serves different\n> purposes from \"gitlink\".\n\nThe more I think about this (and also about how to limit ref\nadvertisements as recently discussed in\nhttps://public-inbox.org/git/20161024132932.i42rqn2vlpocqmkq@sigill.intra.peff.net/),\nthe more I think about Shawn's RefTree:\n\nhttps://public-inbox.org/git/CAJo=hJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg@mail.gmail.com/\n\nCouldn't a RefTree be used to store refs that point to the base\ncommit, the tip commit and the blob that contains the cover letter,\nand maybe also a ref pointing to the RefTree of the previous version\nof the series?\n"},{"id":"305410","messageId":"20161104194907.3yxu2rkayfyic4dr@sigill.intra.peff.net","threadId":"44427","inReplyTo":"CA+P7+xq0LLFBJRNNvCMQ4QR7XBg9H7NSsifiqOYqr+PUBqYRGQ@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-11-04T19:49:07Z","receivedAt":"2016-11-04T20:49:17Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 04, 2016 at 12:19:55PM -0700, Jacob Keller wrote:\n\n> I agree with your assessment here. The main difficulty in implementing\n> gitrefs is to ensure that they actually do get picked up by\n> reachability checks to prevent dropping commits. I'm not sure how easy\n> this is, but I would much rather we go this route rather than\n> continuing along with the hack. This seems like the ideal solution,\n> since it solves the entire problem and doesn't need more hacks bolted\n> on.\n\nI think the main complication is that the reachability rules are used\nduring object transfer. So you'd probably want to introduce some\nprotocol extension to say \"I understand gitrefs\", so that when one side\nsays \"I have sha1 X and its reachable objects\", we know whether they are\nincluding gitrefs there. And likewise receivers with\ntransfer.fsckObjects may complain about the new gitref tree mode\n(fortunately a new object type shouldn't be needed).\n\nYou might also want fallback rules for storing gitrefs on \"old\" servers\n(e.g., backfilling gitrefs you need if the server didn't them in the\ninitial fetch). But I guess storing any gitrefs on such a server is\ninherently dangerous, because the server might prune them at any time.\n\nSo perhaps a related question is: how can gitrefs be designed such that\nexisting servers reject them (rather than accepting the push and then\nlater throwing away half the data). It would be easy to notice in the\nclient during a push that we are sending gitrefs to a server which does\nnot claim that capability. But it seems more robust if it is the server\nwho decides \"I will not accept these bogus objects\".\n\nI haven't thought all that hard about this. That's just my initial\nthoughts on what sound hard. Tweaking the reachability code doesn't seem\nall that bad; we already know all of the spots that care about\nS_ISGITLINK(). It may even be that some of those spots work out of the\nbox (because gitlinks are usually about telling the graph-walking code\nthat we _don't_ care about reachability; we do by default for trees and\nblobs).\n\nI'd be surprised if all such sites work out of the box, though. Even if\nthey see \"ah, sha1 X is referenced by tree Y and isn't a gitlink, and\ntherefore should be reachable\", they need to also note that \"X\" is a\ncommit and recursively walk its objects.\n\n-Peff\n"},{"id":"305413","messageId":"20161104210613.5ax523wrc5robs7l@x","threadId":"44427","inReplyTo":"xmqqa8dfdt6y.fsf@gitster.mtv.corp.google.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-04T21:06:13Z","receivedAt":"2016-11-04T21:06:26Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Fri, Nov 04, 2016 at 10:57:09AM -0700, Junio C Hamano wrote:\n> After your talk at LPC2016, I was thinking about your proposal to\n> give an option to hide certain parents from \"git log\" traversal.\n> \n> While I do not think we would terribly mind a new feature in the\n> core to support third-party additions like \"git series\" better, I\n> think this particular one is a big mistake that we shouldn't take.\n[...]\n> I think this is backwards. The root cause of the issue you have\n> with \"gitk\" is because you added something that is *NOT* a parent to\n> your commit. We shouldn't have to add a mechanism to filter\n> something that shouldn't have been added there in the first place.\n> \n> I am wondering if an alternative approach would work better.\n> \n> Imagine we invent a new tree entry type, \"gitref\", that is similar\n> to \"gitlink\" in that it can record a commit object name in a tree,\n> but unlike \"gitlink\" it does imply reachability. And you do not add\n> phony parents to your commit object. A tree that has \"gitref\"s in\n> it is about annotating the commits in the same repository (e.g. the\n> tree references two commits, \"base\" and \"tip\", to point into a slice\n> of the main history). And it is perfectly sensible for such a\n> pointer to imply reachability---after all it serves different\n> purposes from \"gitlink\".\n\nI absolutely agree with this, and I'd love to have gitref or similar in\ncore git. Given the availability of that mechanism, I'd love to use it\nin git-series. (And in git submodule, as well, for other projects.)\n\nThe one critical issue there, though: that would break backward\ncompatibility with old versions of git. No old version of git could\npush, pull, gc, repack, or otherwise touch a repository that used this\nfeature.\n\nThe advantages of the approach (viewing and manipulating the series with\npure git) seem sufficiently high to make that worth considering, but it\nis a significant downside.\n\n> Another alternative that I am negative about (but is probably a\n> better hack than how you abused the \"parent\" link) might be to add a\n> new commit object header field that behaves similarly to \"parent\"\n> only in that it implies reachability. But recording the extra\n> parent in commit object was not something you wanted to do in the\n> first place (i.e. your series processing is done solely on the\n> contents of the tree, and you do not read this extra parent). If you\n> need to add an in-tree reference to another commit in your future\n> versions of \"git series\", with either this variant or your original\n> implementation, you would end up needing adding more \"parent\" (or\n> pseudo parent) only to preserve reachability. At that point, I\n> think it makes more sense to have entries in the tree to directly\n> ensure reachability, if you want these entries to always point at an\n> in-tree object.\n\nThis would similarly break compatibility with old git, as old git\nwouldn't follow those reachability-only links from commits, so it could\nthrow away the data.\n\nOne approach compatible with old git would be to continue adding the\nrelevant commits as artificial parents, but have a separate commit\nmetadata field that says which parents to ignore; old git would then do\nthe right thing, as long as it doesn't rewrite the commit entirely.\n\nThat does have the same disadvantages of having to duplicate the\ninformation in both the tree and the parent list, though; it's the same\nclass of hack, just with improved usability. I'd much rather use\ngitrefs.\n\n> I am afraid that I probably am two steps ahead of myself, because I\n> am reasonably sure that it is quite possible that I have overlooked\n> something trivially obvious that makes the \"gitref\" approach\n> unworkable.\n\ngitref seems like a good idea to me, as long as we can sort out the\ncompatibility story.\n"},{"id":"305414","messageId":"20161104211959.3532uiud27nhumt7@x","threadId":"44427","inReplyTo":"CAP8UFD2+A0MUKazAfSwCvv61TJRPuoOzH5EkqcrBOUi4TcuoDw@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-04T21:19:59Z","receivedAt":"2016-11-04T21:31:10Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Fri, Nov 04, 2016 at 09:47:41PM +0100, Christian Couder wrote:\n> On Fri, Nov 4, 2016 at 6:57 PM, Junio C Hamano wrote:\n> >\n> > Imagine we invent a new tree entry type, \"gitref\", that is similar\n> > to \"gitlink\" in that it can record a commit object name in a tree,\n> > but unlike \"gitlink\" it does imply reachability. And you do not add\n> > phony parents to your commit object. A tree that has \"gitref\"s in\n> > it is about annotating the commits in the same repository (e.g. the\n> > tree references two commits, \"base\" and \"tip\", to point into a slice\n> > of the main history). And it is perfectly sensible for such a\n> > pointer to imply reachability---after all it serves different\n> > purposes from \"gitlink\".\n> \n> The more I think about this (and also about how to limit ref\n> advertisements as recently discussed in\n> https://public-inbox.org/git/20161024132932.i42rqn2vlpocqmkq@sigill.intra.peff.net/),\n> the more I think about Shawn's RefTree:\n> \n> https://public-inbox.org/git/CAJo=hJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg@mail.gmail.com/\n> \n> Couldn't a RefTree be used to store refs that point to the base\n> commit, the tip commit and the blob that contains the cover letter,\n> and maybe also a ref pointing to the RefTree of the previous version\n> of the series?\n\nThat's really interesting! The Software Heritage project is working on\nsomething similar, because they want to store all the refs as part of\ntheir data model as well. I'll point them to the reftree work.\n\nIf upstream git supported RefTree, I could potentially use that for\ngit-series. However, I do want a commit message and history for the\nseries itself, and using refs in the reftree to refer to the parents\nseems like abusing reftree to recreate commits, in a reversal of the\nhack of using commit parents as a reftree. :)\n\nWhat if, rather than storing a hash reference to a reftree as a single\nreference and replacing it with no history, a reftree could be\nreferenced from a commit and have history? (That would also allow\ntagging a version of the reftree.)\n"},{"id":"305417","messageId":"20161104215538.xmpth6qfuou6nde6@x","threadId":"44427","inReplyTo":"20161104194907.3yxu2rkayfyic4dr@sigill.intra.peff.net","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-04T21:55:39Z","receivedAt":"2016-11-04T21:55:49Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Fri, Nov 04, 2016 at 03:49:07PM -0400, Jeff King wrote:\n> On Fri, Nov 04, 2016 at 12:19:55PM -0700, Jacob Keller wrote:\n> \n> > I agree with your assessment here. The main difficulty in implementing\n> > gitrefs is to ensure that they actually do get picked up by\n> > reachability checks to prevent dropping commits. I'm not sure how easy\n> > this is, but I would much rather we go this route rather than\n> > continuing along with the hack. This seems like the ideal solution,\n> > since it solves the entire problem and doesn't need more hacks bolted\n> > on.\n> \n> I think the main complication is that the reachability rules are used\n> during object transfer. So you'd probably want to introduce some\n> protocol extension to say \"I understand gitrefs\", so that when one side\n> says \"I have sha1 X and its reachable objects\", we know whether they are\n> including gitrefs there. And likewise receivers with\n> transfer.fsckObjects may complain about the new gitref tree mode\n> (fortunately a new object type shouldn't be needed).\n> \n> You might also want fallback rules for storing gitrefs on \"old\" servers\n> (e.g., backfilling gitrefs you need if the server didn't them in the\n> initial fetch). But I guess storing any gitrefs on such a server is\n> inherently dangerous, because the server might prune them at any time.\n> \n> So perhaps a related question is: how can gitrefs be designed such that\n> existing servers reject them (rather than accepting the push and then\n> later throwing away half the data). It would be easy to notice in the\n> client during a push that we are sending gitrefs to a server which does\n> not claim that capability. But it seems more robust if it is the server\n> who decides \"I will not accept these bogus objects\".\n\nThis seems like the critical problem, here. The parent hack I used in\ngit-series might be a hack, but it transparently works with old servers\nand clients. So, for instance, I can push a git-series ref to github,\nwith no changes required on github's part. If git added gitrefs, and I\nstarted using them in git-series, then that'd eliminate parent hack and\nallow many standard git tools to work naturally on git-series commits\nand history, but it'd also mean that people couldn't push git-series\ncommits to any server until that server updates git.\n\nThat said, I'd *love* to have gitrefs available, for a wide variety of\napplications, and I can see an argument for introducing them and waiting\na few years for them to become universally available, similar to the\nprocess gitlinks went through.\n\nBut I'd also love to have a backward-compatible solution.\n\n- Josh Triplett\n"},{"id":"305421","messageId":"CAP8UFD13sDOFuyZMWuoJeLFt_LAsfAHFBHpRwcdAGmA22xNEKQ@mail.gmail.com","threadId":"44427","inReplyTo":"20161104211959.3532uiud27nhumt7@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-11-04T23:04:08Z","receivedAt":"2016-11-04T23:04:16Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Nov 4, 2016 at 10:19 PM, Josh Triplett wrote:\n> On Fri, Nov 04, 2016 at 09:47:41PM +0100, Christian Couder wrote:\n>> On Fri, Nov 4, 2016 at 6:57 PM, Junio C Hamano wrote:\n>> >\n>> > Imagine we invent a new tree entry type, \"gitref\", that is similar\n>> > to \"gitlink\" in that it can record a commit object name in a tree,\n>> > but unlike \"gitlink\" it does imply reachability. And you do not add\n>> > phony parents to your commit object. A tree that has \"gitref\"s in\n>> > it is about annotating the commits in the same repository (e.g. the\n>> > tree references two commits, \"base\" and \"tip\", to point into a slice\n>> > of the main history). And it is perfectly sensible for such a\n>> > pointer to imply reachability---after all it serves different\n>> > purposes from \"gitlink\".\n>>\n>> The more I think about this (and also about how to limit ref\n>> advertisements as recently discussed in\n>> https://public-inbox.org/git/20161024132932.i42rqn2vlpocqmkq@sigill.intra.peff.net/),\n>> the more I think about Shawn's RefTree:\n>>\n>> https://public-inbox.org/git/CAJo=hJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg@mail.gmail.com/\n>>\n>> Couldn't a RefTree be used to store refs that point to the base\n>> commit, the tip commit and the blob that contains the cover letter,\n>> and maybe also a ref pointing to the RefTree of the previous version\n>> of the series?\n>\n> That's really interesting! The Software Heritage project is working on\n> something similar, because they want to store all the refs as part of\n> their data model as well. I'll point them to the reftree work.\n\nYeah, I know them :-) and I think I have already told Stefano\nZacchiroli about this, but I am not sure anymore.\nAnyway I am CC'ing him.\n\n> If upstream git supported RefTree, I could potentially use that for\n> git-series. However, I do want a commit message and history for the\n> series itself, and using refs in the reftree to refer to the parents\n> seems like abusing reftree to recreate commits, in a reversal of the\n> hack of using commit parents as a reftree. :)\n\nYeah, maybe :-) But the properties of the existing Git objects we\nalready use wouldn't change at all.\n\n> What if, rather than storing a hash reference to a reftree as a single\n> reference and replacing it with no history,\n\nIn what I suggest the history is kept because the new reftree has a\nref that points to the old one it is replacing.\n\nYeah, this reftree history maybe seen as \"redundant\" with the commit\nhistory, but in my opinion this can be seen as a \"feature\" that will\nprevent us from \"mucking\" too much with the commit object.\n\n> a reftree could be\n> referenced from a commit and have history? (That would also allow\n> tagging a version of the reftree.)\n\nI think that tags are already allowed to point to any kind of Git\nobject, so tagging a reftree should be allowed anyway if we add a\nreftree object.\n"},{"id":"305424","messageId":"CA+P7+xpwUZscpgzLJYf5vkKKsT6SFkC3TrsyBJXJjGo9cF94nQ@mail.gmail.com","threadId":"44427","inReplyTo":"20161104194907.3yxu2rkayfyic4dr@sigill.intra.peff.net","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-11-04T23:34:34Z","receivedAt":"2016-11-04T23:35:01Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Nov 4, 2016 at 12:49 PM, Jeff King wrote:\n> I think the main complication is that the reachability rules are used\n> during object transfer. So you'd probably want to introduce some\n> protocol extension to say \"I understand gitrefs\", so that when one side\n> says \"I have sha1 X and its reachable objects\", we know whether they are\n> including gitrefs there. And likewise receivers with\n> transfer.fsckObjects may complain about the new gitref tree mode\n> (fortunately a new object type shouldn't be needed).\n>\n> You might also want fallback rules for storing gitrefs on \"old\" servers\n> (e.g., backfilling gitrefs you need if the server didn't them in the\n> initial fetch). But I guess storing any gitrefs on such a server is\n> inherently dangerous, because the server might prune them at any time.\n>\n\nIs it possible currently for a protocol extension to result in \"oh the\nserver doesn't support this so I'm going to stop pushing\"? This would\nbe a rather hard transition, but it would at least ensure that pushing\nto a server which doesn't support gitrefs would fail rather than\nsilently accept objects and then discard them later? I think this is\nthe only real transition unless we can make a change that old servers\nobject to already.\n\n> So perhaps a related question is: how can gitrefs be designed such that\n> existing servers reject them (rather than accepting the push and then\n> later throwing away half the data). It would be easy to notice in the\n> client during a push that we are sending gitrefs to a server which does\n> not claim that capability. But it seems more robust if it is the server\n> who decides \"I will not accept these bogus objects\".\n>\n> I haven't thought all that hard about this. That's just my initial\n> thoughts on what sound hard. Tweaking the reachability code doesn't seem\n> all that bad; we already know all of the spots that care about\n> S_ISGITLINK(). It may even be that some of those spots work out of the\n> box (because gitlinks are usually about telling the graph-walking code\n> that we _don't_ care about reachability; we do by default for trees and\n> blobs).\n\nRight. I'm assuming tree objects don't get checked for invalid mode\nalready? If they do, we could just change the mode to something\nunsupported currently. But... that seems like it might not be the case\nbecause it requires checking every tree object coming in?\n\nI'm not familiar with what sort of checking already exists... Thoughts?\n\n>\n> I'd be surprised if all such sites work out of the box, though. Even if\n> they see \"ah, sha1 X is referenced by tree Y and isn't a gitlink, and\n> therefore should be reachable\", they need to also note that \"X\" is a\n> commit and recursively walk its objects.\n>\n\nThey won't all work out of the box, but it shouldn't be much work to\ndo this part.\n\n> -Peff\n"},{"id":"305425","messageId":"CA+P7+xoKORo6hC2n-E-gHG2OYg3h-m3ZnUQbdopS7S3-5AWoPQ@mail.gmail.com","threadId":"44427","inReplyTo":"20161104215538.xmpth6qfuou6nde6@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-11-04T23:37:34Z","receivedAt":"2016-11-04T23:38:00Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Nov 4, 2016 at 2:55 PM, Josh Triplett wrote:\n> That said, I'd *love* to have gitrefs available, for a wide variety of\n> applications, and I can see an argument for introducing them and waiting\n> a few years for them to become universally available, similar to the\n> process gitlinks went through.\n>\n> But I'd also love to have a backward-compatible solution.\n>\n> - Josh Triplett\n\nI think that you won't really find a backwards compatible solution\nother than something like automatically generating refs for each point\nof history. I know that gerrit does something like this by storing\neach version in \"refs/changes/id/version\" or something along those\nlines. I think this might actually be cleaner than your parent links\nhack, and could be used as a fallback for when gitrefs don't work,\nthough you'd have to code exactly how to tell what to push to a\nrepository when pushing a series?\n\nThanks,\nJake\n"},{"id":"305426","messageId":"20161104234647.xjaf7scdjpppfdop@x","threadId":"44427","inReplyTo":"CA+P7+xoKORo6hC2n-E-gHG2OYg3h-m3ZnUQbdopS7S3-5AWoPQ@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-04T23:46:47Z","receivedAt":"2016-11-04T23:46:58Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Fri, Nov 04, 2016 at 04:37:34PM -0700, Jacob Keller wrote:\n> On Fri, Nov 4, 2016 at 2:55 PM, Josh Triplett wrote:\n> > That said, I'd *love* to have gitrefs available, for a wide variety of\n> > applications, and I can see an argument for introducing them and waiting\n> > a few years for them to become universally available, similar to the\n> > process gitlinks went through.\n> >\n> > But I'd also love to have a backward-compatible solution.\n> >\n> > - Josh Triplett\n> \n> I think that you won't really find a backwards compatible solution\n> other than something like automatically generating refs for each point\n> of history. I know that gerrit does something like this by storing\n> each version in \"refs/changes/id/version\" or something along those\n> lines. I think this might actually be cleaner than your parent links\n> hack, and could be used as a fallback for when gitrefs don't work,\n> though you'd have to code exactly how to tell what to push to a\n> repository when pushing a series?\n\nI'm not sure what the advantage of that would be, and it would mean that\nif you ever have one branch without pushing the other(s), you'd get\nsevere time-delated breakage due to pruning. (And if you pushed the\nseries without the other ref(s), its history would look right but then\nyou couldn't access the underlying versions of the patch series.)\n\nOne of my design goals was to *not* need a special \"git series push\" or\n\"git series pull\"; you should just be able to use git push and git pull,\nand you can set up normal refspecs.\n\nThat said, I could fairly easily generate the existing format with\nartificial parent refs for backward compatibility, and provide a way to\nuse the new gitref-based storage format if you know that all your\nservers and clients can handle it. I'm also open to other suggestions\nfor how to make such a transition while still working with every git\nserver and git client that exists today.\n"},{"id":"305430","messageId":"20161105014817.vm4ush2wfbblzsc7@sigill.intra.peff.net","threadId":"44427","inReplyTo":"CA+P7+xpwUZscpgzLJYf5vkKKsT6SFkC3TrsyBJXJjGo9cF94nQ@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-11-05T01:48:17Z","receivedAt":"2016-11-05T01:48:26Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 04, 2016 at 04:34:34PM -0700, Jacob Keller wrote:\n\n> > You might also want fallback rules for storing gitrefs on \"old\" servers\n> > (e.g., backfilling gitrefs you need if the server didn't them in the\n> > initial fetch). But I guess storing any gitrefs on such a server is\n> > inherently dangerous, because the server might prune them at any time.\n> \n> Is it possible currently for a protocol extension to result in \"oh the\n> server doesn't support this so I'm going to stop pushing\"?\n\nYes, it would be easy for the client to abort if the server fails to\nadvertise a particular extension.\n\nWhat I would worry about more is that \"somehow\" an older client gets\nhold of history with a gitref, and then pushes it. It would be nice if\neven an old server said \"nope, I don't understand this and I won't take\nit\" rather than propagating the data to a server that will throw it\naway.\n\n> Right. I'm assuming tree objects don't get checked for invalid mode\n> already? If they do, we could just change the mode to something\n> unsupported currently. But... that seems like it might not be the case\n> because it requires checking every tree object coming in?\n> \n> I'm not familiar with what sort of checking already exists... Thoughts?\n\nIf the server sets receive.fsckObjects, then fsck_tree() runs and will\nreject any non-standard mode. That option is not the default, though\nsome big hosters set it (GitHub does, but I am actually not that worried\nabout GitHub; if gitrefs support materialized I would probably ship it\nthere fairly promptly).\n\n-Peff\n"},{"id":"305432","messageId":"A269CCA2-5F8C-4AF0-820E-2EA26FEF03D5@joshtriplett.org","threadId":"44427","inReplyTo":"20161105014817.vm4ush2wfbblzsc7@sigill.intra.peff.net","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-05T03:55:23Z","receivedAt":"2016-11-05T03:55:39Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On November 4, 2016 7:48:17 PM MDT, Jeff King wrote:\n>On Fri, Nov 04, 2016 at 04:34:34PM -0700, Jacob Keller wrote:\n>\n>> > You might also want fallback rules for storing gitrefs on \"old\"\n>servers\n>> > (e.g., backfilling gitrefs you need if the server didn't them in\n>the\n>> > initial fetch). But I guess storing any gitrefs on such a server is\n>> > inherently dangerous, because the server might prune them at any\n>time.\n>> \n>> Is it possible currently for a protocol extension to result in \"oh\n>the\n>> server doesn't support this so I'm going to stop pushing\"?\n>\n>Yes, it would be easy for the client to abort if the server fails to\n>advertise a particular extension.\n\nAnd the reverse (old client, new server) should work as well? \n\n- Josh Triplett\n\n"},{"id":"305433","messageId":"xmqq60o2edy5.fsf@gitster.mtv.corp.google.com","threadId":"44427","inReplyTo":"20161104194907.3yxu2rkayfyic4dr@sigill.intra.peff.net","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-05T04:41:06Z","receivedAt":"2016-11-05T04:41:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Fri, Nov 04, 2016 at 12:19:55PM -0700, Jacob Keller wrote:\n>\n>> I agree with your assessment here. The main difficulty in implementing\n>> gitrefs is to ensure that they actually do get picked up by\n>> reachability checks to prevent dropping commits. I'm not sure how easy\n>> this is, but I would much rather we go this route rather than\n>> continuing along with the hack. This seems like the ideal solution,\n>> since it solves the entire problem and doesn't need more hacks bolted\n>> on.\n>\n> I think the main complication is that the reachability rules are used\n> during object transfer. So you'd probably want to introduce some\n> protocol extension to say \"I understand gitrefs\", so that when one side\n> says \"I have sha1 X and its reachable objects\", we know whether they are\n> including gitrefs there. And likewise receivers with\n> transfer.fsckObjects may complain about the new gitref tree mode\n> (fortunately a new object type shouldn't be needed).\n\nQuite honestly I do not think backward compatibility here matters.\nWhen gitlinks were introduced, a repository that was created with\ngitlink capable version of Git would have failed \"git fsck\" that is\nnot gitlink aware, and I think this new \"link with reachability\" is\nthe same deal. No existing implemention understands a tree entry\nwhose mode bits are 140000 or whatever new bit pattern we would\nassign to this thing. You have to wait until both ends understand\nthe new thing, and that is perfectly OK.\n\nBesides, I think the point of having this discussion is that Josh\ndid a good prototyping work of \"git series\" to discover what he can\ndo in that area of \"keeping track of history of history\" and what\noperations are useful, without wasting time on mucking with the\nobject model and traversal machinery that are available to him. \n\nNow his prototyping is at the point where he knows at least one\nenhancement to the core that would help him to redo the prototype in\nthe right way. And I do not mind helping him from the core side.\n"},{"id":"305434","messageId":"20161105044127.tm7tmpcgnw7eeian@sigill.intra.peff.net","threadId":"44427","inReplyTo":"A269CCA2-5F8C-4AF0-820E-2EA26FEF03D5@joshtriplett.org","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-11-05T04:41:27Z","receivedAt":"2016-11-05T04:41:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 04, 2016 at 09:55:23PM -0600, Josh Triplett wrote:\n\n> >> Is it possible currently for a protocol extension to result in \"oh\n> >the\n> >> server doesn't support this so I'm going to stop pushing\"?\n> >\n> >Yes, it would be easy for the client to abort if the server fails to\n> >advertise a particular extension.\n> \n> And the reverse (old client, new server) should work as well?\n\nYes. The server says \"I know about gitrefs\" and if the client does not\nrespond with \"I also know about gitrefs\", then the server can act\nappropriately (e.g., for a fetch, bail if the fetched content includes\ngitrefs).\n\n-Peff\n"},{"id":"305435","messageId":"xmqq1syqedv4.fsf@gitster.mtv.corp.google.com","threadId":"44427","inReplyTo":"CAP8UFD2+A0MUKazAfSwCvv61TJRPuoOzH5EkqcrBOUi4TcuoDw@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-05T04:42:55Z","receivedAt":"2016-11-05T04:43:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Couldn't a RefTree be used to store refs that point to the base\n> commit,\n\nI think it is the other way around. With the new \"gitref\" thing\nthat is a pointer to an in-repository commit, RefTree can be\nnaturally implemented.\n"},{"id":"305436","messageId":"20161105044440.u5imqcrsmpdbtonp@sigill.intra.peff.net","threadId":"44427","inReplyTo":"xmqq60o2edy5.fsf@gitster.mtv.corp.google.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-11-05T04:44:40Z","receivedAt":"2016-11-05T04:44:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 04, 2016 at 09:41:06PM -0700, Junio C Hamano wrote:\n\n> > I think the main complication is that the reachability rules are used\n> > during object transfer. So you'd probably want to introduce some\n> > protocol extension to say \"I understand gitrefs\", so that when one side\n> > says \"I have sha1 X and its reachable objects\", we know whether they are\n> > including gitrefs there. And likewise receivers with\n> > transfer.fsckObjects may complain about the new gitref tree mode\n> > (fortunately a new object type shouldn't be needed).\n> \n> Quite honestly I do not think backward compatibility here matters.\n> When gitlinks were introduced, a repository that was created with\n> gitlink capable version of Git would have failed \"git fsck\" that is\n> not gitlink aware, and I think this new \"link with reachability\" is\n> the same deal. No existing implemention understands a tree entry\n> whose mode bits are 140000 or whatever new bit pattern we would\n> assign to this thing. You have to wait until both ends understand\n> the new thing, and that is perfectly OK.\n\nI'm OK with saying \"if you use the gitref feature, you cannot push or\npull those objects with remotes that do not understand it\". But unlike\ngitlink, if we fail to notice the situation, we run into a case where we\nmight silently lose objects, which is bad. So I think we need to be a\nbit more careful.\n\nI don't think the problems are insurmountable. I just think that's where\nthe real complexity is, not in the changes to teach a single git about\ngitrefs.\n\nI'm happy to stand back and let you or Josh figure out all the corner\ncases. :)\n\n-Peff\n"},{"id":"305437","messageId":"xmqqwpgicyhq.fsf@gitster.mtv.corp.google.com","threadId":"44427","inReplyTo":"xmqq60o2edy5.fsf@gitster.mtv.corp.google.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-05T05:00:17Z","receivedAt":"2016-11-05T05:00:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> I think the main complication is that the reachability rules are used\n>> during object transfer.\n\nOne should not type after spending 20+ waking hours on plane and\nairport. I missed it when I wrote my first response, but yes, the\nreachability that originates from inside a tree object indeed is a\nproblem, as we do not dig into trees while doing the want/have/ack\nexhange.\n\n"},{"id":"305443","messageId":"CAP8UFD0bqNQZ3nuGUDX0qrSo44hf1NL9LeZB_FQcXg3j0mD38A@mail.gmail.com","threadId":"44427","inReplyTo":"xmqq1syqedv4.fsf@gitster.mtv.corp.google.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-11-05T12:17:49Z","receivedAt":"2016-11-05T12:17:57Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Nov 5, 2016 at 5:42 AM, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n>> Couldn't a RefTree be used to store refs that point to the base\n>> commit,\n>\n> I think it is the other way around. With the new \"gitref\" thing\n> that is a pointer to an in-repository commit, RefTree can be\n> naturally implemented.\n\nYeah, I should have read Shawn's RefTree email thread again before\nposting and especially before replying to Josh.\n"},{"id":"305444","messageId":"CAP8UFD1EZ8HBzLAeyFBFgU7n2uJpswqgEgA4XM1YJuRAG_ZAAQ@mail.gmail.com","threadId":"44427","inReplyTo":"CAP8UFD0bqNQZ3nuGUDX0qrSo44hf1NL9LeZB_FQcXg3j0mD38A@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-11-05T12:45:27Z","receivedAt":"2016-11-05T12:45:34Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Nov 5, 2016 at 1:17 PM, Christian Couder\n wrote:\n> On Sat, Nov 5, 2016 at 5:42 AM, Junio C Hamano wrote:\n>> Christian Couder writes:\n>>\n>>> Couldn't a RefTree be used to store refs that point to the base\n>>> commit,\n>>\n>> I think it is the other way around. With the new \"gitref\" thing\n>> that is a pointer to an in-repository commit, RefTree can be\n>> naturally implemented.\n>\n> Yeah, I should have read Shawn's RefTree email thread again before\n> posting and especially before replying to Josh.\n\nBy the way, reading the following email by Peff where gitlink\nreachability was already discussed:\n\nhttps://public-inbox.org/git/20151217221045.GA8150@sigill.intra.peff.net/\n\nand where Peff wrote:\n\n> Of course, the lack of reachability has advantages, too. You can\n> drop commits pointed to by old reflogs without rewriting the ref\n> history. Unfortunately you cannot expunge the reflogs at all. That's\n> good if you like audit trails. Bad if you are worried that your reflogs\n> will grow large. :)\n\nI think that we may not need \"gitref\" at all. We perhaps could just\nhave more ways to configure and tweak how a repo manages commit\nreachability related to gitlinks.\n\nWith shallow clones we already need ways to configure and tweak commit\nreachability anyway.\n\nAnd with what Peff says above it looks like we will need ways\nconfigure and tweak commit reachability with gitlink/gitref anyway. So\nthe point of gitref compared to gitlink would be that they just have a\ndifferent reachability by default. But couldn't that be replaced by a\ndefault rule saying that when a gitlink is reached \"this way or that\nway\" then the commit reachability should be enforced, and otherwise it\nshould not be?\n"},{"id":"305445","messageId":"20161105151836.wztypzrdywyltvrc@x","threadId":"44427","inReplyTo":"CAP8UFD1EZ8HBzLAeyFBFgU7n2uJpswqgEgA4XM1YJuRAG_ZAAQ@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-05T15:18:36Z","receivedAt":"2016-11-05T15:18:51Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Sat, Nov 05, 2016 at 01:45:27PM +0100, Christian Couder wrote:\n> And with what Peff says above it looks like we will need ways\n> configure and tweak commit reachability with gitlink/gitref anyway. So\n> the point of gitref compared to gitlink would be that they just have a\n> different reachability by default. But couldn't that be replaced by a\n> default rule saying that when a gitlink is reached \"this way or that\n> way\" then the commit reachability should be enforced, and otherwise it\n> should not be?\n\nAny version of git unaware of that rule, though, would consider objects\nonly reachable by gitlink as unreachable and delete them, causing data\nloss. Likewise for a server not aware of that rule. And a server\nunaware of that rule would not supply those objects to a client pulling\nsuch a branch.\n\nSo I don't think \"gitlink defined as reachable\" quite works, unless we\nmake some other format-incompatible change that forces clients and\nservers touching that gitlink to know about that reachability rule. (In\nthe absence of a hack such as making the same commit a parent.)\n"},{"id":"305446","messageId":"CAP8UFD3XFHr7POKmZr_6guapC6sme3GvWBV5vPw4XO7FE5HOPw@mail.gmail.com","threadId":"44427","inReplyTo":"20161105151836.wztypzrdywyltvrc@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-11-05T20:21:58Z","receivedAt":"2016-11-05T20:22:06Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Nov 5, 2016 at 4:18 PM, Josh Triplett wrote:\n> On Sat, Nov 05, 2016 at 01:45:27PM +0100, Christian Couder wrote:\n>> And with what Peff says above it looks like we will need ways\n>> configure and tweak commit reachability with gitlink/gitref anyway. So\n>> the point of gitref compared to gitlink would be that they just have a\n>> different reachability by default. But couldn't that be replaced by a\n>> default rule saying that when a gitlink is reached \"this way or that\n>> way\" then the commit reachability should be enforced, and otherwise it\n>> should not be?\n>\n> Any version of git unaware of that rule, though, would consider objects\n> only reachable by gitlink as unreachable and delete them, causing data\n> loss. Likewise for a server not aware of that rule. And a server\n> unaware of that rule would not supply those objects to a client pulling\n> such a branch.\n\nYeah, so you would really need an up-to-date server and client to\nstore the git-series data.\nBut anyway if we create a gitref object, you would also need\nup-to-date servers and clients to make it work.\n\n> So I don't think \"gitlink defined as reachable\" quite works, unless we\n> make some other format-incompatible change that forces clients and\n> servers touching that gitlink to know about that reachability rule. (In\n> the absence of a hack such as making the same commit a parent.)\n\nThere are other tools that would like to tweak reachability rules for objects.\n\nFor example Mike Hommey's git-cinnabar:\n\n https://public-inbox.org/git/20150331100756.GA13377@glandium.org/\n\nand my current work on external object databases:\n\n https://public-inbox.org/git/20160628181933.24620-1-chriscool@tuxfamily.org/\n\nwould be interested in a way to make some blobs not reachable.\n\nSo if we had default rules and a generic way to specify that some\nobjects are, or are not, reachable, that could be used by many tools,\nand the design of these tools would be simplified.\n\nMaybe this could be specified in the attributes files, or in a special\nfile like for shallow clone.\n"},{"id":"305447","messageId":"20161105202553.migx75gfuujakqyk@x","threadId":"44427","inReplyTo":"CAP8UFD3XFHr7POKmZr_6guapC6sme3GvWBV5vPw4XO7FE5HOPw@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-05T20:25:54Z","receivedAt":"2016-11-05T20:26:07Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Sat, Nov 05, 2016 at 09:21:58PM +0100, Christian Couder wrote:\n> On Sat, Nov 5, 2016 at 4:18 PM, Josh Triplett wrote:\n> > On Sat, Nov 05, 2016 at 01:45:27PM +0100, Christian Couder wrote:\n> >> And with what Peff says above it looks like we will need ways\n> >> configure and tweak commit reachability with gitlink/gitref anyway. So\n> >> the point of gitref compared to gitlink would be that they just have a\n> >> different reachability by default. But couldn't that be replaced by a\n> >> default rule saying that when a gitlink is reached \"this way or that\n> >> way\" then the commit reachability should be enforced, and otherwise it\n> >> should not be?\n> >\n> > Any version of git unaware of that rule, though, would consider objects\n> > only reachable by gitlink as unreachable and delete them, causing data\n> > loss. Likewise for a server not aware of that rule. And a server\n> > unaware of that rule would not supply those objects to a client pulling\n> > such a branch.\n> \n> Yeah, so you would really need an up-to-date server and client to\n> store the git-series data.\n> But anyway if we create a gitref object, you would also need\n> up-to-date servers and clients to make it work.\n\nAgreed, but gitrefs have the advantage of failing safe, rather than\nfailing with dataloss.\n\n- Josh Triplett\n"},{"id":"305448","messageId":"CAP8UFD1ipYZNHcWRoX8XwTjpQ4=P+fgwRB5gtbovjU3Gddf8Og@mail.gmail.com","threadId":"44427","inReplyTo":"20161104211959.3532uiud27nhumt7@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-11-05T21:56:14Z","receivedAt":"2016-11-05T21:57:23Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Nov 4, 2016 at 10:19 PM, Josh Triplett wrote:\n> On Fri, Nov 04, 2016 at 09:47:41PM +0100, Christian Couder wrote:\n>> On Fri, Nov 4, 2016 at 6:57 PM, Junio C Hamano wrote:\n>> >\n>> > Imagine we invent a new tree entry type, \"gitref\", that is similar\n>> > to \"gitlink\" in that it can record a commit object name in a tree,\n>> > but unlike \"gitlink\" it does imply reachability. And you do not add\n>> > phony parents to your commit object. A tree that has \"gitref\"s in\n>> > it is about annotating the commits in the same repository (e.g. the\n>> > tree references two commits, \"base\" and \"tip\", to point into a slice\n>> > of the main history). And it is perfectly sensible for such a\n>> > pointer to imply reachability---after all it serves different\n>> > purposes from \"gitlink\".\n>>\n>> The more I think about this (and also about how to limit ref\n>> advertisements as recently discussed in\n>> https://public-inbox.org/git/20161024132932.i42rqn2vlpocqmkq@sigill.intra.peff.net/),\n>> the more I think about Shawn's RefTree:\n>>\n>> https://public-inbox.org/git/CAJo=hJvnAPNAdDcAAwAvU9C4RVeQdoS3Ev9WTguHx4fD0V_nOg@mail.gmail.com/\n\nJust to make things clear, after reading the above link that I posted :-) ...\n\n>> Couldn't a RefTree be used to store refs that point to the base\n>> commit, the tip commit and the blob that contains the cover letter,\n>> and maybe also a ref pointing to the RefTree of the previous version\n>> of the series?\n>\n> That's really interesting! The Software Heritage project is working on\n> something similar, because they want to store all the refs as part of\n> their data model as well. I'll point them to the reftree work.\n>\n> If upstream git supported RefTree, I could potentially use that for\n> git-series. However, I do want a commit message and history for the\n> series itself, and using refs in the reftree to refer to the parents\n> seems like abusing reftree to recreate commits, in a reversal of the\n> hack of using commit parents as a reftree. :)\n>\n> What if, rather than storing a hash reference to a reftree as a single\n> reference and replacing it with no history, a reftree could be\n> referenced from a commit and have history? (That would also allow\n> tagging a version of the reftree.)\n\n... I think that indeed that's what Shawn's reftree proposal is about,\nso I agree that it makes sense.\n\nWe just need to find a good way to specify object reachability.\n"},{"id":"305449","messageId":"CA+P7+xoG3ag8dj7s_NRoqz-EwjVENSJSzE_qj6gnW-SmWt0bgA@mail.gmail.com","threadId":"44427","inReplyTo":"20161105202553.migx75gfuujakqyk@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-11-06T04:50:07Z","receivedAt":"2016-11-06T04:50:36Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Sat, Nov 5, 2016 at 1:25 PM, Josh Triplett wrote:\n> On Sat, Nov 05, 2016 at 09:21:58PM +0100, Christian Couder wrote:\n>> On Sat, Nov 5, 2016 at 4:18 PM, Josh Triplett wrote:\n>> > On Sat, Nov 05, 2016 at 01:45:27PM +0100, Christian Couder wrote:\n>> >> And with what Peff says above it looks like we will need ways\n>> >> configure and tweak commit reachability with gitlink/gitref anyway. So\n>> >> the point of gitref compared to gitlink would be that they just have a\n>> >> different reachability by default. But couldn't that be replaced by a\n>> >> default rule saying that when a gitlink is reached \"this way or that\n>> >> way\" then the commit reachability should be enforced, and otherwise it\n>> >> should not be?\n>> >\n>> > Any version of git unaware of that rule, though, would consider objects\n>> > only reachable by gitlink as unreachable and delete them, causing data\n>> > loss. Likewise for a server not aware of that rule. And a server\n>> > unaware of that rule would not supply those objects to a client pulling\n>> > such a branch.\n>>\n>> Yeah, so you would really need an up-to-date server and client to\n>> store the git-series data.\n>> But anyway if we create a gitref object, you would also need\n>> up-to-date servers and clients to make it work.\n>\n> Agreed, but gitrefs have the advantage of failing safe, rather than\n> failing with dataloss.\n>\n> - Josh Triplett\n\nIsn't the \"failing safe\" only true if the client disconnects when a\nserver doesn't advertise \"i understand gitrefs\"? So couldn't we, as\npart of the rules for reachability advertise a capability that does a\nsimilar thing and fails safe as well?\n\nThanks.\nJake\n"},{"id":"305462","messageId":"20161106163410.ilysej5r6qd3744e@x","threadId":"44427","inReplyTo":"CA+P7+xoG3ag8dj7s_NRoqz-EwjVENSJSzE_qj6gnW-SmWt0bgA@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-06T16:34:10Z","receivedAt":"2016-11-06T16:34:24Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Sat, Nov 05, 2016 at 09:50:07PM -0700, Jacob Keller wrote:\n> On Sat, Nov 5, 2016 at 1:25 PM, Josh Triplett wrote:\n> > On Sat, Nov 05, 2016 at 09:21:58PM +0100, Christian Couder wrote:\n> >> On Sat, Nov 5, 2016 at 4:18 PM, Josh Triplett wrote:\n> >> > On Sat, Nov 05, 2016 at 01:45:27PM +0100, Christian Couder wrote:\n> >> >> And with what Peff says above it looks like we will need ways\n> >> >> configure and tweak commit reachability with gitlink/gitref anyway. So\n> >> >> the point of gitref compared to gitlink would be that they just have a\n> >> >> different reachability by default. But couldn't that be replaced by a\n> >> >> default rule saying that when a gitlink is reached \"this way or that\n> >> >> way\" then the commit reachability should be enforced, and otherwise it\n> >> >> should not be?\n> >> >\n> >> > Any version of git unaware of that rule, though, would consider objects\n> >> > only reachable by gitlink as unreachable and delete them, causing data\n> >> > loss. Likewise for a server not aware of that rule. And a server\n> >> > unaware of that rule would not supply those objects to a client pulling\n> >> > such a branch.\n> >>\n> >> Yeah, so you would really need an up-to-date server and client to\n> >> store the git-series data.\n> >> But anyway if we create a gitref object, you would also need\n> >> up-to-date servers and clients to make it work.\n> >\n> > Agreed, but gitrefs have the advantage of failing safe, rather than\n> > failing with dataloss.\n> >\n> > - Josh Triplett\n> \n> Isn't the \"failing safe\" only true if the client disconnects when a\n> server doesn't advertise \"i understand gitrefs\"? So couldn't we, as\n> part of the rules for reachability advertise a capability that does a\n> similar thing and fails safe as well?\n\nWe could, but if we (or one of the many third-party git implementations)\nmiss a case, gitlinks+reachability may appear to work in many cases with\ndataloss afterward, while gitrefs will fail early and not appear\nfunctional.\n"},{"id":"305463","messageId":"xmqqshr4cyy7.fsf@gitster.mtv.corp.google.com","threadId":"44427","inReplyTo":"20161106163410.ilysej5r6qd3744e@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-06T17:14:56Z","receivedAt":"2016-11-06T17:15:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Triplett writes:\n\n> We could, but if we (or one of the many third-party git implementations)\n> miss a case, gitlinks+reachability may appear to work in many cases with\n> dataloss afterward, while gitrefs will fail early and not appear\n> functional.\n\nI wonder what happens if we do not introduce the \"gitref\" but\ninstead change the behaviour of \"gitlink\" to imply an optional\nreachability. That is, when enumerating what is reachable in your\nrepository, if you see a gitlink and if you notice that you locally\nhave the target of that gitlink, you follow, but if you know you\nlack it, you do not error out. This may be making things too\ncomplex to feasibily implement by simplify them ;-) and I see a few\nimmediate fallout that needs to be thought through (i.e. downsides)\nand a few upsides, too. I am feeling feverish and not thinking\nstraight, so I won't try to weigh pros-and-cons. \n\nThis would definitely need protocol extension when transferring\nobjects across repositories.\n"},{"id":"305465","messageId":"20161106173311.lqoxxgcklx4jlrg7@x","threadId":"44427","inReplyTo":"xmqqshr4cyy7.fsf@gitster.mtv.corp.google.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-06T17:33:11Z","receivedAt":"2016-11-06T17:34:09Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Sun, Nov 06, 2016 at 09:14:56AM -0800, Junio C Hamano wrote:\n> Josh Triplett writes:\n> > We could, but if we (or one of the many third-party git implementations)\n> > miss a case, gitlinks+reachability may appear to work in many cases with\n> > dataloss afterward, while gitrefs will fail early and not appear\n> > functional.\n> \n> I wonder what happens if we do not introduce the \"gitref\" but\n> instead change the behaviour of \"gitlink\" to imply an optional\n> reachability. That is, when enumerating what is reachable in your\n> repository, if you see a gitlink and if you notice that you locally\n> have the target of that gitlink, you follow, but if you know you\n> lack it, you do not error out. This may be making things too\n> complex to feasibily implement by simplify them ;-) and I see a few\n> immediate fallout that needs to be thought through (i.e. downsides)\n> and a few upsides, too. I am feeling feverish and not thinking\n> straight, so I won't try to weigh pros-and-cons. \n> \n> This would definitely need protocol extension when transferring\n> objects across repositories.\n\nIt'd also need a repository format extension locally. Otherwise, if you\never touched that repository with an older git (or a tool built on an\nolder libgit2 or JGit or other library), you could lose data.\n\nIt does seem conceptually appealing, though. In an ideal world, the\noriginal version of gitlink would have had opt-out reachability (and\n.gitmodules with an external repository reference could count as opting\nout).\n\nBut I can't think of any case where it's OK for a git implementation to\nnot know about this reachability extension and still operate on the\ngitlink. And given that, it might as well use a new object type that\nthe old version definitely won't think it understands.\n\n- Josh Triplett\n"},{"id":"305468","messageId":"CA+P7+xoxjwvjXrW0Pwh7ZK-OYBiYamPAxvf_=zqJOsQ8xWDPWw@mail.gmail.com","threadId":"44427","inReplyTo":"20161106173311.lqoxxgcklx4jlrg7@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-11-06T20:17:10Z","receivedAt":"2016-11-06T20:17:40Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Sun, Nov 6, 2016 at 9:33 AM, Josh Triplett wrote:\n> On Sun, Nov 06, 2016 at 09:14:56AM -0800, Junio C Hamano wrote:\n>> Josh Triplett writes:\n>> > We could, but if we (or one of the many third-party git implementations)\n>> > miss a case, gitlinks+reachability may appear to work in many cases with\n>> > dataloss afterward, while gitrefs will fail early and not appear\n>> > functional.\n>>\n>> I wonder what happens if we do not introduce the \"gitref\" but\n>> instead change the behaviour of \"gitlink\" to imply an optional\n>> reachability. That is, when enumerating what is reachable in your\n>> repository, if you see a gitlink and if you notice that you locally\n>> have the target of that gitlink, you follow, but if you know you\n>> lack it, you do not error out. This may be making things too\n>> complex to feasibily implement by simplify them ;-) and I see a few\n>> immediate fallout that needs to be thought through (i.e. downsides)\n>> and a few upsides, too. I am feeling feverish and not thinking\n>> straight, so I won't try to weigh pros-and-cons.\n>>\n>> This would definitely need protocol extension when transferring\n>> objects across repositories.\n>\n> It'd also need a repository format extension locally. Otherwise, if you\n> ever touched that repository with an older git (or a tool built on an\n> older libgit2 or JGit or other library), you could lose data.\n>\n> It does seem conceptually appealing, though. In an ideal world, the\n> original version of gitlink would have had opt-out reachability (and\n> .gitmodules with an external repository reference could count as opting\n> out).\n>\n> But I can't think of any case where it's OK for a git implementation to\n> not know about this reachability extension and still operate on the\n> gitlink. And given that, it might as well use a new object type that\n> the old version definitely won't think it understands.\n>\n> - Josh Triplett\n\nThat's still only true if the receiving end runs fsck, isn't it? I\nsuppose that's a large number of receivers, and at least there are\nways post-push to determine that objects don't make sense to that\nversion of git.\n\nI think using a new mode is the safest way, and it allows easily\nimplementing RefTrees as well as other projects. Additionally, if we\n*wanted* additional \"opt-in / opt-out\" support we could add this by\ndefault to gitrefs,and they could (possibly) replace gitlinks in the\nfuture?\n\nThanks,\nJake\n"},{"id":"305472","messageId":"20161107011841.vy2qfnbefidd2sjf@x","threadId":"44427","inReplyTo":"CA+P7+xoxjwvjXrW0Pwh7ZK-OYBiYamPAxvf_=zqJOsQ8xWDPWw@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-07T01:18:41Z","receivedAt":"2016-11-07T01:20:38Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Sun, Nov 06, 2016 at 12:17:10PM -0800, Jacob Keller wrote:\n> On Sun, Nov 6, 2016 at 9:33 AM, Josh Triplett wrote:\n> > On Sun, Nov 06, 2016 at 09:14:56AM -0800, Junio C Hamano wrote:\n> >> Josh Triplett writes:\n> >> > We could, but if we (or one of the many third-party git implementations)\n> >> > miss a case, gitlinks+reachability may appear to work in many cases with\n> >> > dataloss afterward, while gitrefs will fail early and not appear\n> >> > functional.\n> >>\n> >> I wonder what happens if we do not introduce the \"gitref\" but\n> >> instead change the behaviour of \"gitlink\" to imply an optional\n> >> reachability. That is, when enumerating what is reachable in your\n> >> repository, if you see a gitlink and if you notice that you locally\n> >> have the target of that gitlink, you follow, but if you know you\n> >> lack it, you do not error out. This may be making things too\n> >> complex to feasibily implement by simplify them ;-) and I see a few\n> >> immediate fallout that needs to be thought through (i.e. downsides)\n> >> and a few upsides, too. I am feeling feverish and not thinking\n> >> straight, so I won't try to weigh pros-and-cons.\n> >>\n> >> This would definitely need protocol extension when transferring\n> >> objects across repositories.\n> >\n> > It'd also need a repository format extension locally. Otherwise, if you\n> > ever touched that repository with an older git (or a tool built on an\n> > older libgit2 or JGit or other library), you could lose data.\n> >\n> > It does seem conceptually appealing, though. In an ideal world, the\n> > original version of gitlink would have had opt-out reachability (and\n> > .gitmodules with an external repository reference could count as opting\n> > out).\n> >\n> > But I can't think of any case where it's OK for a git implementation to\n> > not know about this reachability extension and still operate on the\n> > gitlink. And given that, it might as well use a new object type that\n> > the old version definitely won't think it understands.\n> >\n> > - Josh Triplett\n> \n> That's still only true if the receiving end runs fsck, isn't it? I\n> suppose that's a large number of receivers, and at least there are\n> ways post-push to determine that objects don't make sense to that\n> version of git.\n> \n> I think using a new mode is the safest way, and it allows easily\n> implementing RefTrees as well as other projects. Additionally, if we\n> *wanted* additional \"opt-in / opt-out\" support we could add this by\n> default to gitrefs,and they could (possibly) replace gitlinks in the\n> future?\n\nOnce we have gitrefs, you have both alternatives: reachable (gitref) or\nnot reachable (gitlink).\n\nHowever, if you want some way to mark reachable objects as not\nreachable, such as for a sparse checkout, external large-object storage,\nor similar, then you can use a single unified mechanism for that whether\nworking with gitrefs, trees, or blobs.\n"},{"id":"305476","messageId":"CA+P7+xpmC8AdjEL59S_bRbf5QAsk=fO9-d1kmrSppTaJLVL2Fw@mail.gmail.com","threadId":"44427","inReplyTo":"20161107011841.vy2qfnbefidd2sjf@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-11-07T05:35:32Z","receivedAt":"2016-11-07T05:36:23Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Sun, Nov 6, 2016 at 5:18 PM, Josh Triplett wrote:\n> Once we have gitrefs, you have both alternatives: reachable (gitref) or\n> not reachable (gitlink).\n>\n> However, if you want some way to mark reachable objects as not\n> reachable, such as for a sparse checkout, external large-object storage,\n> or similar, then you can use a single unified mechanism for that whether\n> working with gitrefs, trees, or blobs.\n\nFair enough.\n\nThanks,\nJake\n"},{"id":"305480","messageId":"CACsJy8BDySBWyp3iiLinRkBCew5FNXoQo7z9dMb6w6m9a5X=NA@mail.gmail.com","threadId":"44427","inReplyTo":"20161107011841.vy2qfnbefidd2sjf@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-11-07T09:42:04Z","receivedAt":"2016-11-07T09:42:57Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 7, 2016 at 8:18 AM, Josh Triplett wrote:\n> Once we have gitrefs, you have both alternatives: reachable (gitref) or\n> not reachable (gitlink).\n>\n> However, if you want some way to mark reachable objects as not\n> reachable, such as for a sparse checkout, external large-object storage,\n> or similar, then you can use a single unified mechanism for that whether\n> working with gitrefs, trees, or blobs.\n\nHow? Whether an object reachable or not is baked in the definition (of\neither gitlink or gitref). I don't think you can have a \"maybe\nreachable\" type then rely on an external source to determine\nreachability,\n-- \nDuy\n"},{"id":"305487","messageId":"20161107161158.7db3f3gthdlfuhyw@x","threadId":"44427","inReplyTo":"CACsJy8BDySBWyp3iiLinRkBCew5FNXoQo7z9dMb6w6m9a5X=NA@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2016-11-07T16:11:59Z","receivedAt":"2016-11-07T16:12:12Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Mon, Nov 07, 2016 at 04:42:04PM +0700, Duy Nguyen wrote:\n> On Mon, Nov 7, 2016 at 8:18 AM, Josh Triplett wrote:\n> > Once we have gitrefs, you have both alternatives: reachable (gitref) or\n> > not reachable (gitlink).\n> >\n> > However, if you want some way to mark reachable objects as not\n> > reachable, such as for a sparse checkout, external large-object storage,\n> > or similar, then you can use a single unified mechanism for that whether\n> > working with gitrefs, trees, or blobs.\n> \n> How? Whether an object reachable or not is baked in the definition (of\n> either gitlink or gitref). I don't think you can have a \"maybe\n> reachable\" type then rely on an external source to determine\n> reachability,\n\nYou'd have various \"reachable by default\" entries in trees, including\ntrees, blobs, and gitrefs, and then have an external mechanism (likely\nvia .git/config) to say \"ignore objects with these hashes/paths\". For\ninstance, you might say \"ignore all objects only reachable from the path\n'assets/video/*' within a commit's tree\". With the right set of client\nand server extensions, you could then avoid downloading those objects.\n"},{"id":"305652","messageId":"xmqqtwbg8dn1.fsf@gitster.mtv.corp.google.com","threadId":"44427","inReplyTo":"20161106173311.lqoxxgcklx4jlrg7@x","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-11-09T22:57:54Z","receivedAt":"2016-11-09T22:58:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Triplett writes:\n\n>> This would definitely need protocol extension when transferring\n>> objects across repositories.\n>\n> It'd also need a repository format extension locally. Otherwise, if you\n> ever touched that repository with an older git (or a tool built on an\n> older libgit2 or JGit or other library), you could lose data.\n\nTrue. Thanks for sanity-checking me.\n"},{"id":"305877","messageId":"20161113175058.t7s4o7h5chqzazb6@upsilon.cc","threadId":"44427","inReplyTo":"CAP8UFD13sDOFuyZMWuoJeLFt_LAsfAHFBHpRwcdAGmA22xNEKQ@mail.gmail.com","subject":"Re: Regarding \"git log\" on \"git series\" metadata","fromName":"Stefano Zacchiroli","fromEmail":"zack@softwareheritage.org","sentAt":"2016-11-13T17:50:58Z","receivedAt":"2016-11-13T17:58:24Z","isPatch":false,"sender":{"key":"zack@softwareheritage.org","avatar":null},"body":"Hi everyone,\n\nOn Sat, Nov 05, 2016 at 12:04:08AM +0100, Christian Couder wrote:\n> On Fri, Nov 4, 2016 at 10:19 PM, Josh Triplett wrote:\n> > On Fri, Nov 04, 2016 at 09:47:41PM +0100, Christian Couder wrote:\n> >>\n> >> Couldn't a RefTree be used to store refs that point to the base\n> >> commit, the tip commit and the blob that contains the cover letter,\n> >> and maybe also a ref pointing to the RefTree of the previous version\n> >> of the series?\n> >\n> > That's really interesting! The Software Heritage project is working on\n> > something similar, because they want to store all the refs as part of\n> > their data model as well. I'll point them to the reftree work.\n> \n> Yeah, I know them :-) and I think I have already told Stefano\n> Zacchiroli about this, but I am not sure anymore.\n> Anyway I am CC'ing him.\n\nThanks Christian (and Josh, on swh-devel) for pointing me to this.\n\nAs a bit of background, the conceptual data model we have adopted for\nSoftware Heritage [1] is indeed that of a global Merkle DAG, very much\ninspired by Git, but where we deduplicate past the boundaries of\nindividual VCS repositories. This way we can store only once the same\nsoftware artifacts (blobs, trees, commits, etc.) even when they can be\nfound at different software origins [2] (be it due to GitHub-like forks,\nprojects moving around, or simply rogue copies of the same code\nscattered around the Internet).\n\n[1]: https://www.softwareheritage.org/\n\n[2]: \"software origin\" is Software Heritage terminology, which just\n stands for places on the Internet where we can find source code\n\nIn our original design the topmost entries in our Merkle hierarchy used\nto be commits and tags, similar to what Git does. But then we realized\nthat doing so inhibited us from sharing entire repository states across\nmultiple software origins or multiple visits of the same software\norigin. So we decided to add \"repository snapshot objects\" as our\ntopmost entries, which are essentially git-like objects that map refs to\nthe ID of the corresponding (typed-)objects. Rationale and a more\nlengthy description of this is available on our wiki [3]. It is not\nimplemented yet, but we're pretty sold on the design at this point.\n\n[3]: https://wiki.softwareheritage.org/index.php?title=Repository_snapshot_objects\n\nNow, even if my only awareness of what's going on in Git upstream is\nlimited to sporadic chats with Josh and Christian :-), it seems to me\nthat various ideas in the Git ecosystem go in the same direction of our\nsnapshot objects (git-series, RefTree). Which is understandable, given a\nnumber of use cases might be served by this.\n\nI don't think we have much to contribute to discussion or implementation\nhere, and for our needs it doesn't really matter which one gets\nimplemented. That's because we need an implementation of the concept\nwhich is *external* to Git anyhow. But even if it happens to exist\nwithin actual VCS, it's not a big deal for us, as we do have ways to\ndistinguish \"synthetic\" objects in the DAG that we create for our own\nneeds from \"real\" objects coming from actual software origins.\n(Another example of this concept we already have is when we inject\ndistribution source packages or tarballs in our archive. In that case we\ncreate synthetic commits that points to the tree extracted from the\ntarball/package, preserving the ability to distinguish them from real\ncommits coming from VCS out there.)\n\nIf you think we can help in any other way, other than sharing our\nexperiences and design considerations that is, please let me know! (I'm\nnot subscribed to the Git upstream mailing list, but feel free to Cc:-me\nin conversations related to this topic.)\n\nCheers.\n-- \nStefano Zacchiroli . zack@upsilon.cc . upsilon.cc/zack . . o . . . o . o\nComputer Science Professor . CTO Software Heritage . . . . . o . . . o o\nFormer Debian Project Leader . OSI Board Director . . . o o o . . . o .\n« the first rule of tautology club is the first rule of tautology club »\n"}]}