{"thread":{"id":"22545","subject":"A generalization of git notes from blobs to trees - git metadata?","startedAt":"2010-02-06T13:32:52Z","lastAt":"2010-02-10T05:29:02Z","messageCount":19,"participants":["Jon Seymour","Johan Herland","Junio C Hamano","Jeff King","Jakub Narebski","Steven E. Harris"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"133799","messageId":"2cfc40321002060532g4d22dd4dx403bf312708e1424@mail.gmail.com","threadId":"22545","inReplyTo":null,"subject":"A generalization of git notes from blobs to trees - git metadata?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2010-02-06T13:32:52Z","receivedAt":"2010-02-06T13:32:52Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"git notes is a nice innovation - well done to all those involved.\n\nHas consideration ever been given to generalizing the concept to allow\nnote (or more correctly -  metadata) trees with arbitrary sha1s?\n\nFor example, suppose you had reason to cache the distribution that\nresulted from the build of a particular commit, then it'd be nice to\nbe able to do this using a notes like mechanism.\n\n    git metadata import foo-1.1.0 dist ~/foo/dist\n\nwould create a git tree from the contents of ~/foo/dist and then bind\nit to meta item called dist associated with the sha1 corresponding to\nfoo-1.1.0\n\nTo retrieve the contents of the previous build, you'd do something like\n\n   get metadata export foo-1.1.0 dist /tmp/foo-1.1.0\n\nThis would find the metadata tree associated with foo-1.1.0, extract\nthe dist subtree from that tree and write it to disk at /tmp/foo-1.1.0\n\nI've used build outputs as an example here, but really it needn't be\nlimited to that. I can see this facility would be useful for any kind\nof annotation or derived result that is more complex than a single\ntext blob. Metadata trees in combination with a name spacing\ntechnique, could be used to store arbitrary metadata created by an\narbitrary set of tools to arbitrary SHA1 objects.\n\njon.\n"},{"id":"133841","messageId":"201002070236.12711.johan@herland.net","threadId":"22545","inReplyTo":"2cfc40321002060532g4d22dd4dx403bf312708e1424@mail.gmail.com","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2010-02-07T01:36:12Z","receivedAt":"2010-02-07T01:36:12Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Saturday 06 February 2010, Jon Seymour wrote:\n> git notes is a nice innovation - well done to all those involved.\n\nThanks.\n\n> Has consideration ever been given to generalizing the concept to allow\n> note (or more correctly -  metadata) trees with arbitrary sha1s?\n\nNot sure what you mean here. The note infrastructure allows _any_ SHA1 (not \nnecessarily the SHA1 of an existing Git object) to be bound to a note \nobject.\n\nFurthermore, although we currently assume that all note objects are blobs, \nsomeone (who?) has already suggested (as mentioned in the notes TODO list) \nthat a note object could also be a _tree_ object that can be unpacked/read \nto reveal further \"sub-notes\". Hence, in addition to having multiple notes \nrefs (e.g. refs/notes/commits:deadbeef, refs/notes/bugs:deadbeef, etc.) to \ncategorize notes, you could also classify notes _after_ having traversed the \nnotes tree (e.g. refs/notes/bugs:deadbeef/fixes, \nrefs/notes/bugs:deadbeef/causes). Note that support for this has not yet \nbeen written, and AFAIK it is also uncertain how such a change would affect \nthe different use cases for notes (e.g. how to display them in 'git log')\n\n> For example, suppose you had reason to cache the distribution that\n> resulted from the build of a particular commit, then it'd be nice to\n> be able to do this using a notes like mechanism.\n> \n>     git metadata import foo-1.1.0 dist ~/foo/dist\n> \n> would create a git tree from the contents of ~/foo/dist and then bind\n> it to meta item called dist associated with the sha1 corresponding to\n> foo-1.1.0\n\nYou can do this already today by simply using 'git tag':\n\t# Prepare an index with the contents of ~/foo/dist\n\tgit tag foo-1.1.0-dist $(git write-tree)\n\nI don't see why you'd need to add a new metadata command.\n\n> To retrieve the contents of the previous build, you'd do something like\n> \n>    get metadata export foo-1.1.0 dist /tmp/foo-1.1.0\n> \n> This would find the metadata tree associated with foo-1.1.0, extract\n> the dist subtree from that tree and write it to disk at /tmp/foo-1.1.0\n\nOr, if you use a tag instead:\n\tgit --work-tree=/tmp/foo-1.1.0 checkout foo-1.1.0-dist\n\n> I've used build outputs as an example here, but really it needn't be\n> limited to that. I can see this facility would be useful for any kind\n> of annotation or derived result that is more complex than a single\n> text blob. Metadata trees in combination with a name spacing\n> technique, could be used to store arbitrary metadata created by an\n> arbitrary set of tools to arbitrary SHA1 objects.\n\nI still don't see why this provides anything that isn't already supported by \neither using 'git tag', or by implementing support for notes-as-trees in the \nnotes feature.\n\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"133846","messageId":"7v1vgxlr9q.fsf@alter.siamese.dyndns.org","threadId":"22545","inReplyTo":"201002070236.12711.johan@herland.net","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-07T02:21:37Z","receivedAt":"2010-02-07T02:21:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland <johan@herland.net> writes:\n\n> Furthermore, although we currently assume that all note objects are blobs, \n> someone (who?) has already suggested (as mentioned in the notes TODO list) \n> that a note object could also be a _tree_ object that can be unpacked/read \n> to reveal further \"sub-notes\".\n\nI would advice you not to go there.  How would you even _merge_ such a\nthing with other notes attached to the same object?  What determines the\npath in that tree object?\n\nClueless ones can freely make misguided suggestions without thinking\nthings through and make things unnecessarily complex without real gain.\nYou do not have to listen to every one of them.\n"},{"id":"133848","messageId":"2cfc40321002061927m522f0c3aj7d727c47a2f0cb22@mail.gmail.com","threadId":"22545","inReplyTo":"201002070236.12711.johan@herland.net","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2010-02-07T03:27:27Z","receivedAt":"2010-02-07T03:27:27Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"> I still don't see why this provides anything that isn't already supported by\n> either using 'git tag', or by implementing support for notes-as-trees in the\n> notes feature.\n>\n\nThe intent of the metadata facility is to associate derivatives of\nsha1 with the sha1 itself. If I have calculated a derivative of sha1\nin the past, then let me reference that derivative using a metadata\npath which I can look up knowing only the sha1 of the input and\nnothing more. Yes,  I could create tags of the form\n${sha1}/metadata-path for all my derived results but really, this\nseems an abuse of the tag facility.\n\nHere's another motivating example:\n\nSuppose git-svn wrote the SVN id it was synched with into structured\nmetadata associated with a commit, instead of into the commit message,\nthe equivalent of:\n\n    echo ${svn-id} | git metadata write-blob ${sha1} svn-id\n\nWhich means: for the specified sha1, read a blob from stdin and create\na metadata item with a metadata path called svn-id\n\nTo get it out again, you would write:\n\n    git metadata read-blob ${sha1} svn-id\n\nWhich says, for the given object ${sha1}, read the blob from the\nmetadata tree at path svn-id and write its contents to stdout.\n\nThis would avoid cluttering the commit message with the svn-id, avoid\ncluttering the tag space with the info and allow any commit to be\ntagged in this way.\n\nAdmittedly similar function could be achieved a little more clumsily\nnow with appropriate use of GIT_NOTES_REF or with note subtrees, but I\nshare Junio's  reservations about trying to generalize notes from\nblobs to trees, given way notes are currently used by the rest of\ninfrastructure.\n\njon.\n"},{"id":"133849","messageId":"2cfc40321002062032g1e0afd66la7ac4a26295bd817@mail.gmail.com","threadId":"22545","inReplyTo":"2cfc40321002061927m522f0c3aj7d727c47a2f0cb22@mail.gmail.com","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2010-02-07T04:32:05Z","receivedAt":"2010-02-07T04:32:05Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"Another use case could be to store the contents of the man and html\ntrees of git, which are currently published as separate branches.\n\nWith the metadata concept, the man and html trees for each release\ncould be stored as metadata paths (/man, /html) of the associated\ncommit for each release, providing a trivial way to address and access\nthese trees.\n\njon.\n\nOn Sun, Feb 7, 2010 at 2:27 PM, Jon Seymour <jon.seymour@gmail.com> wrote:\n>> I still don't see why this provides anything that isn't already supported by\n>> either using 'git tag', or by implementing support for notes-as-trees in the\n>> notes feature.\n>>\n>\n> The intent of the metadata facility is to associate derivatives of\n> sha1 with the sha1 itself. If I have calculated a derivative of sha1\n> in the past, then let me reference that derivative using a metadata\n> path which I can look up knowing only the sha1 of the input and\n> nothing more. Yes,  I could create tags of the form\n> ${sha1}/metadata-path for all my derived results but really, this\n> seems an abuse of the tag facility.\n>\n> Here's another motivating example:\n>\n> Suppose git-svn wrote the SVN id it was synched with into structured\n> metadata associated with a commit, instead of into the commit message,\n> the equivalent of:\n>\n>    echo ${svn-id} | git metadata write-blob ${sha1} svn-id\n>\n> Which means: for the specified sha1, read a blob from stdin and create\n> a metadata item with a metadata path called svn-id\n>\n> To get it out again, you would write:\n>\n>    git metadata read-blob ${sha1} svn-id\n>\n> Which says, for the given object ${sha1}, read the blob from the\n> metadata tree at path svn-id and write its contents to stdout.\n>\n> This would avoid cluttering the commit message with the svn-id, avoid\n> cluttering the tag space with the info and allow any commit to be\n> tagged in this way.\n>\n> Admittedly similar function could be achieved a little more clumsily\n> now with appropriate use of GIT_NOTES_REF or with note subtrees, but I\n> share Junio's  reservations about trying to generalize notes from\n> blobs to trees, given way notes are currently used by the rest of\n> infrastructure.\n>\n> jon.\n>\n"},{"id":"133852","messageId":"20100207050255.GA17049@coredump.intra.peff.net","threadId":"22545","inReplyTo":"7v1vgxlr9q.fsf@alter.siamese.dyndns.org","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-02-07T05:02:55Z","receivedAt":"2010-02-07T05:02:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Feb 06, 2010 at 06:21:37PM -0800, Junio C Hamano wrote:\n\n> Johan Herland <johan@herland.net> writes:\n> \n> > Furthermore, although we currently assume that all note objects are blobs, \n> > someone (who?) has already suggested (as mentioned in the notes TODO list) \n> > that a note object could also be a _tree_ object that can be unpacked/read \n> > to reveal further \"sub-notes\".\n> \n> I would advice you not to go there.  How would you even _merge_ such a\n> thing with other notes attached to the same object?  What determines the\n> path in that tree object?\n> \n> Clueless ones can freely make misguided suggestions without thinking\n> things through and make things unnecessarily complex without real gain.\n> You do not have to listen to every one of them.\n\nI think I may have been the one to suggest trees or notes at one point.\nBut let me clarify that this is not exactly what the OP is proposing in\nthis thread.\n\nMy suggestion was that some use cases may have many key/value pairs of\nnotes for a single sha1. We basically have two options:\n\n  1. store each in a separate notes ref, with each sha1 mapping to\n     a blob. The note \"name\" is the name of the ref.\n\n  2. store notes in a single notes ref, with each sha1 mapping to a\n     tree with named sub-notes. The note \"name\" is the combination of\n     ref-name and tree entry name.\n\nThe advantage of (1) is that notes are not bound tightly to each other.\nI can distribute the notes tree for one \"name\" independent of the\nothers.  The advantage of (2) is that it is faster and smaller. In (1),\neach note has a separate index, and we must traverse each note index\nseparately.\n\nIn practice, I would expect to use (1) for logically separate datasets.\nFor example, automatic bug-tracking notes would go in a different ref\nfrom human annotations. But I would expect to use (2) if I had, say, 5\ndifferent pieces of bug tracking information and I wanted an easy way to\nrefer to them individually.\n\nAnd a specialized merge for that is straightforward. In the simplest\ncase, you simply say \"notes of this ref are tree-type, or they are\nblob-type\" and then you have no merge problems. But if you want to get\nfancy, you can say that a conflict between \"sha1/blob\" and\n\"sha1/tree/key\" should automatically \"promote\" the first one into\n\"sha1/tree/default\" or some other canonical name.\n\nNote that all of this is my pie-in-the-sky \"here is what I was thinking\nof when I looked at notes a long time ago\". I don't care strongly if it\ngets implemented or not at this point; I just wanted to add some context\nto what Johan had in his notes todo list (or maybe I am wrong, and what\nis in his todo list was based on something totally different said by\nsomebody else, and I have just confused the issue more. :) ).\n\nWith respect to the idea of storing an arbitrary tree, I agree it is\nprobably too complex with respect to merging. In addition, it makes\nthings like \"git log --format=%N\" confusing. I think you would do better\nto simply store a tree sha1 inside the note blob, and callers who were\ninterested in the tree contents could then dereference it and examine as\nthey saw fit.  The only caveat is that you need some way of telling git\nthat the referenced trees are reachable and not to be pruned.\n\n-Peff\n"},{"id":"133854","messageId":"2cfc40321002062136q64f832aesd979c9cb22f3612@mail.gmail.com","threadId":"22545","inReplyTo":"20100207050255.GA17049@coredump.intra.peff.net","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2010-02-07T05:36:59Z","receivedAt":"2010-02-07T05:36:59Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"On Sun, Feb 7, 2010 at 4:02 PM, Jeff King <peff@peff.net> wrote:\n> On Sat, Feb 06, 2010 at 06:21:37PM -0800, Junio C Hamano wrote:\n>\n>> Johan Herland <johan@herland.net> writes:\n>>\n>> > Furthermore, although we currently assume that all note objects are blobs,\n>> > someone (who?) has already suggested (as mentioned in the notes TODO list)\n>> > that a note object could also be a _tree_ object that can be unpacked/read\n>> > to reveal further \"sub-notes\".\n>>\n>> I would advice you not to go there.  How would you even _merge_ such a\n>> thing with other notes attached to the same object?  What determines the\n>> path in that tree object?\n>>\n>> Clueless ones can freely make misguided suggestions without thinking\n>> things through and make things unnecessarily complex without real gain.\n>> You do not have to listen to every one of them.\n>\n> I think I may have been the one to suggest trees or notes at one point.\n> But let me clarify that this is not exactly what the OP is proposing in\n> this thread.\n>\n> My suggestion was that some use cases may have many key/value pairs of\n> notes for a single sha1. We basically have two options:\n>\n>  1. store each in a separate notes ref, with each sha1 mapping to\n>     a blob. The note \"name\" is the name of the ref.\n>\n>  2. store notes in a single notes ref, with each sha1 mapping to a\n>     tree with named sub-notes. The note \"name\" is the combination of\n>     ref-name and tree entry name.\n>\n\nSo, of course, options (1) and (2) need not be exclusive. Use Option\n(1) for different metadata sets and option (2) to partition individual\ndatasets.\n\n>\n> With respect to the idea of storing an arbitrary tree, I agree it is\n> probably too complex with respect to merging. In addition, it makes\n> things like \"git log --format=%N\" confusing. I think you would do better\n> to simply store a tree sha1 inside the note blob, and callers who were\n> interested in the tree contents could then dereference it and examine as\n> they saw fit.  The only caveat is that you need some way of telling git\n> that the referenced trees are reachable and not to be pruned.\n>\n\nAs I see it, the existing use of notes is a special instance of a more\ngeneral metadata capability in which the metadata is constrained to be\na single blob. If notes continued to be constrained in this way, there\nis no reason to change anything with respect to its current userspace\nbehaviour. That said, most of the plumbing which enabled notes could\nbe generalized to enable the arbitrary tree case [ which admittedly, I\nhave yet to sell successfully !]\n\nIn one sense, there is a sense in the merge issue doesn't exist. When\nthe maintainer publishes a tag no-one expects to have to deal with\ndownstream conflicting definitions of the tag. Likewise, if the\nmaintainer were to publish the /man and /html metadata trees (per my\nprevious example) for a release tag, anyone who received\n/refs/metadata/doc would expect to receive the metadata trees as\npublished by the maintainer. Anyone who didn't wouldn't have to pull\n/refs/metadata/doc.\n\nI can see there are use cases where multiple parties might want to\ncontribute metadata and I do not currently have a good solution to\nthat problem, but that is not to say there isn't one - surely it is\njust a question of applying a little intellect creatively?\n\njon.\n"},{"id":"133859","messageId":"m363699zn4.fsf@localhost.localdomain","threadId":"22545","inReplyTo":"2cfc40321002062136q64f832aesd979c9cb22f3612@mail.gmail.com","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-02-07T09:15:10Z","receivedAt":"2010-02-07T09:15:10Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jon Seymour <jon.seymour@gmail.com> writes:\n\n[cut]\n\n> As I see it, the existing use of notes is a special instance of a more\n> general metadata capability in which the metadata is constrained to be\n> a single blob. If notes continued to be constrained in this way, there\n> is no reason to change anything with respect to its current userspace\n> behaviour. That said, most of the plumbing which enabled notes could\n> be generalized to enable the arbitrary tree case [ which admittedly, I\n> have yet to sell successfully !]\n> \n> In one sense, there is a sense in the merge issue doesn't exist. When\n> the maintainer publishes a tag no-one expects to have to deal with\n> downstream conflicting definitions of the tag. Likewise, if the\n> maintainer were to publish the /man and /html metadata trees (per my\n> previous example) for a release tag, anyone who received\n> /refs/metadata/doc would expect to receive the metadata trees as\n> published by the maintainer. Anyone who didn't wouldn't have to pull\n> /refs/metadata/doc.\n> \n> I can see there are use cases where multiple parties might want to\n> contribute metadata and I do not currently have a good solution to\n> that problem, but that is not to say there isn't one - surely it is\n> just a question of applying a little intellect creatively?\n\nAre you trying to repeat fail of Apple's / MacOS / HFS+ filesystem\ndata/resource forks, and Microsoft's Alternate Data Streams in git? :-)\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"133861","messageId":"2cfc40321002070141y36f62679id6ce72f924a635de@mail.gmail.com","threadId":"22545","inReplyTo":"m363699zn4.fsf@localhost.localdomain","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2010-02-07T09:41:02Z","receivedAt":"2010-02-07T09:41:02Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"On Sun, Feb 7, 2010 at 8:15 PM, Jakub Narebski <jnareb@gmail.com> wrote:\n> Jon Seymour <jon.seymour@gmail.com> writes:\n>\n> [cut]\n>\n>> As I see it, the existing use of notes is a special instance of a more\n>> general metadata capability in which the metadata is constrained to be\n>> a single blob. If notes continued to be constrained in this way, there\n>> is no reason to change anything with respect to its current userspace\n>> behaviour. That said, most of the plumbing which enabled notes could\n>> be generalized to enable the arbitrary tree case [ which admittedly, I\n>> have yet to sell successfully !]\n>>\n>> In one sense, there is a sense in the merge issue doesn't exist. When\n>> the maintainer publishes a tag no-one expects to have to deal with\n>> downstream conflicting definitions of the tag. Likewise, if the\n>> maintainer were to publish the /man and /html metadata trees (per my\n>> previous example) for a release tag, anyone who received\n>> /refs/metadata/doc would expect to receive the metadata trees as\n>> published by the maintainer. Anyone who didn't wouldn't have to pull\n>> /refs/metadata/doc.\n>>\n>> I can see there are use cases where multiple parties might want to\n>> contribute metadata and I do not currently have a good solution to\n>> that problem, but that is not to say there isn't one - surely it is\n>> just a question of applying a little intellect creatively?\n>\n> Are you trying to repeat fail of Apple's / MacOS / HFS+ filesystem\n> data/resource forks, and Microsoft's Alternate Data Streams in git? :-)\n>\n\nNo I am not. I don't see why a metadata proposal is any more exposed\nto subversive payloads than say, use of git merge -s ours [ a\nsubversive payload could be made reachable from a commit that\notherwise merges in favour of the legitimate source - who would know?\n]\n\nReally, I can't see why the rationale that makes a single blob used\nfor extending a commit message justified can't be used to justify\nassociating a metadata tree of arbitrary complexity to an arbitrary\nsha1 object. What makes maintaining a mapping to a single blob\nacceptable but maintaining a mapping to a tree unacceptable? Is there\nreally any fundamental difference?\n\njon.\n"},{"id":"133863","messageId":"2cfc40321002070215m79a8da09r2c55dbf1ed74a3ad@mail.gmail.com","threadId":"22545","inReplyTo":"2cfc40321002070141y36f62679id6ce72f924a635de@mail.gmail.com","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2010-02-07T10:15:29Z","receivedAt":"2010-02-07T10:15:29Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"To explain a little further why the metadata concept is different to\nthe resource fork or alternate data stream concept.\n\nThese two concepts were based on the idea of associating metadata with\nthe name of the resource and preserving the metadata along with the\nresource as the resource evolved.\n\nThis is not the intent with the metadata concept. Rather, the idea is\nto annotate the content (whether it be a commit, tree or blob) with\nother content (a tree in the general metadata case, or a blob in the\ngit notes case)\n\nThe use cases I have in mind relate to caching \"expensive (or\nimpractical) to re-derive\" results from an input. So, for example,\nstoring /man and /html  trees for a given commit in a metadata commit\ncalled \"refs/metadata/doc\" would be one case. Storing the foreign SCM\nrevision id that a git repo was pushed into would be another [ storing\nit the commit message isn't an option because the commit has already\nhappened before the push ]. [ And granted: git notes can already be\nused for this scenario ].\n\njon.\n\nOn Sun, Feb 7, 2010 at 8:41 PM, Jon Seymour <jon.seymour@gmail.com> wrote:\n> On Sun, Feb 7, 2010 at 8:15 PM, Jakub Narebski <jnareb@gmail.com> wrote:\n>> Jon Seymour <jon.seymour@gmail.com> writes:\n>>\n>> [cut]\n>>\n>>> As I see it, the existing use of notes is a special instance of a more\n>>> general metadata capability in which the metadata is constrained to be\n>>> a single blob. If notes continued to be constrained in this way, there\n>>> is no reason to change anything with respect to its current userspace\n>>> behaviour. That said, most of the plumbing which enabled notes could\n>>> be generalized to enable the arbitrary tree case [ which admittedly, I\n>>> have yet to sell successfully !]\n>>>\n>>> In one sense, there is a sense in the merge issue doesn't exist. When\n>>> the maintainer publishes a tag no-one expects to have to deal with\n>>> downstream conflicting definitions of the tag. Likewise, if the\n>>> maintainer were to publish the /man and /html metadata trees (per my\n>>> previous example) for a release tag, anyone who received\n>>> /refs/metadata/doc would expect to receive the metadata trees as\n>>> published by the maintainer. Anyone who didn't wouldn't have to pull\n>>> /refs/metadata/doc.\n>>>\n>>> I can see there are use cases where multiple parties might want to\n>>> contribute metadata and I do not currently have a good solution to\n>>> that problem, but that is not to say there isn't one - surely it is\n>>> just a question of applying a little intellect creatively?\n>>\n>> Are you trying to repeat fail of Apple's / MacOS / HFS+ filesystem\n>> data/resource forks, and Microsoft's Alternate Data Streams in git? :-)\n>>\n>\n> No I am not. I don't see why a metadata proposal is any more exposed\n> to subversive payloads than say, use of git merge -s ours [ a\n> subversive payload could be made reachable from a commit that\n> otherwise merges in favour of the legitimate source - who would know?\n> ]\n>\n> Really, I can't see why the rationale that makes a single blob used\n> for extending a commit message justified can't be used to justify\n> associating a metadata tree of arbitrary complexity to an arbitrary\n> sha1 object. What makes maintaining a mapping to a single blob\n> acceptable but maintaining a mapping to a tree unacceptable? Is there\n> really any fundamental difference?\n>\n> jon.\n>\n"},{"id":"133880","messageId":"7vsk9cdgpx.fsf@alter.siamese.dyndns.org","threadId":"22545","inReplyTo":"20100207050255.GA17049@coredump.intra.peff.net","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-07T18:48:58Z","receivedAt":"2010-02-07T18:48:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> ... I think you would do better\n> to simply store a tree sha1 inside the note blob, and callers who were\n> interested in the tree contents could then dereference it and examine as\n> they saw fit.  The only caveat is that you need some way of telling git\n> that the referenced trees are reachable and not to be pruned.\n\nThanks for a good summary.  To paraphrase the idea, for the \"pre-built\nbinaries\" use case, I could update the dodoc.sh script (in 'todo'---that\nis what autobuilds the html and man documentation and updates the\ncorresponding branches at k.org when I push things out to the master\nbranch) to add a note to the commit from 'master' the docs are generated\nfrom, and the note would say which commits on html and man branches\ncorrespond to that commit.  That way, the referenced \"trees\" are of course\nprotected because they are reachable from html/man refs.\n\nRight?\n"},{"id":"133882","messageId":"20100207191836.GA3185@coredump.intra.peff.net","threadId":"22545","inReplyTo":"7vsk9cdgpx.fsf@alter.siamese.dyndns.org","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-02-07T19:18:36Z","receivedAt":"2010-02-07T19:18:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 07, 2010 at 10:48:58AM -0800, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> > ... I think you would do better\n> > to simply store a tree sha1 inside the note blob, and callers who were\n> > interested in the tree contents could then dereference it and examine as\n> > they saw fit.  The only caveat is that you need some way of telling git\n> > that the referenced trees are reachable and not to be pruned.\n> \n> Thanks for a good summary.  To paraphrase the idea, for the \"pre-built\n> binaries\" use case, I could update the dodoc.sh script (in 'todo'---that\n> is what autobuilds the html and man documentation and updates the\n> corresponding branches at k.org when I push things out to the master\n> branch) to add a note to the commit from 'master' the docs are generated\n> from, and the note would say which commits on html and man branches\n> correspond to that commit.  That way, the referenced \"trees\" are of course\n> protected because they are reachable from html/man refs.\n> \n> Right?\n\nYeah, I think that would work fine. I guess there are cases, though,\nwhere somebody might not be keeping a linear history of noted trees in a\nseparate ref (the way you keep html/man refs). In which case they would\nhave to deal with the reachability problem separately. I can't think of\nan example off the top of my head, though.\n\n-Peff\n"},{"id":"133883","messageId":"20100207193320.GB3185@coredump.intra.peff.net","threadId":"22545","inReplyTo":"2cfc40321002062136q64f832aesd979c9cb22f3612@mail.gmail.com","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-02-07T19:33:20Z","receivedAt":"2010-02-07T19:33:20Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 07, 2010 at 04:36:59PM +1100, Jon Seymour wrote:\n\n> As I see it, the existing use of notes is a special instance of a more\n> general metadata capability in which the metadata is constrained to be\n> a single blob. If notes continued to be constrained in this way, there\n> is no reason to change anything with respect to its current userspace\n> behaviour. That said, most of the plumbing which enabled notes could\n> be generalized to enable the arbitrary tree case [ which admittedly, I\n> have yet to sell successfully !]\n\nI do agree that storing trees is a natural generalization of the current\nnotes implementation. Callers have to be made aware that they may see\ntrees, of course, but you could probably \"demote\" trees into their\nrepresentative sha1s for callers who were interested only in a blob\nform.\n\nBut what I am concerned with is that generalizing may violate some\nassumptions made about how notes work. Notes trees can re-balance\nthemselves to some degree, I thought (though I am pretty out of the loop\non current notes developments). So during merges we need to normalize\ntree representations (though we probably already need to do that for the\nblob case). We would also need to do some magic with rename detection\nduring merges.  You would probably want rename detection _within_ a tree\nstored as a note for a particular commit, but not between notes stored\nfor different commits.\n\nOr perhaps you would not even want to do a tree-merge between notes at\nall, and would rather see a conflict if two people noted two different\ntrees. This would make sense to me if you were doing something like\nnoting a build setup. If I note that commit X builds with a tree\npointing to version Y of the build tools, and you note that it builds\nwith version Z of the build tools, what should happen when we merge our\nnotes? I can imagine wanting a conflict, and resolving it to Y or Z\n(perhaps whichever is more desirable). I can also see resolving it to Y\n_and_ Z (iow, treating it like a list). But doing a merge on the two\ntrees of build tools (which are presumably somewhat immutable) is\nprobably not helpful.\n\nWhich to me argues in favor of adding the extra level of indirection.\nThe note should store the tree sha1, and those who want to treat it as a\ntree can do so. Rename and merge issues just go away, as they operate on\nthe tree sha1 and not on the tree itself. And of course the\nrepresentation is just an implementation detail; you could still make a\n\"git metadata\" wrapper to transparently store trees from the user's\nperspective.\n\nThe only complication is that git doesn't know to follow those sha1s for\nreachability analysis. In some cases that won't matter (like Junio's\nhtml/man example), but I suspect in some it will. Perhaps there is some\nway to flag the note entry as \"this stores a sha1 that should be\nfollowed by fsck, but not otherwise dereferenced\".\n\nI dunno. That is all just thinking out loud. It would help if we had\nsome really detailed concrete examples of notes being used in practice.\n\n-Peff\n"},{"id":"133886","messageId":"7v8wb4aj4m.fsf@alter.siamese.dyndns.org","threadId":"22545","inReplyTo":"20100207193320.GB3185@coredump.intra.peff.net","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-07T20:25:13Z","receivedAt":"2010-02-07T20:25:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Or perhaps you would not even want to do a tree-merge between notes at\n> all, and would rather see a conflict if two people noted two different\n> trees.\n\nI've been thinking about the merge issues, and am starting to suspect that\nwe might want a merge strategy quite drastically different even for blob\ncases.  That is one of the reasons why I don't want to see us muddy the\nissues by introducing even more complex \"tree\" case.\n\nAnybody working in the same project can start 'notes' tree with his or her\nown root.  That is the normal use case for annotating commits for your own\nuse.  For merges inside the history of primary contents that people try to\ncollaborate to advance, three-way merge pivoting on a common ancestor is a\nnatural way to reach a satisfactory result.  In notes namespace, on the\nother hand, the norm is to simply overlay the notes trees, adjusting for\nthe fan-out.  You annotated that commit I was not interested in, while I\nannotated this commit you weren't interested in.  We have our notes in the\nend result, and both of us are happy.  If we happen to have annotated the\nsame commit without knowing what the other was doing, then there is no\nsane consolidation---in the most typical case, we would want to keep both,\nperhaps concatenating them together.  Textual merge becomes the exception\nthat triggers two \"notes\" histories happened to have forked from the same\nroot somehow.\n\nAnd for that most typical use case, I suspect even the current \"notes on\nany and all commits for a single purpose are thrown into a one _bag_ that\nis a notes tree, and the growth of that bag is made into a history\" model\ncaptures sets of notes that is too wide.\n\nSuppose Alice, Bob and I are involved in a project, and we annotate\ncommits for some shared purpose (say, tracking regressions).  Alice and\nBob may independently annotate overlapping set of commits (and hopefully\nthey have shared root for their notes history as they are collaborating),\nand they may even be working together on the same issue, but I may not be\ninvolved in the area.  What happens when I pull from Alice and Bob and get\nconflicts in notes they produced, especially the only reason I was\ninterested was because they have new things to say about commits that I am\ninterested in?\n\nYou can end up with conflicts in areas you are not familiar with but Alice\nand Bob are in charge of even in the primary content space, but there is a\nfundamental difference of this type of conflict in the notes space, I\nthink.  The set of contents in the primary content space are supposed to\nmake a consistent whole, and there is a topic branch workflow to partition\nthe work to allow me to easily kick the merge back to them (i.e. I can\ntell Alice and Bob to resolve the conflicts between themselves and trust\nthat what they do between them do not touch outside of their area) without\ngetting blocked.  I don't see a clear workflow to resolve this in the\nnotes space, especially with the set of operations the current \"git notes\"\n(and obvious and straightforward enhancements of what it does).  At least\nnot yet.\n\nIt's like \"keeping track of /etc\" (or \"your home directory\").  It is a\nmisguided thing to do because you are throwing in records of the states of\ntotally unrelated things into a single history (e.g. \"Why does it matter I\nadded new user frotz to /etc/passwd before I futzed with my sendmail\nconfiguration?  ---It shouldn't matter; there shouldn't be ancestry\nrelationships between these two changes\").  I somehow feel that keeping\ntrack of the \"growth of the bag of annotations to any and all commits\" in\na single history may be making the same mistake.\n"},{"id":"133896","messageId":"201002072346.22627.johan@herland.net","threadId":"22545","inReplyTo":"20100207050255.GA17049@coredump.intra.peff.net","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2010-02-07T22:46:22Z","receivedAt":"2010-02-07T22:46:22Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Sunday 07 February 2010, Jeff King wrote:\n> I think I may have been the one to suggest trees or notes at one point.\n> But let me clarify that this is not exactly what the OP is proposing in\n> this thread.\n> \n> My suggestion was that some use cases may have many key/value pairs of\n> notes for a single sha1. We basically have two options:\n> \n>   1. store each in a separate notes ref, with each sha1 mapping to\n>      a blob. The note \"name\" is the name of the ref.\n> \n>   2. store notes in a single notes ref, with each sha1 mapping to a\n>      tree with named sub-notes. The note \"name\" is the combination of\n>      ref-name and tree entry name.\n> \n> The advantage of (1) is that notes are not bound tightly to each other.\n> I can distribute the notes tree for one \"name\" independent of the\n> others.  The advantage of (2) is that it is faster and smaller. In (1),\n> each note has a separate index, and we must traverse each note index\n> separately.\n> \n> In practice, I would expect to use (1) for logically separate datasets.\n> For example, automatic bug-tracking notes would go in a different ref\n> from human annotations. But I would expect to use (2) if I had, say, 5\n> different pieces of bug tracking information and I wanted an easy way to\n> refer to them individually.\n> \n> And a specialized merge for that is straightforward. In the simplest\n> case, you simply say \"notes of this ref are tree-type, or they are\n> blob-type\" and then you have no merge problems. But if you want to get\n> fancy, you can say that a conflict between \"sha1/blob\" and\n> \"sha1/tree/key\" should automatically \"promote\" the first one into\n> \"sha1/tree/default\" or some other canonical name.\n> \n> Note that all of this is my pie-in-the-sky \"here is what I was thinking\n> of when I looked at notes a long time ago\". I don't care strongly if it\n> gets implemented or not at this point; I just wanted to add some context\n> to what Johan had in his notes todo list (or maybe I am wrong, and what\n> is in his todo list was based on something totally different said by\n> somebody else, and I have just confused the issue more. :) ).\n\nNo, My TODO item was indeed based on your suggestion (although poorly \nrepresented by me, both in the TODO list, and in my original answer to Jon). \nHowever, note that I don't feel this specific itch myself, so I'm unlikely \nto scratch it.\n\n> With respect to the idea of storing an arbitrary tree, I agree it is\n> probably too complex with respect to merging. In addition, it makes\n> things like \"git log --format=%N\" confusing. I think you would do better\n> to simply store a tree sha1 inside the note blob, and callers who were\n> interested in the tree contents could then dereference it and examine as\n> they saw fit.  The only caveat is that you need some way of telling git\n> that the referenced trees are reachable and not to be pruned.\n\nAgreed. Arbitrary trees as notes objects is probably not a good idea.\n\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"133922","messageId":"831vgw5vr5.fsf@torus.sehlabs.com","threadId":"22545","inReplyTo":"7v8wb4aj4m.fsf@alter.siamese.dyndns.org","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Steven E. Harris","fromEmail":"seh@panix.com","sentAt":"2010-02-08T02:03:42Z","receivedAt":"2010-02-08T02:03:42Z","isPatch":false,"sender":{"key":"seh@panix.com","avatar":"https://gravatar.com/avatar/d59ec0f7c010ee73cd67db381a5b865206fed17fd4278f12cdb6277db30033fc?d=mp&s=160"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> It's like \"keeping track of /etc\" (or \"your home directory\").  It is a\n> misguided thing to do because you are throwing in records of the\n> states of totally unrelated things into a single history.\n\nI've recently tried doing this again with Git, so this comment piqued my\ninterest. (That is, tracking changes to my various configuration files.)\nI agree that browsing the history in toto is jarring, though the history\nof a particular file may be telling.\n\nIs there an alternative you'd recommend?\n\n-- \nSteven E. Harris\n"},{"id":"134118","messageId":"20100210050902.GD28526@coredump.intra.peff.net","threadId":"22545","inReplyTo":"7v8wb4aj4m.fsf@alter.siamese.dyndns.org","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-02-10T05:09:02Z","receivedAt":"2010-02-10T05:09:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 07, 2010 at 12:25:13PM -0800, Junio C Hamano wrote:\n\n> Suppose Alice, Bob and I are involved in a project, and we annotate\n> commits for some shared purpose (say, tracking regressions).  Alice and\n> Bob may independently annotate overlapping set of commits (and hopefully\n> they have shared root for their notes history as they are collaborating),\n> and they may even be working together on the same issue, but I may not be\n> involved in the area.  What happens when I pull from Alice and Bob and get\n> conflicts in notes they produced, especially the only reason I was\n> interested was because they have new things to say about commits that I am\n> interested in?\n\nHmm. OK, I see the point of Jakub's message a bit more now. You want to\ncreate a new view, inconsistent with that of either Alice or Bob (that\nis, you have taken snippets of each's state, but you cannot in good\nfaith represent this as a history merge, because your state should not\nsupersede either of theirs).\n\nThe standard way to do such a thing in git is to create a new, alternate\nhistory through cherry-picking or rebasing. So I suspect we could do\nsomething like:\n\n  1. git notes pull alice\n\n     We fast-forward (or do the trivial merge) with Alice's work.\n\n  2. git notes pull --ignore-conflicts bob\n\n     We try to merge Bob's work and see that there are conflicts. So we\n     iterate through refs/notes..bob/notes, cherry-picking each one that\n     applies cleanly and ignoring the rest.\n\nAnd then you're at a state inconsistent with Bob, and a superset of what\nAlice has. And that's what your history represents, too: you've branched\nbut done some of the same things as Bob. At that point you can examine\nyour inconsistent state, and then when you're done, you can either:\n\n  3a. Reset back to your pre-ignore-conflicts state.\n\n  3b. Leave it. When you pull from Bob later, your shared changes will\n      be ignored[1], and you will get the conflicts that you ignored\n      earlier.\n\nIt is perhaps a hacky band-aid to handle notes this way, but it is the\n\"most git\" way of doing it. That is, it uses our standard tools and\npractices.  And when all you have is a hammer... :)  And I really expect\nthe \"I am collaborating with these people, but I want an inconsistent\nview of their history\" to be the exception. Most people would _want_ to\nresolve the conflicts (especially if there is a --cat-conflicts\noption to do it automatically) in a collaboration scenario.\n\n-Peff\n\n[1] Actually because history has diverged, you have the usual cherry\npick problems with merging later. If some note is at state A, then I\ncherry-pick Bob's change to B, then Bob changes it to C and I try to\nmerge with him, from the 3-way merge's perspective we have a conflict,\nbecause nothing in the history says that Bob's change to C meant to\nsupersede my cherry-picked version of his history.\n"},{"id":"134121","messageId":"7vvde5irzz.fsf@alter.siamese.dyndns.org","threadId":"22545","inReplyTo":"20100210050902.GD28526@coredump.intra.peff.net","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-10T05:23:12Z","receivedAt":"2010-02-10T05:23:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Sun, Feb 07, 2010 at 12:25:13PM -0800, Junio C Hamano wrote:\n>\n>> Suppose Alice, Bob and I are involved in a project, and we annotate\n>> commits for some shared purpose (say, tracking regressions).  Alice and\n>> Bob may independently annotate overlapping set of commits (and hopefully\n>> they have shared root for their notes history as they are collaborating),\n>> and they may even be working together on the same issue, but I may not be\n>> involved in the area.  What happens when I pull from Alice and Bob and get\n>> conflicts in notes they produced, especially the only reason I was\n>> interested was because they have new things to say about commits that I am\n>> interested in?\n>\n> Hmm. OK, I see the point of Jakub's message a bit more now. You want to\n> create a new view, inconsistent with that of either Alice or Bob (that\n> is, you have taken snippets of each's state, but you cannot in good\n> faith represent this as a history merge, because your state should not\n> supersede either of theirs).\n\nIn the message you are quoting, I am not interested in creating a narrowed\nview.  If I cannot resolve conflicts between Alice and Bob in a merge in\nthe contents space, I would ask either of them (because they are more\nfamiliar with the area) to do the merge.  I however was unsure if asking\nthe same for merges in the notes space is a reasonable thing to do.\n"},{"id":"134122","messageId":"20100210052902.GA28832@coredump.intra.peff.net","threadId":"22545","inReplyTo":"7vvde5irzz.fsf@alter.siamese.dyndns.org","subject":"Re: A generalization of git notes from blobs to trees - git metadata?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-02-10T05:29:02Z","receivedAt":"2010-02-10T05:29:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 09, 2010 at 09:23:12PM -0800, Junio C Hamano wrote:\n\n> > Hmm. OK, I see the point of Jakub's message a bit more now. You want to\n> > create a new view, inconsistent with that of either Alice or Bob (that\n> > is, you have taken snippets of each's state, but you cannot in good\n> > faith represent this as a history merge, because your state should not\n> > supersede either of theirs).\n> \n> In the message you are quoting, I am not interested in creating a narrowed\n> view.  If I cannot resolve conflicts between Alice and Bob in a merge in\n> the contents space, I would ask either of them (because they are more\n> familiar with the area) to do the merge.  I however was unsure if asking\n> the same for merges in the notes space is a reasonable thing to do.\n\nNo, I don't see a problem with asking them to do it. If you are all\ncollaborating as a group, it is something they will need to do\neventually anyway. If they are not, and you are an intermediary, you are\neventually going to share Alice's history with Bob and vice versa. So\nyou pull from Alice, then say to Bob: \"I have some history but I'm not\nsure of the correct merge. Pull from me and merge please\". The only real\nproblem is if you _never_ want to share the history between the two of\nthem. In that case, I think you should keep two parallel branches of\nhistory (refs/notes/alice and refs/notes/bob), and then squash the trees\nat run-time (either concatenating them, or favoring one over the other\nin the case of conflicts).\n\n-Peff\n"}]}