{"thread":{"id":"22705","subject":"Using trees for metatagging","startedAt":"2010-02-18T04:12:40Z","lastAt":"2010-02-19T00:43:08Z","messageCount":9,"participants":["martin f krafft","Avery Pennarun","Johan Herland"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"134924","messageId":"20100218041240.GA4127@lapse.rw.madduck.net","threadId":"22705","inReplyTo":null,"subject":"Using trees for metatagging","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2010-02-18T04:12:40Z","receivedAt":"2010-02-18T04:12:40Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"Git's object store uses trees mainly to represent a hierarchical\nfilesystem. It occurs to me that you could layer additional\nhierarchies on top — specifically, you could use it to track subsets\nof files, i.e. \"tagging\".\n\nFor instance you want some sort of representation for \"the set of\nfiles that need review\". You /could/ create a new tree and reference\nall files in that set as children. Now if you wanted to find out\nwhat to review, you'd list the children of this tree. After\nreviewing a file, you write a new tree with the set less that file's\nref.. Obviously, if you made changes to the file, it should be\nreconnected to all other trees that referenced it.\n\nI have a couple of questions about this:\n\n1. Does Git provide plumbing for me to find out which trees\n reference a given blob? If not, I will have to iterate all trees\n and record which ones have a given message as a child.\n\n2. Is there a way you can fathom by which unlinking a blob from the\n main hierarchy also causes it to be unlinked from this meta tree\n I am speaking of as well? Similarly, if a blob is rewritten, how\n could I make sure it replaces the old blob in all referencing\n trees?\n\n3. Am I right in assuming that I'd have to track a completely\n seperate ancestry for this tree, that is create e.g. a commit\n object, point refs/metatrees/mytree to it, and reference the tree\n from the commit?\n\n4. Since this hierarchy is not really to be mapped into the\n filesystem, how would one resolve conflicts when merging\n ancestries? Of course it would be nice if I could check out this\n meta tree into the filesystem, make changes, and be assured that\n new blobs replace old blobs in other referencing trees, as per\n (2.), but that's a pipedream maybe.\n\n5. Do you know of similar efforts? Are there must-reads out there,\n apart from the design of Git?\n\nThank you,\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \nkill ugly radio\n -- frank zappa\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"134990","messageId":"32541b131002181057gf27538ybf09dbf80b8dbce8@mail.gmail.com","threadId":"22705","inReplyTo":"20100218041240.GA4127@lapse.rw.madduck.net","subject":"Re: Using trees for metatagging","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-02-18T18:57:18Z","receivedAt":"2010-02-18T18:57:18Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Wed, Feb 17, 2010 at 11:12 PM, martin f krafft wrote:\n> Git's object store uses trees mainly to represent a hierarchical\n> filesystem. It occurs to me that you could layer additional\n> hierarchies on top — specifically, you could use it to track subsets\n> of files, i.e. \"tagging\".\n\nI think what you *really* want here is to create a branch containing a\nsingle file, which is the list of all the files you want to review.\nThen when you're done reviewing a file, delete it from your list and\ncommit it. Then just check out that file list branch in another clone\nof your repository and manipulate it however you like.\n\nSorry to be boring.\n\n> 1. Does Git provide plumbing for me to find out which trees\n>   reference a given blob? If not, I will have to iterate all trees\n>   and record which ones have a given message as a child.\n\nNo, you will have to iterate. Also, if *other* people have trees\nreferencing that blob in *their* repositories, you won't know, so you\ncan never be sure that you've successfully found all objects in the\nuniverse that refer to a particular blob.\n\n> 2. Is there a way you can fathom by which unlinking a blob from the\n>   main hierarchy also causes it to be unlinked from this meta tree\n>   I am speaking of as well? Similarly, if a blob is rewritten, how\n>   could I make sure it replaces the old blob in all referencing\n>   trees?\n\nblobs cannot replace other blobs. And a tree that contains a\nparticular blob (indexed by sha1) will never *not* contain that blob,\nbecause the identity of that tree is based on the identitity of the\nblobs it contains. You can create a new tree that doesn't contain the\nblob, but the commit that contained the old tree will never contain\nthe new tree. You would have to create a new commit that contains the\nnew tree, but any commits based on your old commit will never be based\non your new commit. And so on.\n\nThat's just the way content-addressed storage works. Sounds like you\nneed to read more about it.\n\nHave fun,\n\nAvery\n"},{"id":"135009","messageId":"201002182200.09100.johan@herland.net","threadId":"22705","inReplyTo":"20100218041240.GA4127@lapse.rw.madduck.net","subject":"Re: Using trees for metatagging","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2010-02-18T21:00:08Z","receivedAt":"2010-02-18T21:00:08Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Thursday 18 February 2010, martin f krafft wrote:\n> Git's object store uses trees mainly to represent a hierarchical\n> filesystem. It occurs to me that you could layer additional\n> hierarchies on top — specifically, you could use it to track subsets\n> of files, i.e. \"tagging\".\n>\n> For instance you want some sort of representation for \"the set of\n> files that need review\". You /could/ create a new tree and reference\n> all files in that set as children. Now if you wanted to find out\n> what to review, you'd list the children of this tree. After\n> reviewing a file, you write a new tree with the set less that file's\n> ref.. Obviously, if you made changes to the file, it should be\n> reconnected to all other trees that referenced it.\n>\n> I have a couple of questions about this:\n>\n> 1. Does Git provide plumbing for me to find out which trees\n> reference a given blob? If not, I will have to iterate all trees\n> and record which ones have a given message as a child.\n>\n> 2. Is there a way you can fathom by which unlinking a blob from the\n> main hierarchy also causes it to be unlinked from this meta tree\n> I am speaking of as well? Similarly, if a blob is rewritten, how\n> could I make sure it replaces the old blob in all referencing\n> trees?\n>\n> 3. Am I right in assuming that I'd have to track a completely\n> seperate ancestry for this tree, that is create e.g. a commit\n> object, point refs/metatrees/mytree to it, and reference the tree\n> from the commit?\n>\n> 4. Since this hierarchy is not really to be mapped into the\n> filesystem, how would one resolve conflicts when merging\n> ancestries? Of course it would be nice if I could check out this\n> meta tree into the filesystem, make changes, and be assured that\n> new blobs replace old blobs in other referencing trees, as per\n> (2.), but that's a pipedream maybe.\n>\n> 5. Do you know of similar efforts? Are there must-reads out there,\n> apart from the design of Git?\n\nTake a look at the (relatively) new notes feature. (See the jh/notes \nseries in 'pu' and various recent discussions on this mailing list.) \nGit notes probably won't satisfy the exact requirements you list above, \nbut it _does_ tackle some parallel issues (e.g. how to maintain a tree \nthat is not checked out, storing metadata associated with Git objects, \netc.). If you take a step back and reconsider your original problem, \nyou might find that it's solvable by using commit notes.\n\nFor example, you could add a simple note to each blob that has been \nreviewed, on the refs/notes/reviewed notes ref. You could then write a \nsimple script (using \"git notes list\") that lists all blobs (i.e. \nfiles) without a corresponding note in refs/notes/reviewed.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"135018","messageId":"20100218225323.GK9756@lapse.rw.madduck.net","threadId":"22705","inReplyTo":"32541b131002181057gf27538ybf09dbf80b8dbce8@mail.gmail.com","subject":"Re: Using trees for metatagging","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2010-02-18T22:53:23Z","receivedAt":"2010-02-18T22:53:23Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Avery Pennarun [2010.02.19.0757 +1300]:\n> > 1. Does Git provide plumbing for me to find out which trees\n> >   reference a given blob? If not, I will have to iterate all\n> > trees   and record which ones have a given message as a child.\n> \n> No, you will have to iterate. Also, if *other* people have trees\n> referencing that blob in *their* repositories, you won't know, so\n> you can never be sure that you've successfully found all objects\n> in the universe that refer to a particular blob.\n\nThe idea is obviously that you could merge ancestries and thus\npropagate all those changes.\n\n> > 2. Is there a way you can fathom by which unlinking a blob from the\n> >   main hierarchy also causes it to be unlinked from this meta tree\n> >   I am speaking of as well? Similarly, if a blob is rewritten, how\n> >   could I make sure it replaces the old blob in all referencing\n> >   trees?\n> \n> blobs cannot replace other blobs.\n\nIt was a shortcut on my behalf. I meant that a new tree is written\nwith the ref to the old blob removed and the ref to the new blob\nadded.\n\n> And a tree that contains a particular blob (indexed by sha1) will\n> never *not* contain that blob, because the identity of that tree\n> is based on the identitity of the blobs it contains. You can\n> create a new tree that doesn't contain the blob, but the commit\n> that contained the old tree will never contain the new tree. You\n> would have to create a new commit that contains the new tree, but\n> any commits based on your old commit will never be based on your\n> new commit. And so on.\n\nRight, this is the basis of merging. I understand all this.\nI suppose I didn't express myself clearly enough.\n\nSo I am trying to figure out:\n\n1. how to create new trees for all trees that reference a blob that\n is superseeded by a new blob in some sort of scalable way;\n\n2. how to maintain a separate ancestry of commits pointing to those\n trees in a way to be able to harness Git's merging capabilities.\n\nIs this clearer?\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"alle vorurteile kommen aus den eingeweiden.\"\n - friedrich nietzsche\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"135019","messageId":"20100218225758.GL9756@lapse.rw.madduck.net","threadId":"22705","inReplyTo":"201002182200.09100.johan@herland.net","subject":"Re: Using trees for metatagging","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2010-02-18T22:57:59Z","receivedAt":"2010-02-18T22:57:59Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Johan Herland [2010.02.19.1000 +1300]:\n> Take a look at the (relatively) new notes feature. (See the\n> jh/notes series in 'pu' and various recent discussions on this\n> mailing list.) Git notes probably won't satisfy the exact\n> requirements you list above, but it _does_ tackle some parallel\n> issues (e.g. how to maintain a tree that is not checked out,\n> storing metadata associated with Git objects, etc.). If you take\n> a step back and reconsider your original problem, you might find\n> that it's solvable by using commit notes.\n> \n> For example, you could add a simple note to each blob that has\n> been reviewed, on the refs/notes/reviewed notes ref. You could\n> then write a simple script (using \"git notes list\") that lists all\n> blobs (i.e. files) without a corresponding note in\n> refs/notes/reviewed.\n\nI am aware of notes, but so far I stayed away from them, simply\nbecause it seems hackish to represent tag trees as text when dealing\nwith a tool that is essentially all about trees and refs.\n\nCan I use notes to append information to blobs and trees, or just\ncommit objects?\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"without music, life would be a mistake.\"\n - friedrich nietzsche\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"135020","messageId":"32541b131002181506v1d28526cpf8491994868272af@mail.gmail.com","threadId":"22705","inReplyTo":"20100218225758.GL9756@lapse.rw.madduck.net","subject":"Re: Using trees for metatagging","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-02-18T23:06:42Z","receivedAt":"2010-02-18T23:06:42Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Thu, Feb 18, 2010 at 5:57 PM, martin f krafft wrote:\n> I am aware of notes, but so far I stayed away from them, simply\n> because it seems hackish to represent tag trees as text when dealing\n> with a tool that is essentially all about trees and refs.\n\nI think you're using the wrong definition of hacky vs. elegant. A\n\"tree\" is really just a file containing a list of objects. A \"ref\" is\njust a file that contains an object id.\n\nSo checking in a file that contains a list of object ids (or\nfilenames) is perfectly appropriate.\n\nHave fun,\n\nAvery\n"},{"id":"135023","messageId":"20100218232523.GD22371@lapse.rw.madduck.net","threadId":"22705","inReplyTo":"32541b131002181506v1d28526cpf8491994868272af@mail.gmail.com","subject":"Re: Using trees for metatagging","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2010-02-18T23:25:23Z","receivedAt":"2010-02-18T23:25:23Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Avery Pennarun [2010.02.19.1206 +1300]:\n> > I am aware of notes, but so far I stayed away from them, simply\n> > because it seems hackish to represent tag trees as text when dealing\n> > with a tool that is essentially all about trees and refs.\n> \n> I think you're using the wrong definition of hacky vs. elegant. A\n> \"tree\" is really just a file containing a list of objects. A \"ref\" is\n> just a file that contains an object id.\n>\n> So checking in a file that contains a list of object ids (or\n> filenames) is perfectly appropriate.\n\nIndeed. But Git provides a lot of tools to manipulate all those,\nwhich I would not be able to reuse in the text-file approach.\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"good advice is something a man gives\n when he is too old to set a bad example.\n -- la rouchefoucauld\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"135024","messageId":"32541b131002181532h79d3250fid5984f46f82b71f9@mail.gmail.com","threadId":"22705","inReplyTo":"20100218232523.GD22371@lapse.rw.madduck.net","subject":"Re: Using trees for metatagging","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-02-18T23:32:31Z","receivedAt":"2010-02-18T23:32:31Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Thu, Feb 18, 2010 at 6:25 PM, martin f krafft wrote:\n> also sprach Avery Pennarun [2010.02.19.1206 +1300]:\n>> So checking in a file that contains a list of object ids (or\n>> filenames) is perfectly appropriate.\n>\n> Indeed. But Git provides a lot of tools to manipulate all those,\n> which I would not be able to reuse in the text-file approach.\n\nBut you're talking about using a nonstandard approach anyway (unless\nyou use git-notes). So you'd end up rolling your own code with git's\nplumbing anyway, which you can do just as easily with a text file full\nof object ids.\n\nOnce you have an object id, there are plenty of existing git commands\nto do whatever you want.\n\nHave fun,\n\nAvery\n"},{"id":"135027","messageId":"201002190143.08593.johan@herland.net","threadId":"22705","inReplyTo":"20100218225758.GL9756@lapse.rw.madduck.net","subject":"Re: Using trees for metatagging","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2010-02-19T00:43:08Z","receivedAt":"2010-02-19T00:43:08Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Thursday 18 February 2010, martin f krafft wrote:\n> Can I use notes to append information to blobs and trees, or just\n> commit objects?\n\nYes, Since patch #2 in the jh/notes series in 'pu'.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"}]}