{"thread":{"id":"35946","subject":"Re: [RFC/PATCH] Supporting non-blob notes","startedAt":"2014-02-24T10:27:36Z","lastAt":"2014-02-24T13:08:06Z","messageCount":2,"participants":["ydirson@free.fr","Johan Herland"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"235238","messageId":"897120266.591794599.1393237656293.JavaMail.root@zimbra39-e7.priv.proxad.net","threadId":"35946","inReplyTo":"OF15344132.B289BF94-ONC1257C89.003921CB-C1257C89.003E8AF2@local","subject":"Re: [RFC/PATCH] Supporting non-blob notes","fromName":"","fromEmail":"ydirson@free.fr","sentAt":"2014-02-24T10:27:36Z","receivedAt":"2014-02-24T10:27:36Z","isPatch":true,"sender":{"key":"ydirson@free.fr","avatar":null},"body":"Johan Herland wrote on 02/24/2014 02:29:10: \n> On Wed, Feb 19, 2014 at 12:10 AM, Duy Nguyen wrote: \n> > On Tue, Feb 18, 2014 at 9:46 PM, Johan Herland wrote: \n> >> On Mon, Feb 17, 2014 at 11:48 AM, wrote: \n> >>> The recent \"git-note -C changes commit type?\" thread \n> >>> ( http://thread.gmane.org/gmane.comp.version-control.git/241950 ) looks \n> >>> like a good occasion to discuss possible uses of non-blob notes. \n> >>> \n> >>> The use-case we're thinking about is the storage of testrun logs as \n> >>> notes (think: being able to justify that a given set of tests were \n> >>> successfully run on a given revision). \n> >> \n> >> I think this is a good use of notes, and organizing the testrun logs \n> >> into a tree of files seems like a natural way to proceed. \n> > \n> > Notes from the previous attempt to store trees as notes (something to \n> > watch out maybe, when you do it again) \n> > \n> > http://article.gmane.org/gmane.comp.version-control.git/197712 \n> \n> Thanks for that link. It is good to see that these issues have been \n> considered/discussed previously. \n\nYes, it sheds some useful light on the problem, thanks. \n\n> I've been thinking about this for a while now, and I find myself \n> agreeing more and more with Junio's argument in the linked thread. \n> \n> I think notes are fundamentally - like file contents from Git's POV - \n> an unstructured stream of bytes. Any real structure in a git note is \n> imposed by the surrounding application/context, and having Git impose \n> its own object model onto the contents of notes would likely be an \n> unnecessary distraction. \n\nOTOH, it looks like a good idea to allow the surrounding application/context \nto benefit from existing infrastructure. I identified so far: \n(i) diffing/grepping trees \n(ii) efficiency of indexing through notes fanout \n(iii) reachability \n(iv) content packing \n\n> In Yann's example, the testrun logs are probably best structured as a \n> hierarchy of files, but that does not necessarily mean that they MUST \n> be stored as a Git tree object (with accompanying sub-trees and \n> blobs). For example, one could imagine many different solutions for \n> storing the testrun logs: \n> \n> (a) Storing the logs statically on some server, and putting the \n> corresponding URL in a notes blob. Reachability is manual/on-demand \n> (be retrieving the URL). \n\nWould require to redo (ii) and (iv) in a way that does not impait (i) \n\n> (b) Storing the logs in a .tar.gz archive, and adding that archive as \n> a blob note. Reachability is implicit/automatic (by unpacking the \n> archive). \n\nInterferes with (i) and (iv), ie. does not allow to benefit from similarity \nbetween the contents of (unpacked) notes. \n\n> (c) Storing the logs on some ref in an external repo, and putting the \n> repo URL + ref in a notes blob. Reachability is manual/on-demand (by \n> cloning/fetching the repo). \n> (d) Storing the logs on some ref/commit in the same repo, and putting \n> the ref/commit name in a notes blob. Reachability depends on the \n> application/user to sync the ref/commit along with the notes. \n\nBetter than (a), but still does not address (ii). \nAnd indeed, my intent was to let the notes live in a separate \"fork\" repo, \nso ordinary users need not fetch the testrun contents systematically with the \ncode. \n\n> (e) Storing the logs in a commit, putting the commit name in a blob \n> note, and then creating/rewriting the notes history to include the \n> commit in its ancestry. Reachability is automatic (i.e.follows the \n> notes), but the application must control/manipulate the notes history. \n\nAnd finally, that one does address all points in my case. \n\n> Whichever of these (or other) solutions is most appropriate depends on \n> the particular application/context, and (from Git's perspective), none \n> of them are inherently superior to any of the other. Even the question \n> of whether testrun logs should or should not be reachable by default, \n> depends on the surrounding application/context. \n\nWouldn't it make sense to mention these possibilities in the git-notes \nmanpage, to help people use the mechanism as intended ? \n\n> Now, the intention of Yann's RFC is to store the testrun logs directly \n> in a notes _tree_. This is not too different from alternative (e) \n> above, in that reachability is automatic. However, instead of having \n> the surrounding application manipulate the notes history to ensure \n> reachability, the RFC would rather teach Git's notes code to \n> accomodate the (likely rather special) case of having a note that is \n> BOTH structured like (or at least easily mapped to) a Git tree object, \n> AND that should be automatically reachable. \n\nIncidently, proposal (e) would allow the use of commits, although \ndoing so would probably cause problems, not all of the children of the \ncommit used as annotation having the same relationship to their parent. \n\nAre you suggesting using a slightly different mechanism than \nthe \"parent\" relationship ? \n\n> Even though there is a certain elegance to storing such a tree object \n> directly as a notes object, there is AFAICS no other inherent \n> advantage (e.g. performance- or functionality-wise) to following that \n> approach. I'm not at all sure that it justifies increasing the \n> complexity of the notes code. \n> \n> Furthermore, considering the RFC's original intention of also making \n> commit and tag objects directly usable as notes, and realizing the \n> fundamental difficulties in teaching Git to handle this (outlined in \n> my previous email in this thread), I must conclude that the simplicity \n> and flexibility of something like alternative (e) above far outweighs \n> the added code complexity to support allowing any object type to be \n> used as a note. \n\nI was just making the RFC too broad, not having any immediate use for \ncommits or tags. You can consider I was just talking about trees :) \n\n> Maybe we should instead consider making it easier to do alternative \n> (e), by providing a command-line option for supplying additional \n> parents to a notes commit? \n\n-- \nYann Dirson - Bertin Technologies / Bertin IT\n"},{"id":"235239","messageId":"CALKQrgfqYxsCPvksDee=C2z9F7M5WKra4BX19Se-1gVF5TFCAg@mail.gmail.com","threadId":"35946","inReplyTo":"897120266.591794599.1393237656293.JavaMail.root@zimbra39-e7.priv.proxad.net","subject":"Re: [RFC/PATCH] Supporting non-blob notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2014-02-24T13:08:06Z","receivedAt":"2014-02-24T13:08:06Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Mon, Feb 24, 2014 at 11:27 AM, wrote:\n> Johan Herland wrote on 02/24/2014 02:29:10:\n>> I've been thinking about this for a while now, and I find myself\n>> agreeing more and more with Junio's argument in the linked thread.\n>>\n>> I think notes are fundamentally - like file contents from Git's POV -\n>> an unstructured stream of bytes. Any real structure in a git note is\n>> imposed by the surrounding application/context, and having Git impose\n>> its own object model onto the contents of notes would likely be an\n>> unnecessary distraction.\n>\n> OTOH, it looks like a good idea to allow the surrounding application/context\n> to benefit from existing infrastructure. I identified so far:\n>\n> (i) diffing/grepping trees\n> (ii) efficiency of indexing through notes fanout\n\nAll of my proposed alternatives store some sort of reference to the\n\"real\" data in a notes object; even when using a tree object directly\nas a note, the notes tree itself only stores a SHA1 reference to the\ntree object. As such, all alternatives (a) through (e) (even including\nyour RFC) benefit from indexing through the notes fanout, and I'm not\nsure what is gained by attaching the \"real\" data more directly to the\nnotes. In all of (a) through (e), the lookup of a specific commit's\ntestrun logs always start with doing a lookup of the notes associated\nwith a given commit. Once that is done, the remainder of the work is\nabout resolving that reference and retrieving the associated resource,\nWhether the consists of loading an HTTP URL, fetching a remote Git\nrepo, or looking up a local tree object is ultimately an\nimplementation detail, and does not affect the indexing itself.\n\n> (iii) reachability\n> (iv) content packing\n\nThese four criteria/requirements apply to your specific use case, but\nthey do not necessarily apply to _all_ use cases. I can easily imagine\na slightly different scenario: For example, a company setting with\nhighly-available internal servers, and where testrun logs are\nprimarily interesting to a small subset of users (e.g. most developers\nonly look at them very occasionally). Now assume there is already a\n(third-party) system in place for archiving and indexing the testrun\nlogs (i.e. providing (i), (ii) and (iv)), and direct reachability\n(iii) is not desired as including the testrun logs in the repo would\nadd nothing but bloat for most users. In this scenario, simply adding\na note with the appropriate URL to the third-party service would be a\nsufficient and preferable solution.\n\n>> In Yann's example, the testrun logs are probably best structured as a\n>> hierarchy of files, but that does not necessarily mean that they MUST\n>> be stored as a Git tree object (with accompanying sub-trees and\n>> blobs). For example, one could imagine many different solutions for\n>> storing the testrun logs:\n>>\n>> (a) Storing the logs statically on some server, and putting the\n>> corresponding URL in a notes blob. Reachability is manual/on-demand\n>> (be retrieving the URL).\n>\n> Would require to redo (ii) and (iv) in a way that does not impait (i)\n>\n>> (b) Storing the logs in a .tar.gz archive, and adding that archive as\n>> a blob note. Reachability is implicit/automatic (by unpacking the\n>> archive).\n>\n> Interferes with (i) and (iv), ie. does not allow to benefit from similarity\n> between the contents of (unpacked) notes.\n>\n>> (c) Storing the logs on some ref in an external repo, and putting the\n>> repo URL + ref in a notes blob. Reachability is manual/on-demand (by\n>> cloning/fetching the repo).\n>> (d) Storing the logs on some ref/commit in the same repo, and putting\n>> the ref/commit name in a notes blob. Reachability depends on the\n>> application/user to sync the ref/commit along with the notes.\n>\n> Better than (a), but still does not address (ii).\n> And indeed, my intent was to let the notes live in a separate \"fork\" repo,\n> so ordinary users need not fetch the testrun contents systematically with the\n> code.\n\nJust to clarify, my alternatives (except for (e) below) were not\nintended to satisfy the exact criteria for your use case, but only to\ndemonstrate that there exist a variety of solutions for a variety of\nslightly different problems. When we consider adding significant\ncomplexity to the notes code, we must justify that with real and\ntangible benefits, not only for your exact use case, but preferably\nalso for a larger group of related use cases. So far I don't see how\nallowing the direct use of tree objects as notes benefit more than\nyour specific use case...\n\n>> (e) Storing the logs in a commit, putting the commit name in a blob\n>> note, and then creating/rewriting the notes history to include the\n>> commit in its ancestry. Reachability is automatic (i.e.follows the\n>> notes), but the application must control/manipulate the notes history.\n>\n> And finally, that one does address all points in my case.\n>\n>> Whichever of these (or other) solutions is most appropriate depends on\n>> the particular application/context, and (from Git's perspective), none\n>> of them are inherently superior to any of the other. Even the question\n>> of whether testrun logs should or should not be reachable by default,\n>> depends on the surrounding application/context.\n>\n> Wouldn't it make sense to mention these possibilities in the git-notes\n> manpage, to help people use the mechanism as intended ?\n\nI don't disagree, although I think it's hard to provide generic\nsuggestions that will be very useful for all kinds of surrounding\napplications/contexts.\n\n>> Now, the intention of Yann's RFC is to store the testrun logs directly\n>> in a notes _tree_. This is not too different from alternative (e)\n>> above, in that reachability is automatic. However, instead of having\n>> the surrounding application manipulate the notes history to ensure\n>> reachability, the RFC would rather teach Git's notes code to\n>> accomodate the (likely rather special) case of having a note that is\n>> BOTH structured like (or at least easily mapped to) a Git tree object,\n>> AND that should be automatically reachable.\n>\n> Incidently, proposal (e) would allow the use of commits, although\n> doing so would probably cause problems, not all of the children of the\n> commit used as annotation having the same relationship to their parent.\n>\n> Are you suggesting using a slightly different mechanism than\n> the \"parent\" relationship ?\n\nNot really, I'm suggesting that we (ab)use the parent relationship to\nachieve reachability. Sure, this \"pollutes\" the notes history with\n\"pointless\" merges of a totally arbitrary/unrelated history, but the\nnotes history is typically not interesting/important in any case. The\nnotes code itself is only interested in the notes _tree_, and does not\ncare about the history at all. At some point in the past, I believe we\neven considered having the notes ref point to a tree object directly,\nand simply avoid making commit objects for notes altogether.\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"}]}