{"thread":{"id":"35909","subject":"Re: [RFC/PATCH] Supporting non-blob notes","startedAt":"2014-02-18T14:46:28Z","lastAt":"2014-02-24T01:29:10Z","messageCount":3,"participants":["Johan Herland","Duy Nguyen"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"234992","messageId":"CALKQrgff4tnekfkQn9JpJbGrNJzNBYNv=sW6vc3FE+hySkjf3g@mail.gmail.com","threadId":"35909","inReplyTo":"OF23F26261.C9E4823A-ONC1257C82.0032DA0B-C1257C82.0040D57A@local","subject":"Re: [RFC/PATCH] Supporting non-blob notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2014-02-18T14:46:28Z","receivedAt":"2014-02-18T14:46:28Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Mon, Feb 17, 2014 at 11:48 AM, wrote:\n> The recent \"git-note -C changes commit type?\" thread\n> (http://thread.gmane.org/gmane.comp.version-control.git/241950) looks\n> like a good occasion to discuss possible uses of non-blob notes.\n>\n> The use-case we're thinking about is the storage of testrun logs as\n> notes (think: being able to justify that a given set of tests were\n> successfully run on a given revision).\n\nI think this is a good use of notes, and organizing the testrun logs\ninto a tree of files seems like a natural way to proceed.\n\n> Here is a proof-of-concept patch (that applies to 1.8.4.2) I've been\n> playing with. Because of the -C behaviour described in this other\n> thread, I opted for a new -o flag that would not mess with the object\n> argument. This patch is very minimalist, and just allows storing a\n> tree note (currently any type of object, but that's easy to restrict\n> if we want to), and retrieving it.\n\nI think we must think _very_ carefully about which object types we\nallow to be stored in notes trees.\n\nAs far as I can see, you use case (storing testrun logs) is covered\nnicely by allowing tree objects as notes, and I think that's where we\nshould start. The note tree is itself a tree object, and storing\nsub-trees of that is not new or unusual to Git at all. Reachability is\nnicely covered by how Git already handles sub-trees. Obviously we must\nflesh out how the notes-related parts of the code deal with trees (see\nbelow), but that does not really affect the rest of Git, and should\ntherefore be relatively uncontroversial.\n\nIf we go on to _commit_ objects, they are currently only referenced\nfrom tree objects as \"gitlink\"s (with a special \"160000\" mode). If you\nwere to put one of these in a notes tree, you would get the same\nsemantics as a \"gitlink\", i.e. git handles that part of the tree as a\nsubmodule where a different submodule repo is (to be) checked out. The\ncommit is NOT considered/required to be reachable, and would therefore\nnot be automatically communicated by a fetch or push.\n\nSo if you want commits in a notes tree to be handled differently from\ncommits-as-gitlinks, you would have to tweak all the code in Git that\ndeal with gitlinks. You would have to introduce a differentiation\nbetween your \"commits-as-gitlinks\" and \"commits-as-notes\", either by\nreserving another special mode number, or by otherwise making the rest\nof Git notes-aware. All of this comes in addition to teaching the\nnotes-related code how to deal with commits (i.e. how to display them,\netc.).\n\nIn other words, before you embark on this, you need a convincing\nargument for why allowing commits-as-notes is really necessary and\nworth it in the end. Please also consider that you _can_ support\ncommits-as-notes by the mechanism I suggested in the previous thread:\nStore the commit SHA1 in a note-as-blob, and then amend the notes\ncommit to include the commit SHA1 as an additional parent. It's not\nvery elegant, but it solves the reachability problem.\n\nIf we go even further and want to allow ANY git object as a note, then\nwe must also consider tag objects, which AFAIK has never before been\nstored inside a tree. Here we are really entering uncharted\nterritory...\n\nSo for now (and in lieu of a convincing use case for\nnotes-as-commits), I suggest you only look at notes-as-trees. The\nfirst consequence of this is probably that your added -o/--object\noption should be renamed. -t/--tree is not taken, AFAICS...\n\n> Johan Herland wrote:\n>> Obviously, it would not make sense to use refs/notes/history while\n>> displaying the commit log (\"git log --notes=history\"), as the raw\n>> commit object would be shown in the log.\n>\n> Currently, a non-blob commit is just not displayed at all. And rather\n> than displaying the raw object, we have a number of options available,\n> starting with object's sha1, to more elaborate presentations depending\n> on the type of object (commit info, tree hierarchy, etc, as \"git notes\n> show\" already does). This PoC shows that it can be dealt with later.\n\nI'm only considering the notes-as-tree case here...\n\nI assume that if you organize your notes in tree objects, then you\nprobably have more information in there than is useful to display in\nthe textual output from \"git log\". Also, you probably have\nspecial-purpose scripts for initially generating those trees, and\nlater digging into the information stored therein. Hence we should\nconcentrate on getting the basics covered, to allow those scripts to\ndo their thing, and adding bells and whistles to \"git log\" for\ndisplaying notes-as-trees is much less important. For now, \"git log\"\nshould probably show a short summary when encountering a\nnotes-as-tree. Whether that summary consists of merely the tree SHA1,\nor in providing a (relatively short) tree listing, I leave up to you.\nI also agree that this can be dealt with later (as long as the default\nbehaviour is not actively harmful/confusing).\n\n> What I envision, would be viewers like gitk simply show the\n> hyperlinked sha1, and (in the case of a tree note) open a tree view\n> that allows to browse the contents of the note.\n\nAgreed, although not important in v1.\n\n> My (necessarily incomplete) list for what's missing:\n> * more audit of impact (eg. seen \"git fetch\" complain about a non-blob\n> note on HEAD, have to check why; \"git fetch\" refuses to fetch a\n> non-blob note, pack-objects considering it is a corruption)\n> * support in \"notes append/merge\", safeguards for commands\n> that won't know of to deal with non-blobs, tests\n\nFor trees, notes append/edit/merge sort of makes sense, in that is\nshould be possible to edit a notes-as-tree object, and merge it with a\ndifferent notes-as-tree object. Editing a notes-as-tree probably\ninvolved checking out that tree into a temporary directory, and then\nrecording the new tree after the user have mucked about in there. This\nis somewhat similar to how \"git notes merge\" works today, and these\noperations probably will need to expand/reuse that code.\n\n> * let --show-note display the sha1 (make that optional ?)\n\nDisplaying the SHA1 by default is good enough for v1. More options can\nbe added later.\n\n> * extend gitk (and gitweb ?) as described above\n\nNot important for v1. I guess most users of notes-as-trees will have\nsome domain-specific structure/convention imposed on their\nnotes-as-trees, which probably makes general notes-as-trees browsing\nless useful, and domain-specific tools for processing the information\nmore or less required.\n\n> * provide a ^{note} / ^{note:} syntax for easy\n> access to notes contents\n\nYes. This is good, although somewhat orthogonal to the rest of this topic...\n\n\nHope this helps,\n\n...Johan\n"},{"id":"235037","messageId":"CACsJy8C5Hxyj27cX=Jh60ouWpXf-iEeMd9PigKaJTp3rNZrydQ@mail.gmail.com","threadId":"35909","inReplyTo":"CALKQrgff4tnekfkQn9JpJbGrNJzNBYNv=sW6vc3FE+hySkjf3g@mail.gmail.com","subject":"Re: [RFC/PATCH] Supporting non-blob notes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-02-18T23:10:17Z","receivedAt":"2014-02-18T23:10:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Feb 18, 2014 at 9:46 PM, Johan Herland wrote:\n> On Mon, Feb 17, 2014 at 11:48 AM, wrote:\n>> The recent \"git-note -C changes commit type?\" thread\n>> (http://thread.gmane.org/gmane.comp.version-control.git/241950) looks\n>> like a good occasion to discuss possible uses of non-blob notes.\n>>\n>> The use-case we're thinking about is the storage of testrun logs as\n>> notes (think: being able to justify that a given set of tests were\n>> successfully run on a given revision).\n>\n> I think this is a good use of notes, and organizing the testrun logs\n> into a tree of files seems like a natural way to proceed.\n\nNotes from the previous attempt to store trees as notes (something to\nwatch out maybe, when you do it again)\n\nhttp://article.gmane.org/gmane.comp.version-control.git/197712\n-- \nDuy\n"},{"id":"235214","messageId":"CALKQrgeJGD=68=OV499-PZNCiP0vt9YLMgS_WPjSjoTxySggow@mail.gmail.com","threadId":"35909","inReplyTo":"CACsJy8C5Hxyj27cX=Jh60ouWpXf-iEeMd9PigKaJTp3rNZrydQ@mail.gmail.com","subject":"Re: [RFC/PATCH] Supporting non-blob notes","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2014-02-24T01:29:10Z","receivedAt":"2014-02-24T01:29:10Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Wed, Feb 19, 2014 at 12:10 AM, Duy Nguyen wrote:\n> On Tue, Feb 18, 2014 at 9:46 PM, Johan Herland wrote:\n>> On Mon, Feb 17, 2014 at 11:48 AM, wrote:\n>>> The recent \"git-note -C changes commit type?\" thread\n>>> (http://thread.gmane.org/gmane.comp.version-control.git/241950) looks\n>>> like a good occasion to discuss possible uses of non-blob notes.\n>>>\n>>> The use-case we're thinking about is the storage of testrun logs as\n>>> notes (think: being able to justify that a given set of tests were\n>>> successfully run on a given revision).\n>>\n>> I think this is a good use of notes, and organizing the testrun logs\n>> into a tree of files seems like a natural way to proceed.\n>\n> Notes from the previous attempt to store trees as notes (something to\n> watch out maybe, when you do it again)\n>\n> http://article.gmane.org/gmane.comp.version-control.git/197712\n\nThanks for that link. It is good to see that these issues have been\nconsidered/discussed previously.\n\nI've been thinking about this for a while now, and I find myself\nagreeing more and more with Junio's argument in the linked thread.\n\nI think notes are fundamentally - like file contents from Git's POV -\nan unstructured stream of bytes. Any real structure in a git note is\nimposed by the surrounding application/context, and having Git impose\nits own object model onto the contents of notes would likely be an\nunnecessary distraction.\n\nIn Yann's example, the testrun logs are probably best structured as a\nhierarchy of files, but that does not necessarily mean that they MUST\nbe stored as a Git tree object (with accompanying sub-trees and\nblobs). For example, one could imagine many different solutions for\nstoring the testrun logs:\n\n (a) Storing the logs statically on some server, and putting the\ncorresponding URL in a notes blob. Reachability is manual/on-demand\n(be retrieving the URL).\n\n (b) Storing the logs in a .tar.gz archive, and adding that archive as\na blob note. Reachability is implicit/automatic (by unpacking the\narchive).\n\n (c) Storing the logs on some ref in an external repo, and putting the\nrepo URL + ref in a notes blob. Reachability is manual/on-demand (by\ncloning/fetching the repo).\n\n (d) Storing the logs on some ref/commit in the same repo, and putting\nthe ref/commit name in a notes blob. Reachability depends on the\napplication/user to sync the ref/commit along with the notes.\n\n (e) Storing the logs in a commit, putting the commit name in a blob\nnote, and then creating/rewriting the notes history to include the\ncommit in its ancestry. Reachability is automatic (i.e.follows the\nnotes), but the application must control/manipulate the notes history.\n\nWhichever of these (or other) solutions is most appropriate depends on\nthe particular application/context, and (from Git's perspective), none\nof them are inherently superior to any of the other. Even the question\nof whether testrun logs should or should not be reachable by default,\ndepends on the surrounding application/context.\n\nNow, the intention of Yann's RFC is to store the testrun logs directly\nin a notes _tree_. This is not too different from alternative (e)\nabove, in that reachability is automatic. However, instead of having\nthe surrounding application manipulate the notes history to ensure\nreachability, the RFC would rather teach Git's notes code to\naccomodate the (likely rather special) case of having a note that is\nBOTH structured like (or at least easily mapped to) a Git tree object,\nAND that should be automatically reachable.\n\nEven though there is a certain elegance to storing such a tree object\ndirectly as a notes object, there is AFAICS no other inherent\nadvantage (e.g. performance- or functionality-wise) to following that\napproach. I'm not at all sure that it justifies increasing the\ncomplexity of the notes code.\n\nFurthermore, considering the RFC's original intention of also making\ncommit and tag objects directly usable as notes, and realizing the\nfundamental difficulties in teaching Git to handle this (outlined in\nmy previous email in this thread), I must conclude that the simplicity\nand flexibility of something like alternative (e) above far outweighs\nthe added code complexity to support allowing any object type to be\nused as a note.\n\nMaybe we should instead consider making it easier to do alternative\n(e), by providing a command-line option for supplying additional\nparents to a notes commit?\n\n\n...Johan\n\n[1]: The only \"structure\" in notes contents expected by Git is the\ntext format expected when showing notes with \"git log\", or when\nediting/appending notes with your default text editor. However, these\nare typically bypassed and/or customized by an external application\nstoring custom data in notes.\n\n-- \nJohan Herland, \nwww.herland.net\n"}]}