{"thread":{"id":"27987","subject":"Storing additional information in commit headers","startedAt":"2011-08-01T18:20:15Z","lastAt":"2011-08-04T03:41:14Z","messageCount":23,"participants":["martin f krafft","Sverre Rabbelier","Martin Langhoff","Clemens Buchacher","Jeff King","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"172581","messageId":"20110801182015.GA3100@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":null,"subject":"Storing additional information in commit headers","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-01T18:20:15Z","receivedAt":"2011-08-01T18:20:15Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"Dear list,\n\nI've read — with great interest — the recent discussion on\ngeneration numbers[0], mostly because Clemens Buchacher pointed me\nto it as a warning not to mess with commit objects.\n\n0. http://comments.gmane.org/gmane.comp.version-control.git/177146\n\nMy intent was to add an extra commit header to select commits as\na way to store extra information needed to automate the management\nof interdependent branches and patch generation à la TopGit.\n\nHaving read the generation numbers debate, I am not sure that adding\nadditional commit headers is a bad idea per se. From what\nI understand, the main pushback to Linus' idea was that people did\nnot feel it right to store redundant, calculateable information\npermanently in commit objects, where they cannot be altered anymore,\ndespite the non-zero chance of there being an error. Instead, the\nuse of a cache was advocated. I do not want to take a side in this\ndebate with this mail of mine.\n\nInstead, I am investigating ways in which I can store additional\ninformation for a branch, and ideally in a way to make it\ntransparent and automatic for all users of a project's repo.\n\nHence, if I were to store additional information in the commit\nobject headers, this information would by design be correct,\nimmutable, and non-redundant. I am going to reply to my own mail\nwith some implementation details to feed the curious, with the hope\nto keep this debate focused.\n\nAre there any strong reasons against my use of commit headers for\nspecific, well-defined purposes in contained use-cases? E.g. are\nthere tools known to only copy \"known\" headers, which could\npotentially break my assumptions?\n\nThanks,\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"when a gentoo admin tells me that the KISS principle is good for\n 'busy sysadmins', and that it's not an evolutionary step backwards,\n i wonder whether their tape is already running backwards.\"\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172584","messageId":"CAGdFq_h5gimMkwhyOv0PFqVwx4pExL7Zri8Tki_dT5HDtFcBFA@mail.gmail.com","threadId":"27987","inReplyTo":"20110801182015.GA3100@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2011-08-01T18:27:14Z","receivedAt":"2011-08-01T18:27:14Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Mon, Aug 1, 2011 at 20:20, martin f krafft wrote:\n> My intent was to add an extra commit header to select commits as\n> a way to store extra information needed to automate the management\n> of interdependent branches and patch generation à la TopGit.\n\nHave you had a look at git notes?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"172585","messageId":"20110801182845.GA4303@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110801182015.GA3100@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-01T18:28:45Z","receivedAt":"2011-08-01T18:28:45Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach martin f krafft [2011.08.01.2020 +0200]:\n> Hence, if I were to store additional information in the commit\n> object headers, this information would by design be correct,\n> immutable, and non-redundant. I am going to reply to my own mail\n> with some implementation details to feed the curious, with the hope\n> to keep this debate focused.\n\nFor lack of a better idea (cf. [0]), I am currently toying with the\nfollowing approach:\n\nPossibly in addition to the orphan parent pointer to a commit object\nsuggested in [0], and in order to provide a clear means to identify\nsaid orphan parent pointer (holding additional information), I am\nconsidering storing this orphan parent commit's ref in the main\ncommit, using a header like x-topgit-top-base [1].\n\n0. http://permalink.gmane.org/gmane.comp.version-control.git/178349\n1. The use of the x- prefix is obviously intentional to suggest that\n this is a free-form, non-standard extension.\n\nWhenever the extra data need changing, a new x-topgit-top-base ref\nis added to HEAD.\n\nNow, given a commitish, I simply have to walk back in time until\nI find a commit object with such a header, and I have the most\nrecent metadata at my fingertips.\n\nInstead of a ref to the orphan parent commit (which visibily\npollutes the history), I could also just store the information right\nthere.\n\nThis is arguably hackish, but unless I find a better way, it's the\nbest I've come up with thus far.\n\nAnd of course, this could go into the commit message body text,\nbut it being an implementation detail, that's really not the right\nplace for it.\n\nThanks for your consideration,\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"there are two major products that come out of berkeley: lsd and unix.\"\n one caused me an addiction\n -- fyodor\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172588","messageId":"20110801183411.GA5182@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"CAGdFq_h5gimMkwhyOv0PFqVwx4pExL7Zri8Tki_dT5HDtFcBFA@mail.gmail.com","subject":"Re: Storing additional information in commit headers","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-01T18:34:11Z","receivedAt":"2011-08-01T18:34:11Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Sverre Rabbelier [2011.08.01.2027 +0200]:\n> On Mon, Aug 1, 2011 at 20:20, martin f krafft wrote:\n> > My intent was to add an extra commit header to select commits as\n> > a way to store extra information needed to automate the management\n> > of interdependent branches and patch generation à la TopGit.\n> \n> Have you had a look at git notes?\n\nHello, and thanks for taking the time to reply to me!\n\nYes, I have considered git-notes. The issue I have with git-notes is\nthat it requires every contributor to set up refspecs for fetch and\npush, or else the notes will not be exchanged/shared.\n\nI realise this is a minor concern to most of you, or maybe even\na feature (part of the beauty of Git is, after all, that it works\nwithout requiring everyone to have the same local setup), but in our\nuse-case (distro packaging), it's a relatively large burden to new\ncontributors and passerby's (sp?).\n\nAlso, git-notes are mutable (at least from the UI perspectiv) and\nI strive to encode information immutably.\n\nTherefore I am looking for a means to encode this (necessary)\ninformation as part of the main DAG (i.e. not polluting the\nworktree).\n\nI hope this makes sense.\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"first get your facts; then you can distort them at your leisure.\"\n -- mark twain\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172590","messageId":"CACPiFCLPgsC+9cX7r33oCQ2AnuRXMTqOAE5RZLS7hXdHc6B-9Q@mail.gmail.com","threadId":"27987","inReplyTo":"20110801182845.GA4303@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2011-08-01T19:33:38Z","receivedAt":"2011-08-01T19:33:38Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Mon, Aug 1, 2011 at 2:28 PM, martin f krafft wrote:\n> For lack of a better idea (cf. [0]), I am currently toying with the\n> following approach:\n\nHi Martin!\n\nWhat data are you trying to include? Some time ago, I had similar\nideas to yours for a while... and it ended up being that all I needed\nwas to put the additional data /in a file/ and commit that file.\n\nSpeculation: you mention distro packaging, so I assume you're\nimproving the Debian packaging integration, with git tracking\ndebian/rules, perhaps with a wrapper. If you are using a wrapper\nprogram, it is trivial to update this \"metadata\" file, or to ensure\nit's valid/sane, in the preparations to commit, perhaps ensuring that\na pre-commit-hook script is in place and executable.\n\nhth,\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- Software Architect - OLPC\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"172592","messageId":"20110801200149.GA20861@toss","threadId":"27987","inReplyTo":"20110801183411.GA5182@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2011-08-01T20:01:49Z","receivedAt":"2011-08-01T20:01:49Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Mon, Aug 01, 2011 at 08:34:11PM +0200, martin f krafft wrote:\n>\n> Yes, I have considered git-notes. The issue I have with git-notes is\n> that it requires every contributor to set up refspecs for fetch and\n> push, or else the notes will not be exchanged/shared.\n\nNotes are tracked using a 'branch' too. It's just a branch in the\nrefs/notes namespace, the notes ref. You could simply tag your\nnotes ref or point a ref from the refs/heads namespace to it each\ntime you create new notes.\n\n> Also, git-notes are mutable (at least from the UI perspectiv) and\n> I strive to encode information immutably.\n\nNotes are also used by textconv, for example, to cache immutable\ndata. It's not likely a user will end up editing it by accident\nunless you use the default notes ref.\n\nClemens\n"},{"id":"172593","messageId":"20110801201301.GA17111@sigill.intra.peff.net","threadId":"27987","inReplyTo":"20110801182015.GA3100@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-01T20:13:01Z","receivedAt":"2011-08-01T20:13:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 01, 2011 at 08:20:15PM +0200, martin f krafft wrote:\n\n> Instead, I am investigating ways in which I can store additional\n> information for a branch, and ideally in a way to make it\n> transparent and automatic for all users of a project's repo.\n> \n> Hence, if I were to store additional information in the commit\n> object headers, this information would by design be correct,\n> immutable, and non-redundant. I am going to reply to my own mail\n> with some implementation details to feed the curious, with the hope\n> to keep this debate focused.\n> \n> Are there any strong reasons against my use of commit headers for\n> specific, well-defined purposes in contained use-cases? E.g. are\n> there tools known to only copy \"known\" headers, which could\n> potentially break my assumptions?\n\nThis topic has come up several times in the past few years. I think some\nof the relevant questions to consider about your new data are:\n\n 1. Does git actually care about your data? E.g., would it want to use\n it for reachability analysis in git-fsck?\n\n 2. Is it an immutable property of a commit, or can it be changed after\n the fact?\n\nIf (2) is no, then git-notes is probably the best choice.\n\nOtherwise, if (1) is yes, then a commit header makes sense. But then, it\nshould also be something that git is taught about, and your commit\nheader should not be some topgit-specific thing, but a header showing\nthe generalized form.\n\nOtherwise, the usual recommendation is to use a pseudo-header within the\nbody of the commit message (i.e., \"Topgit-Base: ...\" at the end of the\ncommit message). The upside is that it's easy to create, manipulate, and\nexamine using existing git tools. The downside is that it is something\nthat the user is more likely to see in \"git log\" or when editing a\nrebased commit message.\n\nJust about every discussion on this topic ends with the pseudo-header\nrecommendation. The only exceptions AFAIK are \"encoding\" (which git\nitself needs to care about), and \"generation\" (which, as you noted,\nraises other questions).\n\n-Peff\n"},{"id":"172600","messageId":"20110801205143.GA15401@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"CACPiFCLPgsC+9cX7r33oCQ2AnuRXMTqOAE5RZLS7hXdHc6B-9Q@mail.gmail.com","subject":"Re: Storing additional information in commit headers","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-01T20:51:43Z","receivedAt":"2011-08-01T20:51:43Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Martin Langhoff [2011.08.01.2133 +0200]:\n> What data are you trying to include? Some time ago, I had similar\n> ideas to yours for a while... and it ended up being that all I needed\n> was to put the additional data /in a file/ and commit that file.\n\nHi, thanks for taking the time to reply to me!\n\nI am trying to store the top-base of a TopGit branch, which is the\nmerge of all a branch's dependencies.\n\nTopGit uses refs for that, but a ref can only ever point at one such\nmerge, and so it's hard-to-impossible to reconstruct a branch\ndependency in the past.\n\nTopGit does use files in the worktree too. I would love to get rid\nof this as well, since a file like .topmsg (which differs between\nall branches, even related ones), requires to always remember to use\nthe 'ours' merge driver, which requires setup, which makes it harder\nto use.\n\n> If you are using a wrapper program,\n\nI am trying to stay as close as possible to plain Git. All of this\ncould easily be done by a wrapper, but a wrapper always makes too\nmany assumptions to become a viable standard for Debian packaging.\n\n> it's valid/sane, in the preparations to commit, perhaps ensuring\n> that a pre-commit-hook script is in place and executable.\n\nAgain, that requires setup, which increases the barrier of entry to\npasserby's and new contributors.\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"verbing weirds language.\"\n -- calvin\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172601","messageId":"20110801205523.GB15401@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110801200149.GA20861@toss","subject":"Re: Storing additional information in commit headers","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-01T20:55:23Z","receivedAt":"2011-08-01T20:55:23Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Clemens Buchacher [2011.08.01.2201 +0200]:\n> Notes are tracked using a 'branch' too. It's just a branch in the\n> refs/notes namespace, the notes ref. You could simply tag your\n> notes ref or point a ref from the refs/heads namespace to it each\n> time you create new notes.\n\nHi Clemens, thanks for responding!\n\nYou suggest integrating refs/notes/foo into refs/heads by means of\na pointer… at which point we are polluting the branch history space\nagain (think gitk), no?\n\nI appreciate the simplicity of this idea of yours, which I had not\nthought of. Indeed, maintaining a head at the top of\nrefs/notes/topgit-metadata (or whatever) has charm. I do not mean to\ndiscard it at all right now, and will think about this more!\n\ngit-notes was designed to be used for such cases, I was pleased to\nnote the configurability. Maybe it is the ticket.\n\nStill: why not commit headers?\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n... with a plastic cup filled with a liquid that was almost,\nbut not quite, entirely unlike tea.\n -- douglas adams, \"the hitchhiker's guide to the galaxy\"\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172603","messageId":"20110801211104.GC15401@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110801201301.GA17111@sigill.intra.peff.net","subject":"Re: Storing additional information in commit headers","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-01T21:11:04Z","receivedAt":"2011-08-01T21:11:04Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Jeff King [2011.08.01.2213 +0200]:\n> This topic has come up several times in the past few years.\n\nI am sorry that I am bothering the list again. I tried hard to find\nwhatever I could, but after 2–3 hours of web searching, I came here…\n\nThank you for taking the time to answer!\n\n> I think some\n> of the relevant questions to consider about your new data are:\n> \n> 1. Does git actually care about your data? E.g., would it want to use\n> it for reachability analysis in git-fsck?\n> \n> 2. Is it an immutable property of a commit, or can it be changed after\n> the fact?\n\nExcellent points, and I have answers to both:\n\n 1. Ideally, I would like to point to another blob containing\n information. Right now, in order to prevent gc from pruning\n it, that would have to be a commit pointed to with a parent\n pointer, which is just not right (it's not a parent) and causes\n the commit to show up in the history (which it should not, as\n it's an implementation detail).\n\n I'll return to this point further down…\n\n 2. It is immutable. Ideally, I would like to store extra\n information for a ref in ref/heads/*, but there seems to be no\n way of doing this. Hence, I need to store it in commits and\n backtrack for it. Or so I think, at least…\n\n> Otherwise, if (1) is yes, then a commit header makes sense. But\n> then, it should also be something that git is taught about, and\n> your commit header should not be some topgit-specific thing, but\n> a header showing the generalized form.\n\nI agree entirely and would be all too excited to see this happening.\nI already had ideas too:\n\n In addition to the standard tree and parent pointers, there could\n be *-ref and x-*-ref headers, which take a single ref argument,\n presumably to a blob containing more data.\n\n While I cannot conceive a *-ref example, I think it's obvious that\n x-*-ref should be introduced at the same time to keep the *-ref\n namespace clear for future, \"official\" Git use.\n\n In terms of gc and fsck and the like, all *-ref and x-*-ref\n headers would contribute to reachability tests and hence prevent\n pruning of those blobs.\n\n> Otherwise, the usual recommendation is to use a pseudo-header\n> within the body of the commit message (i.e., \"Topgit-Base: ...\" at\n> the end of the commit message). The upside is that it's easy to\n> create, manipulate, and examine using existing git tools. The\n> downside is that it is something that the user is more likely to\n> see in \"git log\" or when editing a rebased commit message.\n\n… to see *and to accidentally mess up*. And while that may even be\nunlikely, it does expose information that really ought to be hidden.\n\n> Just about every discussion on this topic ends with the\n> pseudo-header recommendation. The only exceptions AFAIK are\n> \"encoding\" (which git itself needs to care about), and\n> \"generation\" (which, as you noted, raises other questions).\n\nI can see how it's arguable too why one would want to give git\ncommit objects the ability to reference arbitrary blobs containing\nadditional information. I suppose the answer to this question is\nrelated to the answer to the question of whether Git is\na contained/complete tool as-is, or also serves as\na \"framework\"/\"toolkit\" for advanced/creative use.\n\nThe availability of the porcelain commands seems to suggest that\nextensible/flexible additional features should be welcome! ;)\n\n-- \nmartin; (greetings from the heart of the sun.)\n \\____ echo mailto: !#^.\"<*>\"|tr \"<*> mailto:\" net@madduck\n \nhttp://www.transnationalrepublic.org/\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172625","messageId":"20110802035056.GB17494@sigill.intra.peff.net","threadId":"27987","inReplyTo":"20110801211104.GC15401@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-02T03:50:56Z","receivedAt":"2011-08-02T03:50:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 01, 2011 at 11:11:04PM +0200, martin f krafft wrote:\n\n> > 1. Does git actually care about your data? E.g., would it want to use\n> > it for reachability analysis in git-fsck?\n> > \n> > 2. Is it an immutable property of a commit, or can it be changed after\n> > the fact?\n> \n> Excellent points, and I have answers to both:\n> \n> 1. Ideally, I would like to point to another blob containing\n> information. Right now, in order to prevent gc from pruning\n> it, that would have to be a commit pointed to with a parent\n> pointer, which is just not right (it's not a parent) and causes\n> the commit to show up in the history (which it should not, as\n> it's an implementation detail).\n\nIn that case, notes sound like a nice solution, as that is exactly what\nthey do. Yes, they are mutable, but that might not be that big a deal.\n\n> 2. It is immutable. Ideally, I would like to store extra\n> information for a ref in ref/heads/*, but there seems to be no\n> way of doing this. Hence, I need to store it in commits and\n> backtrack for it. Or so I think, at least…\n\nWait, so you want metadata on a _ref_, not on a commit? That is a very\ndifferent thing, I think. We usually accomplish that with data in\n.git/config. Or if you need to push data between repos, or if it's too\nbig to easily fit in the config, then put it in a blob and keep a\nparallel ref structure (e.g., refs/topgit/bases/refs/heads/master).\n\nOr maybe I'm just misunderstanding.\n\n> > Otherwise, if (1) is yes, then a commit header makes sense. But\n> > then, it should also be something that git is taught about, and\n> > your commit header should not be some topgit-specific thing, but\n> > a header showing the generalized form.\n> \n> I agree entirely and would be all too excited to see this happening.\n> I already had ideas too:\n> \n> In addition to the standard tree and parent pointers, there could\n> be *-ref and x-*-ref headers, which take a single ref argument,\n> presumably to a blob containing more data.\n\nI'm not sure how well-defined that is, though. What does the ref mean?\nWhat does it point to, and what is the meaning with respect to the\noriginal commit? Or are you suggesting that \"*\" would be \"topgit-base\"\nhere, and that git core would understand only that any header matching\nthe pattern \"x-*-ref\" should be followed with respect to\nreachability/pruning. Only the owner of the \"*\" part (topgit in this\ncase) would be able to make sense of the meaning of the ref.\n\nIf that is the case, that does make sense to me. It's basically an\nimmutable version of a note.\n\nHowever, implementing such a thing would mean you have an awkward\ntransition period where some versions of git think the referenced object\nis relevant, and others do not. That's something we can overcome, but\nit's going to require code in git, and possibly a dormant introduction\nperiod.\n\nI suspect you would give git people more warm fuzzies about implementing\nthis by showing a system that is built on git-notes and saying \"this\nworks really well, except that the external note storage is not a good\nreason because { it's mutable, it's not efficient, whatever other reason\nyou find}\". And then we know that the system is proven to work, and that\nmigrating the note-like structure into the object is sensible.\n\nBut I get the impression you're one step back from that now. So it makes\nsense to me to at least prototype it via git-notes, which will give you\nthe same semantic storage (a mapping of commits to some blobs, with\nreachability handled automatically).\n\n> > Otherwise, the usual recommendation is to use a pseudo-header\n> > within the body of the commit message (i.e., \"Topgit-Base: ...\" at\n> > the end of the commit message). The upside is that it's easy to\n> > create, manipulate, and examine using existing git tools. The\n> > downside is that it is something that the user is more likely to\n> > see in \"git log\" or when editing a rebased commit message.\n> \n> … to see *and to accidentally mess up*. And while that may even be\n> unlikely, it does expose information that really ought to be hidden.\n\nI'm not quite sure what the information is, so I can't really judge. Do\nyou have a concrete example?\n\nI got the impression earlier you were wanting to store a human-readable\ntext string. That makes a pseudo-header a reasonable choice. But if you\nare going to reference some blob (which it seems from what you wrote\nabove), and you are interested in proper reachability analysis, then no,\nit probably isn't a good idea.\n\n> I can see how it's arguable too why one would want to give git\n> commit objects the ability to reference arbitrary blobs containing\n> additional information. I suppose the answer to this question is\n> related to the answer to the question of whether Git is\n> a contained/complete tool as-is, or also serves as\n> a \"framework\"/\"toolkit\" for advanced/creative use.\n> \n> The availability of the porcelain commands seems to suggest that\n> extensible/flexible additional features should be welcome! ;)\n\nI think extensibility is welcome. It's just that most discussions so far\nhave ended up realizing that a new header would just be cruft. Maybe\nyours is different. I'm still not 100% sure I understand what you want\nto accomplish, but the idea of an x-*-ref header is a reasonable thing\nfor git to have.\n\n-Peff\n"},{"id":"172637","messageId":"20110802082810.GC29887@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110802035056.GB17494@sigill.intra.peff.net","subject":"Re: Storing additional information in commit headers","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-02T08:28:10Z","receivedAt":"2011-08-02T08:28:10Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Jeff King [2011.08.02.0550 +0200]:\n> > 2. It is immutable. Ideally, I would like to store extra\n> > information for a ref in ref/heads/*, but there seems to be no\n> > way of doing this. Hence, I need to store it in commits and\n> > backtrack for it. Or so I think, at least…\n> \n> Wait, so you want metadata on a _ref_, not on a commit? That is a very\n> different thing, I think. We usually accomplish that with data in\n> .git/config. Or if you need to push data between repos, or if it's too\n> big to easily fit in the config, then put it in a blob and keep a\n> parallel ref structure (e.g., refs/topgit/bases/refs/heads/master).\n> \n> Or maybe I'm just misunderstanding.\n\nYou nailed it perfectly well. Thank you for taking the time again to\nreply to me.\n\nTopGit does what you suggest (a parallel ref structure), but there\nare three problems with this, which I am trying to address:\n\n 1. you need to ensure that these refs are pushed and fetched,\n which requires set up and possible migration issues when things\n change, and can cause big problems for contributors who just so\n happened to forget.\n\n 2. the additional refs confuse people a lot — and I can attest to\n that because I have also at times found myself overwhelmed by\n them when staring at gitk.\n\n 3. once a ref updates, we need to keep a pointer to the previous\n location, since one of the goals is the ability to be able to\n return to a point in history (e.g. for security updates to\n a stable package, or backports). Additional refs enhance the\n aforementioned two problems.\n\nTherefore I thought it would be sensible to store these data in\ncommit. When the data change, there will always be a new commit to\nstore these data, and we do *not* want to update the data in\nprevious commits. Finding the data then becomes backtracking the\nbranch history until a commit is found containing them.\n\n> > In addition to the standard tree and parent pointers, there could\n> > be *-ref and x-*-ref headers, which take a single ref argument,\n> > presumably to a blob containing more data.\n>\n> I'm not sure how well-defined that is, though. What does the ref\n> mean? What does it point to, and what is the meaning with respect\n> to the original commit? Or are you suggesting that \"*\" would be\n> \"topgit-base\" here, and that git core would understand only that\n> any header matching the pattern \"x-*-ref\" should be followed with\n> respect to reachability/pruning. Only the owner of the \"*\" part\n> (topgit in this case) would be able to make sense of the meaning\n> of the ref.\n\nExactly the latter. Sorry for my unannounced use of wildcards in\nthis context.\n\n> If that is the case, that does make sense to me. It's basically an\n> immutable version of a note.\n>\n> However, implementing such a thing would mean you have an awkward\n> transition period where some versions of git think the referenced\n> object is relevant, and others do not. That's something we can\n> overcome, but it's going to require code in git, and possibly\n> a dormant introduction period.\n\nIndeed. This could be adressed by letting a tool like TopGit require\na minimum version of Git. For a while, this will burden developers,\nbut ensure that it works. Over time, this will cease to be\na problem.\n\n> I suspect you would give git people more warm fuzzies about\n> implementing this by showing a system that is built on git-notes\n> and saying \"this works really well, except that the external note\n> storage is not a good reason because { it's mutable, it's not\n> efficient, whatever other reason you find}\". And then we know that\n> the system is proven to work, and that migrating the note-like\n> structure into the object is sensible.\n>\n> But I get the impression you're one step back from that now. So it makes\n> sense to me to at least prototype it via git-notes, which will give you\n> the same semantic storage (a mapping of commits to some blobs, with\n> reachability handled automatically).\n\nI appreciate how you are developing your reasoning, and the advice\nyou give.\n\nIndeed, I am already prototyping using git-notes, and I designed the\ndatastore to be extensible, so that I can use other ways to find the\ndata.\n\nUsing pseudo-headers is another (temporary) way to prove the concept\nworks, but I am afraid that it will become standard too quickly\n(because it's so easy), essentially preventing progress into x-*-ref\ndomain, or forcing us to carry compatibility with us forever.\n\nWhat do you think about using the idea of orphan parent commits\n(OPC) for now? These are conceptually closest to the x-*-ref\npointers, do not require extra setup, pollute history only a little\nbit (IMHO), and slot in with Git and fsck/gc alright.\n\nHere's the idea again, graphically:\n\n o--o--o--●\n /\n #\n\nwhile at HEAD, I would backtrack history until I found HEAD^, which\nhas a parent with a well-defined commit message and holding the data\nI am looking for.\n\nLater, when x-*-ref is mainline, instead of parent pointers, it can\nbe used in place.\n\nWhen there is a merge and the TopGit data need updating, a new\nOPC is slotted into place, on the merge commit. In\nthe following graph, the user then decided also at a later point to\nupdate e.g. the TopGit patch description (.topmsg), which is also\nstored in this OPC:\n\n o--o-o\n / \\ maint master\n o--o--o--o--+--o--O--o--o--o--●\n / / /\n # # #\n\nTo keep things simple, every OPC copies the unchanged data from the\nprevious one as well (compression will reduce the overhead).\n\nLater, I can use the maint branch just in the same way I could use\nmaster when it was at that age.\n\n> > > Otherwise, the usual recommendation is to use a pseudo-header\n> > > within the body of the commit message (i.e., \"Topgit-Base: ...\" at\n> > > the end of the commit message). The upside is that it's easy to\n> > > create, manipulate, and examine using existing git tools. The\n> > > downside is that it is something that the user is more likely to\n> > > see in \"git log\" or when editing a rebased commit message.\n> > \n> > … to see *and to accidentally mess up*. And while that may even be\n> > unlikely, it does expose information that really ought to be hidden.\n> \n> I'm not quite sure what the information is, so I can't really judge. Do\n> you have a concrete example?\n> \n> I got the impression earlier you were wanting to store a human-readable\n> text string. That makes a pseudo-header a reasonable choice. But if you\n> are going to reference some blob (which it seems from what you wrote\n> above), and you are interested in proper reachability analysis, then no,\n> it probably isn't a good idea.\n\nI am not yet sure what information needs storing. Right now, I am\nkeeping five fields:\n\n Depend-Refs A list of the most recent branch points from\n dependency branchs, so that a tool can tell\n when the dependent branch needs an update\n (commits following those refs that are not\n reachable by the branch head).\n Base-Ref The ref to the most recent merge of all\n dependencies, used to create diffs.\n Patch-Branch boolean to suggest whether this branch is\n designed to develop a single patch for\n submission or use in a quilt series.\n Patch-Message Patch description (think git-send-email).\n Integration-Branch boolean to suggest whether instead this branch\n is a branch designed to collect features.\n\nAt the moment, I do now know which of those are necessary, and which\nI am missing.\n\nThe flexibility of being able to store as much as I want, in\nwhatever format I want, without having to fear overloading the\ncommit message or burdening the user, is what makes me want to use\nrefs to blobs.\n\n> I think extensibility is welcome. It's just that most discussions\n> so far have ended up realizing that a new header would just be\n> cruft. Maybe yours is different. I'm still not 100% sure\n> I understand what you want to accomplish, but the idea of an\n> x-*-ref header is a reasonable thing for git to have.\n\nI think there are two questions:\n\n 1. would x-*-ref be a suitable idea for Git core?\n\n I think the answer is yes, as (I think) it's well-defined and\n I cannot see any problems with it, really.\n\n 2. can we prevent abuse?\n\n No, never. But just like you cannot abuse X-* headers in the\n RFC822 format due to their design, x-*-ref abuse would only\n affect those who chose it.\n\nThank you,\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"the question of whether computers can think\n is like the question of whether submarines can swim.\"\n -- edsgar w. dijkstra\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172657","messageId":"4E380148.7010701@alum.mit.edu","threadId":"27987","inReplyTo":"20110801182015.GA3100@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2011-08-02T13:53:12Z","receivedAt":"2011-08-02T13:53:12Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 08/01/2011 08:20 PM, martin f krafft wrote:\n> Are there any strong reasons against my use of commit headers for\n> specific, well-defined purposes in contained use-cases? E.g. are\n> there tools known to only copy \"known\" headers, which could\n> potentially break my assumptions?\n\nBefore you store important information in a git-internal data structure,\nplease consider:\n\n* Some of your developers might prefer using another DVCS (e.g.,\nMercurial via hg-git) and they will not be able to see the information\nat all\n\n* Some day the main project might want to (god forbid!) switch to a\nsuccessor to git, and your extra information might be difficult to migrate.\n\n* Somebody might want to work with your project from a tarball rather\nthan having to install and use git.\n\nTherefore, I recommend a strong bias towards storing information in as\ntransparent, non-system-specific a way as possible. Metadata and\nscripts stored within the file tree part of the repository are typically\na lot easier to work with and more transparent than git-specific hacks.\n\nThat being said, I haven't understood your application well enough to\nknow whether these biases might be trumped by convenience in your\nparticular situation.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"172662","messageId":"20110802150321.GA1390@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110802082810.GC29887@fishbowl.rw.madduck.net","subject":"working prototype of orphan parent commits as datastores (was: Storing additional information in commit headers)","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-02T15:03:21Z","receivedAt":"2011-08-02T15:03:21Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach martin f krafft [2011.08.02.1028 +0200]:\n> What do you think about using the idea of orphan parent commits\n> (OPC) for now? These are conceptually closest to the x-*-ref\n> pointers, do not require extra setup, pollute history only a little\n> bit (IMHO), and slot in with Git and fsck/gc alright.\n> \n> Here's the idea again, graphically:\n> \n> o--o--o--●\n> /\n> #\n> \n> while at HEAD, I would backtrack history until I found HEAD^, which\n> has a parent with a well-defined commit message and holding the data\n> I am looking for.\n> \n> Later, when x-*-ref is mainline, instead of parent pointers, it can\n> be used in place.\n> \n> When there is a merge and the TopGit data need updating, a new\n> OPC is slotted into place, on the merge commit. In\n> the following graph, the user then decided also at a later point to\n> update e.g. the TopGit patch description (.topmsg), which is also\n> stored in this OPC:\n> \n> o--o-o\n> / \\ maint master\n> o--o--o--o--+--o--O--o--o--o--●\n> / / /\n> # # #\n> \n> To keep things simple, every OPC copies the unchanged data from the\n> previous one as well (compression will reduce the overhead).\n\nI have published a working prototype of this kind of datastore, in\ncase people are interested:\n\n http://git.madduck.net/v/code/topgit-ng.git\n\nHere's a bit of synopsis:\n\n\n\n% ./tg-datastore list\nI: returns non-zero if no datastore found at given commit.\nI: prints contents of datastore otherwise.\nmessage: this is a proof-of-concept\n\n% ./tg-datastore find commitref\nI: prints the value of the parameter, or empty if parameter is not found.\nI: returns non-zero if no datastore was found.\ndc58ec49df849ec1aef6929cd40c759a6018e056\n\n% git commit --allow-empty -mone\n[master 78918bb] one\n\n% git commit --allow-empty -mtwo\n[master 7eca0cd] two\n\n% ./tg-datastore find message\nI: prints the value of the parameter, or empty if parameter is not found.\nI: returns non-zero if no datastore was found.\nthis is a proof-of-concept\n\n% ./tg-datastore find commitref\nI: prints the value of the parameter, or empty if parameter is not found.\nI: returns non-zero if no datastore was found.\ndc58ec49df849ec1aef6929cd40c759a6018e056\n\n% ./tg-datastore add message='this is a new message'\nI: returns non-zero if there is already a datastore on HEAD.\nI: adding the following data to the datastore of HEAD:\nI: message: this is a new message\n\n% ./tg-datastore find commitref\nI: prints the value of the parameter, or empty if parameter is not found.\nI: returns non-zero if no datastore was found.\n8e6179050a1aca5485f3e1702780f1b555d8643b\n\n% ./tg-datastore find message\nI: prints the value of the parameter, or empty if parameter is not found.\nI: returns non-zero if no datastore was found.\nthis is a new message\n\n tig output now:\n 2011-08-02 16:52 martin f. krafft M─┐ [master] two\n 2011-08-02 16:54 TopGit │ I TopGit data node\n 2011-08-02 16:52 martin f. krafft I one\n 2011-08-02 16:50 martin f. krafft M─┐ [origin/master] import first prototype\n 2011-08-02 16:50 TopGit │ I TopGit data node\n 2011-08-02 16:48 martin f. krafft I Initial (empty) root commit\n\n% ./tg-datastore remove\nI: always returns zero, even if there was nothing to remove.\n\n% ./tg-datastore find message\nI: prints the value of the parameter, or empty if parameter is not found.\nI: returns non-zero if no datastore was found.\nthis is a proof-of-concept\n\n\n\nNote three things:\n\n 1. I am actually using a x-* header in the TopGit data node commit\n object to help identify it as a commit. This could be done\n differently (e.g. parse the commit message for some magic), but\n I chose to do this on purpose to see how well it fares.\n\n 2. If Git grew x-*-ref headers (refs to objects in general),\n I could use that instead and drop the parent pointer, which\n would make the DAG cleaner.\n\n 3. Right now, you cannot add parent orphan commits to orphans\n themselves, but it would be trivial to enable. I just couldn't\n be bothered.\n\nEnjoy, and comments of course welcome.\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \nwindoze nt crashed.\ni am the blue screen of death.\nno one hears your screams.\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172684","messageId":"20110802185154.GA2499@sigill.intra.peff.net","threadId":"27987","inReplyTo":"20110802082810.GC29887@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-02T18:51:55Z","receivedAt":"2011-08-02T18:51:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 02, 2011 at 10:28:10AM +0200, martin f krafft wrote:\n\n> TopGit does what you suggest (a parallel ref structure), but there\n> are three problems with this, which I am trying to address:\n> \n> 1. you need to ensure that these refs are pushed and fetched,\n> which requires set up and possible migration issues when things\n> change, and can cause big problems for contributors who just so\n> happened to forget.\n\nI agree that is an annoyance, but it is one we can deal with. In the\nnear term, I wonder if a \"tg clone\" would be appropriate to add the\nextra fetch refspecs when cloning (or even a \"tg init\" inside an\nexisting git repo -- I don't actually use topgit, so I'm not sure what\nthe usual initialization process, if any, is).\n\nIn the longer term, it might be nice if git was better at sharing\nthird-party refs. The problem is that we don't know what the refs mean,\nso we don't know which ones are appropriate for sharing. Maybe we could\ndo something like \"refs/shared/topgit/*\", and git by default would push\nand pull items under refs/shared?\n\nThere have also been proposals to have a more mirror-like structure to\nwhat we fetch from remotes. E.g., to put remote refs/tags into\nrefs/remotes/origin/refs/tags, and similar for notes. It may be that it\nis sensible for us to just fetch everything from a remote into\nrefs/remotes, including unknown hierarchies like topgit.\n\n> 2. the additional refs confuse people a lot — and I can attest to\n> that because I have also at times found myself overwhelmed by\n> them when staring at gitk.\n\nUsing \"gitk --all\", I assume? I agree it is annoying, though \"gitk\n--branches\" probably better specifies what you want (unless you stick\nthe parallel ref structure under refs/heads above, which is also a\nsolution to the \"should it be fetched\" plan).\n\n> 3. once a ref updates, we need to keep a pointer to the previous\n> location, since one of the goals is the ability to be able to\n> return to a point in history (e.g. for security updates to\n> a stable package, or backports). Additional refs enhance the\n> aforementioned two problems.\n\nReflogs provide a linear history of the ref updates, but I suspect you\nwant to be able to push and pull these histories. Which reflogs will not\ndo.\n\nIf you want to version the state of refs, then using raw refs isn't the\nright answer. You want a separate commit history with trees that map ref\nnames to commits or other objects. Which is _almost_ what notes are;\nthey map commit sha1s, but you want to map ref names.\n\n> Therefore I thought it would be sensible to store these data in\n> commit. When the data change, there will always be a new commit to\n> store these data, and we do *not* want to update the data in\n> previous commits. Finding the data then becomes backtracking the\n> branch history until a commit is found containing them.\n\nThat seems to me like you are sticking information in a commit that is\nnot actually about the commit, but about the ref that happens to point\nto the commit. What if I have two refs that point to the same commit,\nbut with two different topgit bases? What about years later, when that\ninformation isn't interesting anymore? You're still carrying the cruft\ninside your commit objects.\n\n> > However, implementing such a thing would mean you have an awkward\n> > transition period where some versions of git think the referenced\n> > object is relevant, and others do not. That's something we can\n> > overcome, but it's going to require code in git, and possibly\n> > a dormant introduction period.\n> \n> Indeed. This could be adressed by letting a tool like TopGit require\n> a minimum version of Git. For a while, this will burden developers,\n> but ensure that it works. Over time, this will cease to be\n> a problem.\n\nKeep in mind that your requirement is not just a local thing. Object\nreachability is something that both sides of a transfer need to agree\non. So imagine you use TopGit with a new version of git, and you push to\na site like GitHub. The remote side will take your objects, but it will\nnot send them back to anyone who fetches from your repository (since it\nhas no idea they're relevant). And it will probably prune them after a\nweek or two.\n\n> What do you think about using the idea of orphan parent commits\n> (OPC) for now? These are conceptually closest to the x-*-ref\n> pointers, do not require extra setup, pollute history only a little\n> bit (IMHO), and slot in with Git and fsck/gc alright.\n\nIt doesn't seem like a good idea to me. Parent pointers have a\nwell-defined meaning, and other parts of git (and other tools, even) are\ngoing to assume that's what your parent pointers mean. They are used in\nmerge base calculations, for example. I _think_ you are mostly safe\nhere, because your OPC wouldn't have any real history to it, so finding\na merge base down that path would be fruitless.\n\nBut consider something like \"diff\", which shows a merge commit\ndifferently than a regular commit. Your commits will unexpectedly appear\nas merges to git, and we will show a combined diff versus the OPC, which\nis going to be ugly.\n\n> I am not yet sure what information needs storing. Right now, I am\n> keeping five fields:\n> [...]\n\n\nThanks, that helped with getting a sense of what you're doing.\n\n> I think there are two questions:\n> \n> 1. would x-*-ref be a suitable idea for Git core?\n> \n> I think the answer is yes, as (I think) it's well-defined and\n> I cannot see any problems with it, really.\n\nI think it's a nice idea for extensibility. And if it had been there\nfrom day one, there would be no problems. But now we have to deal with\nthe transition period, and the fact that two different versions of git\nwill have different ideas about the set of objects that are reachable\nfrom a given commit.\n\n> 2. can we prevent abuse?\n> \n> No, never. But just like you cannot abuse X-* headers in the\n> RFC822 format due to their design, x-*-ref abuse would only\n> affect those who chose it.\n\nI don't worry about abuse. You can already stick random cruft in a\ncommit header, and you can already connect objects to a commit via tree\nentries. This idea is just giving git some rules for dealing with it.\n\nI'm still not 100% convinced you want per-commit storage, though, and\nnot per-ref storage.\n\n-Peff\n"},{"id":"172687","messageId":"20110802185708.GB2499@sigill.intra.peff.net","threadId":"27987","inReplyTo":"20110802150321.GA1390@fishbowl.rw.madduck.net","subject":"Re: working prototype of orphan parent commits as datastores (was: Storing additional information in commit headers)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-02T18:57:08Z","receivedAt":"2011-08-02T18:57:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 02, 2011 at 05:03:21PM +0200, martin f krafft wrote:\n\n> tig output now:\n> 2011-08-02 16:52 martin f. krafft M─┐ [master] two\n> 2011-08-02 16:54 TopGit │ I TopGit data node\n> 2011-08-02 16:52 martin f. krafft I one\n> 2011-08-02 16:50 martin f. krafft M─┐ [origin/master] import first prototype\n> 2011-08-02 16:50 TopGit │ I TopGit data node\n> 2011-08-02 16:48 martin f. krafft I Initial (empty) root commit\n\nLook at \"git show origin/master\" here. It ends up as a combined diff.\nWhich is kind of ugly.\n\n-Peff\n"},{"id":"172688","messageId":"20110802190645.GB17322@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110802185154.GA2499@sigill.intra.peff.net","subject":"Re: Storing additional information in commit headers","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-02T19:06:45Z","receivedAt":"2011-08-02T19:06:45Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Jeff King [2011.08.02.2051 +0200]:\n> I agree that is an annoyance, but it is one we can deal with. In the\n> near term, I wonder if a \"tg clone\" would be appropriate to add the\n> extra fetch refspecs when cloning (or even a \"tg init\" inside an\n> existing git repo -- I don't actually use topgit, so I'm not sure what\n> the usual initialization process, if any, is).\n\nHey Jeff, thanks for your response.\n\nTopGit does come with these commands to do the setup for you, but\nthat does not ensure that a new contributor without any idea about\nTopGit won't forget to run them.\n\nThe argument against tg-clone is mainly that I really do not want to\nencapsulate/abstract functionality, but rather stay as close as\npossible to pure Git, and never to mandate anyone to use anything\nelse.\n\n> In the longer term, it might be nice if git was better at sharing\n> third-party refs. The problem is that we don't know what the refs\n> mean, so we don't know which ones are appropriate for sharing.\n> Maybe we could do something like \"refs/shared/topgit/*\", and git\n> by default would push and pull items under refs/shared?\n\nThis could be an interesting and viable approach.\n\n> > Therefore I thought it would be sensible to store these data in\n> > commit. When the data change, there will always be a new commit to\n> > store these data, and we do *not* want to update the data in\n> > previous commits. Finding the data then becomes backtracking the\n> > branch history until a commit is found containing them.\n>\n> That seems to me like you are sticking information in a commit that is\n> not actually about the commit, but about the ref that happens to point\n> to the commit. What if I have two refs that point to the same commit,\n> but with two different topgit bases?\n\nI don't think this can happen, but the point is valid.\n\n> What about years later, when that information isn't interesting\n> anymore? You're still carrying the cruft inside your commit\n> objects.\n[…]\n> I'm still not 100% convinced you want per-commit storage, though,\n> and not per-ref storage.\n\nYes, I do want per-ref storage. Your arguments against my orphan\nparent pointer approach (which could later be a x-*-ref approach)\nare valid.\n\nIt just seems to me that per-ref storage is a lot further away than\nper-commit storage, and I'd really like to move forward with TopGit…\n\nThank you,\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"one should never trust a woman who tells her real age.\n if she tells that, she will tell anything.\"\n -- oscar wilde\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172690","messageId":"20110802190923.GB16674@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110802185708.GB2499@sigill.intra.peff.net","subject":"Re: working prototype of orphan parent commits as datastores (was: Storing additional information in commit headers)","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-02T19:09:23Z","receivedAt":"2011-08-02T19:09:23Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach Jeff King [2011.08.02.2057 +0200]:\n> Look at \"git show origin/master\" here. It ends up as a combined diff.\n> Which is kind of ugly.\n\nYes, absolutely. However, this would no longer be the case if\nx-*-ref could be used. Right now, I am just using orphan parent\ncommits to avoid garbage collection.\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"he gave me his card\n he said, 'call me if they die'\n i shook his hand and said goodbye\n ran out to the street\n when a bowling ball came down the road\n and knocked me off my feet\"\n -- bob dylan\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172692","messageId":"20110802192631.GA20239@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110802190923.GB16674@fishbowl.rw.madduck.net","subject":"Re: working prototype of orphan parent commits as datastores (was: Storing additional information in commit headers)","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-02T19:26:31Z","receivedAt":"2011-08-02T19:26:31Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach martin f krafft [2011.08.02.2109 +0200]:\n> Yes, absolutely. However, this would no longer be the case if\n> x-*-ref could be used. Right now, I am just using orphan parent\n> commits to avoid garbage collection.\n\nrefs/heads/master is a file, containing its payload in the first\nline by format definition, right?\n\nI mean: the storage is right there, isn't it?\n\nOf course this opens a whole new can of worms: merging per-ref data.\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"'oh, that was easy,' says Man, and for an encore goes on to prove\n that black is white and gets himself killed on the next zebra\n crossing.\"\n -- douglas adams, \"the hitchhiker's guide to the galaxy\"\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172693","messageId":"20110802192727.GB20239@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110802190645.GB17322@fishbowl.rw.madduck.net","subject":"per-ref data storage (was: Storing additional information in commit headers)","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-02T19:27:28Z","receivedAt":"2011-08-02T19:27:28Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"[sorry, my previous message was a total reply FAIL]\n\nalso sprach martin f krafft [2011.08.02.2106 +0200]:\n> It just seems to me that per-ref storage is a lot further away than\n> per-commit storage, and I'd really like to move forward with TopGit…\n\nrefs/heads/master is a file, containing its payload in the first\nline by format definition, right?\n\nI mean: the storage is right there, isn't it?\n\nOf course this opens a whole new can of worms: merging per-ref data.\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \n\"nothing can cure the soul but the senses,\n just as nothing can cure the senses but the soul.\"\n -- oscar wilde\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172696","messageId":"20110802211241.GA29029@fishbowl.rw.madduck.net","threadId":"27987","inReplyTo":"20110802192727.GB20239@fishbowl.rw.madduck.net","subject":"Re: per-ref data storage","fromName":"martin f krafft","fromEmail":"madduck@madduck.net","sentAt":"2011-08-02T21:12:41Z","receivedAt":"2011-08-02T21:12:41Z","isPatch":false,"sender":{"key":"madduck@madduck.net","avatar":null},"body":"also sprach martin f krafft [2011.08.02.2127 +0200]:\n> refs/heads/master is a file, containing its payload in the first\n> line by format definition, right?\n> \n> I mean: the storage is right there, isn't it?\n> \n> Of course this opens a whole new can of worms: merging per-ref data.\n\norigin/master can contain a different set of per-ref data than\nmaster, and the consolidation would need to happen during the normal\nmerge.\n\nBut unless there's always a new commit associated with a change of\nthose data, git-push will happily overwrite those data on the\nremote.\n\n… unless the remote refuses to accept a ref update if the data have\nchanged. Conceivably that's could lead into a control path similar\nto what happens on a non-fast-forward push — unless\nreceive.nonFastForwards is on.\n\nWhat then?\n\n-- \nmartin | http://madduck.net/ | http://two.sentenc.es/\n \nseminars, n.:\n from \"semi\" and \"arse\", hence, any half-assed discussion.\n \nspamtraps: madduck.bogus@madduck.net\n"},{"id":"172850","messageId":"20110804033955.GA4534@sigill.intra.peff.net","threadId":"27987","inReplyTo":"20110802190645.GB17322@fishbowl.rw.madduck.net","subject":"Re: Storing additional information in commit headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-04T03:39:56Z","receivedAt":"2011-08-04T03:39:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 02, 2011 at 09:06:45PM +0200, martin f krafft wrote:\n\n> It just seems to me that per-ref storage is a lot further away than\n> per-commit storage, and I'd really like to move forward with TopGit…\n\nI don't think it's that hard. For example:\n\n # our mapping for all refs, and the history of that mapping, will be\n # stored under this ref\n MAP=refs/topgit/metadata\n\n refmap_set() {\n (\n # start with a pristine index based on the current map\n GIT_INDEX_FILE=\"$(git rev-parse --git-dir)/tg-meta-index\"\n export GIT_INDEX_FILE\n if git rev-parse -q --verify $MAP >/dev/null; then\n git read-tree $MAP\n fi\n\n # and then put our new ref and metadata in\n blob=`git hash-object --stdin -w`\n git update-index --add --cacheinfo 100644 $blob $1\n tree=`git write-tree`\n parent=$(git rev-parse -q --verify $MAP)\n commit=`echo 'updated map' | git commit-tree $tree ${parent:+-p $parent}`\n git update-ref $MAP $commit $old\n )\n }\n\n refmap_get() {\n git cat-file blob $MAP:$1\n }\n\n # and some examples of use\n echo some metadata | refmap_set refs/heads/foo\n refmap_get refs/heads/foo |\n sed 's/meta/changed &/' |\n refmap_set refs/heads/foo\n\nIt's a little more clunky than notes, of course, but it's not too bad to\nput into a script. The tricky part is how to handle fetching and merging\nthe metadata ref from other people. But that's not really different from\nnotes. In either case, you're probably going to want to make a custom\nmerge program for combining the meta-information.\n\n-Peff\n"},{"id":"172851","messageId":"20110804034114.GB4534@sigill.intra.peff.net","threadId":"27987","inReplyTo":"20110802192727.GB20239@fishbowl.rw.madduck.net","subject":"Re: per-ref data storage (was: Storing additional information in commit headers)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-08-04T03:41:14Z","receivedAt":"2011-08-04T03:41:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 02, 2011 at 09:27:28PM +0200, martin f krafft wrote:\n\n> [sorry, my previous message was a total reply FAIL]\n> \n> also sprach martin f krafft [2011.08.02.2106 +0200]:\n> > It just seems to me that per-ref storage is a lot further away than\n> > per-commit storage, and I'd really like to move forward with TopGit…\n> \n> refs/heads/master is a file, containing its payload in the first\n> line by format definition, right?\n> \n> I mean: the storage is right there, isn't it?\n\nYes, and I think git will even ignore other stuff in the file. But I\ndon't think you can count on git not obliterating the other stuff when\nit updates the ref. Nor would it be passed over a clone or fetch.\n\n> Of course this opens a whole new can of worms: merging per-ref data.\n\nYes. That's the tricky part. And that's something you'll have to deal\nwith no matter how you store it, I expect.\n\n-Peff\n"}]}