{"thread":{"id":"28928","subject":"Revisiting metadata storage","startedAt":"2011-11-14T00:07:16Z","lastAt":"2011-12-17T00:48:41Z","messageCount":7,"participants":["Richard Hartmann","Jonathan Nieder","Hilco Wijbenga"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"179401","messageId":"CAD77+gQB+0zJG62jrtPn_MwLLR7zgH=5gBtkvPxrKgiLPZsbsw@mail.gmail.com","threadId":"28928","inReplyTo":null,"subject":"Revisiting metadata storage","fromName":"Richard Hartmann","fromEmail":"richih.mailinglist@gmail.com","sentAt":"2011-11-14T00:07:16Z","receivedAt":"2011-11-14T00:07:16Z","isPatch":false,"sender":{"key":"richih.mailinglist@gmail.com","avatar":"https://avatars.githubusercontent.com/u/754723?v=4"},"body":"Hi all,\n\nevery few months, a thread seems to pop up regarding metadata storage,\nbe it owner, mtime, xattr or what have you.\n\nAs of today there is (ttbomk) still no sane way to carry & restore\nmetadata across different repositories.\n\nmetastore[1] is closest to a working solution but its storage format\nis binary and merge-unfriendly and it does not store mtime which is a\ndeal-breaker, for me.\nI tried extend & fix metastore, but failed. Others looked at it,\ndeemed it possible and lost interest and/or their development box.\n\nI know from Joey Hess [2] that the GitTogether 2011 saw some\ndiscussion about metadata, but I was unable to find any follow-up to\nthis issue.\n\n\nTo make a long story short: Does anyone have a working solution,\ntoday? If not, is anyone working on one? If not, is anyone interested\nin working on one? And is there any follow-up to the GitTogether\ndiscussion?\n\nThe feature set of any solution should probably include save, display,\ndiff, and apply on a per-metadata and per-file basis.\n\n\nThanks,\nRichard\n\n[1] http://david.hardeman.nu/software.php\n[2] http://kitenet.net/~joey/blog/entry/GitTogether2011/\n"},{"id":"179923","messageId":"20111124011022.GA14842@elie.hsd1.il.comcast.net","threadId":"28928","inReplyTo":"CAD77+gQB+0zJG62jrtPn_MwLLR7zgH=5gBtkvPxrKgiLPZsbsw@mail.gmail.com","subject":"Re: Revisiting metadata storage","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-11-24T01:10:23Z","receivedAt":"2011-11-24T01:10:23Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Richard Hartmann wrote:\n\n> To make a long story short: Does anyone have a working solution,\n> today?\n\nSure. etckeeper handles metadata such as owner and permissions\nreasonably well.\n"},{"id":"181176","messageId":"CAD77+gR=SjU0Ne9jort91pdHDA=RjkTJUJmnqKBipqoGUmoL_A@mail.gmail.com","threadId":"28928","inReplyTo":"87sjkx8gll.fsf@an-dro.info.enstb.org","subject":"Re: Revisiting metadata storage","fromName":"Richard Hartmann","fromEmail":"richih.mailinglist@gmail.com","sentAt":"2011-12-14T17:59:42Z","receivedAt":"2011-12-14T17:59:42Z","isPatch":false,"sender":{"key":"richih.mailinglist@gmail.com","avatar":"https://avatars.githubusercontent.com/u/754723?v=4"},"body":"On Tue, Dec 6, 2011 at 23:45, Ronan Keryell\n wrote:\n\n> At least I'm interested and began to dig into it but I do not have a lot\n> of time to work on it...\n\nIf we can agree on Perl, I can try to help. I don't think I speak\nenough Python to be of use with that.\n\nOther people who have an interest in this: Please pipe up so we can\nhammer out a rough consensus & roadmap.\n\n\nRichard\n"},{"id":"181258","messageId":"CAE1pOi2TGWmFErcKhQu-a37JjLh22O1zFYoBaVRyfBDFjOTE9Q@mail.gmail.com","threadId":"28928","inReplyTo":"CAD77+gR=SjU0Ne9jort91pdHDA=RjkTJUJmnqKBipqoGUmoL_A@mail.gmail.com","subject":"Re: Revisiting metadata storage","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2011-12-15T21:40:36Z","receivedAt":"2011-12-15T21:40:36Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 14 December 2011 09:59, Richard Hartmann\n wrote:\n> On Tue, Dec 6, 2011 at 23:45, Ronan Keryell\n> wrote:\n>\n>> At least I'm interested and began to dig into it but I do not have a lot\n>> of time to work on it...\n>\n> If we can agree on Perl, I can try to help. I don't think I speak\n> enough Python to be of use with that.\n>\n> Other people who have an interest in this: Please pipe up so we can\n> hammer out a rough consensus & roadmap.\n\nI'd love to have better support for metadata (specifically\ntimestamps). I don't care whether it's Perl, Python, Bash, or C. I\ndon't think I'll be much help coding but I'd like to try.\n\nIn all honesty though, I plan to rewrite our build to use file digests\ninstead of timestamps. Right now every rebase means a full (and almost\ncompletely unnecessary) rebuild. Luckily I'm using the wonderful\ngit-new-workdir so there is no pain when switching branches.\n\nOnce the rewrite is complete (in one or two months) Git's relentless\ntimestamp changes should no longer affect us as much anymore. I would\nstill like to get a better grip on metadata though. Git should be able\nto not touch files that have not changed. But whether that's feasible\nor even in scope of what you have in mind... :-)\n\nCheers,\nHilco\n"},{"id":"181282","messageId":"20111216075251.GA4048@elie.hsd1.il.comcast.net","threadId":"28928","inReplyTo":"CAE1pOi2TGWmFErcKhQu-a37JjLh22O1zFYoBaVRyfBDFjOTE9Q@mail.gmail.com","subject":"Re: Revisiting metadata storage","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-12-16T07:52:51Z","receivedAt":"2011-12-16T07:52:51Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hilco Wijbenga wrote:\n\n> Right now every rebase means a full (and almost\n> completely unnecessary) rebuild.\n\nIt sounds like what you are suffering from is that \"git rebase\" uses\nthe worktree as its workspace instead of doing all that work\nin-memory, right?\n\nIf I were in your situation, I would do the following:\n\n 1. Don't rebase so often. When wanting to take advantage of features\n from a new upstream version, use \"git merge\" to pull it in. Only\n rebase when it is time to make the history presentable for other\n people.\n\n This way, \"git log --first-parent\" will give easy access to\n the intermediate versions you have hacked on and tested recently.\n\n 2. When history gets ugly and you want to rebase to make the series\n easier to make sense of, use a separate workdir:\n\n\t$ git branch tmp; # make a copy to rebase\n\n\t$ cd ..\n\t$ git new-workdir repo rebase-scratch tmp\n\t$ cd rebase-scratch\n\t$ git rebase -i origin/master\n\t...\n\t$ cd ..\n\t$ rm -fr rebase-scratch\n\n\t$ cd repo\n\t$ git diff HEAD tmp;\t# Does the rebased version look better?\n\t$ git reset --keep tmp;\t# Yes. Use it.\n\t$ git branch -d tmp\n\n 3. Once the rebased history looks reasonably good, be sure to rebase\n one final time and test each commit before submitting for other\n people's use.\n\nHope that helps,\nJonathan\n"},{"id":"181318","messageId":"CAE1pOi2GW=3o7QgTEcUYbjif3WokpVdgL6UdKXu9x0yKH-vrGw@mail.gmail.com","threadId":"28928","inReplyTo":"20111216075251.GA4048@elie.hsd1.il.comcast.net","subject":"Re: Revisiting metadata storage","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2011-12-16T18:55:52Z","receivedAt":"2011-12-16T18:55:52Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 15 December 2011 23:52, Jonathan Nieder wrote:\n> Hilco Wijbenga wrote:\n>\n>>                        Right now every rebase means a full (and almost\n>> completely unnecessary) rebuild.\n>\n> It sounds like what you are suffering from is that \"git rebase\" uses\n> the worktree as its workspace instead of doing all that work\n> in-memory, right?\n\nYes, I guess the problem is that it uses the worktree as its workspace.\n\n(I know others disagree but to me it's a bug that Git touches files\nthat it doesn't actually change.)\n\n> If I were in your situation, I would do the following:\n>\n>  1. Don't rebase so often.  When wanting to take advantage of features\n>    from a new upstream version, use \"git merge\" to pull it in.  Only\n>    rebase when it is time to make the history presentable for other\n>    people.\n\nI usually rebase in the morning to get an up-to-date tree. Is that\nconsidered too often? Perhaps it's my Subversion background but I'm\nnot comfortable diverging too much. Is that too paranoid? :-)\n\nSo IIUC, I can do \"git rebase master\" even after multiple \"git merge master\"s?\n\n>    This way, \"git log --first-parent\" will give easy access to\n>    the intermediate versions you have hacked on and tested recently.\n\nWhy is \"git log --first-parent\" important? I read \"git help log\" on\nfirst-parent but that didn't really tell me much. Google was not very\nhelpful either.\n\n>  2. When history gets ugly and you want to rebase to make the series\n>    easier to make sense of, use a separate workdir:\n>\n>        $ git branch tmp; # make a copy to rebase\n\nThis is in my merged branch, right?\n\n>\n>        $ cd ..\n>        $ git new-workdir repo rebase-scratch tmp\n>        $ cd rebase-scratch\n>        $ git rebase -i origin/master\n>        ...\n>        $ cd ..\n>        $ rm -fr rebase-scratch\n>\n>        $ cd repo\n>        $ git diff HEAD tmp;    # Does the rebased version look better?\n>        $ git reset --keep tmp; # Yes.  Use it.\n>        $ git branch -d tmp\n\nInteresting. If I run the rebase after the merge, rebase appears to do\nmuch less work. I.e. it appears to only touch files that have actually\nchanged. Is that true?\n\n>  3. Once the rebased history looks reasonably good, be sure to rebase\n>    one final time and test each commit before submitting for other\n>    people's use.\n>\n> Hope that helps,\n\nYes, thanks for pointing out yet more useful Git options. There seems\nno end to them. :-)\n\nCheers,\nHilco\n"},{"id":"181357","messageId":"20111217004841.GB5709@elie.hsd1.il.comcast.net","threadId":"28928","inReplyTo":"CAE1pOi2GW=3o7QgTEcUYbjif3WokpVdgL6UdKXu9x0yKH-vrGw@mail.gmail.com","subject":"Re: Revisiting metadata storage","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-12-17T00:48:41Z","receivedAt":"2011-12-17T00:48:41Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(+cc: Martin, who has been doing excellent work on \"git rebase\", just\n in case he's curious)\nHi again,\n\nHilco Wijbenga wrote:\n\n> Yes, I guess the problem is that it uses the worktree as its workspace.\n\nThat's a comfort. Thanks for explaining.\n\n> (I know others disagree but to me it's a bug that Git touches files\n> that it doesn't actually change.)\n\nNo, I somewhat agree. If a command is touching more files than it needs\nto, then that is likely to be a bug, or at least an opportunity for\nimprovement.\n\nWhere we might disagree is in how many files \"git rebase\" needs to\ntouch. So let's consider your use case.\n\n[...]\n> I usually rebase in the morning to get an up-to-date tree. Is that\n> considered too often? Perhaps it's my Subversion background but I'm\n> not comfortable diverging too much. Is that too paranoid? :-)\n>\n> So IIUC, I can do \"git rebase master\" even after multiple \"git merge master\"s?\n\nThe second question is easy --- the answer is \"yes\".\n\nYour first question is more a matter of opinion. I will just say a\nlittle about \"git rebase\", to help you decide for yourself.\n\nThe original and still-primary purpose of \"git rebase\" is to refresh and\nclean up a short patch series that is going to be submitted by email\nto some project, by making the series apply to a newer basis version.\nYou can imitate what it does fairly simply by hand:\n\n\t# on master\n\tgit checkout -b master-rebased new-upstream\n\tgit cherry-pick HEAD..master; # [*]\n\tgit branch -M master\n\nThat is, we check out the new basis version and apply any \"local\"\nchanges on top of it one at a time, using human help to resolve\nconflicts as necessary.\n\nThis procedure has the nice property that it is dead simple. It also\nis easily tweaked to produce an \"interactive\" variant that reorders\nthe patches or runs other commands in between applying patches (for\nexample, you can ask git to run the test suite after each commit when\nrebasing by adding \"exec make test\" after each \"pick\" line in the\neditor shown by \"git rebase -i\"). And in the end you have a nice\npatch series that applies without fuzz and doesn't require people\nreading your patches to think about the older code base they were\noriginally written against.\n\nHowever, rebasing has a few downsides.\n\nThe most important one is that each time you rebase, you are making\nnew, untested commits. When you rebase the 300-patch series that\nyou have been debugging in collaboration with other people, you\ncan no longer say \"these changes have been in use and being tested\nfor a few months now; chances are we have already ironed out most\nof the obvious bugs\". The usual heuristic that patches towards the\nbeginning of the series most likely work better and are less likely to\nhave introduced that new crash that makes your program not work at all\nno longer applies, since when you rebase, you can easily introduce\na mistake in conflict resolution. All \"local\" code is new code.\n\nIn the history\n\n A --- o --- o --- B [upstream]\n \\\n P --- Q --- R [master]\n\nafter rebasing\n\n A --- o --- o --- B --- P' --- Q' --- R' [master]\n\nit is even tempting for people to not test the intermediate commits P'\nand Q' before publishing their work, resulting in a history where\nintermediate commits involved in telling the story do not even build.\nSo building and testing old versions to track down a change in\nbehavior (e.g., with \"git bisect\") becomes hard. The history is not\nactual history.\n\nThat is easy to mitigate by only rebasing your small, _private_ patch\nseries that is not part of meaningful history. When asking others to\nincorporate the changes into permanent history, the contributor\nhopefully carefully checks over them for sanity and checks each\nintermediate version before they can be applied. And history on a\nlarge, public scale is still stable.\n\nFor similar reasons, rebasing can make life difficult for people\ntrying to write patches based on your patches. The section RECOVERING\nFROM UPSTREAM REBASE in the git-rebase(1) manual page has more on\nthat.\n\nIf you want to incorporate changes into your branch and preserve the\nhistory of well-tested commits (for example, if you are the upstream\nmaintainer, pulling in changes from other people), a command to do\nthis is git-merge(1). It does not have to rewind or rewrite anything;\nit just uses a 3-way merge algorithm to apply the new changes and\nwrites a commit indicating it has done so and with pointers to the two\nparent commits so history consumers can see the full story.\n\nAnother consideration.\n\nWhen using Subversion and working against the trunk, I find myself\nusing \"svn update\" every day and right before commiting. Otherwise, I\nmay be forced to deal with a painful conflict resolution, or worse,\ncommit a change to one file that uses an API that has been removed in\nanother file.\n\nHowever, when using git, I do not find myself needing to do that.\nInstead, most work pertaining to a particular goal happens on a branch\nspecific to that topic, I pick one version to develop against, and I\nmostly stick to it. This way, I am not distracted by irrelevant\nbreakage or other changes introduced in areas orthogonal to my topic.\n\n\"But how do you make sure your changes work with the current\ncodebase?\" you might ask. Here:\n\n\t# on branch \"topic\"\n\n\t# switch to working on an anonymous branch, or rather no\n\t# branch at all\n\tgit checkout --detach\n\t# grab latest changes to test against\n\tgit merge origin/master\n\t# test!\n\tmake test\n\nThe \"git merge\" step does not present me with the same conflicts it\ndid yesterday thanks to the \"git rerere\" facility (since I have\n[rerere] enabled set to true). If my topic needs some nontrivial\nreconciliation with the wider changes in the project (if there is an\nAPI change, say), I might use \"git merge\" when on branch topic (i.e.,\n_not_ detached) to record the resolution and use the commit message to\ndescribe what happened. Or I might just rebase.\n\nBecause of the nature of patches applied by mail, before sending the\npatches out, I either rebase one last time or very loudly mention\nwhich old version of the codebase the patch series applies to.\nUsually the former. But this step would not be necessary if asking\npeople to pull from me using a protocol that transfers the actual\nobjects.\n\nHope that helps,\nJonathan\n\n[*] Actually, \"git rebase\" is a little smarter than that, in that it\nnotices and skips patches that have already been applied in\nnew-upstream. A better imitation would be to use\n\n\tgit cherry-pick --cherry-pick --right-only HEAD...master\n\nEven better is to look at git-rebase.sh to see what it actually does.\n:)\n"}]}