{"thread":{"id":"38056","subject":"RCS Keywords in Git done right","startedAt":"2014-11-26T16:44:07Z","lastAt":"2014-12-02T17:36:08Z","messageCount":7,"participants":["Derek Moore","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"252583","messageId":"CAMsgyKbTRY5=cHj8Ar8zHDd5WdbcNwZC5caGV-snvZU4aek=YQ@mail.gmail.com","threadId":"38056","inReplyTo":null,"subject":"RCS Keywords in Git done right","fromName":"Derek Moore","fromEmail":"derek.p.moore@gmail.com","sentAt":"2014-11-26T16:44:07Z","receivedAt":"2014-11-26T16:44:07Z","isPatch":false,"sender":{"key":"derek.p.moore@gmail.com","avatar":"https://gravatar.com/avatar/4bf86633cdd04eb5d07180791de5ae0ece9f3d04b34f6e13fdb81d563fc62c23?d=mp&s=160"},"body":"Junio, et al.,\n\nI've completed my first pass at RCS Keywords in Git. I believe I've\ncome up with a solution that is accurate, performant and complete (but\nI have not tested it on big repos yet, I'm doing that today...).\n\nhttps://github.com/derekm/git-keywords\n\nThis work basically takes advantage of all the state-machine\ntransitions in git to surgically perform \"git update-index $(git\narchive $(git log -1 --format=%H @ -- $path) -- $path | tar vx)\"\noverwrites in the work tree. (It also exposes some state transitions\nthat are entirely absent from git, creating a few edge cases, but they\nare relatively unimportant edge cases if your deployed git repos will\nbe managed by an automated system [humans doing development workflows\ncan trigger the edge cases when cancelling certain operations, all\nedge cases just leave you with un-substituted files, which will become\nsubstituted again after checkouts, commits, merges, rewrites, etc.].)\n\nOnly $Author$, $Date$ and $Revision$ can be emulated presently. $Id$\nand other tags requiring filename paths or basenames are possible, but\nwould require changes internal to git allowing \"pretty format\" codes\ninside a file to triangulate filenames from blob hash and commit hash\npairs.\n\nI believe this work fundamentally proves that the theory of RCS\nkeywords is sound in the context of Git, and that full support in\ngit-core is entirely achievable in short order. In fact, other areas\nin git would become improved for several reasons if git devs ingested\nsome of the results of this work.\n\nThere is a lot of gainsaying and kneejerk reaction to the idea of\nkeywords under the assumption of distributed development because of\nthe fallacy of thinking in terms of shared/universal linear history\ninstead of in terms of relative spacetime events.\n\nKeyword substitution can be done accurately relative to the history of\nthe possessor of that history. Last edit timestamps and last authors\nand revision IDs are important to many workflows inside and outside\ndevelopment.\n\nOf the keywords emulated, the only thing I couldn't achieve\n(obviously) were monotonically increasing revision numbers, instead I\nwent with the file's most recent commit short hash (which is more\nproper for git anyway).\n\nTo test it out...\n\n1) clone the repo:\n\ngit clone https://github.com/derekm/git-keywords\n\n2) cd into the repo and setup the hooks:\n\nln -sf ../../post-checkout-filter.pl .git/hooks/post-checkout\nln -sf ../../pre-commit-check.pl .git/hooks/pre-commit\nln -sf ../../post-commit-filter.pl .git/hooks/post-commit\nln -sf ../../post-merge-filter.pl .git/hooks/post-merge\nln -sf ../../post-rewrite-filter.pl .git/hooks/post-rewrite\n\n3) edit .git/config and setup the filters:\n\n[filter \"keywords\"]\n        smudge = ./keyword-smudge.pl %f\n        clean = ./keyword-clean.pl\n\n4) inspect the lack of substitutions:\n\nhead -4 *\n\n5) initialize the repo with first substitutions:\n\nfor i in $(git ls-tree --name-only @); do\n git update-index \\\n  $(git archive \\\n   $(git log -1 --format=%H @ -- $i) -- $i | tar vx)\ndone\n\n6) inspect the presence of substitutions:\n\nhead -4 *\n\n7) ??? (start hacking, try to break it, etc.)\n\n8) PROFIT!\n\nPS: I may consider rewriting the hooks in Bash, but I need to audit\nwhat commands are available under msys-git.\n"},{"id":"252586","messageId":"CAGZ79kZz4_q+p91e7fn8uS--DRqEUPj_eeQPf2WPOWEk=R8fkw@mail.gmail.com","threadId":"38056","inReplyTo":"CAMsgyKbTRY5=cHj8Ar8zHDd5WdbcNwZC5caGV-snvZU4aek=YQ@mail.gmail.com","subject":"Re: RCS Keywords in Git done right","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2014-11-26T18:10:16Z","receivedAt":"2014-11-26T18:10:16Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 26, 2014 at 8:44 AM, Derek Moore <derek.p.moore@gmail.com> wrote:\n> Junio, et al.,\n>\n> I've completed my first pass at RCS Keywords in Git. I believe I've\n> come up with a solution that is accurate, performant and complete (but\n> I have not tested it on big repos yet, I'm doing that today...).\n>\n> https://github.com/derekm/git-keywords\n>\n> This work basically takes advantage of all the state-machine\n> transitions in git to surgically perform \"git update-index $(git\n> archive $(git log -1 --format=%H @ -- $path) -- $path | tar vx)\"\n> overwrites in the work tree. (It also exposes some state transitions\n> that are entirely absent from git, creating a few edge cases, but they\n> are relatively unimportant edge cases if your deployed git repos will\n> be managed by an automated system [humans doing development workflows\n> can trigger the edge cases when cancelling certain operations, all\n> edge cases just leave you with un-substituted files, which will become\n> substituted again after checkouts, commits, merges, rewrites, etc.].)\n\nNow knowing the edge cases won't work, I did not get an idea about the\nstandard case of what should work with this. Would you mind to write\na more detailed example or a more advertising paragraph of what this can do?\nNot getting the big picture may be related to me having not worked with RCS yet.\n\nThanks,\nStefan\n\n> Only $Author$, $Date$ and $Revision$ can be emulated presently. $Id$\n> and other tags requiring filename paths or basenames are possible, but\n> would require changes internal to git allowing \"pretty format\" codes\n> inside a file to triangulate filenames from blob hash and commit hash\n> pairs.\n>\n> I believe this work fundamentally proves that the theory of RCS\n> keywords is sound in the context of Git, and that full support in\n> git-core is entirely achievable in short order. In fact, other areas\n> in git would become improved for several reasons if git devs ingested\n> some of the results of this work.\n>\n> There is a lot of gainsaying and kneejerk reaction to the idea of\n> keywords under the assumption of distributed development because of\n> the fallacy of thinking in terms of shared/universal linear history\n> instead of in terms of relative spacetime events.\n>\n> Keyword substitution can be done accurately relative to the history of\n> the possessor of that history. Last edit timestamps and last authors\n> and revision IDs are important to many workflows inside and outside\n> development.\n>\n> Of the keywords emulated, the only thing I couldn't achieve\n> (obviously) were monotonically increasing revision numbers, instead I\n> went with the file's most recent commit short hash (which is more\n> proper for git anyway).\n>\n> To test it out...\n>\n> 1) clone the repo:\n>\n> git clone https://github.com/derekm/git-keywords\n>\n> 2) cd into the repo and setup the hooks:\n>\n> ln -sf ../../post-checkout-filter.pl .git/hooks/post-checkout\n> ln -sf ../../pre-commit-check.pl .git/hooks/pre-commit\n> ln -sf ../../post-commit-filter.pl .git/hooks/post-commit\n> ln -sf ../../post-merge-filter.pl .git/hooks/post-merge\n> ln -sf ../../post-rewrite-filter.pl .git/hooks/post-rewrite\n>\n> 3) edit .git/config and setup the filters:\n>\n> [filter \"keywords\"]\n>         smudge = ./keyword-smudge.pl %f\n>         clean = ./keyword-clean.pl\n>\n> 4) inspect the lack of substitutions:\n>\n> head -4 *\n>\n> 5) initialize the repo with first substitutions:\n>\n> for i in $(git ls-tree --name-only @); do\n>  git update-index \\\n>   $(git archive \\\n>    $(git log -1 --format=%H @ -- $i) -- $i | tar vx)\n> done\n>\n> 6) inspect the presence of substitutions:\n>\n> head -4 *\n>\n> 7) ??? (start hacking, try to break it, etc.)\n>\n> 8) PROFIT!\n>\n> PS: I may consider rewriting the hooks in Bash, but I need to audit\n> what commands are available under msys-git.\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"252595","messageId":"CAMsgyKZywnac_b2RV0dGs9zOdemJDUQ8oR=bSydBuSxp1VDixQ@mail.gmail.com","threadId":"38056","inReplyTo":"CAGZ79kZz4_q+p91e7fn8uS--DRqEUPj_eeQPf2WPOWEk=R8fkw@mail.gmail.com","subject":"Re: RCS Keywords in Git done right","fromName":"Derek Moore","fromEmail":"derek.p.moore@gmail.com","sentAt":"2014-11-26T19:22:51Z","receivedAt":"2014-11-26T19:22:51Z","isPatch":false,"sender":{"key":"derek.p.moore@gmail.com","avatar":"https://gravatar.com/avatar/4bf86633cdd04eb5d07180791de5ae0ece9f3d04b34f6e13fdb81d563fc62c23?d=mp&s=160"},"body":"> Now knowing the edge cases won't work, I did not get an idea about the\n> standard case of what should work with this. Would you mind to write\n> a more detailed example or a more advertising paragraph of what this can do?\n> Not getting the big picture may be related to me having not worked with RCS yet.\n\nStefan,\n\nRCS Keywords, while originating from RCS, are commonly used in CVS and\nSVN. A lot of LaTeX workflows in the scientific community, for\nexample, use these keyword substitutions, trapping scientists in\nlegacy SCMSes. In my environment, we do builds and deployments from\nwithin pristine working copies or \"checkouts of trunk\", we also have\nsome deployments that are symlinks into live checkouts of trunk, and\nwe have production support workflows where support personnel inspect\nfiles remotely and subsequent escalation procedures rely on the\ncontents of the $Author$ substitutions, etc. As a result of this,\nprojects that have migrated to git are demanding the restoration of\ntheir RCS keyword substitutions.\n\nIn CVS/SVN, keywords are expanded on checkout, placing text related to\nthe most recent history of a give file into that file. RCS has one\nkeyword that takes action on check-in (or commit), being the $Log$\nkeyword, which is a running commit log of the file in the file.\nKeyword expansions are not stored in the repo, but are substituted on\ntheir out of the repo into the working copy, and substitutions are\nreversed on their way from the working copy into the repo.\n\nGit's export-subst feature in git-archive is very similar to RCS\nKeywords. What I'm providing here is a mechanism to enable\nexport-subst functionality throughout normal git workflows and not\njust during builds that employ git-archive, as if export-subst worked\nalongside git's ident feature.\n\nPerhaps described the known issues I've found will also help towards\nunderstanding...\n\n\nKnown Issues\n------------\n\nEdge Case #1 (aka, modified smudge filter)\n\n1. create new branch\n2. edit smudge filter\n3. commit\n4. switch back to previous branch\n5. smudge filter is temporarily disappeared at the moment the smudge\nfilter wants to run\n\nThis edge case is a side-effect of the order in which git performs\ndeletions in the worktree and extractions from the index and\nexecutions of the filters. This edge case is related to a similar to\none seen in older git versions where the smudge is disappeared during\na \"git checkout-index -a -f\", but the sequence of operations has been\nfixed in more recent gits, so the smudge does not disappear during a\ncheckout-index.\n\n\nEdge Case #2\n\n1. create branch B from branch A\n2. make changes in branch A, commit\n3. checkout branch B\n4. git merge A\n5. while editing commit file, file being modified lacks keywords (expected)\n6. delete commit message, cancelling commit\n7. file remains w/o substituted keywords\n8. cancel merge w/ git reset --merge ORIG_HEAD & restored original\nfile is w/o substituted keywords\n\nReason: no available state transition on reset --merge\n\n\nEdge Case #3\n\n1. create branch B from branch A, checkout B\n2. modify file, commit\n3. checkout A\n4. make conflicting edit to same file\n5. git rebase B, rebase will conflict\n6. git rebase --abort\n7. file will be w/o substituted keywords\n\n\nKnown Unissues\n--------------\n\nNot-an-Edge-Case #1\n\n1. create branch B from branch A, checkout B\n2. modify file, commit\n3. checkout A\n4. git merge --squash B\n5. file as modified from B is w/o substituted keywords\nAS EXPECTED - that version of file does not yet contain history in A,\nfile will gain substitutions following commit\n"},{"id":"252605","messageId":"CAGZ79kZLAHDG8h5DMQdOH2cQtaMs_iCtC-xsoKst966a+jaBNA@mail.gmail.com","threadId":"38056","inReplyTo":"CAMsgyKZywnac_b2RV0dGs9zOdemJDUQ8oR=bSydBuSxp1VDixQ@mail.gmail.com","subject":"Re: RCS Keywords in Git done right","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2014-11-26T21:15:14Z","receivedAt":"2014-11-26T21:15:14Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Nov 26, 2014 at 11:22 AM, Derek Moore <derek.p.moore@gmail.com> wrote:\n>> Now knowing the edge cases won't work, I did not get an idea about the\n>> standard case of what should work with this. Would you mind to write\n>> a more detailed example or a more advertising paragraph of what this can do?\n>> Not getting the big picture may be related to me having not worked with RCS yet.\n>\n> Stefan,\n>\n> RCS Keywords, while originating from RCS, are commonly used in CVS and\n> SVN. A lot of LaTeX workflows in the scientific community, for\n> example, use these keyword substitutions, trapping scientists in\n> legacy SCMSes. In my environment, we do builds and deployments from\n> within pristine working copies or \"checkouts of trunk\", we also have\n> some deployments that are symlinks into live checkouts of trunk, and\n> we have production support workflows where support personnel inspect\n> files remotely and subsequent escalation procedures rely on the\n> contents of the $Author$ substitutions, etc. As a result of this,\n> projects that have migrated to git are demanding the restoration of\n> their RCS keyword substitutions.\n>\n> In CVS/SVN, keywords are expanded on checkout, placing text related to\n> the most recent history of a give file into that file. RCS has one\n> keyword that takes action on check-in (or commit), being the $Log$\n> keyword, which is a running commit log of the file in the file.\n> Keyword expansions are not stored in the repo, but are substituted on\n> their out of the repo into the working copy, and substitutions are\n> reversed on their way from the working copy into the repo.\n\nThanks for the explanation!\n\n\n>\n> Git's export-subst feature in git-archive is very similar to RCS\n> Keywords. What I'm providing here is a mechanism to enable\n> export-subst functionality throughout normal git workflows and not\n> just during builds that employ git-archive, as if export-subst worked\n> alongside git's ident feature.\n>\n> Perhaps described the known issues I've found will also help towards\n> understanding...\n>\n>\n> Known Issues\n> ------------\n>\n> Edge Case #1 (aka, modified smudge filter)\n>\n> 1. create new branch\n> 2. edit smudge filter\n> 3. commit\n> 4. switch back to previous branch\n> 5. smudge filter is temporarily disappeared at the moment the smudge\n> filter wants to run\n>\n> This edge case is a side-effect of the order in which git performs\n> deletions in the worktree and extractions from the index and\n> executions of the filters. This edge case is related to a similar to\n> one seen in older git versions where the smudge is disappeared during\n> a \"git checkout-index -a -f\", but the sequence of operations has been\n> fixed in more recent gits, so the smudge does not disappear during a\n> checkout-index.\n>\n>\n> Edge Case #2\n>\n> 1. create branch B from branch A\n> 2. make changes in branch A, commit\n> 3. checkout branch B\n> 4. git merge A\n> 5. while editing commit file, file being modified lacks keywords (expected)\n> 6. delete commit message, cancelling commit\n> 7. file remains w/o substituted keywords\n> 8. cancel merge w/ git reset --merge ORIG_HEAD & restored original\n> file is w/o substituted keywords\n>\n> Reason: no available state transition on reset --merge\n>\n>\n> Edge Case #3\n>\n> 1. create branch B from branch A, checkout B\n> 2. modify file, commit\n> 3. checkout A\n> 4. make conflicting edit to same file\n> 5. git rebase B, rebase will conflict\n> 6. git rebase --abort\n> 7. file will be w/o substituted keywords\n>\n>\n> Known Unissues\n> --------------\n>\n> Not-an-Edge-Case #1\n>\n> 1. create branch B from branch A, checkout B\n> 2. modify file, commit\n> 3. checkout A\n> 4. git merge --squash B\n> 5. file as modified from B is w/o substituted keywords\n> AS EXPECTED - that version of file does not yet contain history in A,\n> file will gain substitutions following commit\n"},{"id":"252868","messageId":"CAMsgyKZWr-1isLvRXMFdzOYu0Yfm3vN_bdk4oRg6UhzSOMq_yQ@mail.gmail.com","threadId":"38056","inReplyTo":"CAGZ79kZLAHDG8h5DMQdOH2cQtaMs_iCtC-xsoKst966a+jaBNA@mail.gmail.com","subject":"Re: RCS Keywords in Git done right","fromName":"Derek Moore","fromEmail":"derek.p.moore@gmail.com","sentAt":"2014-12-02T16:31:02Z","receivedAt":"2014-12-02T16:31:02Z","isPatch":false,"sender":{"key":"derek.p.moore@gmail.com","avatar":"https://gravatar.com/avatar/4bf86633cdd04eb5d07180791de5ae0ece9f3d04b34f6e13fdb81d563fc62c23?d=mp&s=160"},"body":"I've finished testing this work in larger repositories.\n\nWhile the approach is performant and works nicely in small repos, but\nin larger repos one of the requirements for the \"correctness\" of\nsubstitutions slows things down (1 or 2 minutes to perform checkouts\nbetween branches with 10,000+ files).\n\nThe operation that is slowing things down is discovering the relative\ncomplement of commits between the common files of two branches (i.e.,\nwhich files are common between two branches but differ in their latest\ncommit).\n\nMy current approach is:\n1) find files common between @ & @{-1}, \"ls-tree --full-tree\n--name-only -r\" both branches, take the intersection\n2) find current branch's commits for common files, for each file in\nintersection \"log -1 --format=%H $current_branch -- $file\"\n3) find common files where latest commits differ, for each file in\nintersection keep the file if current branche's latest commit does not\nequal prior branch's latest commit\n4) overwrite all kept files with the results of git-archive\n\nIt is steps 2 & 3 that consume the most time in a large repo with\nlarge intersections of common files between branches.\n\nI've tried to conceive of other ways to arriving at the same\n\"filename\"/\"latest current branch commit hash\" pairs where filenames\nare common between branches and latest current branch commit hash\ndiffers from latest prior branch commit hash. I've thought maybe I\ncould traverse commits starting from merge-base instead of traversing\nfiles, but that doesn't seem like it would be a huge improvement.\n\nI'm sure internal to git in C there would be a better/faster way (and\nit would probably look like writing Btrieve queries). Can anyone think\nof a good solution for the intersection of files and complement of\ncommits using only the git CLI tools?\n\nThanks,\n\nDerek\n"},{"id":"252873","messageId":"CAMsgyKbd8Eq07LUktaVy7zyFQwOJ3KOrkJCipp6P9F1_W=OYCg@mail.gmail.com","threadId":"38056","inReplyTo":"CAMsgyKZWr-1isLvRXMFdzOYu0Yfm3vN_bdk4oRg6UhzSOMq_yQ@mail.gmail.com","subject":"Re: RCS Keywords in Git done right","fromName":"Derek Moore","fromEmail":"derek.p.moore@gmail.com","sentAt":"2014-12-02T17:03:05Z","receivedAt":"2014-12-02T17:03:05Z","isPatch":false,"sender":{"key":"derek.p.moore@gmail.com","avatar":"https://gravatar.com/avatar/4bf86633cdd04eb5d07180791de5ae0ece9f3d04b34f6e13fdb81d563fc62c23?d=mp&s=160"},"body":"> My current approach is:\n> 1) find files common between @ & @{-1}, \"ls-tree --full-tree\n> --name-only -r\" both branches, take the intersection\n> 2) find current branch's commits for common files, for each file in\n> intersection \"log -1 --format=%H $current_branch -- $file\"\n> 3) find common files where latest commits differ, for each file in\n> intersection keep the file if current branche's latest commit does not\n> equal prior branch's latest commit\n> 4) overwrite all kept files with the results of git-archive\n\nPS: In large repos, I can dump the entire contents of the repo out of\ngit-archive faster than I can look up the commits of common files\nbetween two branches for a more limited and surgical dump from\ngit-archive (say, 30 seconds to dump everything out of git-archive vs.\n1 minute 30 seconds to find the intersection of files and look up the\nlatest commits).\n"},{"id":"252880","messageId":"CAMsgyKasQ=DZ77e6HJW6u03g9RHsJedG_SQDW0X=-V_9bAYA0w@mail.gmail.com","threadId":"38056","inReplyTo":"CAMsgyKbd8Eq07LUktaVy7zyFQwOJ3KOrkJCipp6P9F1_W=OYCg@mail.gmail.com","subject":"Re: RCS Keywords in Git done right","fromName":"Derek Moore","fromEmail":"derek.p.moore@gmail.com","sentAt":"2014-12-02T17:36:08Z","receivedAt":"2014-12-02T17:36:08Z","isPatch":false,"sender":{"key":"derek.p.moore@gmail.com","avatar":"https://gravatar.com/avatar/4bf86633cdd04eb5d07180791de5ae0ece9f3d04b34f6e13fdb81d563fc62c23?d=mp&s=160"},"body":"PPS: Sounds like I need Peff's git-blame-tree from here:\nhttps://github.com/peff/git/compare/jk/faster-blame-tree\n"}]}