{"thread":{"id":"5718","subject":"[RFC] git-split: Split the history of a git repository by subdirectories and ranges","startedAt":"2006-09-27T08:05:56Z","lastAt":"2006-10-25T02:13:57Z","messageCount":19,"participants":["Josh Triplett","Junio C Hamano","Andy Whitcroft","Linus Torvalds","Jakub Narebski","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"27772","messageId":"451A30E4.50801@freedesktop.org","threadId":"5718","inReplyTo":null,"subject":"[RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Josh Triplett","fromEmail":"josh@freedesktop.org","sentAt":"2006-09-27T08:05:56Z","receivedAt":"2006-09-27T08:05:56Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"Hello,\n\nI co-maintain the X C Binding (XCB) project with Jamey Sharp.\nPreviously, several XCB-related projects all existed under the umbrella\nof a single monolithic GIT repository with per-project subdirectories.\nWe have split this repository into individual per-project repositories.\n\nJamey Sharp and I wrote a script called git-split to accomplish this\nrepository split. git-split reconstructs the history of a sub-project\npreviously stored in a subdirectory of a larger repository. It\nconstructs new commit objects based on the existing tree objects for the\nsubtree in each commit, and discards commits which do not affect the\nhistory of the sub-project, as well as merges made unnecessary due to\nthese discarded commits.  When git-split finishes, it will output the\nsha1 for the new head commit, suitable for redirection into a file in\n.git/refs/heads.  At that point, you can clone the new head, or copy the\nrepository and prune out undesired heads, tags, and objects.\n\nI have attached git-split for review.  If the GIT community has any\ninterest in seeing git-split become a part of GIT, we can write up the\nnecessary documentation and patch.\n\nWe would like to acknowledge the work of the gobby team in creating a\ncollaborative editor which greatly aided the development of git-split.\n\n- Josh Triplett\n\n\n#!/usr/bin/python\n# git-split: Split the history of a git repository by subdirectories and ranges\n# Copyright (C) 2006 Jamey Sharp, Josh Triplett\n#\n# You can redistribute this software and/or modify it under the terms of\n# the GNU General Public License as published by the Free Software\n# Foundation; version 2 dated June, 1991.\n#\n# This program is distributed in the hope that it will be useful, but\n# WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU\n# General Public License for more details.\n\nfrom itertools import izip\nfrom subprocess import Popen, PIPE\nimport os, sys\n\ndef run(cmd, stdin=None, env={}):\n    newenv = os.environ.copy()\n    newenv.update(env)\n    return Popen(cmd, stdin=PIPE, stdout=PIPE, env=newenv).communicate(stdin)[0]\n\ndef parse_author_date(s):\n    \"\"\"Given a GIT author or committer string, return (name, email, date)\"\"\"\n    (name, email, time, timezone) = s.rsplit(None, 3)\n    return (name, email[1:-1], time + \" \" + timezone)\n\ndef get_subtree(tree, name):\n    output = run([\"git-ls-tree\", tree, name])\n    if not output:\n        return None\n    return output.split()[2]\n\ndef is_ancestor(new_commits, cur, other):\n    \"\"\"Return True if cur has other as an ancestor, or False otherwise.\"\"\"\n    return run([\"git-merge-base\", cur, other]).strip() == other\n\ndef walk(commits, new_commits, commit_hash, project):\n    commit = commits[commit_hash]\n    if not(commit.has_key(\"new_hash\")):\n        tree = get_subtree(commit[\"tree\"], project)\n        commit[\"new_tree\"] = tree\n        if not tree:\n            raise Exception(\"Did not find project in tree for commit \" + commit_hash)\n        new_parents = list(set([walk(commits, new_commits, parent, project)\n                                for parent in commit[\"parents\"]]))\n\n        new_hash = None\n        if len(new_parents) == 1:\n            new_hash = new_parents[0]\n        elif len(new_parents) == 2: # Check for unnecessary merge\n            if is_ancestor(new_commits, new_parents[0], new_parents[1]):\n                new_hash = new_parents[0]\n            elif is_ancestor(new_commits, new_parents[1], new_parents[0]):\n                new_hash = new_parents[1]\n        if new_hash and new_commits[new_hash][\"new_tree\"] != tree:\n            new_hash = None\n\n        if not new_hash:\n            args = [\"git-commit-tree\", tree]\n            for new_parent in new_parents:\n                args.extend([\"-p\", new_parent])\n            env = dict(zip([\"GIT_AUTHOR_\"+n for n in [\"NAME\", \"EMAIL\", \"DATE\"]],\n                           parse_author_date(commit[\"author\"]))\n                       +zip([\"GIT_COMMITTER_\"+n for n in [\"NAME\", \"EMAIL\", \"DATE\"]],\n                            parse_author_date(commit[\"committer\"])))\n            new_hash = run(args, commit[\"message\"], env).strip()\n\n        commit[\"new_parents\"] = new_parents\n        commit[\"new_hash\"] = new_hash\n        if new_hash not in new_commits:\n            new_commits[new_hash] = commit\n    return commit[\"new_hash\"]\n\ndef main(args):\n    if not(1 <= len(args) <= 3):\n        print \"Usage: git-split subdir [newest [oldest]]\"\n        return 1\n\n    project = args[0]\n    if len(args) > 1:\n        newest = args[1]\n    else:\n        newest = \"HEAD\"\n    newest_hash = run([\"git-rev-parse\", newest]).strip()\n    if len(args) > 2:\n        oldest = args[2]\n        oldest_hash = run([\"git-rev-parse\", oldest]).strip()\n    else:\n        oldest_hash = None\n\n    grafts = {}\n    try:\n        for line in file(\".git/info/grafts\").read().split(\"\\n\"):\n            if line:\n                child, parents = line.split(None, 1)\n                parents = parents.split()\n                grafts[child] = parents\n    except IOError:\n        pass\n\n    temp = run([\"git-log\", \"--pretty=raw\", newest_hash]).split(\"\\n\\n\")\n    commits = {}\n    for headers,message in izip(temp[0::2], temp[1::2]):\n        commit = {}\n        commit_hash = None\n        headers = [header.split(None, 1) for header in headers.split(\"\\n\")]\n        for key,value in headers:\n            if key == \"parent\":\n                commit.setdefault(\"parents\", []).append(value)\n            elif key == \"commit\":\n                commit_hash = value\n            else:\n                if key in commit:\n                    raise Exception('Duplicate key \"%s\"' % key)\n                commit[key] = value\n        commit[\"message\"] = \"\".join([line[4:]+\"\\n\"\n                                      for line in message.split(\"\\n\")])\n        if commit_hash is None:\n            raise Exception(\"Commit without hash\")\n        if commit_hash in grafts:\n            commit[\"parents\"] = grafts[commit_hash]\n        if commit_hash == oldest_hash or \"parents\" not in commit:\n            commit[\"parents\"] = []\n        commits[commit_hash] = commit\n\n    print walk(commits, dict(), newest_hash, project)\n\ntry:\n    import psyco\n    psyco.full()\nexcept ImportError:\n    pass\n\nif __name__ == \"__main__\": sys.exit(main(sys.argv[1:]))\n"},{"id":"27780","messageId":"7vlko5d3bx.fsf@assigned-by-dhcp.cox.net","threadId":"5718","inReplyTo":"451A30E4.50801@freedesktop.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-27T10:13:54Z","receivedAt":"2006-09-27T10:13:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Triplett <josh@freedesktop.org> writes:\n\n> Jamey Sharp and I wrote a script called git-split to accomplish this\n> repository split. git-split reconstructs the history of a sub-project\n> previously stored in a subdirectory of a larger repository. It\n> constructs new commit objects based on the existing tree objects for the\n> subtree in each commit, and discards commits which do not affect the\n> history of the sub-project, as well as merges made unnecessary due to\n> these discarded commits.\n\nVery nicely done.\n\n> We would like to acknowledge the work of the gobby team in creating a\n> collaborative editor which greatly aided the development of git-split.\n\n> from itertools import izip\n> from subprocess import Popen, PIPE\n> import os, sys\n\nHow recent a Python are we assuming here?  Is late 2.4 recent\nenough?\n\n> def walk(commits, new_commits, commit_hash, project):\n>     commit = commits[commit_hash]\n>     if not(commit.has_key(\"new_hash\")):\n>         tree = get_subtree(commit[\"tree\"], project)\n>         commit[\"new_tree\"] = tree\n>         if not tree:\n>             raise Exception(\"Did not find project in tree for commit \" + commit_hash)\n>         new_parents = list(set([walk(commits, new_commits, parent, project)\n>                                 for parent in commit[\"parents\"]]))\n>\n>         new_hash = None\n>         if len(new_parents) == 1:\n>             new_hash = new_parents[0]\n>         elif len(new_parents) == 2: # Check for unnecessary merge\n>             if is_ancestor(new_commits, new_parents[0], new_parents[1]):\n>                 new_hash = new_parents[0]\n>             elif is_ancestor(new_commits, new_parents[1], new_parents[0]):\n>                 new_hash = new_parents[1]\n>         if new_hash and new_commits[new_hash][\"new_tree\"] != tree:\n>             new_hash = None\n\nThis is a real gem.  I really like reading well-written Python\nprograms.\n\nWhen git-rev-list (or \"git-log --pretty=raw\" that you use in\nyour main()) simplifies the merge history based on subtree, we\nlook at the merge and if the tree matches any of the parent we\ndiscard other parents and make the history a single strand of\npearls.  However for this application that is not what you want,\nso I can see why you run full \"git-log\" and prune history by\nhand here like this.\n\nI wonder if using \"git-log --full-history -- $project\" to let\nthe core side omit commits that do not change the $project (but\nstill give you all merged branches) would have made your job any\neasier?\n\nYou are handling grafts by hand because --pretty=raw is special\nin that it displays the real parents (although traversal does\nuse grafts).  Maybe it would have helped if we had a --pretty\nformat that is similar to raw but rewrites the parents?\n"},{"id":"27784","messageId":"451A6788.5030808@shadowen.org","threadId":"5718","inReplyTo":"7vlko5d3bx.fsf@assigned-by-dhcp.cox.net","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Andy Whitcroft","fromEmail":"apw@shadowen.org","sentAt":"2006-09-27T11:59:04Z","receivedAt":"2006-09-27T11:59:04Z","isPatch":false,"sender":{"key":"apw@shadowen.org","avatar":"https://gravatar.com/avatar/d3088262854661a913ef35cc40fedcc270142d4461791142bc1ea0b2a4e2e147?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> Josh Triplett <josh@freedesktop.org> writes:\n> \n>> Jamey Sharp and I wrote a script called git-split to accomplish this\n>> repository split. git-split reconstructs the history of a sub-project\n>> previously stored in a subdirectory of a larger repository. It\n>> constructs new commit objects based on the existing tree objects for the\n>> subtree in each commit, and discards commits which do not affect the\n>> history of the sub-project, as well as merges made unnecessary due to\n>> these discarded commits.\n> \n> Very nicely done.\n> \n>> We would like to acknowledge the work of the gobby team in creating a\n>> collaborative editor which greatly aided the development of git-split.\n> \n>> from itertools import izip\n>> from subprocess import Popen, PIPE\n>> import os, sys\n> \n> How recent a Python are we assuming here?  Is late 2.4 recent\n> enough?\n> \n>> def walk(commits, new_commits, commit_hash, project):\n>>     commit = commits[commit_hash]\n>>     if not(commit.has_key(\"new_hash\")):\n>>         tree = get_subtree(commit[\"tree\"], project)\n>>         commit[\"new_tree\"] = tree\n>>         if not tree:\n>>             raise Exception(\"Did not find project in tree for commit \" + commit_hash)\n>>         new_parents = list(set([walk(commits, new_commits, parent, project)\n>>                                 for parent in commit[\"parents\"]]))\n>>\n>>         new_hash = None\n>>         if len(new_parents) == 1:\n>>             new_hash = new_parents[0]\n>>         elif len(new_parents) == 2: # Check for unnecessary merge\n>>             if is_ancestor(new_commits, new_parents[0], new_parents[1]):\n>>                 new_hash = new_parents[0]\n>>             elif is_ancestor(new_commits, new_parents[1], new_parents[0]):\n>>                 new_hash = new_parents[1]\n>>         if new_hash and new_commits[new_hash][\"new_tree\"] != tree:\n>>             new_hash = None\n> \n> This is a real gem.  I really like reading well-written Python\n> programs.\n> \n> When git-rev-list (or \"git-log --pretty=raw\" that you use in\n> your main()) simplifies the merge history based on subtree, we\n> look at the merge and if the tree matches any of the parent we\n> discard other parents and make the history a single strand of\n> pearls.  However for this application that is not what you want,\n> so I can see why you run full \"git-log\" and prune history by\n> hand here like this.\n> \n> I wonder if using \"git-log --full-history -- $project\" to let\n> the core side omit commits that do not change the $project (but\n> still give you all merged branches) would have made your job any\n> easier?\n> \n> You are handling grafts by hand because --pretty=raw is special\n> in that it displays the real parents (although traversal does\n> use grafts).  Maybe it would have helped if we had a --pretty\n> format that is similar to raw but rewrites the parents?\n\nI have wondered recently why grafts are hidden in this way.  I feel they\nare something I want to know is occuring in my history as this history\nis being manipulated.  Perhaps we could emit a graft record in the\noutput, indeed have a graft object there?  Someone could commit a\nchange, then graft over it in the history so I'd not see it even though\nits in my working copy.\n\nFor instance in my historical git tree I have the following, note the\nlack of a parent.  If I move the graft up one commit, then I get a\nparent, but not a parent that points at the next commit.\n\n  commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2\n  tree 0bba044c4ce775e45a88a51686b5d9f90697ea9d\n  author Linus Torvalds <torvalds@ppc970.osdl.org> 1113690036 -0700\n  committer Linus Torvalds <torvalds@ppc970.osdl.org> 1113690036 -0700\n\n  [...]\n\n  commit e7e173af42dbf37b1d946f9ee00219cb3b2bea6a\n  tree 0bba044c4ce775e45a88a51686b5d9f90697ea9d\n  parent 607899e17218b485a021c6ebb1cff771fd690eec\n  author Linus Torvalds <torvalds@ppc970.osdl.org> 1112580513 -0700\n  committer Linus Torvalds <torvalds@ppc970.osdl.org> 1112580513 -0700\n\nIt might be nice to have it more like the following, with a graft in\nthere, N*40 would obviously be fakes in the first instance as the object\nisn't modified.  M*40 would refer to the old parent if there was one,\nelse NONE or 0*40 perhaps.\n\n  commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2\n  tree 0bba044c4ce775e45a88a51686b5d9f90697ea9d\n  parent NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n  author Linus Torvalds <torvalds@ppc970.osdl.org> 1113690036 -0700\n  committer Linus Torvalds <torvalds@ppc970.osdl.org> 1113690036 -0700\n\n  [...]\n\n  graft NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\n  parent e7e173af42dbf37b1d946f9ee00219cb3b2bea6a\n\n    OLD: MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM\n\n  commit e7e173af42dbf37b1d946f9ee00219cb3b2bea6a\n  tree 0bba044c4ce775e45a88a51686b5d9f90697ea9d\n  parent 607899e17218b485a021c6ebb1cff771fd690eec\n  author Linus Torvalds <torvalds@ppc970.osdl.org> 1112580513 -0700\n  committer Linus Torvalds <torvalds@ppc970.osdl.org> 1112580513 -0700\n\nI guess the other option would be to annotate the previous commit,\nperhaps on the parent line so we can see the 'right' data in the normal\nplace, but the overridden data is right there and grepable.\n\n  parent e7e173af42dbf37b1d946f9ee00219cb3b2bea6a GRAFT M*40\nor\n  parent e7e173af42dbf37b1d946f9ee00219cb3b2bea6a GRAFT NONE\n\n-apw\n"},{"id":"27805","messageId":"7vr6xxb00m.fsf@assigned-by-dhcp.cox.net","threadId":"5718","inReplyTo":"451A6788.5030808@shadowen.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-27T19:08:25Z","receivedAt":"2006-09-27T19:08:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andy Whitcroft <apw@shadowen.org> writes:\n\n>> You are handling grafts by hand because --pretty=raw is special\n>> in that it displays the real parents (although traversal does\n>> use grafts).  Maybe it would have helped if we had a --pretty\n>> format that is similar to raw but rewrites the parents?\n>\n> I have wondered recently why grafts are hidden in this way.  I feel they\n> are something I want to know is occuring in my history as this history\n> is being manipulated.\n\nJust to make sure we are on the same page, only \"raw\" format\noutput is special and it is special only on output.  Ancestry\ntraversal always honors what you have in grafts.\n\nHowever, you can do:\n\n$ git rev-list --parents --pretty=raw\n\nwhich would give you \"commit $this_commit $its $parents\" lines\nand \"parent $true_parent\" lines at the same time.\n\nAnd they will be inconsistent when you have grafts or path\nlimiter.  The former honor grafts and path limiter, and the\nlatter show the true set of parents.\n"},{"id":"27807","messageId":"7v1wpxayyd.fsf@assigned-by-dhcp.cox.net","threadId":"5718","inReplyTo":"7vr6xxb00m.fsf@assigned-by-dhcp.cox.net","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-27T19:31:22Z","receivedAt":"2006-09-27T19:31:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n> However, you can do:\n>\n> $ git rev-list --parents --pretty=raw\n>\n> which would give you \"commit $this_commit $its $parents\" lines\n> and \"parent $true_parent\" lines at the same time.\n>\n> And they will be inconsistent when you have grafts or path\n> limiter.  The former honor grafts and path limiter, and the\n> latter show the true set of parents.\n\n-- >8 --\nAn illustration of rev-list --parents --pretty=raw\n\nThis script creates two separate histories, A and B, each of\nwhich does:\n\n      (A0, B0): create fileA and subdir/fileB\n      (A1, B1): modify fileA\n      (A2, B2): modify subdir/fileB\n\nand then grafts them together to make B0 a child of A2.  So\nthe final history looks like (time flows from top to bottom):\n\n\t\ttrue parent\ttouches subdir?\n\n\tA0\tnone\t\tyes (creates it)\n        A1      A0\t\tno\n        A2\tA1\t\tyes\n        B0\tnone\t\tyes (different from what's in A2)\n        B1\tB0\t\tno\n        B2\tB1\t\tyes\n\n\"git rev-list --parents --pretty=raw B2\" would give \"fake\"\nparents on the \"commit \" header lines while \"parent \" header\nlines show the parent as recorded in the commit object (i.e. B0\nappears to have A2 as its parent on \"commit \" header but there\nis no \"parent A2\" header line in it).\n\nWhen you have path limiters, we simplify history to omit\ncommits that do not affect the specified paths.  \n\nSo \"git rev-list --parents --pretty=raw B2 subdir\" would return\n\"B2 B0 A2 A0\" (because B1 and A1 do not touch the path).  When\nit does so, the \"commit \" header lines have \"fake\" parents\n(i.e. B2 appears to have B0 as its parent on \"commit \" header),\nbut you can still get the true parents by looking at \"parent \"\nheader.\n\n---\ndiff --git a/t/t6001-rev-list-graft.sh b/t/t6001-rev-list-graft.sh\nnew file mode 100755\nindex 0000000..08a6cff\n--- /dev/null\n+++ b/t/t6001-rev-list-graft.sh\n@@ -0,0 +1,113 @@\n+#!/bin/sh\n+\n+test_description='Revision traversal vs grafts and path limiter'\n+\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\tmkdir subdir &&\n+\techo >fileA fileA &&\n+\techo >subdir/fileB fileB &&\n+\tgit add fileA subdir/fileB &&\n+\tgit commit -a -m \"Initial in one history.\" &&\n+\tA0=`git rev-parse --verify HEAD` &&\n+\n+\techo >fileA fileA modified &&\n+\tgit commit -a -m \"Second in one history.\" &&\n+\tA1=`git rev-parse --verify HEAD` &&\n+\n+\techo >subdir/fileB fileB modified &&\n+\tgit commit -a -m \"Third in one history.\" &&\n+\tA2=`git rev-parse --verify HEAD` &&\n+\n+\trm -f .git/refs/heads/master .git/index &&\n+\t\n+\techo >fileA fileA again &&\n+\techo >subdir/fileB fileB again &&\n+\tgit add fileA subdir/fileB &&\n+\tgit commit -a -m \"Initial in alternate history.\" &&\n+\tB0=`git rev-parse --verify HEAD` &&\n+\n+\techo >fileA fileA modified in alternate history &&\n+\tgit commit -a -m \"Second in alternate history.\" &&\n+\tB1=`git rev-parse --verify HEAD` &&\n+\n+\techo >subdir/fileB fileB modified in alternate history &&\n+\tgit commit -a -m \"Third in alternate history.\" &&\n+\tB2=`git rev-parse --verify HEAD` &&\n+\t: done\n+'\n+\n+check () {\n+\ttype=$1\n+\tshift\n+\n+\targ=\n+\twhich=arg\n+\trm -f test.expect\n+\tfor a\n+\tdo\n+\t\tif test \"z$a\" = z--\n+\t\tthen\n+\t\t\twhich=expect\n+\t\t\tchild=\n+\t\t\tcontinue\n+\t\tfi\n+\t\tif test \"$which\" = arg\n+\t\tthen\n+\t\t\targ=\"$arg$a \"\n+\t\t\tcontinue\n+\t\tfi\n+\t\tif test \"$type\" = basic\n+\t\tthen\n+\t\t\techo \"$a\"\n+\t\telse\n+\t\t\tif test \"z$child\" != z\n+\t\t\tthen\n+\t\t\t\techo \"$child $a\"\n+\t\t\tfi\n+\t\t\tchild=\"$a\"\n+\t\tfi\n+\tdone >test.expect\n+\tif test \"$type\" != basic && test \"z$child\" != z\n+\tthen\n+\t\techo >>test.expect $child\n+\tfi\n+\tif test $type = basic\n+\tthen\n+\t\tgit rev-list $arg >test.actual\n+\telif test $type = parents\n+\tthen\n+\t\tgit rev-list --parents $arg >test.actual\n+\telif test $type = parents-raw\n+\tthen\n+\t\tgit rev-list --parents --pretty=raw $arg |\n+\t\tsed -n -e 's/^commit //p' >test.actual\n+\tfi\n+\tdiff test.expect test.actual\n+}\n+\n+for type in basic parents parents-raw\n+do\n+\ttest_expect_success 'without grafts' \"\n+\t\trm -f .git/info/grafts\n+\t\tcheck $type $B2 -- $B2 $B1 $B0\n+\t\"\n+\n+\ttest_expect_success 'with grafts' \"\n+\t\techo '$B0 $A2' >.git/info/grafts\n+\t\tcheck $type $B2 -- $B2 $B1 $B0 $A2 $A1 $A0\n+\t\"\n+\n+\ttest_expect_success 'without grafts, with pathlimit' \"\n+\t\trm -f .git/info/grafts\n+\t\tcheck $type $B2 subdir -- $B2 $B0\n+\t\"\n+\n+\ttest_expect_success 'with grafts, with pathlimit' \"\n+\t\techo '$B0 $A2' >.git/info/grafts\n+\t\tcheck $type $B2 subdir -- $B2 $B0 $A2 $A0\n+\t\"\n+\n+done\n+test_done\n"},{"id":"29707","messageId":"453C96C9.4010005@freedesktop.org","threadId":"5718","inReplyTo":"7vlko5d3bx.fsf@assigned-by-dhcp.cox.net","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Josh Triplett","fromEmail":"josh@freedesktop.org","sentAt":"2006-10-23T10:17:45Z","receivedAt":"2006-10-23T10:17:45Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"Junio C Hamano wrote:\n> Josh Triplett <josh@freedesktop.org> writes:\n>> Jamey Sharp and I wrote a script called git-split to accomplish this\n>> repository split. git-split reconstructs the history of a sub-project\n>> previously stored in a subdirectory of a larger repository. It\n>> constructs new commit objects based on the existing tree objects for the\n>> subtree in each commit, and discards commits which do not affect the\n>> history of the sub-project, as well as merges made unnecessary due to\n>> these discarded commits.\n> \n> Very nicely done.\n\nThanks!\n\n>> We would like to acknowledge the work of the gobby team in creating a\n>> collaborative editor which greatly aided the development of git-split.\n> \n>> from itertools import izip\n>> from subprocess import Popen, PIPE\n>> import os, sys\n> \n> How recent a Python are we assuming here?  Is late 2.4 recent\n> enough?\n\nWe ran it with 2.4, so yes.  git-split does require at least 2.4,\nthough, because it uses set(), str.rsplit(), and subprocess, none of\nwhich existed in 2.3.\n\n>> def walk(commits, new_commits, commit_hash, project):\n>>     commit = commits[commit_hash]\n>>     if not(commit.has_key(\"new_hash\")):\n>>         tree = get_subtree(commit[\"tree\"], project)\n>>         commit[\"new_tree\"] = tree\n>>         if not tree:\n>>             raise Exception(\"Did not find project in tree for commit \" + commit_hash)\n>>         new_parents = list(set([walk(commits, new_commits, parent, project)\n>>                                 for parent in commit[\"parents\"]]))\n>>\n>>         new_hash = None\n>>         if len(new_parents) == 1:\n>>             new_hash = new_parents[0]\n>>         elif len(new_parents) == 2: # Check for unnecessary merge\n>>             if is_ancestor(new_commits, new_parents[0], new_parents[1]):\n>>                 new_hash = new_parents[0]\n>>             elif is_ancestor(new_commits, new_parents[1], new_parents[0]):\n>>                 new_hash = new_parents[1]\n>>         if new_hash and new_commits[new_hash][\"new_tree\"] != tree:\n>>             new_hash = None\n> \n> This is a real gem.  I really like reading well-written Python\n> programs.\n\nThanks.  We had some fun writing this; git's elegant repository\nstructure made it a joy to work with.\n\n> I wonder if using \"git-log --full-history -- $project\" to let\n> the core side omit commits that do not change the $project (but\n> still give you all merged branches) would have made your job any\n> easier?\n\nI don't think it would.  We still need to know what commit to use as the\nparent of any given commit, so we don't want commits in the log output\nwith parents that don't exist in the log output.  And rewriting parents\nin git-log based on which revisions change the specified subdirectory\nseems like a bad idea.\n\n> You are handling grafts by hand because --pretty=raw is special\n> in that it displays the real parents (although traversal does\n> use grafts).  Maybe it would have helped if we had a --pretty\n> format that is similar to raw but rewrites the parents?\n\nYes, that would help.  We could then avoid dealing with grafts manually.\n\n\nHow would you feel about including git-split in the GIT tree?  We could\ncertainly write up the necessary documentation for it.\n\n- Josh Triplett\n\n"},{"id":"29721","messageId":"Pine.LNX.4.64.0610230846420.3962@g5.osdl.org","threadId":"5718","inReplyTo":"453C96C9.4010005@freedesktop.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-10-23T15:52:46Z","receivedAt":"2006-10-23T15:52:46Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 23 Oct 2006, Josh Triplett wrote:\n> \n> > I wonder if using \"git-log --full-history -- $project\" to let the core \n> > side omit commits that do not change the $project (but still give you \n> > all merged branches) would have made your job any easier?\n> \n> I don't think it would.  We still need to know what commit to use as the\n> parent of any given commit, so we don't want commits in the log output\n> with parents that don't exist in the log output.  And rewriting parents\n> in git-log based on which revisions change the specified subdirectory\n> seems like a bad idea.\n\nUmm.. You didn't realize that git log already _does_ exactly that?\n\nYou need to rewrite the parents in order to get a nice and readable \nhistory, which in turn is needed for any visualizer. So git has long done \nthe parent rewriting in order to be able to do things like\n\n\tgitk drivers/char\n\non the kernel.\n\nAnd yes, that's done by the core revision parsing code, so when you do\n\n\tgit log --full-history --parents -- $project\n\nyou do get the rewritten parent output (of course, it's not actually \n_simplified_, so you get a fair amount of duplicate parents etc which \nyou'd still have to simplify and which don't do anything at all).\n\nWithout the \"--full-history\", you get a simplified history, but it's \nlikely to be _too_ simplified for your use, since it will not only \ncollapse multiple identical parents, it will also totally _remove_ parents \nthat don't introduce any new content.\n\nSo there are multiple levels of history simplification, and right now the \ninternal git revision parser only gives you two choices: \"none\" \n(--full-history) and \"extreme\" (which is the default when you give a set \nof filenames). \n\n\t\t\tLinus\n"},{"id":"29749","messageId":"453D17B5.6070203@freedesktop.org","threadId":"5718","inReplyTo":"Pine.LNX.4.64.0610230846420.3962@g5.osdl.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Josh Triplett","fromEmail":"josh@freedesktop.org","sentAt":"2006-10-23T19:27:49Z","receivedAt":"2006-10-23T19:27:49Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"Linus Torvalds wrote:\n> \n> On Mon, 23 Oct 2006, Josh Triplett wrote:\n>>> I wonder if using \"git-log --full-history -- $project\" to let the core \n>>> side omit commits that do not change the $project (but still give you \n>>> all merged branches) would have made your job any easier?\n>> I don't think it would.  We still need to know what commit to use as the\n>> parent of any given commit, so we don't want commits in the log output\n>> with parents that don't exist in the log output.  And rewriting parents\n>> in git-log based on which revisions change the specified subdirectory\n>> seems like a bad idea.\n> \n> Umm.. You didn't realize that git log already _does_ exactly that?\n\nNo, I didn't, primarily because the git log output I've scrutinized most\ncarefully came from git log --pretty=raw, which doesn't rewrite parents\neven when pointed at a subdirectory.\n\n> You need to rewrite the parents in order to get a nice and readable \n> history, which in turn is needed for any visualizer. So git has long done \n> the parent rewriting in order to be able to do things like\n> \n> \tgitk drivers/char\n> \n> on the kernel.\n>\n> And yes, that's done by the core revision parsing code, so when you do\n> \n> \tgit log --full-history --parents -- $project\n> \n> you do get the rewritten parent output (of course, it's not actually \n> _simplified_, so you get a fair amount of duplicate parents etc which \n> you'd still have to simplify and which don't do anything at all).\n> \n> Without the \"--full-history\", you get a simplified history, but it's \n> likely to be _too_ simplified for your use, since it will not only \n> collapse multiple identical parents, it will also totally _remove_ parents \n> that don't introduce any new content.\n\nConsidering that git-split does exactly that (remove parents that don't\nintroduce new content, assuming they changed things outside the\nsubtree), that might actually work for us.  I just checked, and the\noutput of \"git log --parents -- $project\" on one of my repositories\nseems to show the same sequence of commits as git log --parents on the\nhead commit printed by git-split $project (apart from the rewritten\nsha1s), including elimination of irrelevant merges.\n\n> So there are multiple levels of history simplification, and right now the \n> internal git revision parser only gives you two choices: \"none\" \n> (--full-history) and \"extreme\" (which is the default when you give a set \n> of filenames). \n\nI don't think we need any middle ground here; why might we want less\nsimplification?\n\n- Josh Triplett\n\n\n"},{"id":"29752","messageId":"Pine.LNX.4.64.0610231237080.3962@g5.osdl.org","threadId":"5718","inReplyTo":"453D17B5.6070203@freedesktop.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-10-23T19:50:58Z","receivedAt":"2006-10-23T19:50:58Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 23 Oct 2006, Josh Triplett wrote:\n>\n> > Without the \"--full-history\", you get a simplified history, but it's \n> > likely to be _too_ simplified for your use, since it will not only \n> > collapse multiple identical parents, it will also totally _remove_ parents \n> > that don't introduce any new content.\n> \n> Considering that git-split does exactly that (remove parents that don't\n> introduce new content, assuming they changed things outside the\n> subtree), that might actually work for us.  I just checked, and the\n> output of \"git log --parents -- $project\" on one of my repositories\n> seems to show the same sequence of commits as git log --parents on the\n> head commit printed by git-split $project (apart from the rewritten\n> sha1s), including elimination of irrelevant merges.\n\nOk. In that case, you're good to go, and just use the current \nsimplification entirely.\n\nAlthough I think that somebody (Dscho?) also had a patch to remove \nmultiple identical parents, which he claimed could happen with \nsimplification otherwise. I didn't look any closer at it.\n\n> > So there are multiple levels of history simplification, and right now the \n> > internal git revision parser only gives you two choices: \"none\" \n> > (--full-history) and \"extreme\" (which is the default when you give a set \n> > of filenames). \n> \n> I don't think we need any middle ground here; why might we want less\n> simplification?\n\nThere's really three levels of simplification:\n\n - none at all (\"--full-history\"). This is really annoying, but if you \n   want to guarantee that you see all the changes (even duplicate ones) \n   done along all branches, you currently need to do this one.\n\n   Currently \"git whatchanged\" uses this one (and that ignores merges by\n   default, making it quite palatable). So with \"git whatchanged\", you \n   will get _every_ commit that changed the file, even if there are \n   duplicates alogn different histories.\n\n - extreme (the current default). This one is really nice, in that it \n   shows the simplest history you can make that explains the end result. \n   But it means that if you had two branches that ended up with the same \n   result, we will pick just one of them. And the other one may have done \n   it differently, and the different way of reaching the same result might \n   be interesting. We'll never know.\n\n   As an exmple: the extreme simplification can also throw away branches \n   that had work reverted on them - the branch ended up the _same_ as the \n   one we chose, but it did so because it had some experimental work that \n   was deemed to be bad. Extreme simplification may or may not remove that \n   experiment, simply depending on which branch it _happened_ to pick.\n\n   Currently, this is what most git users see if they ask for pathname \n   simplification, ie \"gitk drivers/char\" or \"git log -p kernel/sched.c\"\n   uses this simplification. It's extremely useful, but it definitely \n   culls real history too.\n\n - The nice one that doesn't throw away potentially interesting \n   duplicate paths to reach the same end result. We don't have this one, \n   so no git commands do this yet.\n\n   The way to do this one would be \"--full-history\", but then removing all \n   parents that are \"redundant\". In other words, for any merge that \n   remains (because of the --full-history), check if one parent is a full \n   superset of another one, and if so, remove the \"dominated\" parent, \n   which simplifies the merge. Continue until nothing can be simplified \n   any more.\n\n   This would _usually_ end up giving the same graph as the \"extreme\" \n   simplification, but if there were two branches that really _did_ \n   generate the same end result using different commits, they'd remain in \n   the end result.\n\nThe problem with the \"nice one\" is that it's expensive as hell. There may \nbe clever tricks to make it less so, though. But I think it's the \nRightThing(tm) to do, at least as an option for when you really want to \nsee a reasonable history that still contains everything that is relevant.\n\n\t\t\tLinus\n"},{"id":"29754","messageId":"ehj7cm$dku$1@sea.gmane.org","threadId":"5718","inReplyTo":"Pine.LNX.4.64.0610231237080.3962@g5.osdl.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-10-23T20:07:11Z","receivedAt":"2006-10-23T20:07:11Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"There is also not-that-obvious result that\n\n  git rev-log --parents --full-history <head> -- <pathspec> \n\ngenerates different result than if either --parents or --full-history are\nabsent.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"29761","messageId":"453D2B98.8010903@freedesktop.org","threadId":"5718","inReplyTo":"Pine.LNX.4.64.0610231237080.3962@g5.osdl.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Josh Triplett","fromEmail":"josh@freedesktop.org","sentAt":"2006-10-23T20:52:40Z","receivedAt":"2006-10-23T20:52:40Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"Linus Torvalds wrote:\n> On Mon, 23 Oct 2006, Josh Triplett wrote:\n>  - The nice one that doesn't throw away potentially interesting \n>    duplicate paths to reach the same end result. We don't have this one, \n>    so no git commands do this yet.\n> \n>    The way to do this one would be \"--full-history\", but then removing all \n>    parents that are \"redundant\". In other words, for any merge that \n>    remains (because of the --full-history), check if one parent is a full \n>    superset of another one, and if so, remove the \"dominated\" parent, \n>    which simplifies the merge. Continue until nothing can be simplified \n>    any more.\n> \n>    This would _usually_ end up giving the same graph as the \"extreme\" \n>    simplification, but if there were two branches that really _did_ \n>    generate the same end result using different commits, they'd remain in \n>    the end result.\n> \n> The problem with the \"nice one\" is that it's expensive as hell. There may \n> be clever tricks to make it less so, though. But I think it's the \n> RightThing(tm) to do, at least as an option for when you really want to \n> see a reasonable history that still contains everything that is relevant.\n\nSo, if a commit has more than one parent (a merge), you want to\neliminate any parents that end up as ancestors to other parents in the\nmerge (including if their head has the same commit ID), but not\neliminate multiple parents with different head commits but the same tree\nobject?  That seems simple enough; I *think* git-split actually already\ndoes that, though I haven't actually tested that particular case.  If\ngit log eliminates all but one of the parents with different commits but\nthe same tree, I believe the commit sequence generated by git-split will\ndiffer from that of git log in that case, by including all such parents.\n\nI do agree that the behavior you describe seems like the best\nsimplification, and I don't think the alternative you describe as\n\"extreme simplification\" makes any sense at all (picking a parent\narbitrarily), nor does it seem any simpler to generate; either way, you\nstill have to figure out if one parent has another as an ancestor, while\nthe additional \"extreme simplification\" just *adds* a comparison of tree\nhashes.\n\nOr have I misunderstood the case you have concerns about?  Why would the\n\"nice\" format incur additional cost?\n\n- Josh Triplett\n\n\n"},{"id":"29765","messageId":"Pine.LNX.4.64.0610231402560.3962@g5.osdl.org","threadId":"5718","inReplyTo":"453D2B98.8010903@freedesktop.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-10-23T21:06:49Z","receivedAt":"2006-10-23T21:06:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 23 Oct 2006, Josh Triplett wrote:\n> \n> Or have I misunderstood the case you have concerns about?  Why would the\n> \"nice\" format incur additional cost?\n\nTry it. The default \"extreme\" simplification is a _hell_ of a lot faster \nthan doing the full history.\n\n\t[torvalds@g5 linux]$ time git-rev-list --full-history --parents HEAD -- kernel/sched.c >/dev/null\n\n\treal    0m4.660s\n\tuser    0m4.612s\n\tsys     0m0.044s\n\n\t[torvalds@g5 linux]$ time git-rev-list --parents HEAD -- kernel/sched.c >/dev/null\n\t\n\treal    0m1.684s\n\tuser    0m1.680s\n\tsys     0m0.004s\n\nand the \"nice\" thing will be much slower still: just trying to figure out \nwhether a commit is a parent of another commit is expensive. Doing so for \n_each_ merge is more expensive still. I think it's O(n^3), but what do I \nknow..\n\n\t\t\tLinus\n"},{"id":"29769","messageId":"Pine.LNX.4.64.0610231411200.3962@g5.osdl.org","threadId":"5718","inReplyTo":"Pine.LNX.4.64.0610231402560.3962@g5.osdl.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-10-23T21:19:45Z","receivedAt":"2006-10-23T21:19:45Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 23 Oct 2006, Linus Torvalds wrote:\n> \n> Try it. The default \"extreme\" simplification is a _hell_ of a lot faster \n> than doing the full history.\n[ timings removed ]\n\nBtw, the reason it is so much faster is that it can be done early, and \nallows us to prune out parts of the history that we don't care about.\n\nFor example, when we hit a merge, and the result of that merge is \nidentical to one of the parents (in the set of filenames that we are \ninterested in), we can simply choose to totally ignore the other parent, \nand we don't need to traverse that history at _all_. Because clearly, all \nthe actual _data_ came from just the other one.\n\nSo the \"extreme\" simplification is way way faster, because in the presense \nof a lot of merges, it can select to go down just one of the paths, and \ntotally ignore the other ones. In practice, for a fairly \"bushy\" history \ntree like the kernel, that can cut down the number of commits you need to \ncompare by a factor of two or more.\n\nIn many ways, it is also actually a _better_ result, in that it's a \n\"closer to minimal\" way of reaching a particular state. So if you're just \ninterested in how something came to be, and want to just cut through the \ncrap, the result extreme simplification really _is_ better.\n\nSo the branches that were dismissed really _aren't_ important - they might \ncontain real work, but from the point of the end result, that real work \nmight as well not have happened, since the simpler history we chose _also_ \nexplain the end result sufficiently.\n\nSo I think the default simplification is really a good default: not only \nbecause it's fundamentally cheaper, but because it is actually more likely \nto be distill what you actually care about if you wonder what happened to \na file or a set of files.\n\nBut if you care about all the \"side efforts\" that didn't actually matter \nfor the end result too, then you'd want the more expensive, and more \ncomplete graph. But it _will_ be a lot more expensive to compute.\n\n\t\tLinus\n"},{"id":"29864","messageId":"Pine.LNX.4.63.0610241654060.2106@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5718","inReplyTo":"Pine.LNX.4.64.0610231237080.3962@g5.osdl.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-10-24T14:56:11Z","receivedAt":"2006-10-24T14:56:11Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 23 Oct 2006, Linus Torvalds wrote:\n\n> Although I think that somebody (Dscho?) also had a patch to remove \n> multiple identical parents, which he claimed could happen with \n> simplification otherwise. I didn't look any closer at it.\n\nIIRC It only happened when full history was wished for, _and_ path \nlimiting. And Junio said that in that case, culling identical parents \nwould be the wrong thing to do.\n\nCiao,\nDscho\n"},{"id":"29866","messageId":"Pine.LNX.4.64.0610240817120.3962@g5.osdl.org","threadId":"5718","inReplyTo":"Pine.LNX.4.63.0610241654060.2106@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-10-24T15:19:16Z","receivedAt":"2006-10-24T15:19:16Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 24 Oct 2006, Johannes Schindelin wrote:\n> \n> On Mon, 23 Oct 2006, Linus Torvalds wrote:\n> \n> > Although I think that somebody (Dscho?) also had a patch to remove \n> > multiple identical parents, which he claimed could happen with \n> > simplification otherwise. I didn't look any closer at it.\n> \n> IIRC It only happened when full history was wished for, _and_ path \n> limiting. And Junio said that in that case, culling identical parents \n> would be the wrong thing to do.\n\nYeah, with full history, you might as well keep the trivially identical \nparents too. In the \"nice rewrite\", you'd get rid of them, but you'd get \nrid of them not because they are trivially identical, but because they are \njust the trivial case of \"one parent totally dominates the other\".\n\nSadly, while it may be the _trivial_ case in --full-history, it's probably \nnot even the common case, at least if you have lots of merges (because you \nend up having one parent point to one merge, and the other to another, and \nyou need to simplify things in multiple passes).\n\n\t\t\tLinus\n"},{"id":"297126","messageId":"7vu01tfe6u.fsf@assigned-by-dhcp.cox.net","threadId":"5718","inReplyTo":"453D17B5.6070203@freedesktop.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-25T00:10:17Z","receivedAt":"2006-10-25T00:10:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Triplett <josh@freedesktop.org> writes:\n\n> Linus Torvalds wrote:\n>> \n>> And yes, that's done by the core revision parsing code, so when you do\n>> \n>> \tgit log --full-history --parents -- $project\n>> \n>> you do get the rewritten parent output (of course, it's not actually \n>> _simplified_, so you get a fair amount of duplicate parents etc which \n>> you'd still have to simplify and which don't do anything at all).\n>> \n>> Without the \"--full-history\", you get a simplified history, but it's \n>> likely to be _too_ simplified for your use, since it will not only \n>> collapse multiple identical parents, it will also totally _remove_ parents \n>> that don't introduce any new content.\n>\n> Considering that git-split does exactly that (remove parents that don't\n> introduce new content, assuming they changed things outside the\n> subtree), that might actually work for us.  I just checked, and the\n> output of \"git log --parents -- $project\" on one of my repositories\n> seems to show the same sequence of commits as git log --parents on the\n> head commit printed by git-split $project (apart from the rewritten\n> sha1s), including elimination of irrelevant merges.\n\nSo one potential action item that came out from this discussion\nfor me is to either modify --pretty=raw (or add --pretty=rawish)\nthat gives the rewritten parents instead of real parents?  With\nthat, you can drop the code to simplify ancestry by hand in your\nloop, and also you do not have to do the grafts inforamation\nyourself either?\n\nIf that is the case I'd be very happy.\n\nThe only thing left for us to decide is if reporting the true\nparenthood like the current --pretty=raw makes sense (if so we\nneed to keep it and introduce --pretty=rawfish).\n\nThe only in-tree user of --pretty=raw seems to be git-svn but it\nonly looks at path-unlimited log/rev-list from one given commit,\nso the only difference between dumping what is recorded in the\ncommit object and listing what parents we _think_ the commit has\nis what we read from grafts.  I think we are safe to just \"fix\"\nthe behaviour of --pretty=raw\n\nComments?\n"},{"id":"298446","messageId":"ehmah3$rkj$2@sea.gmane.org","threadId":"5718","inReplyTo":"7vu01tfe6u.fsf@assigned-by-dhcp.cox.net","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-10-25T00:19:15Z","receivedAt":"2006-10-25T00:19:15Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> So one potential action item that came out from this discussion\n> for me is to either modify --pretty=raw (or add --pretty=rawish)\n> that gives the rewritten parents instead of real parents?  With\n> that, you can drop the code to simplify ancestry by hand in your\n> loop, and also you do not have to do the grafts inforamation\n> yourself either?\n> \n> If that is the case I'd be very happy.\n> \n> The only thing left for us to decide is if reporting the true\n> parenthood like the current --pretty=raw makes sense (if so we\n> need to keep it and introduce --pretty=rawfish).\n> \n> The only in-tree user of --pretty=raw seems to be git-svn but it\n> only looks at path-unlimited log/rev-list from one given commit,\n> so the only difference between dumping what is recorded in the\n> commit object and listing what parents we _think_ the commit has\n> is what we read from grafts.  I think we are safe to just \"fix\"\n> the behaviour of --pretty=raw\n> \n> Comments?\n\nThe name --pretty=raw suggest output of info directly from commit object,\nbut perhaps that just me (--pretty=rawish or ==pretty=headers).\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"296685","messageId":"453EC505.7060807@freedesktop.org","threadId":"5718","inReplyTo":"7vu01tfe6u.fsf@assigned-by-dhcp.cox.net","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Josh Triplett","fromEmail":"josh@freedesktop.org","sentAt":"2006-10-25T01:59:33Z","receivedAt":"2006-10-25T01:59:33Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"Junio C Hamano wrote:\n> Josh Triplett <josh@freedesktop.org> writes:\n>> Linus Torvalds wrote:\n>>> And yes, that's done by the core revision parsing code, so when you do\n>>>\n>>> \tgit log --full-history --parents -- $project\n>>>\n>>> you do get the rewritten parent output (of course, it's not actually \n>>> _simplified_, so you get a fair amount of duplicate parents etc which \n>>> you'd still have to simplify and which don't do anything at all).\n>>>\n>>> Without the \"--full-history\", you get a simplified history, but it's \n>>> likely to be _too_ simplified for your use, since it will not only \n>>> collapse multiple identical parents, it will also totally _remove_ parents \n>>> that don't introduce any new content.\n>> Considering that git-split does exactly that (remove parents that don't\n>> introduce new content, assuming they changed things outside the\n>> subtree), that might actually work for us.  I just checked, and the\n>> output of \"git log --parents -- $project\" on one of my repositories\n>> seems to show the same sequence of commits as git log --parents on the\n>> head commit printed by git-split $project (apart from the rewritten\n>> sha1s), including elimination of irrelevant merges.\n> \n> So one potential action item that came out from this discussion\n> for me is to either modify --pretty=raw (or add --pretty=rawish)\n> that gives the rewritten parents instead of real parents?  With\n> that, you can drop the code to simplify ancestry by hand in your\n> loop, and also you do not have to do the grafts inforamation\n> yourself either?\n> \n> If that is the case I'd be very happy.\n> \n> The only thing left for us to decide is if reporting the true\n> parenthood like the current --pretty=raw makes sense (if so we\n> need to keep it and introduce --pretty=rawfish).\n> \n> The only in-tree user of --pretty=raw seems to be git-svn but it\n> only looks at path-unlimited log/rev-list from one given commit,\n> so the only difference between dumping what is recorded in the\n> commit object and listing what parents we _think_ the commit has\n> is what we read from grafts.  I think we are safe to just \"fix\"\n> the behaviour of --pretty=raw\n\nI actually think I want to look further into the idea of just using git\n--pretty=raw --parents -- $project, and see if I can find any corner cases\nwhere it generates a different history than what we want.  This combination of\noptions seems like it provides everything we need: redundant history\nsimplification, parent rewriting based on simplification and grafts, and easy\nparsing.  If the only case in which it differs occurs when you have two\ndistinct commits with identical trees, I don't know that I care too much; that\nparticular scenario seems unlikely to occur in any of the trees I care about,\nand any sane simplification behavior for it seems OK. :) As long as it runs\ncorrectly with various ancestor/descendant/cousin/unrelated relationships\nbetween merged branches (which I want to test further), I think it will do the\njob nicely.\n\n- Josh Triplett\n\n"},{"id":"296408","messageId":"7viri9f8gq.fsf@assigned-by-dhcp.cox.net","threadId":"5718","inReplyTo":"453EC505.7060807@freedesktop.org","subject":"Re: [RFC] git-split: Split the history of a git repository by subdirectories and ranges","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-25T02:13:57Z","receivedAt":"2006-10-25T02:13:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Triplett <josh@freedesktop.org> writes:\n\n> Junio C Hamano wrote:\n>\n>> The only thing left for us to decide is if reporting the true\n>> parenthood like the current --pretty=raw makes sense (if so we\n>> need to keep it and introduce --pretty=rawfish).\n>> \n>> The only in-tree user of --pretty=raw seems to be git-svn but it\n>> only looks at path-unlimited log/rev-list from one given commit,\n>> so the only difference between dumping what is recorded in the\n>> commit object and listing what parents we _think_ the commit has\n>> is what we read from grafts.  I think we are safe to just \"fix\"\n>> the behaviour of --pretty=raw\n>\n> I actually think I want to look further into the idea of just using git\n> --pretty=raw --parents -- $project, and see if I can find any corner cases\n> where it generates a different history than what we want.\n\nI do not mind _coding_ the --pretty=rawfish change if needed but\nI do not think it is necessary, which is pretty good news.\n\nAfter I wrote the message I realized that I probably do not have\nto do anything, since --parents would give you the rewritten\nparents already anyway.\n"}]}