{"thread":{"id":"8950","subject":"how to do directory renames in fast-import","startedAt":"2007-07-10T01:09:02Z","lastAt":"2007-07-11T23:11:41Z","messageCount":10,"participants":["David Frech","Shawn O. Pearce","Rogan Dawes","Uwe Kleine-König"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"46899","messageId":"7154c5c60707091809y7e0b67d5u3f94658b7e814325@mail.gmail.com","threadId":"8950","inReplyTo":null,"subject":"how to do directory renames in fast-import","fromName":"David Frech","fromEmail":"david@nimblemachines.com","sentAt":"2007-07-10T01:09:02Z","receivedAt":"2007-07-10T01:09:02Z","isPatch":false,"sender":{"key":"david@nimblemachines.com","avatar":null},"body":"Git can track file renames implicitly. If I delete and then add (under\na different name) the same content, git will figure that out.\n\nBut if a directory was renamed, I have no way to tell fast-import\nabout it. I can't delete the directory (using a 'D' command) and then\nadd it back (with a different name) with all its contents, because my\nsource material (an svn dump file) doesn't tell me, at that point,\nabout all the files involved because nothing about them has changed.\n\nfast-import knows about the contents of the directory I want to\nrename, but doesn't give me a primitive to do the rename. Is this\nsomething we need to add? My frontend could keep track of this, but I\nwould duplicating work that fast-import is already doing.\n\nCheers,\n\n- David\n\n-- \nIf I have not seen farther, it is because I have stood in the\nfootsteps of giants.\n"},{"id":"46907","messageId":"20070710031036.GA9045@spearce.org","threadId":"8950","inReplyTo":"7154c5c60707091809y7e0b67d5u3f94658b7e814325@mail.gmail.com","subject":"[PATCH] Support wholesale directory renames in fast-import","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-07-10T03:10:36Z","receivedAt":"2007-07-10T03:10:36Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Some source material (e.g. Subversion dump files) perform directory\nrenames without telling us exactly which files in that subdirectory\nwere moved.  This makes it hard for a frontend to convert such data\nformats to a fast-import stream, as all the frontend has on hand\nis \"Rename a/ to b/\" with no details about what files are in a/,\nunless the frontend also kept track of all files.\n\nThe new 'R' subcommand within a commit allows the frontend to\nrename either a file or an entire subdirectory, without needing to\nknow the object's SHA-1 or the specific files contained within it.\nThe rename is performed as efficiently as possible internally,\nmaking it cheaper than a 'D'/'M' pair for a file rename.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n\n David Frech <david@nimblemachines.com> wrote:\n > Git can track file renames implicitly. If I delete and then add (under\n > a different name) the same content, git will figure that out.\n > \n > But if a directory was renamed, I have no way to tell fast-import\n > about it. I can't delete the directory (using a 'D' command) and then\n > add it back (with a different name) with all its contents, because my\n > source material (an svn dump file) doesn't tell me, at that point,\n > about all the files involved because nothing about them has changed.\n > \n > fast-import knows about the contents of the directory I want to\n > rename, but doesn't give me a primitive to do the rename. Is this\n > something we need to add? My frontend could keep track of this, but I\n > would duplicating work that fast-import is already doing.\n\n Does the following do the trick for you?  It is also available\n from my fastimport.git master branch:\n\n\tgit://repo.or.cz/git/fastimport.git      master\n\thttp://repo.or.cz/r/git/fastimport.git   master\n\n Yes, it passes all tests...\n\n Documentation/git-fast-import.txt |   28 ++++++++++-\n fast-import.c                     |   91 ++++++++++++++++++++++++++++++-------\n t/t9300-fast-import.sh            |   68 +++++++++++++++++++++++++++\n 3 files changed, 168 insertions(+), 19 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex c66af7c..80a8ee0 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -302,7 +302,7 @@ change to the project.\n \tdata\n \t('from' SP <committish> LF)?\n \t('merge' SP <committish> LF)?\n-\t(filemodify | filedelete | filedeleteall)*\n+\t(filemodify | filedelete | filerename | filedeleteall)*\n \tLF\n ....\n \n@@ -325,11 +325,13 @@ commit message use a 0 length data.  Commit messages are free-form\n and are not interpreted by Git.  Currently they must be encoded in\n UTF-8, as fast-import does not permit other encodings to be specified.\n \n-Zero or more `filemodify`, `filedelete` and `filedeleteall` commands\n+Zero or more `filemodify`, `filedelete`, `filename` and\n+`filedeleteall` commands\n may be included to update the contents of the branch prior to\n creating the commit.  These commands may be supplied in any order.\n However it is recommended that a `filedeleteall` command preceed\n-all `filemodify` commands in the same commit, as `filedeleteall`\n+all `filemodify` and `filerename` commands in the same commit, as\n+`filedeleteall`\n wipes the branch clean (see below).\n \n `author`\n@@ -495,6 +497,26 @@ here `<path>` is the complete path of the file or subdirectory to\n be removed from the branch.\n See `filemodify` above for a detailed description of `<path>`.\n \n+`filerename`\n+^^^^^^^^^^^^\n+Renames an existing file or subdirectory to a different location\n+within the branch.  The existing file or directory must exist. If\n+the destination exists it will be replaced by the source directory.\n+\n+....\n+\t'R' SP <path> SP <path> LF\n+....\n+\n+here the first `<path>` is the source location and the second\n+`<path>` is the destination.  See `filemodify` above for a detailed\n+description of what `<path>` may look like.  To use a source path\n+that contains SP the path must be quoted.\n+\n+A `filerename` command takes effect immediately.  Once the source\n+location has been renamed to the destination any future commands\n+applied to the source location will create new files there and not\n+impact the destination of the rename.\n+\n `filedeleteall`\n ^^^^^^^^^^^^^^^\n Included in a `commit` command to remove all files (and also all\ndiff --git a/fast-import.c b/fast-import.c\nindex f9bfcc7..a1cb13f 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -26,9 +26,10 @@ Format of STDIN stream:\n     lf;\n   commit_msg ::= data;\n \n-  file_change ::= file_clr | file_del | file_obm | file_inm;\n+  file_change ::= file_clr | file_del | file_rnm | file_obm | file_inm;\n   file_clr ::= 'deleteall' lf;\n   file_del ::= 'D' sp path_str lf;\n+  file_rnm ::= 'R' sp path_str sp path_str lf;\n   file_obm ::= 'M' sp mode sp (hexsha1 | idnum) sp path_str lf;\n   file_inm ::= 'M' sp mode sp 'inline' sp path_str lf\n     data;\n@@ -1154,7 +1155,8 @@ static int tree_content_set(\n \tstruct tree_entry *root,\n \tconst char *p,\n \tconst unsigned char *sha1,\n-\tconst uint16_t mode)\n+\tconst uint16_t mode,\n+\tstruct tree_content *subtree)\n {\n \tstruct tree_content *t = root->tree;\n \tconst char *slash1;\n@@ -1168,20 +1170,22 @@ static int tree_content_set(\n \t\tn = strlen(p);\n \tif (!n)\n \t\tdie(\"Empty path component found in input\");\n+\tif (!slash1 && !S_ISDIR(mode) && subtree)\n+\t\tdie(\"Non-directories cannot have subtrees\");\n \n \tfor (i = 0; i < t->entry_count; i++) {\n \t\te = t->entries[i];\n \t\tif (e->name->str_len == n && !strncmp(p, e->name->str_dat, n)) {\n \t\t\tif (!slash1) {\n-\t\t\t\tif (e->versions[1].mode == mode\n+\t\t\t\tif (!S_ISDIR(mode)\n+\t\t\t\t\t\t&& e->versions[1].mode == mode\n \t\t\t\t\t\t&& !hashcmp(e->versions[1].sha1, sha1))\n \t\t\t\t\treturn 0;\n \t\t\t\te->versions[1].mode = mode;\n \t\t\t\thashcpy(e->versions[1].sha1, sha1);\n-\t\t\t\tif (e->tree) {\n+\t\t\t\tif (e->tree)\n \t\t\t\t\trelease_tree_content_recursive(e->tree);\n-\t\t\t\t\te->tree = NULL;\n-\t\t\t\t}\n+\t\t\t\te->tree = subtree;\n \t\t\t\thashclr(root->versions[1].sha1);\n \t\t\t\treturn 1;\n \t\t\t}\n@@ -1191,7 +1195,7 @@ static int tree_content_set(\n \t\t\t}\n \t\t\tif (!e->tree)\n \t\t\t\tload_tree(e);\n-\t\t\tif (tree_content_set(e, slash1 + 1, sha1, mode)) {\n+\t\t\tif (tree_content_set(e, slash1 + 1, sha1, mode, subtree)) {\n \t\t\t\thashclr(root->versions[1].sha1);\n \t\t\t\treturn 1;\n \t\t\t}\n@@ -1209,9 +1213,9 @@ static int tree_content_set(\n \tif (slash1) {\n \t\te->tree = new_tree_content(8);\n \t\te->versions[1].mode = S_IFDIR;\n-\t\ttree_content_set(e, slash1 + 1, sha1, mode);\n+\t\ttree_content_set(e, slash1 + 1, sha1, mode, subtree);\n \t} else {\n-\t\te->tree = NULL;\n+\t\te->tree = subtree;\n \t\te->versions[1].mode = mode;\n \t\thashcpy(e->versions[1].sha1, sha1);\n \t}\n@@ -1219,7 +1223,10 @@ static int tree_content_set(\n \treturn 1;\n }\n \n-static int tree_content_remove(struct tree_entry *root, const char *p)\n+static int tree_content_remove(\n+\tstruct tree_entry *root,\n+\tconst char *p,\n+\tstruct tree_entry *backup_leaf)\n {\n \tstruct tree_content *t = root->tree;\n \tconst char *slash1;\n@@ -1239,13 +1246,14 @@ static int tree_content_remove(struct tree_entry *root, const char *p)\n \t\t\t\tgoto del_entry;\n \t\t\tif (!e->tree)\n \t\t\t\tload_tree(e);\n-\t\t\tif (tree_content_remove(e, slash1 + 1)) {\n+\t\t\tif (tree_content_remove(e, slash1 + 1, backup_leaf)) {\n \t\t\t\tfor (n = 0; n < e->tree->entry_count; n++) {\n \t\t\t\t\tif (e->tree->entries[n]->versions[1].mode) {\n \t\t\t\t\t\thashclr(root->versions[1].sha1);\n \t\t\t\t\t\treturn 1;\n \t\t\t\t\t}\n \t\t\t\t}\n+\t\t\t\tbackup_leaf = NULL;\n \t\t\t\tgoto del_entry;\n \t\t\t}\n \t\t\treturn 0;\n@@ -1254,10 +1262,11 @@ static int tree_content_remove(struct tree_entry *root, const char *p)\n \treturn 0;\n \n del_entry:\n-\tif (e->tree) {\n+\tif (backup_leaf)\n+\t\tmemcpy(backup_leaf, e, sizeof(*backup_leaf));\n+\telse if (e->tree)\n \t\trelease_tree_content_recursive(e->tree);\n-\t\te->tree = NULL;\n-\t}\n+\te->tree = NULL;\n \te->versions[1].mode = 0;\n \thashclr(e->versions[1].sha1);\n \thashclr(root->versions[1].sha1);\n@@ -1629,7 +1638,7 @@ static void file_change_m(struct branch *b)\n \t\t\t    typename(type), command_buf.buf);\n \t}\n \n-\ttree_content_set(&b->branch_tree, p, sha1, S_IFREG | mode);\n+\ttree_content_set(&b->branch_tree, p, sha1, S_IFREG | mode, NULL);\n \tfree(p_uq);\n }\n \n@@ -1645,10 +1654,58 @@ static void file_change_d(struct branch *b)\n \t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n \t\tp = p_uq;\n \t}\n-\ttree_content_remove(&b->branch_tree, p);\n+\ttree_content_remove(&b->branch_tree, p, NULL);\n \tfree(p_uq);\n }\n \n+static void file_change_r(struct branch *b)\n+{\n+\tconst char *s, *d;\n+\tchar *s_uq, *d_uq;\n+\tconst char *endp;\n+\tstruct tree_entry leaf;\n+\n+\ts = command_buf.buf + 2;\n+\ts_uq = unquote_c_style(s, &endp);\n+\tif (s_uq) {\n+\t\tif (*endp != ' ')\n+\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n+\t}\n+\telse {\n+\t\tendp = strchr(s, ' ');\n+\t\tif (!endp)\n+\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n+\t\ts_uq = xmalloc(endp - s + 1);\n+\t\tmemcpy(s_uq, s, endp - s);\n+\t\ts_uq[endp - s] = 0;\n+\t}\n+\ts = s_uq;\n+\n+\tendp++;\n+\tif (!*endp)\n+\t\tdie(\"Missing dest: %s\", command_buf.buf);\n+\n+\td = endp;\n+\td_uq = unquote_c_style(d, &endp);\n+\tif (d_uq) {\n+\t\tif (*endp)\n+\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n+\t\td = d_uq;\n+\t}\n+\n+\tmemset(&leaf, 0, sizeof(leaf));\n+\ttree_content_remove(&b->branch_tree, s, &leaf);\n+\tif (!leaf.versions[1].mode)\n+\t\tdie(\"Path %s not in branch\", s);\n+\ttree_content_set(&b->branch_tree, d,\n+\t\tleaf.versions[1].sha1,\n+\t\tleaf.versions[1].mode,\n+\t\tleaf.tree);\n+\n+\tfree(s_uq);\n+\tfree(d_uq);\n+}\n+\n static void file_change_deleteall(struct branch *b)\n {\n \trelease_tree_content_recursive(b->branch_tree.tree);\n@@ -1816,6 +1873,8 @@ static void cmd_new_commit(void)\n \t\t\tfile_change_m(b);\n \t\telse if (!prefixcmp(command_buf.buf, \"D \"))\n \t\t\tfile_change_d(b);\n+\t\telse if (!prefixcmp(command_buf.buf, \"R \"))\n+\t\t\tfile_change_r(b);\n \t\telse if (!strcmp(\"deleteall\", command_buf.buf))\n \t\t\tfile_change_deleteall(b);\n \t\telse\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 53774c8..bf3720d 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -580,4 +580,72 @@ test_expect_success \\\n \t git diff --raw L^ L >output &&\n \t git diff expect output'\n \n+###\n+### series M\n+###\n+\n+test_tick\n+cat >input <<INPUT_END\n+commit refs/heads/M1\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+data <<COMMIT\n+file rename\n+COMMIT\n+\n+from refs/heads/branch^0\n+R file2/newf file2/n.e.w.f\n+\n+INPUT_END\n+\n+cat >expect <<EOF\n+:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100\tfile2/newf\tfile2/n.e.w.f\n+EOF\n+test_expect_success \\\n+\t'M: rename file in same subdirectory' \\\n+\t'git-fast-import <input &&\n+\t git diff-tree -M -r M1^ M1 >actual &&\n+\t compare_diff_raw expect actual'\n+\n+cat >input <<INPUT_END\n+commit refs/heads/M2\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+data <<COMMIT\n+file rename\n+COMMIT\n+\n+from refs/heads/branch^0\n+R file2/newf i/am/new/to/you\n+\n+INPUT_END\n+\n+cat >expect <<EOF\n+:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100\tfile2/newf\ti/am/new/to/you\n+EOF\n+test_expect_success \\\n+\t'M: rename file to new subdirectory' \\\n+\t'git-fast-import <input &&\n+\t git diff-tree -M -r M2^ M2 >actual &&\n+\t compare_diff_raw expect actual'\n+\n+cat >input <<INPUT_END\n+commit refs/heads/M3\n+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+data <<COMMIT\n+file rename\n+COMMIT\n+\n+from refs/heads/M2^0\n+R i other/sub\n+\n+INPUT_END\n+\n+cat >expect <<EOF\n+:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc R100\ti/am/new/to/you\tother/sub/am/new/to/you\n+EOF\n+test_expect_success \\\n+\t'M: rename subdirectory to new subdirectory' \\\n+\t'git-fast-import <input &&\n+\t git diff-tree -M -r M3^ M3 >actual &&\n+\t compare_diff_raw expect actual'\n+\n test_done\n-- \n1.5.3.rc0.879.g64b8\n"},{"id":"46910","messageId":"7154c5c60707092116p70aaeb8l90cda9265311b999@mail.gmail.com","threadId":"8950","inReplyTo":"20070710031036.GA9045@spearce.org","subject":"Re: [PATCH] Support wholesale directory renames in fast-import","fromName":"David Frech","fromEmail":"nimblemachines@gmail.com","sentAt":"2007-07-10T04:16:41Z","receivedAt":"2007-07-10T04:16:41Z","isPatch":true,"sender":{"key":"nimblemachines@gmail.com","avatar":null},"body":"This should do nicely! Thank you!\n\nNow my challenge is that the svn dump doesn't *actually* say \"rename\na/ to b/\"; it says \"copy a/ to b/; delete a/\", so I have to infer the\nrename.\n\nBut your patch makes my import possible, and it wasn't before!\n\nCheers,\n\n- David\n\nOn 7/9/07, Shawn O. Pearce <spearce@spearce.org> wrote:\n> Some source material (e.g. Subversion dump files) perform directory\n> renames without telling us exactly which files in that subdirectory\n> were moved.  This makes it hard for a frontend to convert such data\n> formats to a fast-import stream, as all the frontend has on hand\n> is \"Rename a/ to b/\" with no details about what files are in a/,\n> unless the frontend also kept track of all files.\n>\n> The new 'R' subcommand within a commit allows the frontend to\n> rename either a file or an entire subdirectory, without needing to\n> know the object's SHA-1 or the specific files contained within it.\n> The rename is performed as efficiently as possible internally,\n> making it cheaper than a 'D'/'M' pair for a file rename.\n>\n> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>\n> ---\n>\n>  David Frech <david@nimblemachines.com> wrote:\n>  > Git can track file renames implicitly. If I delete and then add (under\n>  > a different name) the same content, git will figure that out.\n>  >\n>  > But if a directory was renamed, I have no way to tell fast-import\n>  > about it. I can't delete the directory (using a 'D' command) and then\n>  > add it back (with a different name) with all its contents, because my\n>  > source material (an svn dump file) doesn't tell me, at that point,\n>  > about all the files involved because nothing about them has changed.\n>  >\n>  > fast-import knows about the contents of the directory I want to\n>  > rename, but doesn't give me a primitive to do the rename. Is this\n>  > something we need to add? My frontend could keep track of this, but I\n>  > would duplicating work that fast-import is already doing.\n>\n>  Does the following do the trick for you?  It is also available\n>  from my fastimport.git master branch:\n>\n>         git://repo.or.cz/git/fastimport.git      master\n>         http://repo.or.cz/r/git/fastimport.git   master\n>\n>  Yes, it passes all tests...\n>\n[patch elided]\n"},{"id":"46927","messageId":"469346D8.4080906@dawes.za.net","threadId":"8950","inReplyTo":"20070710031036.GA9045@spearce.org","subject":"Re: [PATCH] Support wholesale directory renames in fast-import","fromName":"Rogan Dawes","fromEmail":"lists@dawes.za.net","sentAt":"2007-07-10T08:44:08Z","receivedAt":"2007-07-10T08:44:08Z","isPatch":true,"sender":{"key":"lists@dawes.za.net","avatar":null},"body":"Shawn O. Pearce wrote:\n> -Zero or more `filemodify`, `filedelete` and `filedeleteall` commands\n> +Zero or more `filemodify`, `filedelete`, `filename` and\n                                                 ^^ filerename\n\nRogan\n"},{"id":"46942","messageId":"20070710135554.GK4436@spearce.org","threadId":"8950","inReplyTo":"469346D8.4080906@dawes.za.net","subject":"Re: [PATCH] Support wholesale directory renames in fast-import","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-07-10T13:55:54Z","receivedAt":"2007-07-10T13:55:54Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Rogan Dawes <lists@dawes.za.net> wrote:\n> Shawn O. Pearce wrote:\n> >-Zero or more `filemodify`, `filedelete` and `filedeleteall` commands\n> >+Zero or more `filemodify`, `filedelete`, `filename` and\n>                                                 ^^ filerename\n\nUgh.  Thanks.  I just pushed out a corrected version.\n\n-- \nShawn.\n"},{"id":"46945","messageId":"20070710140338.GA18450@informatik.uni-freiburg.de","threadId":"8950","inReplyTo":"7154c5c60707092116p70aaeb8l90cda9265311b999@mail.gmail.com","subject":"Re: [PATCH] Support wholesale directory renames in fast-import","fromName":"Uwe Kleine-König","fromEmail":"ukleinek@informatik.uni-freiburg.de","sentAt":"2007-07-10T14:03:39Z","receivedAt":"2007-07-10T14:03:39Z","isPatch":true,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hallo David,\n\nDavid Frech wrote:\n> Now my challenge is that the svn dump doesn't *actually* say \"rename\n> a/ to b/\"; it says \"copy a/ to b/; delete a/\", so I have to infer the\n> rename.\nI don't know fast-import very well, but why not doing exactly what the\ndump file suggests:  copy a b; delete a ?\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=12+divided+by+3\n"},{"id":"46947","messageId":"20070710141442.GM4436@spearce.org","threadId":"8950","inReplyTo":"20070710140338.GA18450@informatik.uni-freiburg.de","subject":"Re: [PATCH] Support wholesale directory renames in fast-import","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-07-10T14:14:42Z","receivedAt":"2007-07-10T14:14:42Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Uwe Kleine-K??nig <ukleinek@informatik.uni-freiburg.de> wrote:\n> David Frech wrote:\n> > Now my challenge is that the svn dump doesn't *actually* say \"rename\n> > a/ to b/\"; it says \"copy a/ to b/; delete a/\", so I have to infer the\n> > rename.\n> \n> I don't know fast-import very well, but why not doing exactly what the\n> dump file suggests:  copy a b; delete a ?\n\nBecause there is no copy operator in fast-import.  So you cannot\ndo \"copy a b\".  Apparently that's what I should have implemented,\nas rename in Git really is as simple as the copy/delete pair.  Ugh.\n\nCopy isn't really that hard, it just can't be nearly as efficient as\nrename, as copying a subtree will force me to either duplicate data\nin memory or reload trees from disk to duplicate data in memory.\nBut its a copy, so data duplication is expected.  ;-)\n\nI'll implement a copy opertor soon.  Shouldn't be too difficult.\nMaybe someone else would like to take a shot at implementing it...\n\n-- \nShawn.\n"},{"id":"46981","messageId":"7154c5c60707101255k7fcd207fg62f1c59518ba5039@mail.gmail.com","threadId":"8950","inReplyTo":"20070710141442.GM4436@spearce.org","subject":"Re: [PATCH] Support wholesale directory renames in fast-import","fromName":"David Frech","fromEmail":"david@nimblemachines.com","sentAt":"2007-07-10T19:55:46Z","receivedAt":"2007-07-10T19:55:46Z","isPatch":true,"sender":{"key":"david@nimblemachines.com","avatar":null},"body":"Hmm. I think Uwe is right. Copy is probably the \"right\" primitive, and\nrename can always be synthesized from copy+delete.\n\nSince Subversion is built around the idea of \"cheap copies\" there is\nno incentive for them to represent renames other than as \"copy, then\ndelete\".\n\nBut isn't the same true in a way of git? If I copy a directory (a\ntree), then the new tree is the same tree - it has the same SHA-1\nhash, so I can simply refer to the existing object. Same for file\nblobs.\n\nSubversion dump files have *lots* of copies. Might be nice to be able\nto feed these directly into fast-import and have it DTRT, esp if it\nwas smart about sharing identical data structures.\n\n- David\n\nOn 7/10/07, Shawn O. Pearce <spearce@spearce.org> wrote:\n> Uwe Kleine-K??nig <ukleinek@informatik.uni-freiburg.de> wrote:\n> > David Frech wrote:\n> > > Now my challenge is that the svn dump doesn't *actually* say \"rename\n> > > a/ to b/\"; it says \"copy a/ to b/; delete a/\", so I have to infer the\n> > > rename.\n> >\n> > I don't know fast-import very well, but why not doing exactly what the\n> > dump file suggests:  copy a b; delete a ?\n>\n> Because there is no copy operator in fast-import.  So you cannot\n> do \"copy a b\".  Apparently that's what I should have implemented,\n> as rename in Git really is as simple as the copy/delete pair.  Ugh.\n>\n> Copy isn't really that hard, it just can't be nearly as efficient as\n> rename, as copying a subtree will force me to either duplicate data\n> in memory or reload trees from disk to duplicate data in memory.\n> But its a copy, so data duplication is expected.  ;-)\n>\n> I'll implement a copy opertor soon.  Shouldn't be too difficult.\n> Maybe someone else would like to take a shot at implementing it...\n>\n> --\n> Shawn.\n>\n\n\n-- \nIf I have not seen farther, it is because I have stood in the\nfootsteps of giants.\n"},{"id":"47012","messageId":"20070711075744.GO4436@spearce.org","threadId":"8950","inReplyTo":"7154c5c60707101255k7fcd207fg62f1c59518ba5039@mail.gmail.com","subject":"Re: [PATCH] Support wholesale directory renames in fast-import","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-07-11T07:57:44Z","receivedAt":"2007-07-11T07:57:44Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"David Frech <david@nimblemachines.com> wrote:\n> Hmm. I think Uwe is right. Copy is probably the \"right\" primitive, and\n> rename can always be synthesized from copy+delete.\n> \n> Since Subversion is built around the idea of \"cheap copies\" there is\n> no incentive for them to represent renames other than as \"copy, then\n> delete\".\n> \n> But isn't the same true in a way of git? If I copy a directory (a\n> tree), then the new tree is the same tree - it has the same SHA-1\n> hash, so I can simply refer to the existing object. Same for file\n> blobs.\n> \n> Subversion dump files have *lots* of copies. Might be nice to be able\n> to feed these directly into fast-import and have it DTRT, esp if it\n> was smart about sharing identical data structures.\n\nYes.  All of that is true.  ;-)\n\nI'm tired.  I just worked an 18 hour day.  I need to go do it all\nover again in about 4 hours.  So I'm going to head off to bed.  But\nI did manage to implement this (I think).  Its totally untested.\nBut feel free to poke at it:\n\n  git://repo.or.cz/git/fastimport.git copy-wip\n\nI'll write documentation and unit tests tomorrow.  And fix any bugs,\nif any get identified.\n\nThe implementation should copy as little memory as possible to do the\nactual copy.  This should make a C/D pair about as efficient as an\nR command if the directory being copied has not yet been modified\nas part of the current commit (this is probably typical for an\nSVN dump file).  The only difference should be a slight increase\nin running time for the C/D pair, as directory entry lookup in\nfast-import is O(n).\n\nOh, and as always, C works with both files and directories...\n\nHmm.  Quickly reading this diff I can actually do it shorter with\na bit of refactoring.  I'll clean that up tomorrow night.  I blame\nit on the lack of sleep that I'm suffering from right now.  ;-)\n\n\n-->8--\nWIP Teach fast import how to copy\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n fast-import.c |  121 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 files changed, 120 insertions(+), 1 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex a1cb13f..41c0352 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -26,10 +26,16 @@ Format of STDIN stream:\n     lf;\n   commit_msg ::= data;\n \n-  file_change ::= file_clr | file_del | file_rnm | file_obm | file_inm;\n+  file_change ::= file_clr\n+    | file_del\n+    | file_rnm\n+    | file_cpy\n+    | file_obm\n+    | file_inm;\n   file_clr ::= 'deleteall' lf;\n   file_del ::= 'D' sp path_str lf;\n   file_rnm ::= 'R' sp path_str sp path_str lf;\n+  file_cpy ::= 'C' sp path_str sp path_str lf;\n   file_obm ::= 'M' sp mode sp (hexsha1 | idnum) sp path_str lf;\n   file_inm ::= 'M' sp mode sp 'inline' sp path_str lf\n     data;\n@@ -623,6 +629,33 @@ static void release_tree_entry(struct tree_entry *e)\n \tavail_tree_entry = e;\n }\n \n+static struct tree_content *dup_tree_content(struct tree_content *s)\n+{\n+\tstruct tree_content *d;\n+\tstruct tree_entry *a, *b;\n+\tunsigned int i, j;\n+\n+\tif (!s)\n+\t\treturn NULL;\n+\td = new_tree_content(s->entry_count);\n+\tfor (i = 0, j = 0; i < s->entry_count; i++) {\n+\t\ta = s->entries[i];\n+\t\tif (a->versions[1].mode) {\n+\t\t\tb = new_tree_entry();\n+\t\t\tmemcpy(b, a, sizeof(*a));\n+\t\t\tif (is_null_sha1(b->versions[1].sha1))\n+\t\t\t\tb->tree = dup_tree_content(b->tree);\n+\t\t\telse\n+\t\t\t\tb->tree = NULL;\n+\t\t\td->entries[j++] = a;\n+\t\t}\n+\t}\n+\td->entry_count = j;\n+\td->delta_depth = s->delta_depth;\n+\n+\treturn d;\n+}\n+\n static void start_packfile(void)\n {\n \tstatic char tmpfile[PATH_MAX];\n@@ -1273,6 +1306,43 @@ del_entry:\n \treturn 1;\n }\n \n+static int tree_content_get(\n+\tstruct tree_entry *root,\n+\tconst char *p,\n+\tstruct tree_entry *leaf)\n+{\n+\tstruct tree_content *t = root->tree;\n+\tconst char *slash1;\n+\tunsigned int i, n;\n+\tstruct tree_entry *e;\n+\n+\tslash1 = strchr(p, '/');\n+\tif (slash1)\n+\t\tn = slash1 - p;\n+\telse\n+\t\tn = strlen(p);\n+\n+\tfor (i = 0; i < t->entry_count; i++) {\n+\t\te = t->entries[i];\n+\t\tif (e->name->str_len == n && !strncmp(p, e->name->str_dat, n)) {\n+\t\t\tif (!slash1) {\n+\t\t\t\tmemcpy(leaf, e, sizeof(*leaf));\n+\t\t\t\tif (is_null_sha1(e->versions[1].sha1))\n+\t\t\t\t\tleaf->tree = dup_tree_content(leaf->tree);\n+\t\t\t\telse\n+\t\t\t\t\tleaf->tree = NULL;\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tif (!S_ISDIR(e->versions[1].mode))\n+\t\t\t\treturn 0;\n+\t\t\tif (!e->tree)\n+\t\t\t\tload_tree(e);\n+\t\t\treturn tree_content_get(e, slash1 + 1, leaf);\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n static int update_branch(struct branch *b)\n {\n \tstatic const char *msg = \"fast-import\";\n@@ -1706,6 +1776,53 @@ static void file_change_r(struct branch *b)\n \tfree(d_uq);\n }\n \n+static void file_change_c(struct branch *b)\n+{\n+\tconst char *s, *d;\n+\tchar *s_uq, *d_uq;\n+\tconst char *endp;\n+\tstruct tree_entry leaf;\n+\n+\ts = command_buf.buf + 2;\n+\ts_uq = unquote_c_style(s, &endp);\n+\tif (s_uq) {\n+\t\tif (*endp != ' ')\n+\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n+\t}\n+\telse {\n+\t\tendp = strchr(s, ' ');\n+\t\tif (!endp)\n+\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n+\t\ts_uq = xmalloc(endp - s + 1);\n+\t\tmemcpy(s_uq, s, endp - s);\n+\t\ts_uq[endp - s] = 0;\n+\t}\n+\ts = s_uq;\n+\n+\tendp++;\n+\tif (!*endp)\n+\t\tdie(\"Missing dest: %s\", command_buf.buf);\n+\n+\td = endp;\n+\td_uq = unquote_c_style(d, &endp);\n+\tif (d_uq) {\n+\t\tif (*endp)\n+\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n+\t\td = d_uq;\n+\t}\n+\n+\tmemset(&leaf, 0, sizeof(leaf));\n+\tif (!tree_content_get(&b->branch_tree, s, &leaf))\n+\t\tdie(\"Path %s not in branch\", s);\n+\ttree_content_set(&b->branch_tree, d,\n+\t\tleaf.versions[1].sha1,\n+\t\tleaf.versions[1].mode,\n+\t\tleaf.tree);\n+\n+\tfree(s_uq);\n+\tfree(d_uq);\n+}\n+\n static void file_change_deleteall(struct branch *b)\n {\n \trelease_tree_content_recursive(b->branch_tree.tree);\n@@ -1875,6 +1992,8 @@ static void cmd_new_commit(void)\n \t\t\tfile_change_d(b);\n \t\telse if (!prefixcmp(command_buf.buf, \"R \"))\n \t\t\tfile_change_r(b);\n+\t\telse if (!prefixcmp(command_buf.buf, \"C \"))\n+\t\t\tfile_change_c(b);\n \t\telse if (!strcmp(\"deleteall\", command_buf.buf))\n \t\t\tfile_change_deleteall(b);\n \t\telse\n-- \n1.5.3.rc0.879.g64b8\n\n\n-- \nShawn.\n"},{"id":"47093","messageId":"7154c5c60707111611t7692230ah71883dfd6867c52e@mail.gmail.com","threadId":"8950","inReplyTo":"20070711075744.GO4436@spearce.org","subject":"Re: [PATCH] Support wholesale directory renames in fast-import","fromName":"David Frech","fromEmail":"nimblemachines@gmail.com","sentAt":"2007-07-11T23:11:41Z","receivedAt":"2007-07-11T23:11:41Z","isPatch":true,"sender":{"key":"nimblemachines@gmail.com","avatar":null},"body":"On 7/11/07, Shawn O. Pearce <spearce@spearce.org> wrote:\n> I'm tired.  I just worked an 18 hour day.  I need to go do it all\n> over again in about 4 hours.  So I'm going to head off to bed.  But\n> I did manage to implement this (I think).  Its totally untested.\n> But feel free to poke at it:\n>\n>   git://repo.or.cz/git/fastimport.git copy-wip\n>\n> I'll write documentation and unit tests tomorrow.  And fix any bugs,\n> if any get identified.\n\nShawn, don't knock yourself out. ;-)\n\nI think getting the semantics right (of the command set in\nfast-import) is important, but it doesn't have to be right\n*yesterday*. I won't need to point my command stream at fast-import\nfor at least another day or two. ;-) I have some subtle code to write\nfirst that'll take me a bit to get right...\n\nThanks for your enthusiasm though!\n\nCheers,\n\n- David\n\n> --\n> Shawn.\n>\n\n-- \nIf I have not seen farther, it is because I have stood in the\nfootsteps of giants.\n"}]}