{"thread":{"id":"27472","subject":"[PATCH v2] git-svn: New flag to emulate empty directories","startedAt":"2011-05-26T06:38:23Z","lastAt":"2011-06-28T04:23:24Z","messageCount":3,"participants":["Ray Chen","Eric Wong"],"isPatch":true,"patchVersion":2,"patchTotal":null},"messages":[{"id":"168799","messageId":"1306391903-22092-1-git-send-email-rchen@cs.umd.edu","threadId":"27472","inReplyTo":null,"subject":"[PATCH v2] git-svn: New flag to emulate empty directories","fromName":"Ray Chen","fromEmail":"rchen@cs.umd.edu","sentAt":"2011-05-26T06:38:23Z","receivedAt":"2011-05-26T06:38:23Z","isPatch":true,"sender":{"key":"rchen@cs.umd.edu","avatar":"https://avatars.githubusercontent.com/u/1909064?v=4"},"body":"Adds the --preserve-empty-dirs flag to the clone operation that will\ndetect empty directories in the target Subversion repository and create\nplaceholder files in the corresponding local Git directories.  This allows\n\"empty\" directories to exist in the history of a Git repository.\n\nAlso adds the --placeholder-file flag to control the name of any\nplaceholder files created.  Default value is \".gitignore\".\n\nSigned-off-by: Ray Chen <rchen@cs.umd.edu>\n---\n\nThis version of the patch allows --preserve-empty-dirs and --placeholder-file\nto be specified only during the clone operation.  This removes the ability to\nuse this feature for long-term tracking.\n\nThe main problem with the long-term tracking case occurs when all files are\ndeleted from a directory in a Subversion-based Git repository, and dcommitted\nback to Subversion.  Currently, this causes the directory to be removed from\nthe local Git repo, but leaves an empty directory in the Subversion repo.\n\nTo correctly deal with this situation, dcommit would need to modify the local\ncommit object by re-adding a tree object that contains the placeholder file\nobject.  This would effectively be something like an automatic rebase/squash,\nwhich I'm not sure is possible from the high-level git-svn.perl file.  Does\nanybody know a way to do this?\n\n(BUG REPORT: If somebody then commits a new file to the empty directory using\n Subversion, the next \"git-svn fetch\" will transfer the new file to the local\n Git repo, but it won't show up in the working directory.)\n\nAdditionally, placeholder files are now tracked and removed if they become\nunnecessary.\n\nFinally, this patch includes documentation and a test case.\n\n Documentation/git-svn.txt              |   11 +++\n git-svn.perl                           |   97 ++++++++++++++++++++++++++-\n t/t9159-git-svn-preserve-empty-dirs.sh |  117 ++++++++++++++++++++++++++++++++\n 3 files changed, 223 insertions(+), 2 deletions(-)\n create mode 100755 t/t9159-git-svn-preserve-empty-dirs.sh\n\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex 713e523..fbe4325 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -156,6 +156,17 @@ Skip \"branches\" and \"tags\" of first level directories;;\n \taffecting the working tree; and the 'rebase' command will be\n \table to update the working tree with the latest changes.\n \n+--preserve-empty-dirs;;\n+\tCreate a placeholder file in the local Git repository for each\n+\tempty directory fetched from Subversion.  This includes directories\n+\tthat become empty by removing all entries in the Subversion\n+\trepository (but not the directory itself).  The placeholder files\n+\tare also removed when no longer necessary.\n+\n+--placeholder-filename=<filename>;;\n+\tSet the name of placeholder files created by --preserve-empty-dirs.\n+\tDefault: \".gitignore\"\n+\n 'rebase'::\n \tThis fetches revisions from the SVN parent of the current HEAD\n \tand rebases the current (uncommitted to SVN) work against it.\ndiff --git a/git-svn.perl b/git-svn.perl\nindex 7849cfc..c658f30 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -89,6 +89,7 @@ my ($_stdin, $_help, $_edit,\n \t$_prefix, $_no_checkout, $_url, $_verbose,\n \t$_git_format, $_commit_url, $_tag, $_merge_info);\n $Git::SVN::_follow_parent = 1;\n+$SVN::Git::Fetcher::_placeholder_filename = \".gitignore\";\n $_q ||= 0;\n my %remote_opts = ( 'username=s' => \\$Git::SVN::Prompt::_username,\n                     'config-dir=s' => \\$Git::SVN::Ra::config_dir,\n@@ -139,6 +140,10 @@ my %cmd = (\n \t\t\t   %fc_opts } ],\n \tclone => [ \\&cmd_clone, \"Initialize and fetch revisions\",\n \t\t\t{ 'revision|r=s' => \\$_revision,\n+\t\t\t  'preserve-empty-dirs' =>\n+\t\t\t\t\\$SVN::Git::Fetcher::_preserve_empty_dirs,\n+\t\t\t  'placeholder-filename=s' =>\n+\t\t\t\t\\$SVN::Git::Fetcher::_placeholder_filename,\n \t\t\t   %fc_opts, %init_opts } ],\n \tinit => [ \\&cmd_init, \"Initialize a repo for tracking\" .\n \t\t\t  \" (requires URL argument)\",\n@@ -4076,12 +4081,13 @@ sub _read_password {\n }\n \n package SVN::Git::Fetcher;\n-use vars qw/@ISA/;\n+use vars qw/@ISA $_ignore_regex $_preserve_empty_dirs $_placeholder_filename\n+            @deleted_gpath %added_placeholder/;\n use strict;\n use warnings;\n use Carp qw/croak/;\n+use File::Basename qw/dirname/;\n use IO::File qw//;\n-use vars qw/$_ignore_regex/;\n \n # file baton members: path, mode_a, mode_b, pool, fh, blob, base\n sub new {\n@@ -4223,6 +4229,7 @@ sub delete_entry {\n \t\t$self->{gii}->remove($gpath);\n \t\tprint \"\\tD\\t$gpath\\n\" unless $::_q;\n \t}\n+\tpush @deleted_gpath, $gpath unless $added_placeholder{$gpath};\n \t$self->{empty}->{$path} = 0;\n \tundef;\n }\n@@ -4255,7 +4262,14 @@ sub add_file {\n \t\tmy ($dir, $file) = ($path =~ m#^(.*?)/?([^/]+)$#);\n \t\tdelete $self->{empty}->{$dir};\n \t\t$mode = '100644';\n+\n+\t\tif ($added_placeholder{$dir}) {\n+\t\t\tmy $fn = \"$dir/$_placeholder_filename\";\n+\t\t\tdelete_entry($self, $fn) unless $path eq $fn;\n+\t\t\tdelete $added_placeholder{$dir}\n+\t\t}\n \t}\n+\n \t{ path => $path, mode_a => $mode, mode_b => $mode,\n \t  pool => SVN::Pool->new, action => 'A' };\n }\n@@ -4273,6 +4287,7 @@ sub add_directory {\n \t\t\tchomp;\n \t\t\t$self->{gii}->remove($_);\n \t\t\tprint \"\\tD\\t$_\\n\" unless $::_q;\n+\t\t\tpush @deleted_gpath, $gpath;\n \t\t}\n \t\tcommand_close_pipe($ls, $ctx);\n \t\t$self->{empty}->{$path} = 0;\n@@ -4280,6 +4295,12 @@ sub add_directory {\n \tmy ($dir, $file) = ($path =~ m#^(.*?)/?([^/]+)$#);\n \tdelete $self->{empty}->{$dir};\n \t$self->{empty}->{$path} = 1;\n+\n+\tif ($added_placeholder{$dir}) {\n+\t\tdelete_entry($self, \"$dir/$_placeholder_filename\");\n+\t\tdelete $added_placeholder{$dir}\n+\t}\n+\n out:\n \t{ path => $path };\n }\n@@ -4443,12 +4464,84 @@ sub abort_edit {\n \n sub close_edit {\n \tmy $self = shift;\n+\n+\tif ($_preserve_empty_dirs) {\n+\t\tmy @empty_dirs;\n+\n+\t\t# Any entry flagged as empty that also has an associated\n+\t\t# dir_prop represents a newly created empty directory.\n+\t\tforeach my $i (keys %{$self->{empty}}) {\n+\t\t\tpush @empty_dirs, $i if exists $self->{dir_prop}->{$i};\n+\t\t}\n+\n+\t\t# Search for directories that have become empty due subsequent\n+\t\t# file deletes.\n+\t\tpush @empty_dirs, $self->find_empty_directories();\n+\n+\t\t# Finally, add a placeholder file to each empty directory.\n+\t\t$self->add_placeholder_file($_) foreach (@empty_dirs);\n+\t}\n+\n \t$self->{git_commit_ok} = 1;\n \t$self->{nr} = $self->{gii}->{nr};\n \tdelete $self->{gii};\n \t$self->SUPER::close_edit(@_);\n }\n \n+sub find_empty_directories {\n+\tmy ($self) = @_;\n+\tmy @empty_dirs;\n+\tmy %dirs = map { dirname($_) => 1 } @deleted_gpath;\n+\n+\tforeach my $dir (sort keys %dirs) {\n+\t\tnext if $dir eq \".\";\n+\n+\t\t# If there have been any additions to this directory, there is\n+\t\t# no reason to check if it is empty.\n+\t\tmy $skip_added = 0;\n+\t\tforeach my $t (qw/dir_prop file_prop/) {\n+\t\t\tforeach my $path (keys %{ $self->{$t} }) {\n+\t\t\t\tif (exists $self->{$t}->{dirname($path)}) {\n+\t\t\t\t\t$skip_added = 1;\n+\t\t\t\t\tlast;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tlast if $skip_added;\n+\t\t}\n+\t\tnext if $skip_added;\n+\n+\t\t# Use `git ls-tree` to get the filenames of this directory\n+\t\t# that existed prior to this particular commit.\n+\t\tmy $ls = command('ls-tree', '-z', '--name-only',\n+\t\t\t\t $self->{c}, \"$dir/\");\n+\t\tmy %files = map { $_ => 1 } split(/\\0/, $ls);\n+\n+\t\t# Remove the filenames that were deleted during this commit.\n+\t\tdelete $files{$_} foreach (@deleted_gpath);\n+\n+\t\t# Report the directory if there are no filenames left.\n+\t\tpush @empty_dirs, $dir unless (scalar %files);\n+\t}\n+\t@empty_dirs;\n+}\n+\n+sub add_placeholder_file {\n+\tmy ($self, $dir) = @_;\n+\tmy $path = \"$dir/$_placeholder_filename\";\n+\tmy $gpath = $self->git_path($path);\n+\n+\tmy $fh = $::_repository->temp_acquire($gpath);\n+\tmy $hash = $::_repository->hash_and_insert_object(Git::temp_path($fh));\n+\tGit::temp_release($fh, 1);\n+\t$self->{gii}->update('100644', $hash, $gpath) or croak $!;\n+\n+\t# The directory should no longer be considered empty.\n+\tdelete $self->{empty}->{$dir} if exists $self->{empty}->{$dir};\n+\n+\t# Keep track of any placeholder files we create.\n+\t$added_placeholder{$dir} = 1;\n+}\n+\n package SVN::Git::Editor;\n use vars qw/@ISA $_rmdir $_cp_similarity $_find_copies_harder $_rename_limit/;\n use strict;\ndiff --git a/t/t9159-git-svn-preserve-empty-dirs.sh b/t/t9159-git-svn-preserve-empty-dirs.sh\nnew file mode 100755\nindex 0000000..50cbf51\n--- /dev/null\n+++ b/t/t9159-git-svn-preserve-empty-dirs.sh\n@@ -0,0 +1,117 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2011 Ray Chen\n+#\n+\n+test_description='git svn test (option --preserve-empty-dirs)\n+\n+This test uses git to clone a Subversion repository that contains empty\n+directories, and checks that corresponding directories are created in the\n+local Git repository with placeholder files.'\n+\n+. ./lib-git-svn.sh\n+\n+say 'define NO_SVN_TESTS to skip git svn tests'\n+GIT_REPO=git-svn-repo\n+\n+test_expect_success 'initialize source svn repo containing empty dirs' '\n+\tsvn_cmd mkdir -m x \"$svnrepo\"/trunk &&\n+\tsvn_cmd co \"$svnrepo\"/trunk \"$SVN_TREE\" &&\n+\t(\n+\t\tcd \"$SVN_TREE\" &&\n+\t\tmkdir -p 1 2 3/a 3/b 4 5 6 &&\n+\t\techo \"First non-empty file\"  > 2/file1.txt &&\n+\t\techo \"Second non-empty file\" > 2/file2.txt &&\n+\t\techo \"Third non-empty file\"  > 3/a/file1.txt &&\n+\t\techo \"Fourth non-empty file\" > 3/b/file1.txt &&\n+\t\tsvn_cmd add 1 2 3 4 5 6 &&\n+\t\tsvn_cmd commit -m \"initial commit\" &&\n+\n+\t\tmkdir 4/a &&\n+\t\tsvn_cmd add 4/a &&\n+\t\tsvn_cmd commit -m \"nested empty directory\" &&\n+\t\tmkdir 4/a/b &&\n+\t\tsvn_cmd add 4/a/b &&\n+\t\tsvn_cmd commit -m \"deeply nested empty directory\" &&\n+\t\tmkdir 4/a/b/c &&\n+\t\tsvn_cmd add 4/a/b/c &&\n+\t\tsvn_cmd commit -m \"really deeply nested empty directory\" &&\n+\t\techo \"Kill the placeholder file\" > 4/a/b/c/foo &&\n+\t\tsvn_cmd add 4/a/b/c/foo &&\n+\t\tsvn_cmd commit -m \"Regular file to remove placeholder\" &&\n+\n+\t\tsvn_cmd del 2/file2.txt &&\n+\t\tsvn_cmd del 3/b &&\n+\t\tsvn_cmd commit -m \"delete non-last entry in directory\" &&\n+\n+\t\tsvn_cmd del 2/file1.txt &&\n+\t\tsvn_cmd del 3/a &&\n+\t\tsvn_cmd commit -m \"delete last entry in directory\" &&\n+\n+\t\techo \"Conflict file\" > 5/.placeholder &&\n+\t\tmkdir 6/.placeholder &&\n+\t\tsvn_cmd add 5/.placeholder 6/.placeholder &&\n+\t\tsvn_cmd commit -m \"Placeholder Namespace conflict\"\n+\t) &&\n+\trm -rf \"$SVN_TREE\"\n+'\n+\n+test_expect_success 'clone svn repo with --preserve-empty-dirs' '\n+\tgit svn clone \"$svnrepo\"/trunk --preserve-empty-dirs \"$GIT_REPO\"\n+'\n+\n+# \"$GIT_REPO\"/1 should only contain the placeholder file.\n+test_expect_success 'directory empty from inception' '\n+\ttest -f \"$GIT_REPO\"/1/.gitignore &&\n+\ttest `find \"$GIT_REPO\"/1 -type f | wc -l` = \"1\"\n+'\n+\n+# \"$GIT_REPO\"/2 and \"$GIT_REPO\"/3 should only contain the placeholder file.\n+test_expect_success 'directory empty from subsequent commit' '\n+\ttest -f \"$GIT_REPO\"/2/.gitignore &&\n+\ttest `find \"$GIT_REPO\"/2 -type f | wc -l` = \"1\" &&\n+\ttest -f \"$GIT_REPO\"/3/.gitignore &&\n+\ttest `find \"$GIT_REPO\"/3 -type f | wc -l` = \"1\"\n+'\n+\n+# No placeholder files should exist in \"$GIT_REPO\"/4, even though one was\n+# generated for every sub-directory at some point in the repo's history.\n+test_expect_success 'add entry to previously empty directory' '\n+\ttest `find \"$GIT_REPO\"/4 -type f | wc -l` = \"1\" &&\n+\ttest -f \"$GIT_REPO\"/4/a/b/c/foo\n+'\n+\n+# The HEAD~2 commit should not have introduced .gitignore placeholder files.\n+test_expect_success 'remove non-last entry from directory' '\n+\t(\n+\t\tcd \"$GIT_REPO\" &&\n+\t\tgit checkout HEAD~2\n+\t) &&\n+\ttest_must_fail test -f \"$GIT_REPO\"/2/.gitignore &&\n+\ttest_must_fail test -f \"$GIT_REPO\"/3/.gitignore\n+'\n+\n+# After re-cloning the repository with --placeholder-file specified, there\n+# should be 5 files named \".placeholder\" in the local Git repo.\n+test_expect_success 'clone svn repo with --placeholder-file specified' '\n+\trm -rf \"$GIT_REPO\" &&\n+\tgit svn clone \"$svnrepo\"/trunk --preserve-empty-dirs \\\n+\t\t--placeholder-file=.placeholder \"$GIT_REPO\" &&\n+\tfind \"$GIT_REPO\" -type f -name \".placeholder\" &&\n+\ttest `find \"$GIT_REPO\" -type f -name \".placeholder\" | wc -l` = \"5\"\n+'\n+\n+# \"$GIT_REPO\"/5/.placeholder should be a file, and non-empty.\n+test_expect_success 'placeholder namespace conflict with file' '\n+\ttest -s \"$GIT_REPO\"/5/.placeholder\n+'\n+\n+# \"$GIT_REPO\"/6/.placeholder should be a directory, and the \"$GIT_REPO\"/6 tree\n+# should only contain one file: the placeholder.\n+test_expect_success 'placeholder namespace conflict with directory' '\n+\ttest -d \"$GIT_REPO\"/6/.placeholder &&\n+\ttest -f \"$GIT_REPO\"/6/.placeholder/.placeholder &&\n+\ttest `find \"$GIT_REPO\"/6 -type f | wc -l` = \"1\"\n+'\n+\n+test_done\n-- \n1.7.4.4\n"},{"id":"169150","messageId":"loom.20110601T231627-922@post.gmane.org","threadId":"27472","inReplyTo":"1306391903-22092-1-git-send-email-rchen@cs.umd.edu","subject":"Re: [PATCH v2] git-svn: New flag to emulate empty directories","fromName":"Ray Chen","fromEmail":"rchen@cs.umd.edu","sentAt":"2011-06-01T21:22:05Z","receivedAt":"2011-06-01T21:22:05Z","isPatch":true,"sender":{"key":"rchen@cs.umd.edu","avatar":"https://avatars.githubusercontent.com/u/1909064?v=4"},"body":"Ray Chen <rchen <at> cs.umd.edu> writes:\n\n> \n> Adds the --preserve-empty-dirs flag to the clone operation that will\n> detect empty directories in the target Subversion repository and create\n> placeholder files in the corresponding local Git directories.  This allows\n> \"empty\" directories to exist in the history of a Git repository.\n> \n> Also adds the --placeholder-file flag to control the name of any\n> placeholder files created.  Default value is \".gitignore\".\n> \n\nIs there anything I can do to make this patch more palatable for inclusion?\n\nIt'd also be fine if this patch got left by the wayside.  I recognize that\ncloning Subversion repositories with empty directories is something of a corner\ncase.\n\n- Ray\n"},{"id":"170605","messageId":"20110628042324.GA29123@dcvr.yhbt.net","threadId":"27472","inReplyTo":"loom.20110601T231627-922@post.gmane.org","subject":"Re: [PATCH v2] git-svn: New flag to emulate empty directories","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2011-06-28T04:23:24Z","receivedAt":"2011-06-28T04:23:24Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Sorry, I got tied up with other things for a while.  Feel free to\nremind me after a week or so if I don't respond, too.\n\nRay Chen <rchen@cs.umd.edu> wrote:\n> Is there anything I can do to make this patch more palatable for inclusion?\n\nI would like the config option to be \"sticky\" like the \"--ignore-paths\"\noption when used with clone and saved to the config file.  This can be\ndone inside do_git_init_db().\n\nIn shell scripts, use $(command) instead of `command` in shell scripts\nas suggested in Documentation/CodingStyle.\n\nOtherwise, things look mostly alright and I'm inclined to accept your\npatch.  An extra reviewer would also be helpful since I very rarely use\ngit-svn nowadays.\n\n-- \nEric Wong\n"}]}