{"thread":{"id":"20159","subject":"[PATCH] Add git svn gc command","startedAt":"2009-07-19T23:00:52Z","lastAt":"2009-07-21T03:25:15Z","messageCount":8,"participants":["Robert Zeh","Eric Wong","demerphq","Thomas Rast"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"118275","messageId":"8E0EFA3B-E8C0-4030-8BAE-E8EF65694031@gmail.com","threadId":"20159","inReplyTo":null,"subject":"[PATCH] Add git svn gc command","fromName":"Robert Zeh","fromEmail":"robert.a.zeh@gmail.com","sentAt":"2009-07-19T23:00:52Z","receivedAt":"2009-07-19T23:00:52Z","isPatch":true,"sender":{"key":"robert.a.zeh@gmail.com","avatar":"https://avatars.githubusercontent.com/u/174737?v=4"},"body":"Add a git svn gc command that gzips all unhandled.log files, and \nremoves all index files under .git/svn.\n\nSigned-off-by: Robert Allan Zeh \n---\n Documentation/git-svn.txt | 4 ++++\n git-svn.perl | 37 +++++++++++++++++++++++++++++++++++++\n t/t9140-git-svn-gc.sh | 45 ++++++++++++++++++++++++++++++++++++ \n+++++++++\n 3 files changed, 86 insertions(+), 0 deletions(-)\n create mode 100755 t/t9140-git-svn-gc.sh\n\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex 10af599..6d0753e 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -348,6 +348,10 @@ Any other arguments are passed directly to 'git \nlog'\n \"checksum mismatch\" (missed a modification). If the problem\n file cannot be ignored forever (with --ignore-paths) the only\n way to repair the repo is to use 'reset'.\n+\n+'gc':: Compress $GIT_DIR/svn//unhandled.log files in .git/svn\n+ and remove $GIT_DIR/svn/index files in .git/svn.\n+\n +\n Only the rev_map and refs/remotes/git-svn are changed. Follow 'reset'\n with a 'fetch' and then 'git reset' or 'git rebase' to move local\ndiff --git a/git-svn.perl b/git-svn.perl\nindex cfade63..022efc0 100755\n--- a/git-svn.perl\n+++ b/git-svn.perl\n@@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';\n push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';\n push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';\n use Carp qw/croak/;\n+use Compress::Zlib;\n use Digest::MD5;\n use IO::File qw//;\n use File::Basename qw/dirname basename/;\n use File::Path qw/mkpath/;\n use File::Spec;\n+use File::Find;\n+use File::Basename;\n use Getopt::Long qw/:config gnu_getopt no_ignore_case auto_abbrev/;\n use IPC::Open3;\n use Git;\n@@ -217,6 +220,9 @@ my %cmd = (\n \"Undo fetches back to the specified SVN revision\",\n { 'revision|r=s' => \\$_revision,\n 'parent|p' => \\$_fetch_parent } ],\n+ 'gc' => [ \\&cmd_gc,\n+ \"Compress unhandled.log files in .git/svn and remove \nindex files in .git/svn\",\n+ {} ],\n );\n\n my $cmd;\n@@ -1107,6 +1113,18 @@ sub cmd_reset {\n print \"r$r = $c ($gs->{ref_id})\\n\";\n }\n\n+sub cmd_gc\n+{\n+ use Cwd;\n+ my $git_dir = $ENV{GIT_DIR};\n+ my $svn_dir = \"$git_dir/svn\";\n+ print getcwd;\n+ print \"\\n\";\n+ find({ wanted => \\&gc_directory,\n+ no_chdir => 1},\n+ \"$git_dir/svn\");\n+}\n+\n ########################### utility functions \n#########################\n\n sub rebase_cmd {\n@@ -1527,6 +1545,25 @@ sub md5sum {\n return $md5->hexdigest();\n }\n\n+sub gc_directory {\n+ if (-f $_ && basename($_) eq \"unhandled.log\") {\n+ my $out_filename = $_ . \".gz\";\n+ print $out_filename, \"\\n\";\n+ open my $in_fh, \"<\", $_ or die \"Unable to open $_: $!\\n\";\n+ binmode $in_fh;\n+ my $gz = gzopen($out_filename, \"wb\") or die \"Unable to open \n$out_filename: $!\\n\";\n+\n+ my $res;\n+ while ($res = sysread($in_fh, my $str, 1024)) {\n+ $gz->gzwrite($str) or die \"Unable to write: $gzerrno\\n\";\n+ }\n+ unlink $_ or die \"unlink $File::Find::name: $!\\n\";\n+ } elsif (-f $_ && basename($_) eq \"index\") {\n+ unlink $_ or die \"unlink %_: $!\\n\";\n+ }\n+}\n+\n+\n package Git::SVN;\n use strict;\n use warnings;\ndiff --git a/t/t9140-git-svn-gc.sh b/t/t9140-git-svn-gc.sh\nnew file mode 100755\nindex 0000000..545b6c7\n--- /dev/null\n+++ b/t/t9140-git-svn-gc.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2009 Robert Allan Zeh\n+\n+test_description='git svn gc basic tests'\n+\n+. ./lib-git-svn.sh\n+\n+mkdir import\n+mkdir tmp\n+cd import\n+ cat >> test.txt <<\\EOF\n+Sample text for Subversion repository.\n+EOF\n+ svn_cmd import -m 'import for git svn' . \"$svnrepo\" > /dev/null\n+cd ..\n+test_expect_success 'checkout working copy from svn' 'svn co \n\"$svnrepo\" test_wc'\n+test_expect_success 'set some properties to create an unhandled.log \nfile' \\\n+ 'cd test_wc &&\n+ svn_cmd propset foo bar test.txt &&\n+ svn_cmd commit -m \"property set\"\n+ cd ..'\n+\n+test_expect_success \\\n+ 'Setup repo and fetch' '\n+ git svn init \"$svnrepo\" &&\n+ git svn fetch'\n+\n+test_expect_success \\\n+ 'make backup copy of unhandled.log' '\n+ cp .git/svn/git-svn/unhandled.log tmp'\n+\n+test_expect_success \\\n+ 'git svn gc runs' '\n+ git svn gc'\n+\n+test_expect_success \\\n+ 'git svn gc produces a valid gzip file' '\n+ gunzip .git/svn/git-svn/unhandled.log.gz'\n+\n+test_expect_success \\\n+ 'git svn gc does not change unhandled.log files' '\n+ test_cmp .git/svn/git-svn/unhandled.log tmp/unhandled.log'\n+\n+test_done\n--\n1.6.4.rc1.dirty\n"},{"id":"118289","messageId":"20090720075929.GA5591@dcvr.yhbt.net","threadId":"20159","inReplyTo":"8E0EFA3B-E8C0-4030-8BAE-E8EF65694031@gmail.com","subject":"Re: [PATCH] Add git svn gc command","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-07-20T07:59:29Z","receivedAt":"2009-07-20T07:59:29Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Robert Zeh wrote:\n> Add a git svn gc command that gzips all unhandled.log files, and removes \n> all index files under .git/svn.\n>\n> Signed-off-by: Robert Allan Zeh \n> ---\n> Documentation/git-svn.txt | 4 ++++\n> git-svn.perl | 37 +++++++++++++++++++++++++++++++++++++\n> t/t9140-git-svn-gc.sh | 45 ++++++++++++++++++++++++++++++++++++ \n> +++++++++\n> 3 files changed, 86 insertions(+), 0 deletions(-)\n> create mode 100755 t/t9140-git-svn-gc.sh\n\nHi Robert,\n\nYour mailer is mangling whitespace badly so the patch isn't applying\nat all. Make sure indents are real tabs like the rest of the code.\n\nSome more comments below, but I think this will be a good addition\nto git svn.\n\n> --- a/git-svn.perl\n> +++ b/git-svn.perl\n> @@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';\n> push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';\n> push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';\n> use Carp qw/croak/;\n> +use Compress::Zlib;\n\nI'd \"require\" Compress::Zlib lazilly so it's not loaded at startup.\nIt's not a stock component of Perl and not needed for the majority of\ncommands.\n\n> +sub cmd_gc\n> +{\n> + use Cwd;\n> + my $git_dir = $ENV{GIT_DIR};\n> + my $svn_dir = \"$git_dir/svn\";\n> + print getcwd;\n> + print \"\\n\";\n\nI'm not sure why you need to print getcwd there, leftover code from\ntesting/debugging?\n\n> +sub gc_directory {\n> + if (-f $_ && basename($_) eq \"unhandled.log\") {\n> + my $out_filename = $_ . \".gz\";\n> + print $out_filename, \"\\n\";\n> + open my $in_fh, \"<\", $_ or die \"Unable to open $_: $!\\n\";\n> + binmode $in_fh;\n> + my $gz = gzopen($out_filename, \"wb\") or die \"Unable to open \n> $out_filename: $!\\n\";\n\ngzopen with \"ab\" might be a better idea than \"wb\". I'm not sure if\nclobbering the existing log every time we run this command is a good\nidea.\n\n-- \nEric Wong\n"},{"id":"118290","messageId":"9b18b3110907200115p4c0c3b45nc6c7f31b34ceff0c@mail.gmail.com","threadId":"20159","inReplyTo":"20090720075929.GA5591@dcvr.yhbt.net","subject":"Re: [PATCH] Add git svn gc command","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-07-20T08:15:28Z","receivedAt":"2009-07-20T08:15:28Z","isPatch":true,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/7/20 Eric Wong :\n> Robert Zeh wrote:\n>> Add a git svn gc command that gzips all unhandled.log files, and removes\n>> all index files under .git/svn.\n>>\n>> Signed-off-by: Robert Allan Zeh \n>> ---\n>>  Documentation/git-svn.txt |    4 ++++\n>>  git-svn.perl              |   37 +++++++++++++++++++++++++++++++++++++\n>>  t/t9140-git-svn-gc.sh     |   45 ++++++++++++++++++++++++++++++++++++\n>> +++++++++\n>>  3 files changed, 86 insertions(+), 0 deletions(-)\n>>  create mode 100755 t/t9140-git-svn-gc.sh\n>\n> Hi Robert,\n>\n> Your mailer is mangling whitespace badly so the patch isn't applying\n> at all.  Make sure indents are real tabs like the rest of the code.\n>\n> Some more comments below, but I think this will be a good addition\n> to git svn.\n>\n>> --- a/git-svn.perl\n>> +++ b/git-svn.perl\n>> @@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';\n>>  push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';\n>>  push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';\n>>  use Carp qw/croak/;\n>> +use Compress::Zlib;\n>\n> I'd \"require\" Compress::Zlib lazilly so it's not loaded at startup.\n> It's not a stock component of Perl and not needed for the majority of\n> commands.\n\nActually, it has been a core component since 5.9.3\n\ncheers,\nYves\n\n\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"118291","messageId":"20090720083621.GA3237@dcvr.yhbt.net","threadId":"20159","inReplyTo":"9b18b3110907200115p4c0c3b45nc6c7f31b34ceff0c@mail.gmail.com","subject":"Re: [PATCH] Add git svn gc command","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-07-20T08:36:21Z","receivedAt":"2009-07-20T08:36:21Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"demerphq wrote:\n> 2009/7/20 Eric Wong :\n> > Robert Zeh wrote:\n> >> --- a/git-svn.perl\n> >> +++ b/git-svn.perl\n> >> @@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';\n> >>  push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';\n> >>  push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';\n> >>  use Carp qw/croak/;\n> >> +use Compress::Zlib;\n> >\n> > I'd \"require\" Compress::Zlib lazilly so it's not loaded at startup.\n> > It's not a stock component of Perl and not needed for the majority of\n> > commands.\n> \n> Actually, it has been a core component since 5.9.3\n\nAh thanks for pointing that out, I didn't notice my 5.10.x install had\nit. Nevertheless, git svn needs to continue supporting 5.8.x for a\nwhile longer.\n\n-- \nEric Wong\n"},{"id":"118292","messageId":"9b18b3110907200202kbb78c8dm43e0451900e4f2fb@mail.gmail.com","threadId":"20159","inReplyTo":"20090720083621.GA3237@dcvr.yhbt.net","subject":"Re: [PATCH] Add git svn gc command","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-07-20T09:02:15Z","receivedAt":"2009-07-20T09:02:15Z","isPatch":true,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/7/20 Eric Wong :\n> demerphq wrote:\n>> 2009/7/20 Eric Wong :\n>> > Robert Zeh wrote:\n>> >> --- a/git-svn.perl\n>> >> +++ b/git-svn.perl\n>> >> @@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';\n>> >>  push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';\n>> >>  push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';\n>> >>  use Carp qw/croak/;\n>> >> +use Compress::Zlib;\n>> >\n>> > I'd \"require\" Compress::Zlib lazilly so it's not loaded at startup.\n>> > It's not a stock component of Perl and not needed for the majority of\n>> > commands.\n>>\n>> Actually, it has been a core component since 5.9.3\n>\n> Ah thanks for pointing that out, I didn't notice my 5.10.x install had\n> it.  Nevertheless, git svn needs to continue supporting 5.8.x for a\n> while longer.\n\nI guess something like:\n\nmy $can_compress= eval \"require Compress::Zlib; 1\";\n\nwould be the right solution.\n\n\n\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"118298","messageId":"200907201342.01912.trast@student.ethz.ch","threadId":"20159","inReplyTo":"9b18b3110907200202kbb78c8dm43e0451900e4f2fb@mail.gmail.com","subject":"Re: [PATCH] Add git svn gc command","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2009-07-20T11:41:59Z","receivedAt":"2009-07-20T11:41:59Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"demerphq wrote:\n> \n> my $can_compress= eval \"require Compress::Zlib; 1\";\n\nWouldn't\n\n my $can_compress = eval { require Compress::Zlib; 1 };\n\nbe better on the grounds that this will catch compilation errors in\nthe eval'd statement at compilation (of the main program) time?\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"118309","messageId":"9b18b3110907200518p57a8b3gdfb51004e274b2bc@mail.gmail.com","threadId":"20159","inReplyTo":"200907201342.01912.trast@student.ethz.ch","subject":"Re: [PATCH] Add git svn gc command","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-07-20T12:18:48Z","receivedAt":"2009-07-20T12:18:48Z","isPatch":true,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/7/20 Thomas Rast :\n> demerphq wrote:\n>>\n>> my $can_compress= eval \"require Compress::Zlib; 1\";\n>\n> Wouldn't\n>\n>  my $can_compress = eval { require Compress::Zlib; 1 };\n>\n> be better on the grounds that this will catch compilation errors in\n> the eval'd statement at compilation (of the main program) time?\n\nYeah, thats a good catch. In this case there is no need for the eval\nSTRING usage.\n\ncheers,\nYves\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"118354","messageId":"20090721032515.GA4636@dcvr.yhbt.net","threadId":"20159","inReplyTo":"1030C457-44CB-4D78-A692-8F3B34A3E9BD@gmail.com","subject":"Re: [PATCH] Add git svn gc command","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2009-07-21T03:25:15Z","receivedAt":"2009-07-21T03:25:15Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Robert Zeh wrote:\n> Hi Eric,\n>\n> Thanks for all of the suggestions. I've attached a patch that takes all \n> of the list suggestions into account. I hope I've implemented the \n> can_compress changes properly.\n\nHi Robert, that looks fine to me.\n\n> I believe I've fixed the white space issues; I tried a round trip and my \n> tabs were preserved. The getcwd stuff was indeed leftover code from \n> testing; I found another left over debug statement.\n\nThe tabs seem alright, but spaces are still mangled. Perhaps\nDocumentation/SubmittingPatches has tips for your mailer? (or check out\ngit send-email)\n\n> I have one remaining question --- does git svn gc need to avoid \n> removing index files if --no-metadata was set? If it does, I'd \n> appreciate a tip on how to see if it was set.\n\nNope, not at all, index files are very safely removable unless there's a\nclone/fetch actively running. The rev_map files on the other hand are\nnot removable iff --no-metadata or --use-svm-props is used.\n\n-- \nEric Wong\n"}]}