# git svn mkdirs ignores compressed unhandled.log files 7 messages from 2009-12-17 to 2009-12-23. Participants: Robert Zeh, Eric Wong, Junio C Hamano. Thread: https://gitlist.dev/t/21977 ## Robert Zeh, 2009-12-17 17:10 Subject: git svn mkdirs ignores compressed unhandled.log files Message-ID: <8BB233FB-4269-4B14-8703-A4FF1E25FB0D@gmail.com> URL: https://gitlist.dev/e/8BB233FB-4269-4B14-8703-A4FF1E25FB0D%40gmail.com ``` It looks like there is a conflict between git svn gc and git svn mkdirs. The git svn mkdirs command only looks at unhandled.log files. Shouldn't it also look at any compressed unhandled.log files too? Robert ``` ## Eric Wong, 2009-12-17 20:08 Subject: Re: git svn mkdirs ignores compressed unhandled.log files Message-ID: <20091217200852.GA5797@dcvr.yhbt.net> URL: https://gitlist.dev/e/20091217200852.GA5797%40dcvr.yhbt.net In-Reply-To: <8BB233FB-4269-4B14-8703-A4FF1E25FB0D@gmail.com> ``` Robert Zeh wrote: > It looks like there is a conflict between git svn gc and git svn > mkdirs. The git svn mkdirs command only looks at unhandled.log files. > Shouldn't it also look at any compressed unhandled.log files too? Hi Robert, Yes, an oversight. Does this patch work for you? (Highly untested) Would you mind writing a test case, been a bit busy with other stuff. Thanks. diff --git a/git-svn.perl b/git-svn.perl index a4b052c..d362de7 100755 --- a/git-svn.perl +++ b/git-svn.perl @@ -2740,21 +2740,44 @@ sub do_fetch { sub mkemptydirs { my ($self, $r) = @_; + + sub scan { + my ($r, $empty_dirs, $line) = @_; + if (defined $r && $line =~ /^r(\d+)$/) { + return 0 if $1 > $r; + } elsif ($line =~ /^ \+empty_dir: (.+)$/) { + $empty_dirs->{$1} = 1; + } elsif ($line =~ /^ \-empty_dir: (.+)$/) { + my @d = grep {m[^\Q$1\E(/|$)]} (keys %$empty_dirs); + delete @$empty_dirs{@d}; + } + 1; # continue + }; + my %empty_dirs = (); + my $gz_file = "$self->{dir}/unhandled.log.gz"; + if (-f $gz_file) { + if (!$can_compress) { + warn "Compress::Zlib could not be found; ", + "empty directories in $gz_file will not be read\n"; + } else { + my $gz = Compress::Zlib::gzopen($gz_file, "rb") or + die "Unable to open $gz_file: $!\n"; + my $line; + while ($gz->gzreadline($line) > 0) { + scan($r, \%empty_dirs, $line) or last; + } + $gz->gzclose; + } + } - open my $fh, '<', "$self->{dir}/unhandled.log" or return; - binmode $fh or croak "binmode: $!"; - while (<$fh>) { - if (defined $r && /^r(\d+)$/) { - last if $1 > $r; - } elsif (/^ \+empty_dir: (.+)$/) { - $empty_dirs{$1} = 1; - } elsif (/^ \-empty_dir: (.+)$/) { - my @d = grep {m[^\Q$1\E(/|$)]} (keys %empty_dirs); - delete @empty_dirs{@d}; + if (open my $fh, '<', "$self->{dir}/unhandled.log") { + binmode $fh or croak "binmode: $!"; + while (<$fh>) { + scan($r, \%empty_dirs, $_) or last; } + close $fh; } - close $fh; my $strip = qr/\A\Q$self->{path}\E(?:\/|$)/; foreach my $d (sort keys %empty_dirs) { -- Eric Wong ``` ## Eric Wong, 2009-12-19 22:27 Subject: [PATCH] git svn: make empty directory creation gc-aware Message-ID: <20091219222738.GA20331@dcvr.yhbt.net> URL: https://gitlist.dev/e/20091219222738.GA20331%40dcvr.yhbt.net In-Reply-To: <20091217200852.GA5797@dcvr.yhbt.net> ``` The "git svn gc" command creates and appends to unhandled.log.gz files which should be parsed before the uncompressed unhandled.log files. Reported-by: Robert Zeh Signed-off-by: Eric Wong --- Eric Wong wrote: > Robert Zeh wrote: > > It looks like there is a conflict between git svn gc and git svn > > mkdirs. The git svn mkdirs command only looks at unhandled.log files. > > Shouldn't it also look at any compressed unhandled.log files too? > > Hi Robert, > > Yes, an oversight. Does this patch work for you? (Highly untested) Test case included and pushed out to git://git.bogomips.org/git-svn More pushes hopefully coming as Sam and Andrew work out the mergeinfo performance problems and I look into crossing svn-remote boundaries for parent lookups. git-svn.perl | 45 +++++++++++++++++++++++++++++++---------- t/t9146-git-svn-empty-dirs.sh | 24 +++++++++++++++++++++ 2 files changed, 58 insertions(+), 11 deletions(-) diff --git a/git-svn.perl b/git-svn.perl index a4b052c..d362de7 100755 --- a/git-svn.perl +++ b/git-svn.perl @@ -2740,21 +2740,44 @@ sub do_fetch { sub mkemptydirs { my ($self, $r) = @_; + + sub scan { + my ($r, $empty_dirs, $line) = @_; + if (defined $r && $line =~ /^r(\d+)$/) { + return 0 if $1 > $r; + } elsif ($line =~ /^ \+empty_dir: (.+)$/) { + $empty_dirs->{$1} = 1; + } elsif ($line =~ /^ \-empty_dir: (.+)$/) { + my @d = grep {m[^\Q$1\E(/|$)]} (keys %$empty_dirs); + delete @$empty_dirs{@d}; + } + 1; # continue + }; + my %empty_dirs = (); + my $gz_file = "$self->{dir}/unhandled.log.gz"; + if (-f $gz_file) { + if (!$can_compress) { + warn "Compress::Zlib could not be found; ", + "empty directories in $gz_file will not be read\n"; + } else { + my $gz = Compress::Zlib::gzopen($gz_file, "rb") or + die "Unable to open $gz_file: $!\n"; + my $line; + while ($gz->gzreadline($line) > 0) { + scan($r, \%empty_dirs, $line) or last; + } + $gz->gzclose; + } + } - open my $fh, '<', "$self->{dir}/unhandled.log" or return; - binmode $fh or croak "binmode: $!"; - while (<$fh>) { - if (defined $r && /^r(\d+)$/) { - last if $1 > $r; - } elsif (/^ \+empty_dir: (.+)$/) { - $empty_dirs{$1} = 1; - } elsif (/^ \-empty_dir: (.+)$/) { - my @d = grep {m[^\Q$1\E(/|$)]} (keys %empty_dirs); - delete @empty_dirs{@d}; + if (open my $fh, '<', "$self->{dir}/unhandled.log") { + binmode $fh or croak "binmode: $!"; + while (<$fh>) { + scan($r, \%empty_dirs, $_) or last; } + close $fh; } - close $fh; my $strip = qr/\A\Q$self->{path}\E(?:\/|$)/; foreach my $d (sort keys %empty_dirs) { diff --git a/t/t9146-git-svn-empty-dirs.sh b/t/t9146-git-svn-empty-dirs.sh index 9b8d046..3f2d719 100755 --- a/t/t9146-git-svn-empty-dirs.sh +++ b/t/t9146-git-svn-empty-dirs.sh @@ -114,5 +114,29 @@ test_expect_success 'removed top-level directory does not exist' ' test ! -e removed/d ' +unhandled=.git/svn/refs/remotes/git-svn/unhandled.log +test_expect_success 'git svn gc-ed files work' ' + ( + cd removed && + git svn gc && + : Compress::Zlib may not be available && + if test -f "$unhandled".gz + then + svn mkdir -m gz "$svnrepo"/gz && + git reset --hard $(git rev-list HEAD | tail -1) && + git svn rebase && + test -f "$unhandled".gz && + test -f "$unhandled" && + for i in a b c "weird file name" gz "! !" + do + if ! test -d "$i" + then + echo >&2 "$i does not exist" + exit 1 + fi + done + fi + ) +' test_done -- Eric Wong ``` ## Junio C Hamano, 2009-12-20 07:08 Subject: Re: [PATCH] git svn: make empty directory creation gc-aware Message-ID: <7vzl5e3yx3.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vzl5e3yx3.fsf%40alter.siamese.dyndns.org In-Reply-To: <20091219222738.GA20331@dcvr.yhbt.net> ``` Eric Wong writes: > The "git svn gc" command creates and appends to unhandled.log.gz > files which should be parsed before the uncompressed > unhandled.log files. > > Reported-by: Robert Zeh > Signed-off-by: Eric Wong > --- > Eric Wong wrote: > > Robert Zeh wrote: > > > It looks like there is a conflict between git svn gc and git svn > > > mkdirs. The git svn mkdirs command only looks at unhandled.log files. > > > Shouldn't it also look at any compressed unhandled.log files too? > > > > Hi Robert, > > > > Yes, an oversight. Does this patch work for you? (Highly untested) > > Test case included and pushed out to git://git.bogomips.org/git-svn > > More pushes hopefully coming as Sam and Andrew work out the mergeinfo > performance problems and I look into crossing svn-remote boundaries > for parent lookups. Thanks. This particular patch should be in 1.6.6 final, because mkdirs first appeared in 1.6.6-rc0 at 6111b93 (git svn: attempt to create empty dirs on clone+rebase, 2009-11-15), and 1.6.5.X series does not have the command, so this seems like a new feature that never existed in any tagged release, and if we shipped 1.6.6 without this patch, we will be shipping it with a know breakage, while if we shipped it with this, even if this patch somehow had an unintended side effect, at worst we'd be exchanging a bug with some other bug, so it wouldn't be worse. Is mkdirs the only "noteworthy" feature that should be mentioned in the Release Notes in your area? It would be really nice if you can give a patch to Documentation/RelNotes-1.6.6.txt in a few days to turn a single liner I have there to something more helpful. The current shortlog since 1.6.5 indicates there weren't that much activity during this release. Alex Vandiver (3): git-svn: sort svk merge tickets to account for minimal parents git-svn: Set svn.authorsfile to an absolute path when cloning git-svn: set svn.authorsfile earlier when cloning Eric Wong (7): git svn: fix fetch where glob is on the top-level URL git svn: read global+system config for clone+init git svn: attempt to create empty dirs on clone+rebase git svn: always reuse existing remotes on fetch git svn: strip leading path when making empty dirs git svn: log removals of empty directories git svn: make empty directory creation gc-aware Greg Price (1): git svn: Don't create empty directories whose parents were deleted Jonathan Nieder (2): add -i, send-email, svn, p4, etc: use "git var GIT_EDITOR" am -i, git-svn: use "git var GIT_PAGER" Sam Vilain (2): git-svn: convert SVK merge tickets to extra parents git-svn: convert SVN 1.5+ / svnmerge.py svn:mergeinfo props to parents Thomas Rast (1): Document git-svn's first-parent rule Toby Allsopp (1): git svn: handle SVN merges from revisions past the tip of the branch ``` ## Eric Wong, 2009-12-20 07:09 Subject: [PATCH 2/1] t9146: use 'svn_cmd' wrapper Message-ID: <20091220070959.GA8141@dcvr.yhbt.net> URL: https://gitlist.dev/e/20091220070959.GA8141%40dcvr.yhbt.net In-Reply-To: <20091219222738.GA20331@dcvr.yhbt.net> ``` Using 'svn' directly may not work for all users. Signed-off-by: Eric Wong --- > Test case included and pushed out to git://git.bogomips.org/git-svn Junio: Not sure if you've merged yet, but feel free to squash this with the other one. Thanks. t/t9146-git-svn-empty-dirs.sh | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/t/t9146-git-svn-empty-dirs.sh b/t/t9146-git-svn-empty-dirs.sh index 3f2d719..565365c 100755 --- a/t/t9146-git-svn-empty-dirs.sh +++ b/t/t9146-git-svn-empty-dirs.sh @@ -122,7 +122,7 @@ test_expect_success 'git svn gc-ed files work' ' : Compress::Zlib may not be available && if test -f "$unhandled".gz then - svn mkdir -m gz "$svnrepo"/gz && + svn_cmd mkdir -m gz "$svnrepo"/gz && git reset --hard $(git rev-list HEAD | tail -1) && git svn rebase && test -f "$unhandled".gz && -- Eric Wong ``` ## Eric Wong, 2009-12-20 07:21 Subject: Re: [PATCH] git svn: make empty directory creation gc-aware Message-ID: <20091220072015.GB8141@dcvr.yhbt.net> URL: https://gitlist.dev/e/20091220072015.GB8141%40dcvr.yhbt.net In-Reply-To: <7vzl5e3yx3.fsf@alter.siamese.dyndns.org> ``` Junio C Hamano wrote: > > More pushes hopefully coming as Sam and Andrew work out the mergeinfo > > performance problems and I look into crossing svn-remote boundaries > > for parent lookups. > > Thanks. > > This particular patch should be in 1.6.6 final, because mkdirs first > appeared in 1.6.6-rc0 at 6111b93 (git svn: attempt to create empty dirs on > clone+rebase, 2009-11-15), and 1.6.5.X series does not have the command, > so this seems like a new feature that never existed in any tagged release, > and if we shipped 1.6.6 without this patch, we will be shipping it with a > know breakage, while if we shipped it with this, even if this patch > somehow had an unintended side effect, at worst we'd be exchanging a bug > with some other bug, so it wouldn't be worse. I agree completely. > Is mkdirs the only "noteworthy" feature that should be mentioned in the > Release Notes in your area? It would be really nice if you can give a > patch to Documentation/RelNotes-1.6.6.txt in a few days to turn a single > liner I have there to something more helpful. The current shortlog since > 1.6.5 indicates there weren't that much activity during this release. Sam's merge handling work is definitely noteworthy, but it's already in the release notes, hopefully the performance regression there is worked out. I'll definitely send you a patch to the release notes after I get a chance to figure out the other issue with multiple svn-remotes tonight/tomorrow. -- Eric Wong ``` ## Robert Zeh, 2009-12-23 04:12 Subject: Re: git svn mkdirs ignores compressed unhandled.log files Message-ID: <661FC27C-6C5D-4703-93BE-48B1812C3800@gmail.com> URL: https://gitlist.dev/e/661FC27C-6C5D-4703-93BE-48B1812C3800%40gmail.com In-Reply-To: <20091217200852.GA5797@dcvr.yhbt.net> ``` On Dec 17, 2009, at 2:08 PM, Eric Wong wrote: > Robert Zeh wrote: >> It looks like there is a conflict between git svn gc and git svn >> mkdirs. The git svn mkdirs command only looks at unhandled.log files. >> Shouldn't it also look at any compressed unhandled.log files too? > > Hi Robert, > > Yes, an oversight. Does this patch work for you? (Highly untested) > > Would you mind writing a test case, been a bit busy with other stuff. > Thanks. Eric, Your patch works for the existing t9146-git-svn-empty-dirs.sh test, and the test I've sent as a patch in another email. Robert ```