# git-svn and u-boot broken. 9 messages from 2006-10-13 to 2006-10-14. Participants: Joakim Tjernlund, Randal L. Schwartz, Eric Wong. Thread: https://gitlist.dev/t/5916 ## Joakim Tjernlund, 2006-10-13 19:19 Subject: git-svn and u-boot broken. Message-ID: <00ad01c6eefc$84298280$1267a8c0@Jocke> URL: https://gitlist.dev/e/00ad01c6eefc%2484298280%241267a8c0%40Jocke ``` I am trying to create a SVN tree using git-svn from the u-boot git tree and it doesn't work. First, I had to change this (from memory) in git-svn: my $got = SVN::TxDelta::send_stream($fh, @$atd, $self->{pool}); to my $got; if ( $got ) { $got = SVN::TxDelta::send_stream($fh, @$atd, $self->{pool}); } else { $got = $exp } I am no perl programmer so please change as you se fit. Secondly I ran out of open file descriptors which I "fixed" with ulimit -n 10000, maybe there is a leak in git-svn? Then it still failed, but this I could not fix. Don't have the error msg atm as I am home now, but it should be easy enough to reproduce, just clone current u-boot tree and try :) What I really would like to do is merge all u-boot commits up to a point into one commit and then commit that single git commit to a clean SVN repo. Then I want merge the remaining git commits(my own stuff). Tried to import a tar tree into the SVN repo with svn import and then merge the remain commits on top of that using git-svn but that didn't work, lots of conflicts. Jocke ``` ## Randal L. Schwartz, 2006-10-13 19:39 Subject: Re: git-svn and u-boot broken. Message-ID: <86slhsgg3f.fsf@blue.stonehenge.com> URL: https://gitlist.dev/e/86slhsgg3f.fsf%40blue.stonehenge.com In-Reply-To: <00ad01c6eefc$84298280$1267a8c0@Jocke> ``` >>>>> "Joakim" == Joakim Tjernlund writes: Joakim> First, I had to change this (from memory) in git-svn: Joakim> my $got = SVN::TxDelta::send_stream($fh, @$atd, $self->{pool}); Joakim> to Joakim> my $got; Joakim> if ( $got ) { Joakim> $got = SVN::TxDelta::send_stream($fh, @$atd, $self->{pool}); Joakim> } else { Joakim> $got = $exp Joakim> } Joakim> I am no perl programmer so please change as you se fit. That doesn't make any sense. You'll never run the if-true branch there. The value of $got immediately following "my $got;" is always undef. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training! ``` ## Joakim Tjernlund, 2006-10-13 19:43 Subject: RE: git-svn and u-boot broken. Message-ID: <00ae01c6eeff$d6c454e0$1267a8c0@Jocke> URL: https://gitlist.dev/e/00ae01c6eeff%24d6c454e0%241267a8c0%40Jocke In-Reply-To: <86slhsgg3f.fsf@blue.stonehenge.com> ``` > > >>>>> "Joakim" == Joakim Tjernlund > writes: > > Joakim> First, I had to change this (from memory) in git-svn: > Joakim> my $got = SVN::TxDelta::send_stream($fh, @$atd, > $self->{pool}); > Joakim> to > Joakim> my $got; > Joakim> if ( $got ) { > Joakim> $got = SVN::TxDelta::send_stream($fh, @$atd, > $self->{pool}); > Joakim> } else { > Joakim> $got = $exp > Joakim> } > Joakim> I am no perl programmer so please change as you se fit. > > That doesn't make any sense. You'll never run the if-true > branch there. > The value of $got immediately following "my $got;" is always undef. Typo, should be 'if( $atd )', sorry ``` ## Joakim Tjernlund, 2006-10-14 01:21 Subject: RE: git-svn and u-boot broken. Message-ID: <00e301c6ef2f$16993ac0$1267a8c0@Jocke> URL: https://gitlist.dev/e/00e301c6ef2f%2416993ac0%241267a8c0%40Jocke In-Reply-To: <86slhsgg3f.fsf@blue.stonehenge.com> ``` > -----Original Message----- > From: Randal L. Schwartz [mailto:merlyn@stonehenge.com] > Sent: den 13 oktober 2006 21:40 > To: Joakim Tjernlund > Cc: git@vger.kernel.org > Subject: Re: git-svn and u-boot broken. > > >>>>> "Joakim" == Joakim Tjernlund > writes: > > Joakim> First, I had to change this (from memory) in git-svn: > Joakim> my $got = SVN::TxDelta::send_stream($fh, @$atd, > $self->{pool}); > Joakim> to > Joakim> my $got; > Joakim> if ( $got ) { > Joakim> $got = SVN::TxDelta::send_stream($fh, @$atd, > $self->{pool}); > Joakim> } else { > Joakim> $got = $exp > Joakim> } > Joakim> I am no perl programmer so please change as you se fit. > > That doesn't make any sense. You'll never run the if-true > branch there. > The value of $got immediately following "my $got;" is always undef. git-pull --squash seems to do what I want but know I get this when committing to svn: git-svn commit remotes/git-svn..svn-branch ... A cpu/mpc824x/drivers/i2c_export.h A cpu/mpc824x/drivers/i2o.h A cpu/mpc824x/drivers/i2o/Makefile A cpu/mpc824x/drivers/i2o/Makefile_pc Cannot allocate memory at /usr/bin/git-svn line 1462 3072 at /usr/bin/git-svn line 547 main::commit_lib('2ff9bcb7908d2752f643695ab3a28f9ababfafab') called at /usr/bin/git-svn line 457 main::commit('remotes/git-svn..svn-branch') called at /usr/bin/git-svn line 149 Jocke ``` ## Eric Wong, 2006-10-14 06:28 Subject: Re: git-svn and u-boot broken. Message-ID: <20061014062812.GC29149@localdomain> URL: https://gitlist.dev/e/20061014062812.GC29149%40localdomain In-Reply-To: <00e301c6ef2f$16993ac0$1267a8c0@Jocke> ``` Joakim Tjernlund wrote: > > > > -----Original Message----- > > From: Randal L. Schwartz [mailto:merlyn@stonehenge.com] > > Sent: den 13 oktober 2006 21:40 > > To: Joakim Tjernlund > > Cc: git@vger.kernel.org > > Subject: Re: git-svn and u-boot broken. > > > > >>>>> "Joakim" == Joakim Tjernlund > > writes: > > > > Joakim> First, I had to change this (from memory) in git-svn: > > Joakim> my $got = SVN::TxDelta::send_stream($fh, @$atd, > > $self->{pool}); > > Joakim> to > > Joakim> my $got; > > Joakim> if ( $got ) { > > Joakim> $got = SVN::TxDelta::send_stream($fh, @$atd, > > $self->{pool}); > > Joakim> } else { > > Joakim> $got = $exp > > Joakim> } > > Joakim> I am no perl programmer so please change as you se fit. With your correction following merlyn's comments, it could work[1]. I'm curious as to what change git-svn printed right before hitting an error here, and would like to see the files (before and after) that were printed by git-svn. I'm not sure what change would show up in git but cause apply_textdelta to return undef... [1] - I am tempted to apply a change based on yours, but this could just be working around another bug somewhere else (either in git-svn or subversion). > > That doesn't make any sense. You'll never run the if-true > > branch there. > > The value of $got immediately following "my $got;" is always undef. > > git-pull --squash seems to do what I want but know I get this when > committing to svn: > git-svn commit remotes/git-svn..svn-branch > ... > A cpu/mpc824x/drivers/i2c_export.h > A cpu/mpc824x/drivers/i2o.h > A cpu/mpc824x/drivers/i2o/Makefile > A cpu/mpc824x/drivers/i2o/Makefile_pc > Cannot allocate memory at /usr/bin/git-svn line 1462 > 3072 at /usr/bin/git-svn line 547 > main::commit_lib('2ff9bcb7908d2752f643695ab3a28f9ababfafab') called at /usr/bin/git-svn line 457 > main::commit('remotes/git-svn..svn-branch') called at /usr/bin/git-svn line 149 How large is the changeset you're committing? (git show --stat 2ff9bcb7908d2752f643695ab3a28f9ababfafab) I actually have a fair amount of fork() hacks in git-svn to workaround leaks in the SVN:: libraries (hard to avoid otherwise in Perl). I haven't noticed (or looked for) file-descriptor leakage very heard, though. Also, which version of git-svn are you running? Also, which version of Subversion (client library, and server (if possible) are you running? There seems to be a lot of new bugreports to git-svn lately and very little has changed in git-svn the past few months... -- Eric Wong ``` ## Joakim Tjernlund, 2006-10-14 09:51 Subject: RE: git-svn and u-boot broken. Message-ID: <00f301c6ef76$62172060$1267a8c0@Jocke> URL: https://gitlist.dev/e/00f301c6ef76%2462172060%241267a8c0%40Jocke In-Reply-To: <20061014062812.GC29149@localdomain> ``` > -----Original Message----- > From: Eric Wong [mailto:normalperson@yhbt.net] > Sent: den 14 oktober 2006 08:28 > To: Joakim Tjernlund > Cc: 'Randal L. Schwartz'; git@vger.kernel.org > Subject: Re: git-svn and u-boot broken. > > Joakim Tjernlund wrote: > > > > > > > -----Original Message----- > > > From: Randal L. Schwartz [mailto:merlyn@stonehenge.com] > > > Sent: den 13 oktober 2006 21:40 > > > To: Joakim Tjernlund > > > Cc: git@vger.kernel.org > > > Subject: Re: git-svn and u-boot broken. > > > > > > >>>>> "Joakim" == Joakim Tjernlund > > > writes: > > > > > > Joakim> First, I had to change this (from memory) in git-svn: > > > Joakim> my $got = SVN::TxDelta::send_stream($fh, @$atd, > > > $self->{pool}); > > > Joakim> to > > > Joakim> my $got; > > > Joakim> if ( $got ) { > > > Joakim> $got = SVN::TxDelta::send_stream($fh, @$atd, > > > $self->{pool}); > > > Joakim> } else { > > > Joakim> $got = $exp > > > Joakim> } > > > Joakim> I am no perl programmer so please change as you se fit. > > With your correction following merlyn's comments, it could > work[1]. I'm > curious as to what change git-svn printed right before > hitting an error from memory: Usage: svn_txdelta_send_stream(x,y,z,m) > here, and would like to see the files (before and after) that were > printed by git-svn. Sorry, don't have that hande ATM > > I'm not sure what change would show up in git but cause > apply_textdelta > to return undef... > > [1] - I am tempted to apply a change based on yours, but this > could just > be working around another bug somewhere else (either in git-svn or > subversion). > > > > That doesn't make any sense. You'll never run the if-true > > > branch there. > > > The value of $got immediately following "my $got;" is > always undef. > > > > git-pull --squash seems to do what I want but know I get this when > > committing to svn: > > git-svn commit remotes/git-svn..svn-branch > > ... > > A cpu/mpc824x/drivers/i2c_export.h > > A cpu/mpc824x/drivers/i2o.h > > A cpu/mpc824x/drivers/i2o/Makefile > > A cpu/mpc824x/drivers/i2o/Makefile_pc > > Cannot allocate memory at /usr/bin/git-svn line 1462 > > 3072 at /usr/bin/git-svn line 547 > > > main::commit_lib('2ff9bcb7908d2752f643695ab3a28f9ababfafab') > called at /usr/bin/git-svn line 457 > > main::commit('remotes/git-svn..svn-branch') called > at /usr/bin/git-svn line 149 > > How large is the changeset you're committing? > (git show --stat 2ff9bcb7908d2752f643695ab3a28f9ababfafab) That gives me a huge listing that ends in: 4306 files changed, 1194819 insertions(+), 0 deletions(-) got some entries that look like this: board/MAI/bios_emulator/scitech/makedefs/qnxnto.mk | 157 .../bios_emulator/scitech/makedefs/rules/bc16.mk | 69 .../bios_emulator/scitech/makedefs/rules/bc3.mk whats the ... for? > > I actually have a fair amount of fork() hacks in git-svn to workaround > leaks in the SVN:: libraries (hard to avoid otherwise in Perl). > I haven't noticed (or looked for) file-descriptor leakage very heard, > though. > > Also, which version of git-svn are you running? Also, which > version of > Subversion (client library, and server (if possible) are you running? git 1.4.2.3 subversion 1.3.2-r1 I will try upgrading to newer subverison. distribution: Gentoo ``` ## Eric Wong, 2006-10-14 22:46 Subject: Re: git-svn and u-boot broken. Message-ID: <20061014224636.GA12565@soma> URL: https://gitlist.dev/e/20061014224636.GA12565%40soma In-Reply-To: <00f301c6ef76$62172060$1267a8c0@Jocke> ``` Joakim Tjernlund wrote: > > > -----Original Message----- > > From: Eric Wong [mailto:normalperson@yhbt.net] > > Sent: den 14 oktober 2006 08:28 > > To: Joakim Tjernlund > > Cc: 'Randal L. Schwartz'; git@vger.kernel.org > > Subject: Re: git-svn and u-boot broken. > > > > Joakim Tjernlund wrote: > > > > > > > > > > -----Original Message----- > > > > From: Randal L. Schwartz [mailto:merlyn@stonehenge.com] > > > > Sent: den 13 oktober 2006 21:40 > > > > To: Joakim Tjernlund > > > > Cc: git@vger.kernel.org > > > > Subject: Re: git-svn and u-boot broken. > > > > > > > > >>>>> "Joakim" == Joakim Tjernlund > > > > writes: > > > > > > > > Joakim> First, I had to change this (from memory) in git-svn: > > > > Joakim> my $got = SVN::TxDelta::send_stream($fh, @$atd, > > > > $self->{pool}); > > > > Joakim> to > > > > Joakim> my $got; > > > > Joakim> if ( $got ) { > > > > Joakim> $got = SVN::TxDelta::send_stream($fh, @$atd, > > > > $self->{pool}); > > > > Joakim> } else { > > > > Joakim> $got = $exp > > > > Joakim> } > > > > Joakim> I am no perl programmer so please change as you se fit. > > > > With your correction following merlyn's comments, it could > > work[1]. I'm > > curious as to what change git-svn printed right before > > hitting an error > > from memory: > Usage: svn_txdelta_send_stream(x,y,z,m) > > here, and would like to see the files (before and after) that were > > printed by git-svn. > > Sorry, don't have that hande ATM Could you publish your git repository for u-boot and also the exact commands (if you have them) you used to attempt the import into SVN? Thanks. > > I'm not sure what change would show up in git but cause > > apply_textdelta > > to return undef... > > > > [1] - I am tempted to apply a change based on yours, but this > > could just > > be working around another bug somewhere else (either in git-svn or > > subversion). > > > > > > That doesn't make any sense. You'll never run the if-true > > > > branch there. > > > > The value of $got immediately following "my $got;" is > > always undef. > > > > > > git-pull --squash seems to do what I want but know I get this when > > > committing to svn: > > > git-svn commit remotes/git-svn..svn-branch > > > ... > > > A cpu/mpc824x/drivers/i2c_export.h > > > A cpu/mpc824x/drivers/i2o.h > > > A cpu/mpc824x/drivers/i2o/Makefile > > > A cpu/mpc824x/drivers/i2o/Makefile_pc > > > Cannot allocate memory at /usr/bin/git-svn line 1462 > > > 3072 at /usr/bin/git-svn line 547 > > > > > main::commit_lib('2ff9bcb7908d2752f643695ab3a28f9ababfafab') > > called at /usr/bin/git-svn line 457 > > > main::commit('remotes/git-svn..svn-branch') called > > at /usr/bin/git-svn line 149 > > > > How large is the changeset you're committing? > > (git show --stat 2ff9bcb7908d2752f643695ab3a28f9ababfafab) > > That gives me a huge listing that ends in: > 4306 files changed, 1194819 insertions(+), 0 deletions(-) Yikes, that's a huge commit. git-svn memory usage was bounded for the size of each changeset, so yes, this could get nasty. Patch coming in a reply to this message. > got some entries that look like this: > board/MAI/bios_emulator/scitech/makedefs/qnxnto.mk | 157 > .../bios_emulator/scitech/makedefs/rules/bc16.mk | 69 > .../bios_emulator/scitech/makedefs/rules/bc3.mk > > whats the ... for? Long filename that couldn't fit on the (80 char width?) diffstat prints out. -- Eric Wong ``` ## Eric Wong, 2006-10-14 22:48 Subject: [PATCH] git-svn: reduce memory usage for large commits Message-ID: <20061014224834.GB12565@soma> URL: https://gitlist.dev/e/20061014224834.GB12565%40soma In-Reply-To: <20061014224636.GA12565@soma> ``` apply_textdelta and send_stream can use a separate pool from the rest of the editor interface, so we'll use a separate SVN::Pool for them and clear the pool after each file is sent to SVN. This drastically reduces memory usage per-changeset committed, and makes large commits (and initial imports) of several thousand files possible. Signed-off-by: Eric Wong --- git-svn.perl | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/git-svn.perl b/git-svn.perl index 0f968c8..54d2356 100755 --- a/git-svn.perl +++ b/git-svn.perl @@ -3354,9 +3354,11 @@ sub chg_file { seek $fh, 0, 0 or croak $!; my $exp = $md5->hexdigest; - my $atd = $self->apply_textdelta($fbat, undef, $self->{pool}); - my $got = SVN::TxDelta::send_stream($fh, @$atd, $self->{pool}); + my $pool = SVN::Pool->new; + my $atd = $self->apply_textdelta($fbat, undef, $pool); + my $got = SVN::TxDelta::send_stream($fh, @$atd, $pool); die "Checksum mismatch\nexpected: $exp\ngot: $got\n" if ($got ne $exp); + $pool->clear; close $fh or croak $!; } -- 1.4.3.rc2.ge931 ``` ## Joakim Tjernlund, 2006-10-14 23:54 Subject: RE: git-svn and u-boot broken. Message-ID: <00f901c6efec$07fdc630$1267a8c0@Jocke> URL: https://gitlist.dev/e/00f901c6efec%2407fdc630%241267a8c0%40Jocke In-Reply-To: <20061014224636.GA12565@soma> ``` > Joakim Tjernlund wrote: > > > > > -----Original Message----- > > > From: Eric Wong [mailto:normalperson@yhbt.net] > > > Sent: den 14 oktober 2006 08:28 > > > To: Joakim Tjernlund > > > Cc: 'Randal L. Schwartz'; git@vger.kernel.org > > > Subject: Re: git-svn and u-boot broken. > > > > > > Joakim Tjernlund wrote: > > > > > > > > > > > > > -----Original Message----- > > > > > From: Randal L. Schwartz [mailto:merlyn@stonehenge.com] > > > > > Sent: den 13 oktober 2006 21:40 > > > > > To: Joakim Tjernlund > > > > > Cc: git@vger.kernel.org > > > > > Subject: Re: git-svn and u-boot broken. > > > > > > > > > > >>>>> "Joakim" == Joakim Tjernlund > > > > > writes: > > > > > > > > > > Joakim> First, I had to change this (from memory) in git-svn: > > > > > Joakim> my $got = SVN::TxDelta::send_stream($fh, @$atd, > > > > > $self->{pool}); > > > > > Joakim> to > > > > > Joakim> my $got; > > > > > Joakim> if ( $got ) { > > > > > Joakim> $got = SVN::TxDelta::send_stream($fh, @$atd, > > > > > $self->{pool}); > > > > > Joakim> } else { > > > > > Joakim> $got = $exp > > > > > Joakim> } > > > > > Joakim> I am no perl programmer so please change as > you se fit. > > > > > > With your correction following merlyn's comments, it could > > > work[1]. I'm > > > curious as to what change git-svn printed right before > > > hitting an error > > > > from memory: > > Usage: svn_txdelta_send_stream(x,y,z,m) > > > > here, and would like to see the files (before and after) that were > > > printed by git-svn. > > > > Sorry, don't have that hande ATM > > Could you publish your git repository for u-boot and also the > exact commands (if you have them) you used to attempt the import > into SVN? Thanks. The git repo is a bit complicated as I don't have have anywhere to put it. However I don't think you need it, just clone DENX u-boot repo at git://www.denx.de/git/u-boot.git, it bails out before it gets to my changes. Here is the script I use to test with REPO="file:///tmp/SVNuboot" REPO_PATH="/tmp/SVNuboot" GIT_REPO="/tmp/mygituboot" ORG_REPO="/usr/local/src/u-boot" rm -rf "$REPO_PATH" rm -rf "$GIT_REPO" svnadmin create "$REPO_PATH" svn mkdir -m "initial repo layout" "$REPO"/trunk "$REPO"/branches "$REPO"/tags mkdir -p "$GIT_REPO" cd "$GIT_REPO" git-svn init "$REPO"/trunk echo [user] >> .git/config echo name="jocke" >> .git/config echo email="Joakim.Tjernlund@transmode.se" >> .git/config git-svn fetch git checkout -b svn-branch remotes/git-svn git-fetch "$ORG_REPO" tmcu2:tmcu git-branch initial-uboot f5e0d03970409feb3c77ab0107d5dece6b7d45c9 git pull . initial-uboot # --squash makes one large commit. #git pull --squash . initial-uboot #cg-commit -m "merge up to f5e0d03970409feb3c77ab0107d5dece6b7d45c9" # without --squash one gets a SVN repo with lots of small commits(2) git pull . initial-uboot git-svn commit -q remotes/git-svn..svn-branch (2) This fails currently with git-svn commit -q remotes/git-svn..svn-branch diff-tree 9030f4be0b324cb312cfecd86d4a7da04ffa99ee 8e9655f863246db60c51140153186acc2afdc855 Inappropriate ioctl for device at /usr/bin/git-svn line 1462 6400 at /usr/bin/git-svn line 1462 6400 at /usr/bin/git-svn line 547 main::commit_lib('8e9655f863246db60c51140153186acc2afdc855', 'ef04a0aa2676738847bd69914cadac0b96d6e731', '4d03a21f2329bb13a9661a23742ac0b77a7a76d5', 'f190c11b1f22ff766e046588e5a7bb55f28ae305', '182e10691f378987b53c64ee0347d542e4924ef6', '527b5a511c804074eebfc08c9ae7860b6717da85', 'ea339205a94706d8b9170bbd89e8c1373df9a92f', '3ada834e30a0742991209140049035c626ccea85', 'feaedfcf84c92599c0b805f66cc8a628d82411f0', ...) called at /usr/bin/git-svn line 457 main::commit('remotes/git-svn..svn-branch') called at /usr/bin/git-svn line 149 > > > > I'm not sure what change would show up in git but cause > > > apply_textdelta > > > to return undef... > > > > > > [1] - I am tempted to apply a change based on yours, but this > > > could just > > > be working around another bug somewhere else (either in git-svn or > > > subversion). > > > > > > > > That doesn't make any sense. You'll never run the if-true > > > > > branch there. > > > > > The value of $got immediately following "my $got;" is > > > always undef. > > > > > > > > git-pull --squash seems to do what I want but know I > get this when > > > > committing to svn: > > > > git-svn commit remotes/git-svn..svn-branch > > > > ... > > > > A cpu/mpc824x/drivers/i2c_export.h > > > > A cpu/mpc824x/drivers/i2o.h > > > > A cpu/mpc824x/drivers/i2o/Makefile > > > > A cpu/mpc824x/drivers/i2o/Makefile_pc > > > > Cannot allocate memory at /usr/bin/git-svn line 1462 > > > > 3072 at /usr/bin/git-svn line 547 > > > > > > > main::commit_lib('2ff9bcb7908d2752f643695ab3a28f9ababfafab') > > > called at /usr/bin/git-svn line 457 > > > > main::commit('remotes/git-svn..svn-branch') called > > > at /usr/bin/git-svn line 149 > > > > > > How large is the changeset you're committing? > > > (git show --stat 2ff9bcb7908d2752f643695ab3a28f9ababfafab) > > > > That gives me a huge listing that ends in: > > 4306 files changed, 1194819 insertions(+), 0 deletions(-) > > Yikes, that's a huge commit. git-svn memory usage was bounded > for the size of each changeset, so yes, this could get nasty. > Patch coming in a reply to this message. Thanks, will try that patch next. Jocke ```