git-svn: Missing files

Hi all,

I'm probably confused, but it seems like git-svn has missed some files while importing the buildroot sources. I did this:

  git-svn init svn://uclibc.org/trunk/buildroot
  git-svn fetch

Then I used gitk to find the revision corresponding to svn's r15711, and checked that out:

  git-checkout -b test c84f9463b67bdb57c93bec571b1118b37cb597ce

commit c84f9463b67bdb57c93bec571b1118b37cb597ce
Author: vapier <vapier@69ca8d6d-28ef-0310-b511-8ec308f3f277>
Date:   Mon Jul 17 03:53:12 2006 +0000

    fix whitespace

    git-svn-id: svn://uclibc.org/trunk/buildroot@15711
69ca8d6d-28ef-0310-b511-8ec308f3f277

The resulting tree is missing a bunch of files, compared to what I checked out with svn:

  svn export -r15711 svn://uclibc.org/trunk/buildroot buildroot

Also, if I use git-svn to only fetch r15711 then I get what I expect, with no missing files. The tree is the same as the 'svn export' minus a few empty directories:

  git-svn init svn://uclibc.org/trunk/buildroot
  git-svn fetch -r15711

I've listed the missing files below. There are no other differences - all the files that did turn up are identical in all three trees ("svn export", "git-svn fetch" and "git-svn fetch -r15711").

Disclaimer: I'm no subversion expert, and I'm fairly new to git. But
git rocks, and I'd love somewhere to locally commit my hacks to
subversion projects like buildroot.

Thanks!

- Ben.

diff -ur --exclude .git svn-export/buildroot git-svn-fetch-all/buildroot Only in svn-export/buildroot/package: linux Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 003_kbuild_fixes.diff.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 004_386_emu.diff.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 005_modularize_vesafb.diff.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 006_init_unshare.diff.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 008-ieee1394-fix.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 009-always-inline.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 010-optimize-for-size.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 012-x86-check_gcc.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 015_cramfs_initrd.diff.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 017-printk.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 018-slab-loop-init.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 041-changeloop.patch.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 042-loopfixes.patch.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 062-silence-blk-queue.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 063-silence.kbd.patch.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 064-shutup-md.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 076-nmap-freak.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 077-orinoco-0.13e.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 078-hostap.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 079-jiffies64.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 089-no-touch-makedep.bz2 Only in svn-export/buildroot/target/device/Soekris/net4521/kernel-patches: 100_VERSION.bz2 Only in svn-export/buildroot/target/generic: access_point Only in svn-export/buildroot/target/generic: firewall Only in svn-export/buildroot/target/generic/target_skeleton/etc: fstab Only in svn-export/buildroot/target/generic/target_skeleton/etc: hostname Only in svn-export/buildroot/target/generic/target_skeleton/etc: hosts Only in svn-export/buildroot/target/generic/target_skeleton/etc/init.d: rcS Only in svn-export/buildroot/target/generic/target_skeleton/etc/init.d: S40network Only in svn-export/buildroot/target/generic/target_skeleton/etc: inittab Only in svn-export/buildroot/target/generic/target_skeleton/etc: inputrc Only in svn-export/buildroot/target/generic/target_skeleton/etc: issue Only in svn-export/buildroot/target/generic/target_skeleton/etc: network Only in svn-export/buildroot/target/generic/target_skeleton/etc: passwd Only in svn-export/buildroot/target/generic/target_skeleton/etc: profile Only in svn-export/buildroot/target/generic/target_skeleton/etc: protocols Only in svn-export/buildroot/target/generic/target_skeleton/etc: random-seed Only in svn-export/buildroot/target/generic/target_skeleton/etc: securetty Only in svn-export/buildroot/target/generic/target_skeleton/etc: services Only in svn-export/buildroot/target/generic/target_skeleton/etc: shadow Only in svn-export/buildroot/target/generic/target_skeleton/etc: TZ Only in svn-export/buildroot/target/generic/target_skeleton: root Only in svn-export/buildroot/target/generic/target_skeleton: usr Only in svn-export/buildroot/toolchain/binutils: 2.16.91.0.6 Only in svn-export/buildroot/toolchain/binutils/2.16.91.0.7: 300-001_ld_makefile_patch.patch Only in svn-export/buildroot/toolchain/binutils/2.16.91.0.7: 300-006_better_file_error.patch Only in svn-export/buildroot/toolchain/binutils/2.16.91.0.7: 300-012_check_ldrunpath_length.patch Only in svn-export/buildroot/toolchain/binutils/2.16.91.0.7: 400-mips-ELF_MAXPAGESIZE-4K.patch Only in svn-export/buildroot/toolchain/binutils/2.16.91.0.7: 702-binutils-skip-comments.patch Only in svn-export/buildroot/toolchain/binutils: 2.17 Only in svn-export/buildroot/toolchain/binutils: 2.17.50.0.2 Only in svn-export/buildroot/toolchain/binutils: 2.17.50.0.3 Only in svn-export/buildroot/toolchain/gcc/3.4.6: 100-uclibc-conf.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 200-uclibc-locale.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 300-libstdc++-pic.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 600-gcc34-arm-ldm-peephole.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 601-gcc34-arm-ldm.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 601-gcc34-arm-ldm-peephole2.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 602-sdk-libstdc++-includes.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 700-pr15068-fix.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 71_all_sh-pr16665-fix.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 72_all_sh-no-reorder-blocks.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 73_all_sh-pr20617.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 800-powerpc-libc_stack_end-uclibc.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: 900-nios2.patch Only in svn-export/buildroot/toolchain/gcc/3.4.6: arm-softfloat.patch.conditional Only in svn-export/buildroot/toolchain/gcc/4.0.3: 100-uclibc-conf.patch Only in svn-export/buildroot/toolchain/gcc/4.0.3: 200-uclibc-locale.patch Only in svn-export/buildroot/toolchain/gcc/4.0.3: 300-libstdc++-pic.patch Only in svn-export/buildroot/toolchain/gcc/4.0.3: 301-missing-execinfo_h.patch Only in svn-export/buildroot/toolchain/gcc/4.0.3: 302-c99-snprintf.patch Only in svn-export/buildroot/toolchain/gcc/4.0.3: 303-c99-complex-ugly-hack.patch Only in svn-export/buildroot/toolchain/gcc/4.0.3: 602-sdk-libstdc++-includes.patch Only in svn-export/buildroot/toolchain/gcc/4.1.1: 100-uclibc-conf.patch Only in svn-export/buildroot/toolchain/gcc/4.1.1: 301-missing-execinfo_h.patch Only in svn-export/buildroot/toolchain/gcc/4.1.1: 302-c99-snprintf.patch Only in svn-export/buildroot/toolchain/gcc/4.1.1: 303-c99-complex-ugly-hack.patch Only in svn-export/buildroot/toolchain/gcc/4.1.1: 304-index_macro.patch Only in svn-export/buildroot/toolchain/gcc/4.1.1: 740-sh-pr24836.patch Only in svn-export/buildroot/toolchain/gcc/4.1.1: 800-arm-bigendian.patch Only in svn-export/buildroot/toolchain/gcc/4.2: 103-uclibc-conf-noupstream.patch Only in svn-export/buildroot/toolchain/gcc/4.2: 203-uclibc-locale-no__x.patch Only in svn-export/buildroot/toolchain/gcc/4.2: 204-uclibc-locale-wchar_fix.patch Only in svn-export/buildroot/toolchain/gcc/4.2: 301-missing-execinfo_h.patch Only in svn-export/buildroot/toolchain/gcc/4.2: 302-c99-snprintf.patch Only in svn-export/buildroot/toolchain/gcc/4.2: 303-c99-complex-ugly-hack.patch Only in svn-export/buildroot/toolchain/gdb: 6.5

Oh. I just looked in git-svn and found this:

$VERSION = '1.1.1-broken';

Fair enough. So far I haven't explored other branches in git.git, I've no idea what "pu" stands for. Can someone point me in the right direction?

Thanks,

- Ben.

On 7/20/06, Ben Williamson <benw@pobox.com> wrote:

Ben Williamson <benw@pobox.com> wrote:

I removed the version tag and started using GIT_VERSION when git-svn moved out of contrib/ a few weeks ago.

As far as I remember, I don't remember git-svn having problems with missing files. There has been a bug where it got extra files from other places in the repository, but that's fixed.

May I ask if you have the Perl SVN:: library bindings installed? If so 1.1.1-broken (and all versions afterwards) will automatically. use them (if the SVN library version is >= 1.1).

Nevertheless, I'm running an import right now (with the SVN:: libraries enabled) and will make another run with them disabled (which is kind of slow). I'll keep you posted...

I've actually been getting a lot of real-world git-svn usage in the past few weeks (and hence the lack of git-related work) and haven't noticed any major problems.

pu is "potential updates", it's very bleeding edge. next is a few steps ahead of master, which should be the safest of the three.

Eric Wong <normalperson@yhbt.net> wrote:

Ok, I think I've found the problem. This problem only happens if you're using the SVN:: libraries, right?

Wow, I just got back from lunch to three replies and a patch! I'll try to keep up here...

On 7/20/06, Eric Wong <normalperson@yhbt.net> wrote:

Yes I have Perl SVN:: installed, because I started off trying git-svnimport. (Most of my attempts there resulted in lots of commits of the same empty tree, until someone else on this list mentioned that "trunk/projectname" should be removed from the url. Then it started working, but it seems to be a large repo and was taking forever, so I tried git-svn.) I didn't realise git-svn didn't need the Perl stuff.

$ rpm -qa | grep subversion subversion-1.2.1-0.1.1.fc3.rf subversion-perl-1.2.1-0.1.1.fc3.rf

Me too, with your latest script - will let you know. Thanks!

- Ben.

Ben, can you please try this patch against the git-svn.perl file in master? You can also find the full patched file here:

	http://git-svn.bogomips.org/git-svn.perl

From: Eric Wong <normalperson@yhbt.net>
Date: Wed, 19 Jul 2006 20:42:35 -0700
Subject: [PATCH] git-svn: fix fetching new directories copies when using SVN:: libs

Log output from SVN doesn't list all the new files that were added if a new directory was copied from an existing place in the repository. This means we'll have to do some extra work and traverse new directories ourselves.

Thanks to Ben Williamson for the excellent bug report.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 git-svn.perl |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

On 7/20/06, Eric Wong <normalperson@yhbt.net> wrote:

Just finished, and the results were heaps better, but it seems package/dhcp/Config.in was still missed. (The others below are empty directories.)

$ diff -ur --exclude=.git --exclude=.svn svn-export/buildroot git-svn-fetch-fixed/buildroot Only in svn-export/buildroot/package/dhcp: Config.in Only in svn-export/buildroot/package: linux Only in svn-export/buildroot/target/generic: access_point Only in svn-export/buildroot/target/generic: firewall

Hey, good enough for me, but then I'm not using dhcp...  ;)

- Ben.

Log output from SVN doesn't list all the new files that were added if a new directory was copied from an existing place in the repository. This means we'll have to do some extra work and traverse new directories ourselves.

This has been updated from the original patch to defer traversed adds until all removals have been done. Please disregard the original.

Thanks to Ben Williamson for the excellent bug report and testing.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 git-svn.perl |   23 ++++++++++++++++-------
 1 files changed, 16 insertions(+), 7 deletions(-)

Thanks.