# Re: git-svn: why fetching files is so slow 8 messages from 2006-11-24 to 2006-11-28. Participants: Seth Falcon, Eric Wong, Junio C Hamano, Pazu. Thread: https://gitlist.dev/t/43450 ## Pazu, 2006-11-24 13:36 Subject: git-svn: why fetching files is so slow Message-ID: URL: https://gitlist.dev/e/loom.20061124T143148-286%40post.gmane.org ``` ... compared to the standalone svn client. I'm working with repositories over the internet, using not-so-fast links, but still, a svn checkout takes somewhere around 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to fetch the initial revision. Later fetches also take *a lot* more time than a svn update would. Cheers, -- Pazu ``` ## Seth Falcon, 2006-11-24 17:10 Subject: Re: git-svn: why fetching files is so slow Message-ID: URL: https://gitlist.dev/e/m2odqwlqi8.fsf%40ziti.fhcrc.org In-Reply-To: ``` Pazu writes: > ... compared to the standalone svn client. I'm working with repositories over > the internet, using not-so-fast links, but still, a svn checkout takes somewhere > around 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to > fetch the initial revision. Later fetches also take *a lot* more time than a svn > update would. [warning: I _think_ this is how it works, but not 100% sure] When you use git-svn to fetch from an svn repository, you make a separate request for each commit that occurred on the remote svn repos. When you use the svn client, it only needs to compute and download one delta . If you are not already using the Perl SVN bindings (you will need to build svn from source), you should give them a try. They are much faster. My experience has often been the opposite, but I think that is because I work with an svn repository where I track a directory that has many many subdirs. The svn working copy traversal is so slow that even with the extra network overhead, git + git-svn ends up being faster for fetch (and much faster for any local operation). ``` ## Eric Wong, 2006-11-24 19:16 Subject: Re: git-svn: why fetching files is so slow Message-ID: <20061124191609.GA32506@localdomain> URL: https://gitlist.dev/e/20061124191609.GA32506%40localdomain In-Reply-To: ``` Pazu wrote: > ... compared to the standalone svn client. I'm working with repositories over > the internet, using not-so-fast links, but still, a svn checkout takes somewhere > around 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to > fetch the initial revision. Later fetches also take *a lot* more time than a svn > update would. git-svn transfers full files, and not deltas. I'll hopefully have a chance to look into improving the situation for slow links this weekend. -- ``` ## Pazu, 2006-11-24 19:28 Subject: Re: git-svn: why fetching files is so slow Message-ID: URL: https://gitlist.dev/e/loom.20061124T202153-512%40post.gmane.org In-Reply-To: <20061124191609.GA32506@localdomain> ``` Eric Wong yhbt.net> writes: > git-svn transfers full files, and not deltas. I'll hopefully have a > chance to look into improving the situation for slow links this weekend. Yes, but why would that make fetching the first revision slower? In this situation, both svn and git-svn would have to fetch full files. Maybe git-svn isn't using gzip compression or http pipelining? -- Pazu ``` ## Eric Wong, 2006-11-24 20:33 Subject: Re: git-svn: why fetching files is so slow Message-ID: <20061124203320.GA21654@soma> URL: https://gitlist.dev/e/20061124203320.GA21654%40soma In-Reply-To: ``` Pazu wrote: > Eric Wong yhbt.net> writes: > > > git-svn transfers full files, and not deltas. I'll hopefully have a > > chance to look into improving the situation for slow links this weekend. > > Yes, but why would that make fetching the first revision slower? In this > situation, both svn and git-svn would have to fetch full files. Maybe git-svn > isn't using gzip compression or http pipelining? Even for the initial transfer, the tree is bundled into one big delta (at least over https). -- ``` ## Junio C Hamano, 2006-11-24 20:42 Subject: Re: git-svn: why fetching files is so slow Message-ID: <7vy7q0a85p.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vy7q0a85p.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <20061124203320.GA21654@soma> ``` Eric Wong writes: > Pazu wrote: >> Eric Wong yhbt.net> writes: >> >> > git-svn transfers full files, and not deltas. I'll hopefully have a >> > chance to look into improving the situation for slow links this weekend. >> >> Yes, but why would that make fetching the first revision slower? In this >> situation, both svn and git-svn would have to fetch full files. Maybe git-svn >> isn't using gzip compression or http pipelining? > > Even for the initial transfer, the tree is bundled into one big delta > (at least over https). Do you mean that "one big delta" saves duplicates across copies inside the tree (e.g. svn tags and branches can be expressed as a mostly identical copies of each other), or do you mean "one full file at a time" requests are killing us, compared to a such single transfer of "one big delta"? ``` ## Eric Wong, 2006-11-24 22:14 Subject: Re: git-svn: why fetching files is so slow Message-ID: <20061124221435.GA21072@localdomain> URL: https://gitlist.dev/e/20061124221435.GA21072%40localdomain In-Reply-To: <7vy7q0a85p.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano wrote: > Eric Wong writes: > > > Pazu wrote: > >> Eric Wong yhbt.net> writes: > >> > >> > git-svn transfers full files, and not deltas. I'll hopefully have a > >> > chance to look into improving the situation for slow links this weekend. > >> > >> Yes, but why would that make fetching the first revision slower? In this > >> situation, both svn and git-svn would have to fetch full files. Maybe git-svn > >> isn't using gzip compression or http pipelining? > > > > Even for the initial transfer, the tree is bundled into one big delta > > (at least over https). > > Do you mean that "one big delta" saves duplicates across copies > inside the tree (e.g. svn tags and branches can be expressed as > a mostly identical copies of each other), or do you mean "one > full file at a time" requests are killing us, compared to a such > single transfer of "one big delta"? One full file at a time requests are definitely killing us (over slow links, at least). I'm not sure how/if duplicates inside a requested tree are optimized on the server side. -- ``` ## Eric Wong, 2006-11-28 05:46 Subject: [PATCH 2/2] git-svn: update tests for recent changes Message-ID: <20061128054650.GB396@soma> URL: https://gitlist.dev/e/20061128054650.GB396%40soma In-Reply-To: ``` * Enable test for delta transfers in full-svn-test. * Run tests against the root of the repository so we won't have to revisit 308906fa6e98132cab839a4f42701386fba368ef and efe4631def181d32f932672a7ea31e52ee0ab308 again. The graft-branches test still runs as before. Signed-off-by: Eric Wong --- t/Makefile | 3 ++- t/lib-git-svn.sh | 2 +- t/t9100-git-svn-basic.sh | 5 +++++ t/t9103-git-svn-graft-branches.sh | 2 ++ 4 files changed, 10 insertions(+), 2 deletions(-) diff --git a/t/Makefile b/t/Makefile index 8983509..0abe66d 100644 --- a/t/Makefile +++ b/t/Makefile @@ -27,8 +27,9 @@ clean: # we can test NO_OPTIMIZE_COMMITS independently of LC_ALL full-svn-test: + $(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_DELTA_FETCH=1 \ + GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C $(MAKE) $(TSVN) GIT_SVN_NO_LIB=1 GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C - $(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C $(MAKE) $(TSVN) GIT_SVN_NO_LIB=1 GIT_SVN_NO_OPTIMIZE_COMMITS=0 \ LC_ALL=en_US.UTF-8 $(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_NO_OPTIMIZE_COMMITS=0 \ diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh index 29a1e72..63c6703 100644 --- a/t/lib-git-svn.sh +++ b/t/lib-git-svn.sh @@ -45,6 +45,6 @@ else svnadmin create "$svnrepo" fi -svnrepo="file://$svnrepo/test-git-svn" +svnrepo="file://$svnrepo" diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh index 34a3ccd..f9de232 100755 --- a/t/t9100-git-svn-basic.sh +++ b/t/t9100-git-svn-basic.sh @@ -228,6 +228,11 @@ tree 56a30b966619b863674f5978696f4a3594f tree d667270a1f7b109f5eb3aaea21ede14b56bfdd6e tree 8f51f74cf0163afc9ad68a4b1537288c4558b5a4 EOF + +if test -z "$GIT_SVN_NO_LIB" || test "$GIT_SVN_NO_LIB" -eq 0; then + echo tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904 >> expected +fi + test_expect_success "$name" "diff -u a expected" test_done diff --git a/t/t9103-git-svn-graft-branches.sh b/t/t9103-git-svn-graft-branches.sh index cc62d4e..293b98f 100755 --- a/t/t9103-git-svn-graft-branches.sh +++ b/t/t9103-git-svn-graft-branches.sh @@ -1,6 +1,8 @@ test_description='git-svn graft-branches' . ./lib-git-svn.sh +svnrepo="$svnrepo/test-git-svn" + test_expect_success 'initialize repo' " mkdir import && cd import && -- 1.4.4.1.g22a08 ```