threads / discuss / 43450
Subject: Re: git-svn: why fetching files is so slow
git-svn: why fetching files is so slow
... compared to the standalone svn client. I'm working with repositories over the internet, using not-so-fast links, but still, a svn checkout takes somewhere around 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to fetch the initial revision. Later fetches also take *a lot* more time than a svn update would.
Cheers,
-- Pazu
Seth FalconPazu <pazu@pazu.com.br> writes:
> ... compared to the standalone svn client. I'm working with repositories over > the internet, using not-so-fast links, but still, a svn checkout takes somewhere > around 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to > fetch the initial revision. Later fetches also take *a lot* more time than a svn > update would.
[warning: I _think_ this is how it works, but not 100% sure] When you use git-svn to fetch from an svn repository, you make a separate request for each commit that occurred on the remote svn repos. When you use the svn client, it only needs to compute and download one delta .
If you are not already using the Perl SVN bindings (you will need to build svn from source), you should give them a try. They are much faster.
My experience has often been the opposite, but I think that is because I work with an svn repository where I track a directory that has many many subdirs. The svn working copy traversal is so slow that even with the extra network overhead, git + git-svn ends up being faster for fetch (and much faster for any local operation).
Pazu <pazu@pazu.com.br> wrote:
> ... compared to the standalone svn client. I'm working with repositories over > the internet, using not-so-fast links, but still, a svn checkout takes somewhere > around 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to > fetch the initial revision. Later fetches also take *a lot* more time than a svn > update would.
git-svn transfers full files, and not deltas. I'll hopefully have a chance to look into improving the situation for slow links this weekend.
Eric Wong <normalperson <at> yhbt.net> writes:
> git-svn transfers full files, and not deltas. I'll hopefully have a > chance to look into improving the situation for slow links this weekend.
Yes, but why would that make fetching the first revision slower? In this situation, both svn and git-svn would have to fetch full files. Maybe git-svn isn't using gzip compression or http pipelining?
-- Pazu
Pazu <pazu@pazu.com.br> wrote:
> Eric Wong <normalperson <at> yhbt.net> writes: > > > git-svn transfers full files, and not deltas. I'll hopefully have a > > chance to look into improving the situation for slow links this weekend. > > Yes, but why would that make fetching the first revision slower? In this > situation, both svn and git-svn would have to fetch full files. Maybe git-svn > isn't using gzip compression or http pipelining?
Even for the initial transfer, the tree is bundled into one big delta (at least over https).
Junio C HamanoEric Wong <normalperson@yhbt.net> writes:
> Pazu <pazu@pazu.com.br> wrote: >> Eric Wong <normalperson <at> yhbt.net> writes: >> >> > git-svn transfers full files, and not deltas. I'll hopefully have a >> > chance to look into improving the situation for slow links this weekend. >> >> Yes, but why would that make fetching the first revision slower? In this >> situation, both svn and git-svn would have to fetch full files. Maybe git-svn >> isn't using gzip compression or http pipelining? > > Even for the initial transfer, the tree is bundled into one big delta > (at least over https).
Do you mean that "one big delta" saves duplicates across copies inside the tree (e.g. svn tags and branches can be expressed as a mostly identical copies of each other), or do you mean "one full file at a time" requests are killing us, compared to a such single transfer of "one big delta"?
Junio C Hamano <junkio@cox.net> wrote:
> Eric Wong <normalperson@yhbt.net> writes: > > > Pazu <pazu@pazu.com.br> wrote: > >> Eric Wong <normalperson <at> yhbt.net> writes: > >> > >> > git-svn transfers full files, and not deltas. I'll hopefully have a > >> > chance to look into improving the situation for slow links this weekend. > >> > >> Yes, but why would that make fetching the first revision slower? In this > >> situation, both svn and git-svn would have to fetch full files. Maybe git-svn > >> isn't using gzip compression or http pipelining? > > > > Even for the initial transfer, the tree is bundled into one big delta > > (at least over https). > > Do you mean that "one big delta" saves duplicates across copies > inside the tree (e.g. svn tags and branches can be expressed as > a mostly identical copies of each other), or do you mean "one > full file at a time" requests are killing us, compared to a such > single transfer of "one big delta"?
One full file at a time requests are definitely killing us (over slow links, at least). I'm not sure how/if duplicates inside a requested tree are optimized on the server side.
[PATCH 2/2] git-svn: update tests for recent changes
* Enable test for delta transfers in full-svn-test.
* Run tests against the root of the repository so we won't have to revisit 308906fa6e98132cab839a4f42701386fba368ef and efe4631def181d32f932672a7ea31e52ee0ab308 again. The graft-branches test still runs as before.
Signed-off-by: Eric Wong <normalperson@yhbt.net> --- t/Makefile | 3 ++- t/lib-git-svn.sh | 2 +- t/t9100-git-svn-basic.sh | 5 +++++ t/t9103-git-svn-graft-branches.sh | 2 ++ 4 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/t/Makefile b/t/Makefile index 8983509..0abe66d 100644 --- a/t/Makefile +++ b/t/Makefile @@ -27,8 +27,9 @@ clean: # we can test NO_OPTIMIZE_COMMITS independently of LC_ALL full-svn-test: + $(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_DELTA_FETCH=1 \ + GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C $(MAKE) $(TSVN) GIT_SVN_NO_LIB=1 GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C - $(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C $(MAKE) $(TSVN) GIT_SVN_NO_LIB=1 GIT_SVN_NO_OPTIMIZE_COMMITS=0 \ LC_ALL=en_US.UTF-8 $(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_NO_OPTIMIZE_COMMITS=0 \ diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh index 29a1e72..63c6703 100644 --- a/t/lib-git-svn.sh +++ b/t/lib-git-svn.sh @@ -45,6 +45,6 @@ else svnadmin create "$svnrepo" fi -svnrepo="file://$svnrepo/test-git-svn" +svnrepo="file://$svnrepo" diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh index 34a3ccd..f9de232 100755 --- a/t/t9100-git-svn-basic.sh +++ b/t/t9100-git-svn-basic.sh @@ -228,6 +228,11 @@ tree 56a30b966619b863674f5978696f4a3594f tree d667270a1f7b109f5eb3aaea21ede14b56bfdd6e tree 8f51f74cf0163afc9ad68a4b1537288c4558b5a4 EOF + +if test -z "$GIT_SVN_NO_LIB" || test "$GIT_SVN_NO_LIB" -eq 0; then + echo tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904 >> expected +fi + test_expect_success "$name" "diff -u a expected" test_done diff --git a/t/t9103-git-svn-graft-branches.sh b/t/t9103-git-svn-graft-branches.sh index cc62d4e..293b98f 100755 --- a/t/t9103-git-svn-graft-branches.sh +++ b/t/t9103-git-svn-graft-branches.sh @@ -1,6 +1,8 @@ test_description='git-svn graft-branches' . ./lib-git-svn.sh +svnrepo="$svnrepo/test-git-svn" + test_expect_success 'initialize repo' " mkdir import && cd import &&
-- 1.4.4.1.g22a08