{"thread":{"id":"43450","subject":"Re: git-svn: why fetching files is so slow","startedAt":"2006-11-24T13:36:35Z","lastAt":"2006-11-28T05:46:50Z","messageCount":8,"participants":["Seth Falcon","Eric Wong","Junio C Hamano","Pazu"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"297994","messageId":"loom.20061124T143148-286@post.gmane.org","threadId":"43450","inReplyTo":null,"subject":"git-svn: why fetching files is so slow","fromName":"Pazu","fromEmail":"pazu@pazu.com.br","sentAt":"2006-11-24T13:36:35Z","receivedAt":"2006-11-24T13:36:35Z","isPatch":false,"sender":{"key":"pazu@pazu.com.br","avatar":null},"body":"... compared to the standalone svn client. I'm working with repositories over\nthe internet, using not-so-fast links, but still, a svn checkout takes somewhere\naround 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to\nfetch the initial revision. Later fetches also take *a lot* more time than a svn\nupdate would.\n\nCheers,\n\n-- Pazu\n"},{"id":"294842","messageId":"m2odqwlqi8.fsf@ziti.fhcrc.org","threadId":"43450","inReplyTo":"loom.20061124T143148-286@post.gmane.org","subject":"Re: git-svn: why fetching files is so slow","fromName":"Seth Falcon","fromEmail":"sethfalcon@gmail.com","sentAt":"2006-11-24T17:10:39Z","receivedAt":"2006-11-24T17:10:39Z","isPatch":false,"sender":{"key":"sethfalcon@gmail.com","avatar":"https://gravatar.com/avatar/fd62fe73d3013b12fce71d5269ec52eaca3c4cee479efc6eea3603d6f1d8bf8d?d=mp&s=160"},"body":"Pazu writes:\n> ... compared to the standalone svn client. I'm working with repositories over\n> the internet, using not-so-fast links, but still, a svn checkout takes somewhere\n> around 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to\n> fetch the initial revision. Later fetches also take *a lot* more time than a svn\n> update would.\n\n[warning: I _think_ this is how it works, but not 100% sure]\nWhen you use git-svn to fetch from an svn repository, you make a\nseparate request for each commit that occurred on the remote svn\nrepos. When you use the svn client, it only needs to compute and\ndownload one delta .\n\nIf you are not already using the Perl SVN bindings (you will need to\nbuild svn from source), you should give them a try. They are much\nfaster.\n\nMy experience has often been the opposite, but I think that is because\nI work with an svn repository where I track a directory that has many\nmany subdirs. The svn working copy traversal is so slow that even\nwith the extra network overhead, git + git-svn ends up being faster\nfor fetch (and much faster for any local operation).\n\n"},{"id":"297777","messageId":"20061124191609.GA32506@localdomain","threadId":"43450","inReplyTo":"loom.20061124T143148-286@post.gmane.org","subject":"Re: git-svn: why fetching files is so slow","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2006-11-24T19:16:09Z","receivedAt":"2006-11-24T19:16:09Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Pazu wrote:\n> ... compared to the standalone svn client. I'm working with repositories over\n> the internet, using not-so-fast links, but still, a svn checkout takes somewhere\n> around 5 to 10 minutes, while git-svn fetch takes at least 10 times that just to\n> fetch the initial revision. Later fetches also take *a lot* more time than a svn\n> update would.\n\ngit-svn transfers full files, and not deltas. I'll hopefully have a\nchance to look into improving the situation for slow links this weekend.\n\n-- \n"},{"id":"298483","messageId":"loom.20061124T202153-512@post.gmane.org","threadId":"43450","inReplyTo":"20061124191609.GA32506@localdomain","subject":"Re: git-svn: why fetching files is so slow","fromName":"Pazu","fromEmail":"pazu@pazu.com.br","sentAt":"2006-11-24T19:28:48Z","receivedAt":"2006-11-24T19:28:48Z","isPatch":false,"sender":{"key":"pazu@pazu.com.br","avatar":null},"body":"Eric Wong yhbt.net> writes:\n\n> git-svn transfers full files, and not deltas. I'll hopefully have a\n> chance to look into improving the situation for slow links this weekend.\n\nYes, but why would that make fetching the first revision slower? In this\nsituation, both svn and git-svn would have to fetch full files. Maybe git-svn\nisn't using gzip compression or http pipelining?\n\n-- Pazu\n"},{"id":"295537","messageId":"20061124203320.GA21654@soma","threadId":"43450","inReplyTo":"loom.20061124T202153-512@post.gmane.org","subject":"Re: git-svn: why fetching files is so slow","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2006-11-24T20:33:20Z","receivedAt":"2006-11-24T20:33:20Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Pazu wrote:\n> Eric Wong yhbt.net> writes:\n> \n> > git-svn transfers full files, and not deltas. I'll hopefully have a\n> > chance to look into improving the situation for slow links this weekend.\n> \n> Yes, but why would that make fetching the first revision slower? In this\n> situation, both svn and git-svn would have to fetch full files. Maybe git-svn\n> isn't using gzip compression or http pipelining?\n\nEven for the initial transfer, the tree is bundled into one big delta\n(at least over https).\n\n-- \n"},{"id":"295573","messageId":"7vy7q0a85p.fsf@assigned-by-dhcp.cox.net","threadId":"43450","inReplyTo":"20061124203320.GA21654@soma","subject":"Re: git-svn: why fetching files is so slow","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-24T20:42:26Z","receivedAt":"2006-11-24T20:42:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> Pazu wrote:\n>> Eric Wong yhbt.net> writes:\n>> \n>> > git-svn transfers full files, and not deltas. I'll hopefully have a\n>> > chance to look into improving the situation for slow links this weekend.\n>> \n>> Yes, but why would that make fetching the first revision slower? In this\n>> situation, both svn and git-svn would have to fetch full files. Maybe git-svn\n>> isn't using gzip compression or http pipelining?\n>\n> Even for the initial transfer, the tree is bundled into one big delta\n> (at least over https).\n\nDo you mean that \"one big delta\" saves duplicates across copies\ninside the tree (e.g. svn tags and branches can be expressed as\na mostly identical copies of each other), or do you mean \"one\nfull file at a time\" requests are killing us, compared to a such\nsingle transfer of \"one big delta\"?\n\n"},{"id":"296424","messageId":"20061124221435.GA21072@localdomain","threadId":"43450","inReplyTo":"7vy7q0a85p.fsf@assigned-by-dhcp.cox.net","subject":"Re: git-svn: why fetching files is so slow","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2006-11-24T22:14:35Z","receivedAt":"2006-11-24T22:14:35Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano wrote:\n> Eric Wong writes:\n> \n> > Pazu wrote:\n> >> Eric Wong yhbt.net> writes:\n> >> \n> >> > git-svn transfers full files, and not deltas. I'll hopefully have a\n> >> > chance to look into improving the situation for slow links this weekend.\n> >> \n> >> Yes, but why would that make fetching the first revision slower? In this\n> >> situation, both svn and git-svn would have to fetch full files. Maybe git-svn\n> >> isn't using gzip compression or http pipelining?\n> >\n> > Even for the initial transfer, the tree is bundled into one big delta\n> > (at least over https).\n> \n> Do you mean that \"one big delta\" saves duplicates across copies\n> inside the tree (e.g. svn tags and branches can be expressed as\n> a mostly identical copies of each other), or do you mean \"one\n> full file at a time\" requests are killing us, compared to a such\n> single transfer of \"one big delta\"?\n\nOne full file at a time requests are definitely killing us (over slow\nlinks, at least). I'm not sure how/if duplicates inside a requested\ntree are optimized on the server side.\n\n-- \n"},{"id":"296338","messageId":"20061128054650.GB396@soma","threadId":"43450","inReplyTo":"loom.20061124T143148-286@post.gmane.org","subject":"[PATCH 2/2] git-svn: update tests for recent changes","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2006-11-28T05:46:50Z","receivedAt":"2006-11-28T05:46:50Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"* Enable test for delta transfers in full-svn-test.\n\n* Run tests against the root of the repository so we won't have\n to revisit 308906fa6e98132cab839a4f42701386fba368ef and\n efe4631def181d32f932672a7ea31e52ee0ab308 again.\n The graft-branches test still runs as before.\n\nSigned-off-by: Eric Wong \n---\n t/Makefile | 3 ++-\n t/lib-git-svn.sh | 2 +-\n t/t9100-git-svn-basic.sh | 5 +++++\n t/t9103-git-svn-graft-branches.sh | 2 ++\n 4 files changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/t/Makefile b/t/Makefile\nindex 8983509..0abe66d 100644\n--- a/t/Makefile\n+++ b/t/Makefile\n@@ -27,8 +27,9 @@ clean:\n \n # we can test NO_OPTIMIZE_COMMITS independently of LC_ALL\n full-svn-test:\n+\t$(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_DELTA_FETCH=1 \\\n+\t\t\t\t\tGIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C\n \t$(MAKE) $(TSVN) GIT_SVN_NO_LIB=1 GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C\n-\t$(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_NO_OPTIMIZE_COMMITS=1 LC_ALL=C\n \t$(MAKE) $(TSVN) GIT_SVN_NO_LIB=1 GIT_SVN_NO_OPTIMIZE_COMMITS=0 \\\n \t\t\t\t\t\t\tLC_ALL=en_US.UTF-8\n \t$(MAKE) $(TSVN) GIT_SVN_NO_LIB=0 GIT_SVN_NO_OPTIMIZE_COMMITS=0 \\\ndiff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh\nindex 29a1e72..63c6703 100644\n--- a/t/lib-git-svn.sh\n+++ b/t/lib-git-svn.sh\n@@ -45,6 +45,6 @@ else\n \tsvnadmin create \"$svnrepo\"\n fi\n \n-svnrepo=\"file://$svnrepo/test-git-svn\"\n+svnrepo=\"file://$svnrepo\"\n \n \ndiff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh\nindex 34a3ccd..f9de232 100755\n--- a/t/t9100-git-svn-basic.sh\n+++ b/t/t9100-git-svn-basic.sh\n@@ -228,6 +228,11 @@ tree 56a30b966619b863674f5978696f4a3594f\n tree d667270a1f7b109f5eb3aaea21ede14b56bfdd6e\n tree 8f51f74cf0163afc9ad68a4b1537288c4558b5a4\n EOF\n+\n+if test -z \"$GIT_SVN_NO_LIB\" || test \"$GIT_SVN_NO_LIB\" -eq 0; then\n+\techo tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904 >> expected\n+fi\n+\n test_expect_success \"$name\" \"diff -u a expected\"\n \n test_done\ndiff --git a/t/t9103-git-svn-graft-branches.sh b/t/t9103-git-svn-graft-branches.sh\nindex cc62d4e..293b98f 100755\n--- a/t/t9103-git-svn-graft-branches.sh\n+++ b/t/t9103-git-svn-graft-branches.sh\n@@ -1,6 +1,8 @@\n test_description='git-svn graft-branches'\n . ./lib-git-svn.sh\n \n+svnrepo=\"$svnrepo/test-git-svn\"\n+\n test_expect_success 'initialize repo' \"\n \tmkdir import &&\n \tcd import &&\n-- \n1.4.4.1.g22a08\n"}]}