{"thread":{"id":"20229","subject":"[PATCH 1/3] Add support for external programs for handling native fetches","startedAt":"2009-07-25T17:51:29Z","lastAt":"2009-07-26T19:05:35Z","messageCount":13,"participants":["Daniel Barkalow","Johannes Schindelin","Linus Torvalds","Shawn O. Pearce","Steven Noonan","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"118738","messageId":"alpine.LNX.2.00.0907251330240.2147@iabervon.org","threadId":"20229","inReplyTo":null,"subject":"[PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-07-25T17:51:29Z","receivedAt":"2009-07-25T17:51:29Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"transport_get() can call transport_shim_init() to have list and\nfetch-ref operations handled by running a separate program as:\n\n git shim- []\n\nThis program then accepts, on its stdin, \"list\" and \"fetch \n\" commands; the former prints out a list of available refs and\neither their hashes or what they are symreefs to, while the latter\nfetches them into the local object database and prints a newline when done.\n\nSigned-off-by: Daniel Barkalow \n---\nThis is similar to, but different from, my git-vcs-* stuff; this one \nexpects the helper to be able to look up sha1s for refs efficiently, and \nto be able to fetch objects directly into the local object database \nwithout fast-import. On the otherhand, it should be sufficiently general \nto allow arbitrary methods for moving the data around.\n\n Documentation/git-shim.txt | 37 ++++++++++++\n Makefile | 1 +\n transport-shim.c | 140 ++++++++++++++++++++++++++++++++++++++++++++\n transport.h | 3 +\n 4 files changed, 181 insertions(+), 0 deletions(-)\n create mode 100644 Documentation/git-shim.txt\n create mode 100644 transport-shim.c\n\ndiff --git a/Documentation/git-shim.txt b/Documentation/git-shim.txt\nnew file mode 100644\nindex 0000000..26a611f\n--- /dev/null\n+++ b/Documentation/git-shim.txt\n@@ -0,0 +1,37 @@\n+git-shim(1)\n+============\n+\n+NAME\n+----\n+git-shim - Helper programs for interoperation with remote git\n+\n+SYNOPSIS\n+--------\n+'git shim-' \n+\n+DESCRIPTION\n+-----------\n+\n+These programs are normally not used directly by end users, but are\n+invoked by various git programs that interact with remote repositories\n+when the repository they would operate on will be accessed using\n+transport code not linked into the main git binary.\n+\n+COMMANDS\n+--------\n+\n+Commands are given by the caller on the helper's standard input, one per line.\n+\n+'list'::\n+\tOutputs the names of refs followed by the hex of their values\n+\tor \"@\" for symrefs, one per line. After the complete\n+\tlist, outputs a blank line.\n+\n+'fetch' ref::\n+\tFetches the given ref, writing the necessary objects to the\n+\tdatabase. Outputs a blank line when the fetch is complete.\n++\n+If a fatal error occurs, the program writes the error message to\n+stderr and exits. The caller should expect that a suitable error\n+message has been printed if the child closes the connection without\n+completing a valid response for the current command.\ndiff --git a/Makefile b/Makefile\nindex bde27ed..01efc73 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -549,6 +549,7 @@ LIB_OBJS += symlinks.o\n LIB_OBJS += tag.o\n LIB_OBJS += trace.o\n LIB_OBJS += transport.o\n+LIB_OBJS += transport-shim.o\n LIB_OBJS += tree-diff.o\n LIB_OBJS += tree.o\n LIB_OBJS += tree-walk.o\ndiff --git a/transport-shim.c b/transport-shim.c\nnew file mode 100644\nindex 0000000..3fc6abf\n--- /dev/null\n+++ b/transport-shim.c\n@@ -0,0 +1,140 @@\n+#include \"cache.h\"\n+#include \"transport.h\"\n+\n+#include \"run-command.h\"\n+#include \"commit.h\"\n+#include \"diff.h\"\n+#include \"revision.h\"\n+\n+struct shim_data\n+{\n+\tconst char *name;\n+\tstruct child_process *shim;\n+};\n+\n+static struct child_process *get_shim(struct transport *transport)\n+{\n+\tstruct shim_data *data = transport->data;\n+\tif (!data->shim) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstruct child_process *shim = xcalloc(1, sizeof(*shim));\n+\t\tshim->in = -1;\n+\t\tshim->out = -1;\n+\t\tshim->err = 0;\n+\t\tshim->argv = xcalloc(4, sizeof(*shim->argv));\n+\t\tstrbuf_addf(&buf, \"shim-%s\", data->name);\n+\t\tshim->argv[0] = buf.buf;\n+\t\tshim->argv[1] = transport->remote->name;\n+\t\tshim->argv[2] = transport->url;\n+\t\tshim->git_cmd = 1;\n+\t\tstart_command(shim);\n+\t\tdata->shim = shim;\n+\t}\n+\treturn data->shim;\n+}\n+\n+static int disconnect_shim(struct transport *transport)\n+{\n+\tstruct shim_data *data = transport->data;\n+\tif (data->shim) {\n+\t\twrite(data->shim->in, \"\\n\", 1);\n+\t\tclose(data->shim->in);\n+\t\tfinish_command(data->shim);\n+\t\tfree(data->shim);\n+\t\ttransport->data = NULL;\n+\t}\n+\treturn 0;\n+}\n+\n+static int fetch_refs_via_shim(struct transport *transport,\n+\t\t\t int nr_heads, const struct ref **to_fetch)\n+{\n+\tstruct child_process *shim;\n+\tconst struct ref *posn;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint i, count;\n+\tFILE *file;\n+\n+\tcount = 0;\n+\tfor (i = 0; i < nr_heads; i++) {\n+\t\tposn = to_fetch[i];\n+\t\tif (posn->status & REF_STATUS_UPTODATE)\n+\t\t\tcontinue;\n+\t\tcount++;\n+\t}\n+\n+\tif (count) {\n+\t\tshim = get_shim(transport);\n+\t\tfor (i = 0; i < nr_heads; i++) {\n+\t\t\tposn = to_fetch[i];\n+\t\t\tif (posn->status & REF_STATUS_UPTODATE)\n+\t\t\t\tcontinue;\n+\t\t\twrite(shim->in, \"fetch \", 6);\n+\t\t\twrite(shim->in, sha1_to_hex(posn->old_sha1), 40);\n+\t\t\twrite(shim->in, \" \", 1);\n+\t\t\twrite(shim->in, posn->name, strlen(posn->name));\n+\t\t\twrite(shim->in, \"\\n\", 1);\n+\t\t}\n+\t\tfile = fdopen(shim->out, \"r\");\n+\t\twhile (count) {\n+\t\t\tif (strbuf_getline(&buf, file, '\\n') == EOF)\n+\t\t\t\texit(128); // child died, message supplied already\n+\n+\t\t\tcount--;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static struct ref *get_refs_via_shim(struct transport *transport, int for_push)\n+{\n+\tstruct child_process *shim;\n+\tstruct ref *ret = NULL;\n+\tstruct ref **end = &ret;\n+\tstruct ref *posn;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tFILE *file;\n+\n+\tshim = get_shim(transport);\n+\twrite(shim->in, \"list\\n\", 5);\n+\n+\tfile = fdopen(shim->out, \"r\");\n+\twhile (1) {\n+\t\tchar *eon;\n+\t\tif (strbuf_getline(&buf, file, '\\n') == EOF)\n+\t\t\texit(128); // child died, message supplied already\n+\n+\t\tif (!*buf.buf)\n+\t\t\tbreak;\n+\n+\t\teon = strchr(buf.buf, ' ');\n+\t\tif (eon)\n+\t\t\t*eon = '\\0';\n+\t\t*end = alloc_ref(buf.buf);\n+\t\tif (eon) {\n+\t\t\tif (eon[1] == '@')\n+\t\t\t\t(*end)->symref = xstrdup(eon + 2);\n+\t\t\telse\n+\t\t\t\tget_sha1_hex(eon + 1, (*end)->old_sha1);\n+\t\t}\n+\t\tend = &((*end)->next);\n+\t\tstrbuf_reset(&buf);\n+\t}\n+\tstrbuf_release(&buf);\n+\n+\tfor (posn = ret; posn; posn = posn->next)\n+\t\tresolve_remote_symref(posn, ret);\n+\n+\treturn ret;\n+}\n+\n+void transport_shim_init(struct transport *transport, const char *name)\n+{\n+\tstruct shim_data *data = xmalloc(sizeof(*data));\n+\tdata->shim = NULL;\n+\tdata->name = name;\n+\ttransport->data = data;\n+\ttransport->get_refs_list = get_refs_via_shim;\n+\ttransport->fetch = fetch_refs_via_shim;\n+\ttransport->disconnect = disconnect_shim;\n+}\ndiff --git a/transport.h b/transport.h\nindex 51b5397..01f650d 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -77,4 +77,7 @@ void transport_unlock_pack(struct transport *transport);\n int transport_disconnect(struct transport *transport);\n char *transport_anonymize_url(const char *url);\n \n+/* Transport methods defined outside transport.c */\n+void transport_shim_init(struct transport *transport, const char *name);\n+\n #endif\n-- \n1.6.3.2.425.g3dedc\n"},{"id":"118742","messageId":"alpine.DEB.1.00.0907252017060.8306@pacific.mpi-cbg.de","threadId":"20229","inReplyTo":"alpine.LNX.2.00.0907251330240.2147@iabervon.org","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-25T18:19:41Z","receivedAt":"2009-07-25T18:19:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 25 Jul 2009, Daniel Barkalow wrote:\n\n> transport_get() can call transport_shim_init() to have list and\n> fetch-ref operations handled by running a separate program as:\n> \n> git shim- []\n> \n> This program then accepts, on its stdin, \"list\" and \"fetch \n> \" commands; the former prints out a list of available refs and\n> either their hashes or what they are symreefs to, while the latter\n> fetches them into the local object database and prints a newline when done.\n> \n> Signed-off-by: Daniel Barkalow \n> ---\n> This is similar to, but different from, my git-vcs-* stuff; this one \n> expects the helper to be able to look up sha1s for refs efficiently, and \n> to be able to fetch objects directly into the local object database \n> without fast-import. On the otherhand, it should be sufficiently general \n> to allow arbitrary methods for moving the data around.\n\nI wonder if this cannot be integrated into the git-vcs stuff; after all, \nthey do pretty much the same things, except that the vcs helpers are \npretty dumb, and the shim helpers are not as dumb.\n\nI could imagine that the name of the helper could reveal its capability to \nact as a shim helper: git-vcs-shim-http\n\nCiao,\nDscho\n"},{"id":"118743","messageId":"alpine.LFD.2.01.0907251107500.3960@localhost.localdomain","threadId":"20229","inReplyTo":"alpine.LNX.2.00.0907251330240.2147@iabervon.org","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-25T18:25:33Z","receivedAt":"2009-07-25T18:25:33Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 25 Jul 2009, Daniel Barkalow wrote:\n>\n> This is similar to, but different from, my git-vcs-* stuff; this one \n> expects the helper to be able to look up sha1s for refs efficiently, and \n> to be able to fetch objects directly into the local object database \n> without fast-import. On the otherhand, it should be sufficiently general \n> to allow arbitrary methods for moving the data around.\n\nWonderful. \n\nI tested the series, and now there is (not surprisingly, but I made sure \nto test anyway) no difference what-so-ever between NO_CURL and default, \nand 'ldd' looks nice.\n\nPlus it looks like that whole \"shim\" thing is a good idea in general, in \nthat it allows a much more flexible model for fetching/pushing.\n\nSo a very big Acked-by: from me for the series. I didn't test that http: \nworks with it, but I don't personally even care, so I'd ack it even \nwithout that ;)\n\nBtw, some real timing numbers for 'time make -j64 test':\n\n - before:\n\treal\t1m16.070s\n\tuser\t2m47.046s\n\tsys\t2m34.698s\n\n - after:\n\treal\t0m58.851s\n\tuser\t1m57.087s\n\tsys\t1m44.671s\n\nso that's actually a real-world example of the whole 'scripting \nperformance'. Not an insignificant speedup on my machine (with an \nobligatory \"nyaah, nyaah, I can do the git test-suite under a minute\" just \nto rub peoples noses in the fact that my desktop computer is disgustingly \nfast).\n\nThat's an almost 30% performance improvement, despite the fact that parts \nof the test suite didn't actually change (shell costs are the same, the \nsvn tests are quite perl-intensive etc).\n\n\t\t\tLinus\n"},{"id":"118744","messageId":"20090725184526.GB11191@spearce.org","threadId":"20229","inReplyTo":"alpine.LNX.2.00.0907251330240.2147@iabervon.org","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-07-25T18:45:26Z","receivedAt":"2009-07-25T18:45:26Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Daniel Barkalow wrote:\n> +Commands are given by the caller on the helper's standard input, one per line.\n> +\n> +'list'::\n> +\tOutputs the names of refs followed by the hex of their values\n> +\tor \"@\" for symrefs, one per line. After the complete\n> +\tlist, outputs a blank line.\n\nI hate to be a nit, but why this format and not the one that is\nproduced by `git ls-remote` and the native protocol? I know its\npretty arbitrary either way, but since we already have a history\nof using \"SHA1 ref\\n\" why not continue that tradition here?\n\n-- \nShawn.\n"},{"id":"118767","messageId":"alpine.LNX.2.00.0907251446460.2147@iabervon.org","threadId":"20229","inReplyTo":"20090725184526.GB11191@spearce.org","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-07-25T19:09:13Z","receivedAt":"2009-07-25T19:09:13Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sat, 25 Jul 2009, Shawn O. Pearce wrote:\n\n> Daniel Barkalow wrote:\n> > +Commands are given by the caller on the helper's standard input, one per line.\n> > +\n> > +'list'::\n> > +\tOutputs the names of refs followed by the hex of their values\n> > +\tor \"@\" for symrefs, one per line. After the complete\n> > +\tlist, outputs a blank line.\n> \n> I hate to be a nit, but why this format and not the one that is\n> produced by `git ls-remote` and the native protocol? I know its\n> pretty arbitrary either way, but since we already have a history\n> of using \"SHA1 ref\\n\" why not continue that tradition here?\n\nIn the VCS protocol, the SHA1 is generally not known when simply listing \nrefs, so I didn't have a useless initial field in that protocol; I just \ninherited having the line start with the name for this protocol.\n\nIn any case, I want to be able to support symrefs, so an extension to the \ntradition is required, but putting @ before the name instead of \nafter is easy enough.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"118768","messageId":"alpine.LFD.2.01.0907251151270.3960@localhost.localdomain","threadId":"20229","inReplyTo":"alpine.LFD.2.01.0907251107500.3960@localhost.localdomain","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-25T19:09:23Z","receivedAt":"2009-07-25T19:09:23Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 25 Jul 2009, Linus Torvalds wrote:\n> \n> Btw, some real timing numbers for 'time make -j64 test':\n> \n> - before:\n> \treal\t1m16.070s\n> \tuser\t2m47.046s\n> \tsys\t2m34.698s\n> \n> - after:\n> \treal\t0m58.851s\n> \tuser\t1m57.087s\n> \tsys\t1m44.671s\n> \n> so that's actually a real-world example of the whole 'scripting \n> performance'. Not an insignificant speedup on my machine (with an \n> obligatory \"nyaah, nyaah, I can do the git test-suite under a minute\" just \n> to rub peoples noses in the fact that my desktop computer is disgustingly \n> fast).\n> \n> That's an almost 30% performance improvement, despite the fact that parts \n> of the test suite didn't actually change (shell costs are the same, the \n> svn tests are quite perl-intensive etc).\n\nJust in case people wonder _why_, here is a profile from before and after. \nNote how big a deal the page faulting, unmapping (TLB flushes etc), and \nfork() is (copy_page_range()).\n\nAnd notice how the biggest user space cost - even after the change - is \ndo_lookup_x() in the dynamic loader. But before the change it was the very \ntop entry, and you had things like strcmp and _dl_relocate_object pretty \nhigh too. Not to mention that you got just a lot _more_ munmap's and page \nfaults:\n\n - before:\n\n 4.51% git /lib64/ld-2.10.1.so [.] do_lookup_x\n 3.17% git [kernel] [k] unmap_vmas\n 2.75% git [kernel] [k] page_fault\n 1.48% git [kernel] [k] copy_page_c\n 1.43% git /lib64/ld-2.10.1.so [.] strcmp\n 1.30% git [kernel] [k] _spin_lock\n 1.12% git /lib64/ld-2.10.1.so [.] _dl_relocate_object\n 0.99% git-svn [kernel] [k] copy_page_range\n 0.99% git [kernel] [k] kmem_cache_alloc\n 0.97% git [kernel] [k] get_page_from_freelist\n 0.92% git [kernel] [k] copy_page_range\n 0.88% git [kernel] [k] clear_page_c\n 0.80% git [kernel] [k] find_vma\n 0.79% git /lib64/ld-2.10.1.so [.] _dl_lookup_symbol_x\n 0.68% git [kernel] [k] handle_mm_fault\n 0.68% git /lib64/libc-2.10.1.so [.] _int_malloc\n 0.63% git /bin/bash 0x00000000046e96\n 0.57% git /lib64/libc-2.10.1.so [.] __GI__dl_addr\n 0.51% git [kernel] [k] release_pages\n\n - after:\n\n 3.02% git [kernel] [k] unmap_vmas\n 2.74% git [kernel] [k] page_fault\n 1.32% git [kernel] [k] copy_page_c\n 1.23% git [kernel] [k] _spin_lock\n 1.17% git-svn [kernel] [k] copy_page_range\n 1.06% git [kernel] [k] copy_page_range\n 0.99% git /lib64/ld-2.10.1.so [.] do_lookup_x\n 0.95% git /lib64/libc-2.10.1.so [.] _int_malloc\n 0.83% git [kernel] [k] get_page_from_freelist\n 0.83% git /lib64/libc-2.10.1.so [.] __GI__dl_addr\n 0.82% git [kernel] [k] clear_page_c\n 0.70% git [kernel] [k] kmem_cache_alloc\n 0.65% git [kernel] [k] handle_mm_fault\n 0.62% git /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so [.] Perl_yyparse\n 0.60% git-svn /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so [.] Perl_yyparse\n 0.59% git /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so [.] Perl_yylex\n 0.58% git [kernel] [k] release_pages\n 0.58% git [kernel] [k] page_remove_rmap\n 0.57% git /bin/bash 0x0000000004c2df\n 0.55% git-svn /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so [.] Perl_yylex\n 0.52% git [kernel] [k] find_vma\n 0.52% git [kernel] [k] strnlen_user\n 0.52% git-svn /lib64/libc-2.10.1.so [.] _int_malloc\n\nInteresting to see how after the change, perl is now looking like a fairly \nbig part.\n\nThe big picture (not per-function, but per-program split by code segment: \nkernel, executable, library) shows the same thing. git does have a high \nkernel component in general, but something like \"make test\" makes it even \nbigger, since most of the costs are really forking a _lot_ of git \nprograms:\n\n- before:\n\n 33.23% git [kernel] \n 11.93% git /lib64/ld-2.10.1.so \n 7.55% git-svn /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so\n 6.82% git /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so\n 4.83% git /lib64/libc-2.10.1.so \n 3.28% git-svn [kernel] \n 1.82% sh [kernel] \n 1.57% git /bin/bash \n 1.52% perl /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so\n 1.37% git-svn /lib64/libc-2.10.1.so \n 1.28% tput [kernel] \n 1.26% git-filter-bran [kernel] \n 0.98% rm [kernel] \n 0.97% sed [kernel] \n 0.82% git-rebase--int [kernel] \n 0.71% git-bisect [kernel] \n 0.64% git ./git \n 0.62% grep [kernel] \n 0.55% cat [kernel] \n\n - after:\n\n 30.30% git [kernel] \n 10.62% git-svn /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so\n 9.77% git /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so\n 6.31% git /lib64/libc-2.10.1.so \n 4.31% git-svn [kernel] \n 3.49% git /lib64/ld-2.10.1.so \n 2.17% git /bin/bash \n 2.10% perl /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so\n 1.93% git-svn /lib64/libc-2.10.1.so \n 1.90% sh [kernel] \n 1.40% git-filter-bran [kernel] \n 1.24% tput [kernel] \n 0.95% sed [kernel] \n 0.91% rm [kernel] \n 0.89% git ./git \n 0.84% git-rebase--int [kernel] \n 0.82% git-filter-bran /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so\n 0.75% sh /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi/CORE/libperl.so\n 0.59% sh /lib64/libc-2.10.1.so \n 0.57% grep [kernel] \n 0.57% git-bisect [kernel] \n 0.55% cat [kernel] \n\nNote how the biggest user-space component used to be the dynamic loader. \nNow it's down there way below the perl overhead.\n\nAnd notice how while the dynamic loader was \"just\" 11% of all overhead \n(and is still 3.5% after the fix), the reason performance has improved by \n30% is that the dynamic loader has a _huge_ kernel overhead due to the \nwhole mmap/munmap/mprotect/page-fault-to-COW/etc code. \n\n\t\tLinus\n"},{"id":"118783","messageId":"f488382f0907251408q383a7789l5342c7ec41b402a@mail.gmail.com","threadId":"20229","inReplyTo":"alpine.LFD.2.01.0907251107500.3960@localhost.localdomain","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Steven Noonan","fromEmail":"steven@uplinklabs.net","sentAt":"2009-07-25T21:08:49Z","receivedAt":"2009-07-25T21:08:49Z","isPatch":true,"sender":{"key":"steven@uplinklabs.net","avatar":"https://gravatar.com/avatar/b0cd397a10638433f76e084531aa0af3bef85f8fdb59b1ebe2ddaf168cd100e9?d=mp&s=160"},"body":"On Sat, Jul 25, 2009 at 12:25 PM, Linus\nTorvalds wrote:\n>\n>\n> On Sat, 25 Jul 2009, Daniel Barkalow wrote:\n>>\n>> This is similar to, but different from, my git-vcs-* stuff; this one\n>> expects the helper to be able to look up sha1s for refs efficiently, and\n>> to be able to fetch objects directly into the local object database\n>> without fast-import. On the otherhand, it should be sufficiently general\n>> to allow arbitrary methods for moving the data around.\n>\n> Wonderful.\n>\n> I tested the series, and now there is (not surprisingly, but I made sure\n> to test anyway) no difference what-so-ever between NO_CURL and default,\n> and 'ldd' looks nice.\n>\n> Plus it looks like that whole \"shim\" thing is a good idea in general, in\n> that it allows a much more flexible model for fetching/pushing.\n>\n> So a very big Acked-by: from me for the series. I didn't test that http:\n> works with it, but I don't personally even care, so I'd ack it even\n> without that ;)\n>\n> Btw, some real timing numbers for 'time make -j64 test':\n>\n>  - before:\n>        real    1m16.070s\n>        user    2m47.046s\n>        sys     2m34.698s\n>\n>  - after:\n>        real    0m58.851s\n>        user    1m57.087s\n>        sys     1m44.671s\n>\n> so that's actually a real-world example of the whole 'scripting\n> performance'. Not an insignificant speedup on my machine (with an\n> obligatory \"nyaah, nyaah, I can do the git test-suite under a minute\" just\n> to rub peoples noses in the fact that my desktop computer is disgustingly\n> fast).\n\nBut more importantly, how fast can your machine compile the kernel?\n(without ccache or any \"cheats\" like that, of course)\n\n>\n> That's an almost 30% performance improvement, despite the fact that parts\n> of the test suite didn't actually change (shell costs are the same, the\n> svn tests are quite perl-intensive etc).\n>\n>                        Linus\n"},{"id":"118785","messageId":"alpine.LFD.2.01.0907251414140.3960@localhost.localdomain","threadId":"20229","inReplyTo":"f488382f0907251408q383a7789l5342c7ec41b402a@mail.gmail.com","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-25T21:14:33Z","receivedAt":"2009-07-25T21:14:33Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 25 Jul 2009, Steven Noonan wrote:\n> \n> But more importantly, how fast can your machine compile the kernel?\n> (without ccache or any \"cheats\" like that, of course)\n\nWithout ccache? One minute. With ccache? 16 seconds.\n\n\t\tLinus\n"},{"id":"118813","messageId":"7vmy6srngw.fsf@alter.siamese.dyndns.org","threadId":"20229","inReplyTo":"alpine.LFD.2.01.0907251107500.3960@localhost.localdomain","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-26T06:02:23Z","receivedAt":"2009-07-26T06:02:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> ... Not an insignificant speedup on my machine (with an \n> obligatory \"nyaah, nyaah, I can do the git test-suite under a minute\" just \n> to rub peoples noses in the fact that my desktop computer is disgustingly \n> fast).\n\nWith git-svn tests?\n"},{"id":"118815","messageId":"7viqhgrm1h.fsf@alter.siamese.dyndns.org","threadId":"20229","inReplyTo":"alpine.LFD.2.01.0907251107500.3960@localhost.localdomain","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-26T06:33:14Z","receivedAt":"2009-07-26T06:33:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> So a very big Acked-by: from me for the series. I didn't test that http: \n> works with it, but I don't personally even care, so I'd ack it even \n> without that ;)\n\nHeh, \"git ls-remote http://...\" segfaulting for me is not a very good\nsign, but I like the concept.\n"},{"id":"118834","messageId":"alpine.LFD.2.01.0907260901100.3960@localhost.localdomain","threadId":"20229","inReplyTo":"7viqhgrm1h.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-26T16:06:58Z","receivedAt":"2009-07-26T16:06:58Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 25 Jul 2009, Junio C Hamano wrote:\n\n> Linus Torvalds writes:\n> \n> > So a very big Acked-by: from me for the series. I didn't test that http: \n> > works with it, but I don't personally even care, so I'd ack it even \n> > without that ;)\n> \n> Heh, \"git ls-remote http://...\" segfaulting for me is not a very good\n> sign, but I like the concept.\n\nHmm. Just tested. Works for me (git and kernel repos on kernel.org). Which \nrepo did you test against?\n\n\t\tLinus\n"},{"id":"118836","messageId":"alpine.LFD.2.01.0907260925020.3960@localhost.localdomain","threadId":"20229","inReplyTo":"7vmy6srngw.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-26T16:27:12Z","receivedAt":"2009-07-26T16:27:12Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 25 Jul 2009, Junio C Hamano wrote:\n> Linus Torvalds writes:\n> \n> > ... Not an insignificant speedup on my machine (with an \n> > obligatory \"nyaah, nyaah, I can do the git test-suite under a minute\" just \n> > to rub peoples noses in the fact that my desktop computer is disgustingly \n> > fast).\n> \n> With git-svn tests?\n\nYup. There's a few tests I'm skipping, but not many:\n\n\t* skipping test, network testing disabled by default\n\t* skipping test, network testing disabled by default\n\t* skipping svnserve test. (set $SVNSERVE_PORT to enable)\n\t* skipping svn-info test (SVN version: 1.6.1 not supported)\n\t* skipping svnserve test. (set $SVNSERVE_PORT to enable)\n\t* skipping git-cvsserver tests, Perl SQLite interface unavailable\n\t* skipping git-cvsserver tests, Perl SQLite interface unavailable\n\t* skipping cvsimport tests, cvsps not found\n\n(those first two ones are the GIT_TEST_HTTPD ones).\n\n\t\t\tLinus\n"},{"id":"118844","messageId":"7vljmbp8n4.fsf@alter.siamese.dyndns.org","threadId":"20229","inReplyTo":"alpine.LFD.2.01.0907260901100.3960@localhost.localdomain","subject":"Re: [PATCH 1/3] Add support for external programs for handling native fetches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-26T19:05:35Z","receivedAt":"2009-07-26T19:05:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Sat, 25 Jul 2009, Junio C Hamano wrote:\n>\n>> Linus Torvalds writes:\n>> \n>> > So a very big Acked-by: from me for the series. I didn't test that http: \n>> > works with it, but I don't personally even care, so I'd ack it even \n>> > without that ;)\n>> \n>> Heh, \"git ls-remote http://...\" segfaulting for me is not a very good\n>> sign, but I like the concept.\n>\n> Hmm. Just tested. Works for me (git and kernel repos on kernel.org).\n\nInteresting.\n\nThe version from 'pu' seems to work just fine (and I just double checked\nthat ldd says I do not link the main git binary with cURL library with\nthat version). I saw the breakage when running the version from the topic\nbranch itself, but I do not seem to be able to reproduce it anymore.\n"}]}