{"thread":{"id":"29906","subject":"--progress for git submodule update?","startedAt":"2012-03-10T07:17:28Z","lastAt":"2012-03-15T22:34:31Z","messageCount":7,"participants":["Chris Kees","Jens Lehmann","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"186616","messageId":"CAOVFbFiBOH2MyC3HUtf=hA_PssSRENW7uwpBVxh0TwnO7h90XA@mail.gmail.com","threadId":"29906","inReplyTo":null,"subject":"--progress for git submodule update?","fromName":"Chris Kees","fromEmail":"cekees@gmail.com","sentAt":"2012-03-10T07:17:28Z","receivedAt":"2012-03-10T07:17:28Z","isPatch":false,"sender":{"key":"cekees@gmail.com","avatar":"https://gravatar.com/avatar/e641423366bea929f162ec7953487481490d7e716b9489061f0744585fd122f3?d=mp&s=160"},"body":"Hi,\n\nWould it be reasonable to have a --progress option for 'git submodule\nupdate'? I'm using buildbot with a git repository containing large\nsubmodules, and buildbot times out on the submodule update\noccasionally because there is no output for long periods of time.\nAdjusting buildbot's timeout factor will do for me in the short run.\n\nThanks,\nChris\n"},{"id":"186667","messageId":"4F5D0FFD.1020509@web.de","threadId":"29906","inReplyTo":"CAOVFbFiBOH2MyC3HUtf=hA_PssSRENW7uwpBVxh0TwnO7h90XA@mail.gmail.com","subject":"Re: --progress for git submodule update?","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-03-11T20:50:05Z","receivedAt":"2012-03-11T20:50:05Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 10.03.2012 08:17, schrieb Chris Kees:\n> Would it be reasonable to have a --progress option for 'git submodule\n> update'? I'm using buildbot with a git repository containing large\n> submodules, and buildbot times out on the submodule update\n> occasionally because there is no output for long periods of time.\n> Adjusting buildbot's timeout factor will do for me in the short run.\n\nAs cloning a submodule talks a lot about its progress am I right\nsuspecting it is the checkout part that is taking so long for you?\nThe submodule script always uses the -q option for git checkout\n(which also gets rid of the unwanted \"detached HEAD\" messages). So\nAFAICS before a --progress option could be added to the submodule\nscript, git checkout would have to learn an option to show progress\nbut not the detached HEAD message (or to just suppress that advice).\n\nWhat times are we talking about here?\n"},{"id":"186805","messageId":"CAOVFbFhMfpFa5=a0Z50H7nHdQFHn9Y4ApUnQJq6GCOFP+AKy5A@mail.gmail.com","threadId":"29906","inReplyTo":"4F5D0FFD.1020509@web.de","subject":"Re: --progress for git submodule update?","fromName":"Chris Kees","fromEmail":"cekees@gmail.com","sentAt":"2012-03-13T02:17:29Z","receivedAt":"2012-03-13T02:17:29Z","isPatch":false,"sender":{"key":"cekees@gmail.com","avatar":"https://gravatar.com/avatar/e641423366bea929f162ec7953487481490d7e716b9489061f0744585fd122f3?d=mp&s=160"},"body":"It's 'git submodule update --recursive' that is taking so long\nsilently.  The problem is mainly on the first time. There are about 10\nsubmodules that together have taken more than 30 minutes. It's not\nreally just the amount of data, I think there are also network traffic\nissues that slow things down on some systems.\n\nChris\n\nOn Sun, Mar 11, 2012 at 3:50 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:\n> Am 10.03.2012 08:17, schrieb Chris Kees:\n>> Would it be reasonable to have a --progress option for 'git submodule\n>> update'? I'm using buildbot with a git repository containing large\n>> submodules, and buildbot times out on the submodule update\n>> occasionally because there is no output for long periods of time.\n>> Adjusting buildbot's timeout factor will do for me in the short run.\n>\n> As cloning a submodule talks a lot about its progress am I right\n> suspecting it is the checkout part that is taking so long for you?\n> The submodule script always uses the -q option for git checkout\n> (which also gets rid of the unwanted \"detached HEAD\" messages). So\n> AFAICS before a --progress option could be added to the submodule\n> script, git checkout would have to learn an option to show progress\n> but not the detached HEAD message (or to just suppress that advice).\n>\n> What times are we talking about here?\n"},{"id":"186986","messageId":"4F60F4A6.1070507@web.de","threadId":"29906","inReplyTo":"CAOVFbFhMfpFa5=a0Z50H7nHdQFHn9Y4ApUnQJq6GCOFP+AKy5A@mail.gmail.com","subject":"Re: --progress for git submodule update?","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-03-14T19:42:30Z","receivedAt":"2012-03-14T19:42:30Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 13.03.2012 03:17, schrieb Chris Kees:\n> It's 'git submodule update --recursive' that is taking so long\n> silently.  The problem is mainly on the first time. There are about 10\n> submodules that together have taken more than 30 minutes. It's not\n> really just the amount of data, I think there are also network traffic\n> issues that slow things down on some systems.\n\nI suppose with \"first time\" you mean right after \"git submodule init\",\nwhen the submodules have to be cloned initially? Thinking about that\nagain, you mentioned a buildbot doing all that. When the submodules\nare updated from a script, no progress output is shown at all and only\nthe line \"Cloning into 'xxx'...\" will appear for each submodule, which\nexplains why you don't see output for quite some time.\n\nSo I suspect increasing the timeout on your buildbot is the way to go,\nas progress output is intended for humans.\n"},{"id":"186996","messageId":"7vd38eoo0u.fsf@alter.siamese.dyndns.org","threadId":"29906","inReplyTo":"4F60F4A6.1070507@web.de","subject":"Re: --progress for git submodule update?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-03-14T21:45:05Z","receivedAt":"2012-03-14T21:45:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> Am 13.03.2012 03:17, schrieb Chris Kees:\n>> It's 'git submodule update --recursive' that is taking so long\n>> silently.  The problem is mainly on the first time. There are about 10\n>> submodules that together have taken more than 30 minutes. It's not\n>> really just the amount of data, I think there are also network traffic\n>> issues that slow things down on some systems.\n>\n> I suppose with \"first time\" you mean right after \"git submodule init\",\n> when the submodules have to be cloned initially? Thinking about that\n> again, you mentioned a buildbot doing all that. When the submodules\n> are updated from a script, no progress output is shown at all and only\n> the line \"Cloning into 'xxx'...\" will appear for each submodule, which\n> explains why you don't see output for quite some time.\n>\n> So I suspect increasing the timeout on your buildbot is the way to go,\n> as progress output is intended for humans.\n\nConsidering that more often than not, people who run stuff from cronjob\nrequest us to be quiet when nothing wrong happens, I think that is a\nsane thing to suggest.\n\nI do not mind \"submodule $subcmd --progress\" to pass \"--progress\" down to\nwhatever underlying git commands it uses, or squelch \"-q\" that it uses\nwhen running them by default, though.\n\nWe may even want to make \"git submodule init -q\" to squelch the \"Cloning\ninto...\" message, but that is a separate topic.\n"},{"id":"187005","messageId":"CAOVFbFj4nF1GmqdDSmb4z_JXXTxsNktRGaaReF8=msbOwaRt=g@mail.gmail.com","threadId":"29906","inReplyTo":"7vd38eoo0u.fsf@alter.siamese.dyndns.org","subject":"Re: --progress for git submodule update?","fromName":"Chris Kees","fromEmail":"cekees@gmail.com","sentAt":"2012-03-15T03:43:44Z","receivedAt":"2012-03-15T03:43:44Z","isPatch":false,"sender":{"key":"cekees@gmail.com","avatar":"https://gravatar.com/avatar/e641423366bea929f162ec7953487481490d7e716b9489061f0744585fd122f3?d=mp&s=160"},"body":"Yes, I meant when they have to be cloned initially. It was actually\nthe builtbot developers who asked me to post a request because the\nlack of output is used to trigger the timeout and the output is\nconveniently stored and posted to the web so you can see what is\nhappening on a given platform. Here is the snippet from the log of a\nmachine failing because the python submodule of the project is taking\ntoo long--clearly a slow network issue since python generally\nshouldn't take more than 20 minutes to clone:\n\nCloning into externalPackages/clawpack/clawutil...\nSubmodule path 'externalPackages/clawpack/clawutil': checked out\n'48f6be675b6fe40be62c39a48e1c948903e5c43d'\nCloning into externalPackages/clawpack/pyclaw...\nSubmodule path 'externalPackages/clawpack/pyclaw': checked out\n'921b1796b9bb0d11db1ced7996f95b9b24ce0b23'\nCloning into externalPackages/clawpack/riemann...\nSubmodule path 'externalPackages/clawpack/riemann': checked out\n'a16d64d750fc414ce131455734ae7d68d6c0f46b'\nCloning into externalPackages/clawpack/visclaw...\nSubmodule path 'externalPackages/clawpack/visclaw': checked out\n'2b6ab68330c4498eade4b2e7dabc5de9b593ecac'\nCloning into externalPackages/cython...\nSubmodule path 'externalPackages/cython': checked out\n'77fd4991fe270b1097318c8ccdca9740df68d9e6'\nCloning into externalPackages/daetk...\nSubmodule path 'externalPackages/daetk': checked out\n'781f732fbc206cfdfa6da170fbc161a80d4133f8'\nCloning into externalPackages/hdf5...\nSubmodule path 'externalPackages/hdf5': checked out\n'e853fa711d4ecc71ba7025c64db5bbe41558d635'\nCloning into externalPackages/ipython...\nSubmodule path 'externalPackages/ipython': checked out\n'698e9abebe0d6e0f261e7050c48680c4608a0985'\nCloning into externalPackages/matplotlib...\nSubmodule path 'externalPackages/matplotlib': checked out\n'd6ec786c28b9ddbcb0704a1a6e1fb7233eb3212b'\nCloning into externalPackages/mpi4py...\nSubmodule path 'externalPackages/mpi4py': checked out\n'eac3d3602e8272ab8eff46988fde396f0a255d90'\nCloning into externalPackages/nose...\nSubmodule path 'externalPackages/nose': checked out\n'ccb3878af99138301c612d5bb92e7a27d3e39d22'\nCloning into externalPackages/numexpr...\nSubmodule path 'externalPackages/numexpr': checked out\n'ed3ca13b58548d68fbc8b9a380990aac1e4897c9'\nCloning into externalPackages/numpy...\nSubmodule path 'externalPackages/numpy': checked out\n'3f685a1a990f7b6e5149c80b52436fb4207e49f5'\nCloning into externalPackages/petsc...\nSubmodule path 'externalPackages/petsc': checked out\n'4d5fdb033efc8bf2cdfe228f993ec9305f98854b'\nCloning into externalPackages/petsc-BuildSystem...\nSubmodule path 'externalPackages/petsc-BuildSystem': checked out\n'e9f7e22d13f5d12da05c8f5d34d574e4bfe053ff'\nCloning into externalPackages/petsc4py...\nSubmodule path 'externalPackages/petsc4py': checked out\n'adae544de17fd8cdcf2d51290ddaecfbe2107180'\nCloning into externalPackages/pytables...\nSubmodule path 'externalPackages/pytables': checked out\n'467ba035ba72acd7df28c648b1a52ecadebe4935'\nCloning into externalPackages/python...\n\ncommand timed out: 1200 seconds without output, attempting to kill\nprocess killed by signal 9\nprogram finished with exit code -1\nelapsedTime=2641.623387\n\n\nOn Wed, Mar 14, 2012 at 4:45 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n>\n>> Am 13.03.2012 03:17, schrieb Chris Kees:\n>>> It's 'git submodule update --recursive' that is taking so long\n>>> silently.  The problem is mainly on the first time. There are about 10\n>>> submodules that together have taken more than 30 minutes. It's not\n>>> really just the amount of data, I think there are also network traffic\n>>> issues that slow things down on some systems.\n>>\n>> I suppose with \"first time\" you mean right after \"git submodule init\",\n>> when the submodules have to be cloned initially? Thinking about that\n>> again, you mentioned a buildbot doing all that. When the submodules\n>> are updated from a script, no progress output is shown at all and only\n>> the line \"Cloning into 'xxx'...\" will appear for each submodule, which\n>> explains why you don't see output for quite some time.\n>>\n>> So I suspect increasing the timeout on your buildbot is the way to go,\n>> as progress output is intended for humans.\n>\n> Considering that more often than not, people who run stuff from cronjob\n> request us to be quiet when nothing wrong happens, I think that is a\n> sane thing to suggest.\n>\n> I do not mind \"submodule $subcmd --progress\" to pass \"--progress\" down to\n> whatever underlying git commands it uses, or squelch \"-q\" that it uses\n> when running them by default, though.\n>\n> We may even want to make \"git submodule init -q\" to squelch the \"Cloning\n> into...\" message, but that is a separate topic.\n>\n"},{"id":"187067","messageId":"4F626E77.1050309@web.de","threadId":"29906","inReplyTo":"CAOVFbFj4nF1GmqdDSmb4z_JXXTxsNktRGaaReF8=msbOwaRt=g@mail.gmail.com","subject":"Re: --progress for git submodule update?","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-03-15T22:34:31Z","receivedAt":"2012-03-15T22:34:31Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 15.03.2012 04:43, schrieb Chris Kees:\n> Yes, I meant when they have to be cloned initially. It was actually\n> the builtbot developers who asked me to post a request because the\n> lack of output is used to trigger the timeout and the output is\n> conveniently stored and posted to the web so you can see what is\n> happening on a given platform. Here is the snippet from the log of a\n> machine failing because the python submodule of the project is taking\n> too long--clearly a slow network issue since python generally\n> shouldn't take more than 20 minutes to clone:\n[snip]\n> '467ba035ba72acd7df28c648b1a52ecadebe4935'\n> Cloning into externalPackages/python...\n> \n> command timed out: 1200 seconds without output, attempting to kill\n> process killed by signal 9\n> program finished with exit code -1\n> elapsedTime=2641.623387\n\nOk. Another option you have is to avoid the long clone times. Clone\nthe external repo locally (but outside your build environment) and\ninstead of cloning the whole repo each time over the net just do a\nfetch into the local copy (to get the latest updates over the slow\nnetwork). Then clone from there into your build environment (after\nthe buildbot edited the submodule url entry in .git/config to point\nto the local copy). That should work substantially faster.\n"}]}