{"thread":{"id":"46295","subject":"speeding up git pull from a busy gerrit instance over a slow link?","startedAt":"2017-06-30T12:38:04Z","lastAt":"2017-07-03T07:01:49Z","messageCount":4,"participants":["Noel Grandin","Jonathan Tan","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"323566","messageId":"19ee7852-efcc-66d0-24ad-3462a4d5eaf6@peralex.com","threadId":"46295","inReplyTo":null,"subject":"speeding up git pull from a busy gerrit instance over a slow link?","fromName":"Noel Grandin","fromEmail":"noel@peralex.com","sentAt":"2017-06-30T12:28:15Z","receivedAt":"2017-06-30T12:38:04Z","isPatch":false,"sender":{"key":"noel@peralex.com","avatar":null},"body":"Hi\n\nI'm running git version 2.13.1 on Ubuntu 16.04 (x64)\n\nI'm connecting over a very slow (international link) to a very busy gerrit server (gerrit.libreoffice.org) using ssh.\nPing types are on the order of 200ms.\n\nUsing GIT_TRACE_PACKET=true, what I am seeing is that the bulk of the time is spent retrieving packets having to do with \nthings which I have no interest in, i.e. the refs/changes/* stuff (see below).\n\nIs there any way to deliberately exclude these from the pull process?\n\nMy git config looks like:\n    remote.origin.url=ssh://logerrit/core\n    remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*\n    branch.master.remote=origin\n    branch.master.merge=refs/heads/master\n    branch.sdrshapeid.remote=origin\n    branch.sdrshapeid.merge=refs/heads/master\n\nThanks, Noel Grandin\n\n---------------------\nsnippet of packet trace\n-------------------\n\n14:20:45.705091 pkt-line.c:80           packet:        fetch< c5b026801c729ab37e2af6a610f31ca2e28b51fe \nrefs/changes/99/29099/2\n14:20:45.705093 pkt-line.c:80           packet:        fetch< 931e2c40aeb4cf4591ae9fcfea1b352b966f0a32 \nrefs/changes/99/29199/1\n14:20:45.705096 pkt-line.c:80           packet:        fetch< 373b44a2fcbe78e8a3ff14cd410826af151a6adf \nrefs/changes/99/29199/2\n14:20:45.705099 pkt-line.c:80           packet:        fetch< 90e0db5edf83595121818627f590cc6b776f9f45 \nrefs/changes/99/29299/1\n14:20:45.705102 pkt-line.c:80           packet:        fetch< 0f23647484e969fcd3c1b7e7ae72249026b20a80 \nrefs/changes/99/29299/2\n14:20:45.705104 pkt-line.c:80           packet:        fetch< c6b6e54eeb0878a63179d2827d9c1623fe6f54e9 \nrefs/changes/99/29299/3\n14:20:45.705107 pkt-line.c:80           packet:        fetch< 8e466c431e6d61911b2b15895a4b63df12422057 \nrefs/changes/99/29299/4\n14:20:45.705110 pkt-line.c:80           packet:        fetch< 168bb711950f175c0fc4a889b9c8d41cf35bedec \nrefs/changes/99/29399/1\n14:20:45.705114 pkt-line.c:80           packet:        fetch< 6c76ade9516ada09222a18af69eb381b2e859b24 \nrefs/changes/99/29399/2\n14:20:45.705116 pkt-line.c:80           packet:        fetch< ff88d1ba10b2bfc7af7cbb518d3a3c1122d6dffc \nrefs/changes/99/29599/1\n14:20:45.705119 pkt-line.c:80           packet:        fetch< 0d93900801224b797741e9a1abf305109fa35665 \nrefs/changes/99/29599/2\n14:20:45.705122 pkt-line.c:80           packet:        fetch< 46f34d1b27f2056bb145cbdb526d4ed48b426f97 \nrefs/changes/99/29699/1\n14:20:45.705124 pkt-line.c:80           packet:        fetch< 90d0a3d08c07d22b9a2b85ee08c72a0c047ed2d5 \nrefs/changes/99/29699/2\n14:20:45.705127 pkt-line.c:80           packet:        fetch< 8821675d8e5a8b9c6f7fa5139a973394bfa67294 \nrefs/changes/99/29699/3\n\nDisclaimer: http://www.peralex.com/disclaimer.html\n\n\n"},{"id":"323649","messageId":"20170630145912.1047890d@twelve2.svl.corp.google.com","threadId":"46295","inReplyTo":"19ee7852-efcc-66d0-24ad-3462a4d5eaf6@peralex.com","subject":"Re: speeding up git pull from a busy gerrit instance over a slow link?","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-30T21:59:12Z","receivedAt":"2017-06-30T21:59:22Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 30 Jun 2017 14:28:15 +0200\nNoel Grandin <noel@peralex.com> wrote:\n\n> ---------------------\n> snippet of packet trace\n> -------------------\n> \n> 14:20:45.705091 pkt-line.c:80           packet:        fetch< c5b026801c729ab37e2af6a610f31ca2e28b51fe \n> refs/changes/99/29099/2\n> 14:20:45.705093 pkt-line.c:80           packet:        fetch< 931e2c40aeb4cf4591ae9fcfea1b352b966f0a32 \n> refs/changes/99/29199/1\n\nOut of curiosity, what is the timestamp difference between the first and\nlast GIT_TRACE_PACKET log message containing \"refs/changes\"?\n\nThis is an issue for the Android repository too (which also uses\nGerrit). I have some work in progress to avoid the extra refs from being\nsent [1], but haven't been working on it recently.\n\n[1] https://public-inbox.org/git/cover.1491851452.git.jonathantanmy@google.com/\n"},{"id":"323650","messageId":"CAGZ79kaYYSeyeHpnMuK=+pV+JA-97a=if-AyELvSA4Y5yGhmmg@mail.gmail.com","threadId":"46295","inReplyTo":"19ee7852-efcc-66d0-24ad-3462a4d5eaf6@peralex.com","subject":"Re: speeding up git pull from a busy gerrit instance over a slow link?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-30T22:00:25Z","receivedAt":"2017-06-30T22:00:30Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Jun 30, 2017 at 5:28 AM, Noel Grandin <noel@peralex.com> wrote:\n> Hi\n>\n> I'm running git version 2.13.1 on Ubuntu 16.04 (x64)\n>\n> I'm connecting over a very slow (international link) to a very busy gerrit\n> server (gerrit.libreoffice.org) using ssh.\n> Ping types are on the order of 200ms.\n>\n> Using GIT_TRACE_PACKET=true, what I am seeing is that the bulk of the time\n> is spent retrieving packets having to do with things which I have no\n> interest in, i.e. the refs/changes/* stuff (see below).\n>\n> Is there any way to deliberately exclude these from the pull process?\n>\n> My git config looks like:\n>    remote.origin.url=ssh://logerrit/core\n>    remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*\n>    branch.master.remote=origin\n>    branch.master.merge=refs/heads/master\n>    branch.sdrshapeid.remote=origin\n>    branch.sdrshapeid.merge=refs/heads/master\n>\n> Thanks, Noel Grandin\n\nFrom here I figured, you're talking about gerrit.libreoffice.org/core\nMy initial clone is taking rather long, it was stuck in the\n\"Counting objects\" phase for quite some time. Maybe ask the\nGerrit administrator if they have pack.useBitmaps enabled?\n\n(This is a tangent, and most likely related to the initial clone only,\nyou are asking for everyday fetches.)\n"},{"id":"323746","messageId":"e68b650f-7e16-6935-9867-f54e391b9eaf@peralex.com","threadId":"46295","inReplyTo":"20170630145912.1047890d@twelve2.svl.corp.google.com","subject":"Re: speeding up git pull from a busy gerrit instance over a slow link?","fromName":"Noel Grandin","fromEmail":"noel@peralex.com","sentAt":"2017-07-03T07:01:14Z","receivedAt":"2017-07-03T07:01:49Z","isPatch":false,"sender":{"key":"noel@peralex.com","avatar":null},"body":"On 2017/06/30 11:59 PM, Jonathan Tan wrote:\n>\n> Out of curiosity, what is the timestamp difference between the first and\n> last GIT_TRACE_PACKET log message containing \"refs/changes\"?\n>\n\nCut down log looks like:\n\n08:37:17.734527 pkt-line.c:80           packet:        fetch< baeb5486c43d39b063371f91cfaae8efc2c8700b \nrefs/changes/00/1000/1\n... 3 minutes ..\n08:40:10.983005 pkt-line.c:80           packet:        fetch< b0dd80238089dfcc0b3bf7bed99564adce649397 \nrefs/changes/99/6799/2\n08:40:10.983054 pkt-line.c:80           packet:        fetch< 5c5dd57b54e3107b3069dc8f82df74df63d13555 \nrefs/heads/distro/collabora/cp-4.0\n..\n08:40:11.134355 pkt-line.c:80           packet:        fetch< f8c345808ceafe87be6207e5ae304a9fa6c4cd16 refs/heads/master\n08:40:11.134404 pkt-line.c:80           packet:        fetch< 3cacadc5c9e6a0780a4c75cd3614eddc8db8e933 \nrefs/remotes/origin/HEAD\n...\n08:40:11.173013 pkt-line.c:80           packet:        fetch< 6c1b76c38f0fc469a2cbf41ffacde4d76df11ead \nrefs/remotes/origin/origin/feature/submodules\n08:40:11.173025 pkt-line.c:80           packet:        fetch< a7d2fe68fd1db70f16a827f828dc541954f9d0f2 refs/tags/COOL_1.0\n...\n08:40:11.814181 pkt-line.c:80           packet:        fetch< 9125509a7c6e65336330b8ac42a293aa77b18ee3 \nrefs/tags/windows_build_successful_2011_11_08^{}\n...\n08:40:12.724809 pkt-line.c:80           packet:     sideband< 0000\n\n\nDisclaimer: http://www.peralex.com/disclaimer.html\n\n\n"}]}