# speeding up git pull from a busy gerrit instance over a slow link? 4 messages from 2017-06-30 to 2017-07-03. Participants: Noel Grandin, Jonathan Tan, Stefan Beller. Thread: https://gitlist.dev/t/46295 ## Noel Grandin, 2017-06-30 12:28 Subject: speeding up git pull from a busy gerrit instance over a slow link? Message-ID: <19ee7852-efcc-66d0-24ad-3462a4d5eaf6@peralex.com> URL: https://gitlist.dev/e/19ee7852-efcc-66d0-24ad-3462a4d5eaf6%40peralex.com ``` Hi I'm running git version 2.13.1 on Ubuntu 16.04 (x64) I'm connecting over a very slow (international link) to a very busy gerrit server (gerrit.libreoffice.org) using ssh. Ping types are on the order of 200ms. Using GIT_TRACE_PACKET=true, what I am seeing is that the bulk of the time is spent retrieving packets having to do with things which I have no interest in, i.e. the refs/changes/* stuff (see below). Is there any way to deliberately exclude these from the pull process? My git config looks like: remote.origin.url=ssh://logerrit/core remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master branch.sdrshapeid.remote=origin branch.sdrshapeid.merge=refs/heads/master Thanks, Noel Grandin --------------------- snippet of packet trace ------------------- 14:20:45.705091 pkt-line.c:80 packet: fetch< c5b026801c729ab37e2af6a610f31ca2e28b51fe refs/changes/99/29099/2 14:20:45.705093 pkt-line.c:80 packet: fetch< 931e2c40aeb4cf4591ae9fcfea1b352b966f0a32 refs/changes/99/29199/1 14:20:45.705096 pkt-line.c:80 packet: fetch< 373b44a2fcbe78e8a3ff14cd410826af151a6adf refs/changes/99/29199/2 14:20:45.705099 pkt-line.c:80 packet: fetch< 90e0db5edf83595121818627f590cc6b776f9f45 refs/changes/99/29299/1 14:20:45.705102 pkt-line.c:80 packet: fetch< 0f23647484e969fcd3c1b7e7ae72249026b20a80 refs/changes/99/29299/2 14:20:45.705104 pkt-line.c:80 packet: fetch< c6b6e54eeb0878a63179d2827d9c1623fe6f54e9 refs/changes/99/29299/3 14:20:45.705107 pkt-line.c:80 packet: fetch< 8e466c431e6d61911b2b15895a4b63df12422057 refs/changes/99/29299/4 14:20:45.705110 pkt-line.c:80 packet: fetch< 168bb711950f175c0fc4a889b9c8d41cf35bedec refs/changes/99/29399/1 14:20:45.705114 pkt-line.c:80 packet: fetch< 6c76ade9516ada09222a18af69eb381b2e859b24 refs/changes/99/29399/2 14:20:45.705116 pkt-line.c:80 packet: fetch< ff88d1ba10b2bfc7af7cbb518d3a3c1122d6dffc refs/changes/99/29599/1 14:20:45.705119 pkt-line.c:80 packet: fetch< 0d93900801224b797741e9a1abf305109fa35665 refs/changes/99/29599/2 14:20:45.705122 pkt-line.c:80 packet: fetch< 46f34d1b27f2056bb145cbdb526d4ed48b426f97 refs/changes/99/29699/1 14:20:45.705124 pkt-line.c:80 packet: fetch< 90d0a3d08c07d22b9a2b85ee08c72a0c047ed2d5 refs/changes/99/29699/2 14:20:45.705127 pkt-line.c:80 packet: fetch< 8821675d8e5a8b9c6f7fa5139a973394bfa67294 refs/changes/99/29699/3 Disclaimer: http://www.peralex.com/disclaimer.html ``` ## Jonathan Tan, 2017-06-30 21:59 Subject: Re: speeding up git pull from a busy gerrit instance over a slow link? Message-ID: <20170630145912.1047890d@twelve2.svl.corp.google.com> URL: https://gitlist.dev/e/20170630145912.1047890d%40twelve2.svl.corp.google.com In-Reply-To: <19ee7852-efcc-66d0-24ad-3462a4d5eaf6@peralex.com> ``` On Fri, 30 Jun 2017 14:28:15 +0200 Noel Grandin wrote: > --------------------- > snippet of packet trace > ------------------- > > 14:20:45.705091 pkt-line.c:80 packet: fetch< c5b026801c729ab37e2af6a610f31ca2e28b51fe > refs/changes/99/29099/2 > 14:20:45.705093 pkt-line.c:80 packet: fetch< 931e2c40aeb4cf4591ae9fcfea1b352b966f0a32 > refs/changes/99/29199/1 Out of curiosity, what is the timestamp difference between the first and last GIT_TRACE_PACKET log message containing "refs/changes"? This is an issue for the Android repository too (which also uses Gerrit). I have some work in progress to avoid the extra refs from being sent [1], but haven't been working on it recently. [1] https://public-inbox.org/git/cover.1491851452.git.jonathantanmy@google.com/ ``` ## Stefan Beller, 2017-06-30 22:00 Subject: Re: speeding up git pull from a busy gerrit instance over a slow link? Message-ID: URL: https://gitlist.dev/e/CAGZ79kaYYSeyeHpnMuK%3D%2BpV%2BJA-97a%3Dif-AyELvSA4Y5yGhmmg%40mail.gmail.com In-Reply-To: <19ee7852-efcc-66d0-24ad-3462a4d5eaf6@peralex.com> ``` On Fri, Jun 30, 2017 at 5:28 AM, Noel Grandin wrote: > Hi > > I'm running git version 2.13.1 on Ubuntu 16.04 (x64) > > I'm connecting over a very slow (international link) to a very busy gerrit > server (gerrit.libreoffice.org) using ssh. > Ping types are on the order of 200ms. > > Using GIT_TRACE_PACKET=true, what I am seeing is that the bulk of the time > is spent retrieving packets having to do with things which I have no > interest in, i.e. the refs/changes/* stuff (see below). > > Is there any way to deliberately exclude these from the pull process? > > My git config looks like: > remote.origin.url=ssh://logerrit/core > remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* > branch.master.remote=origin > branch.master.merge=refs/heads/master > branch.sdrshapeid.remote=origin > branch.sdrshapeid.merge=refs/heads/master > > Thanks, Noel Grandin From here I figured, you're talking about gerrit.libreoffice.org/core My initial clone is taking rather long, it was stuck in the "Counting objects" phase for quite some time. Maybe ask the Gerrit administrator if they have pack.useBitmaps enabled? (This is a tangent, and most likely related to the initial clone only, you are asking for everyday fetches.) ``` ## Noel Grandin, 2017-07-03 07:01 Subject: Re: speeding up git pull from a busy gerrit instance over a slow link? Message-ID: URL: https://gitlist.dev/e/e68b650f-7e16-6935-9867-f54e391b9eaf%40peralex.com In-Reply-To: <20170630145912.1047890d@twelve2.svl.corp.google.com> ``` On 2017/06/30 11:59 PM, Jonathan Tan wrote: > > Out of curiosity, what is the timestamp difference between the first and > last GIT_TRACE_PACKET log message containing "refs/changes"? > Cut down log looks like: 08:37:17.734527 pkt-line.c:80 packet: fetch< baeb5486c43d39b063371f91cfaae8efc2c8700b refs/changes/00/1000/1 ... 3 minutes .. 08:40:10.983005 pkt-line.c:80 packet: fetch< b0dd80238089dfcc0b3bf7bed99564adce649397 refs/changes/99/6799/2 08:40:10.983054 pkt-line.c:80 packet: fetch< 5c5dd57b54e3107b3069dc8f82df74df63d13555 refs/heads/distro/collabora/cp-4.0 .. 08:40:11.134355 pkt-line.c:80 packet: fetch< f8c345808ceafe87be6207e5ae304a9fa6c4cd16 refs/heads/master 08:40:11.134404 pkt-line.c:80 packet: fetch< 3cacadc5c9e6a0780a4c75cd3614eddc8db8e933 refs/remotes/origin/HEAD ... 08:40:11.173013 pkt-line.c:80 packet: fetch< 6c1b76c38f0fc469a2cbf41ffacde4d76df11ead refs/remotes/origin/origin/feature/submodules 08:40:11.173025 pkt-line.c:80 packet: fetch< a7d2fe68fd1db70f16a827f828dc541954f9d0f2 refs/tags/COOL_1.0 ... 08:40:11.814181 pkt-line.c:80 packet: fetch< 9125509a7c6e65336330b8ac42a293aa77b18ee3 refs/tags/windows_build_successful_2011_11_08^{} ... 08:40:12.724809 pkt-line.c:80 packet: sideband< 0000 Disclaimer: http://www.peralex.com/disclaimer.html ```