# [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack 12 messages from 2017-11-27 to 2017-11-28. Participants: Takuto Ikuta, Junio C Hamano, Jeff King, Johannes Schindelin. Thread: https://gitlist.dev/t/47326 ## Takuto Ikuta, 2017-11-27 04:27 Subject: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: <0102015ffbbb2905-570eadd1-6b5c-46af-a3a9-bddfbd01c242-000000@eu-west-1.amazonses.com> URL: https://gitlist.dev/e/0102015ffbbb2905-570eadd1-6b5c-46af-a3a9-bddfbd01c242-000000%40eu-west-1.amazonses.com ``` Do not call prepare_packed_git for every refs. In fetch-pack, git list ups entries in .git/objects/pack directory for each refs. Such behavior makes git fetch-pack slow for the repository having the large number of refs, especially for windows. For chromium repository, having more than 300000 remote refs, git fetch takes more than 3 minutes if fetch-pack runs on my windows workstation. This patch improves the time to around 17 seconds in the same machine. Signed-off-by: Takuto Ikuta --- fetch-pack.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fetch-pack.c b/fetch-pack.c index 008b25d3db087..0184584e80599 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -716,7 +716,7 @@ static int everything_local(struct fetch_pack_args *args, for (ref = *refs; ref; ref = ref->next) { struct object *o; - if (!has_object_file(&ref->old_oid)) + if (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK)) continue; o = parse_object(&ref->old_oid); -- https://github.com/git/git/pull/437 ``` ## Junio C Hamano, 2017-11-27 04:35 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/xmqqo9noe3u0.fsf%40gitster.mtv.corp.google.com In-Reply-To: <0102015ffbbb2905-570eadd1-6b5c-46af-a3a9-bddfbd01c242-000000@eu-west-1.amazonses.com> ``` Takuto Ikuta writes: > diff --git a/fetch-pack.c b/fetch-pack.c > index 008b25d3db087..0184584e80599 100644 > --- a/fetch-pack.c > +++ b/fetch-pack.c > @@ -716,7 +716,7 @@ static int everything_local(struct fetch_pack_args *args, > for (ref = *refs; ref; ref = ref->next) { > struct object *o; > > - if (!has_object_file(&ref->old_oid)) > + if (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK)) > continue; > It appears that great minds think alike? cf. https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/ The 5-patch series that contains the same change as this one is cooking and will hopefully be in the released version before the end of the year. ``` ## Jeff King, 2017-11-27 04:37 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: <20171127043740.GA5994@sigill> URL: https://gitlist.dev/e/20171127043740.GA5994%40sigill In-Reply-To: ``` On Mon, Nov 27, 2017 at 01:35:35PM +0900, Junio C Hamano wrote: > Takuto Ikuta writes: > > > diff --git a/fetch-pack.c b/fetch-pack.c > > index 008b25d3db087..0184584e80599 100644 > > --- a/fetch-pack.c > > +++ b/fetch-pack.c > > @@ -716,7 +716,7 @@ static int everything_local(struct fetch_pack_args *args, > > for (ref = *refs; ref; ref = ref->next) { > > struct object *o; > > > > - if (!has_object_file(&ref->old_oid)) > > + if (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK)) > > continue; > > > > It appears that great minds think alike? > > cf. https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/ It's funny that we'd get two patches so close together. AFAIK the slowness here has been with us for years, and I just happened to investigate it recently. > The 5-patch series that contains the same change as this one is > cooking and will hopefully be in the released version before the end > of the year. I'd be curious if the 5th patch there provides an additional speedup for Takuto's case. -Peff ``` ## Junio C Hamano, 2017-11-27 04:53 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/xmqqk1yce301.fsf%40gitster.mtv.corp.google.com In-Reply-To: <20171127043740.GA5994@sigill> ``` Jeff King writes: >> cf. https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/ > > It's funny that we'd get two patches so close together. AFAIK the > slowness here has been with us for years, and I just happened to > investigate it recently. > >> The 5-patch series that contains the same change as this one is >> cooking and will hopefully be in the released version before the end >> of the year. > > I'd be curious if the 5th patch there provides an additional speedup for > Takuto's case. Indeed, it is a very good point. IIUC, the 5th one is about fetching tons of refs that you have never seen, right? If a repository that has trouble with everything-local is suffering because it right now has 300k remote-tracking branches, I'd imagine that these remote-tracking branches are being added at a considerable rate, so I'd not be surprised if these "new" refs benefits from that patch. And it would be nice to know how much a real life scenario actually does improve. Thanks. ``` ## Takuto Ikuta, 2017-11-27 05:01 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/CALNjmMo3KH4QXY00ViVaiHdy%3DpSZ5j0BWRueO1T%3DsC_r0%3D3c3g%40mail.gmail.com In-Reply-To: ``` 2017-11-27 13:53 GMT+09:00 Junio C Hamano : > Jeff King writes: > >>> cf. https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/ >> >> It's funny that we'd get two patches so close together. AFAIK the >> slowness here has been with us for years, and I just happened to >> investigate it recently. Yes, thank you for let me know. Please ignore my patch, sorry. >> >>> The 5-patch series that contains the same change as this one is >>> cooking and will hopefully be in the released version before the end >>> of the year. >> >> I'd be curious if the 5th patch there provides an additional speedup for >> Takuto's case. > > Indeed, it is a very good point. > > IIUC, the 5th one is about fetching tons of refs that you have never > seen, right? If a repository that has trouble with everything-local > is suffering because it right now has 300k remote-tracking branches, > I'd imagine that these remote-tracking branches are being added at a > considerable rate, so I'd not be surprised if these "new" refs > benefits from that patch. And it would be nice to know how much a > real life scenario actually does improve. > > Thanks. In chromium repository, your 5th patch does not improve performance, took more than 5 minutes to run fetch on windows. 4th patch is very important for the repository in daily fetch. I hope your 4th patch will be merged. Thanks. -- Takuto Ikuta Software Engineer in Tokyo Chrome Infrastructure (goma team) ``` ## Jeff King, 2017-11-27 05:04 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: <20171127050420.GB6858@sigill> URL: https://gitlist.dev/e/20171127050420.GB6858%40sigill In-Reply-To: ``` On Mon, Nov 27, 2017 at 01:53:34PM +0900, Junio C Hamano wrote: > > I'd be curious if the 5th patch there provides an additional speedup for > > Takuto's case. > > Indeed, it is a very good point. > > IIUC, the 5th one is about fetching tons of refs that you have never > seen, right? If a repository that has trouble with everything-local > is suffering because it right now has 300k remote-tracking branches, > I'd imagine that these remote-tracking branches are being added at a > considerable rate, so I'd not be surprised if these "new" refs > benefits from that patch. And it would be nice to know how much a > real life scenario actually does improve. Right, I think it will only kick in for new refs (and maybe deleted ones, but I didn't check). I actually did have a real life scenario that was improved, but it's not very typical. As I've mentioned before, we keep a big shared-object repository for forks of a single repo, so a fairly common operation is git -C network.git fetch ../$fork.git +refs/*:refs/remotes/$fork/* The first time that's run on a fork, everything looks like a new ref. So "real life", but thankfully not life for most git users. :) -Peff ``` ## Johannes Schindelin, 2017-11-27 21:24 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.21.1.1711272222100.6482%40virtualbox In-Reply-To: ``` Hi, On Mon, 27 Nov 2017, Takuto Ikuta wrote: > 2017-11-27 13:53 GMT+09:00 Junio C Hamano : > > Jeff King writes: > > > >>> The 5-patch series that contains the same change as this one is > >>> cooking and will hopefully be in the released version before the end > >>> of the year. > >> > >> I'd be curious if the 5th patch there provides an additional speedup > >> for Takuto's case. > > > > Indeed, it is a very good point. > > > > IIUC, the 5th one is about fetching tons of refs that you have never > > seen, right? If a repository that has trouble with everything-local > > is suffering because it right now has 300k remote-tracking branches, > > I'd imagine that these remote-tracking branches are being added at a > > considerable rate, so I'd not be surprised if these "new" refs > > benefits from that patch. And it would be nice to know how much a > > real life scenario actually does improve. > > > > Thanks. > > In chromium repository, your 5th patch does not improve performance, > took more than 5 minutes to run fetch on windows. > 4th patch is very important for the repository in daily fetch. > I hope your 4th patch will be merged. If you want it in Git for Windows early (where performance is traditionally worse than on Linux because Git is really optimized for Linux), I would not mind a Pull Request at https://github.com/git-for-windows/git. My current plan is to release a new Git for Windows version on Wednesday, to allow for a new cURL version to be bundled. Ciao, Johannes ``` ## Junio C Hamano, 2017-11-27 23:52 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/xmqqbmjn9t48.fsf%40gitster.mtv.corp.google.com In-Reply-To: ``` Johannes Schindelin writes: > My current plan is to release a new Git for Windows version on Wednesday, > to allow for a new cURL version to be bundled. Is this an updated 2.15.0 or are you planning to package 2.15.1? I am asking because you earlier asked me to hold 2.15.1 while you were away last week and there are accumulated changes on 'maint'. As I won't be online later this week, unless I cut 2.15.1 today or early tomorrow, it will have to be done early next week, and I'd prefer not to do 2.15.1 _immediately_ after a platform announces an updated release of 2.15.0 for obvious reasons. ``` ## Johannes Schindelin, 2017-11-28 00:33 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.21.1.1711280132480.6482%40virtualbox In-Reply-To: ``` Hi Junio, On Tue, 28 Nov 2017, Junio C Hamano wrote: > Johannes Schindelin writes: > > > My current plan is to release a new Git for Windows version on Wednesday, > > to allow for a new cURL version to be bundled. > > Is this an updated 2.15.0 or are you planning to package 2.15.1? If there is a 2.15.1 to pick up for me, I'll take it. Otherwise it'll be Git for Windows v2.15.0(2). Ciao, Dscho ``` ## Takuto Ikuta, 2017-11-28 01:58 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/CALNjmMpc2F4OSrQPBtRGwu6RM%3DV81PXJxOrP2w9dLdtL1%3DF%2Bkw%40mail.gmail.com In-Reply-To: ``` Hi Johannes, As long as this PR is included in next Git for Windows release, I won't suffer from slow git fetch. https://github.com/git-for-windows/git/pull/1372 But I sent you 2 PRs to follow right way. https://github.com/git-for-windows/git/pull/1379 https://github.com/git-for-windows/git/pull/1380 Feel free to merge these PRs. Thanks. 2017-11-28 9:33 GMT+09:00 Johannes Schindelin : > Hi Junio, > > On Tue, 28 Nov 2017, Junio C Hamano wrote: > >> Johannes Schindelin writes: >> >> > My current plan is to release a new Git for Windows version on Wednesday, >> > to allow for a new cURL version to be bundled. >> >> Is this an updated 2.15.0 or are you planning to package 2.15.1? > > If there is a 2.15.1 to pick up for me, I'll take it. Otherwise it'll be > Git for Windows v2.15.0(2). > > Ciao, > Dscho -- Takuto Ikuta Software Engineer in Tokyo Chrome Infrastructure (goma team) ``` ## Johannes Schindelin, 2017-11-28 11:27 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.21.1.1711281226490.6482%40virtualbox In-Reply-To: ``` Hi, On Tue, 28 Nov 2017, Takuto Ikuta wrote: > As long as this PR is included in next Git for Windows release, I > won't suffer from slow git fetch. > https://github.com/git-for-windows/git/pull/1372 > > But I sent you 2 PRs to follow right way. > https://github.com/git-for-windows/git/pull/1379 > https://github.com/git-for-windows/git/pull/1380 > Feel free to merge these PRs. I amended them slightly, and merged them. Thank you, Dscho ``` ## Takuto Ikuta, 2017-11-28 11:35 Subject: Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack Message-ID: URL: https://gitlist.dev/e/CALNjmMo2jVNXGGuP4t1LTgSLpSKRcYz%3DN_wiVwZotnnSvPFiTQ%40mail.gmail.com In-Reply-To: ``` Hi, Thank you for merging! Takuto 2017-11-28 20:27 GMT+09:00 Johannes Schindelin : > Hi, > > On Tue, 28 Nov 2017, Takuto Ikuta wrote: > >> As long as this PR is included in next Git for Windows release, I >> won't suffer from slow git fetch. >> https://github.com/git-for-windows/git/pull/1372 >> >> But I sent you 2 PRs to follow right way. >> https://github.com/git-for-windows/git/pull/1379 >> https://github.com/git-for-windows/git/pull/1380 >> Feel free to merge these PRs. > > I amended them slightly, and merged them. > > Thank you, > Dscho -- Takuto Ikuta Software Engineer in Tokyo Chrome Infrastructure (goma team) ```