{"thread":{"id":"47326","subject":"[PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","startedAt":"2017-11-27T04:27:53Z","lastAt":"2017-11-28T11:36:38Z","messageCount":12,"participants":["Takuto Ikuta","Junio C Hamano","Jeff King","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"333572","messageId":"0102015ffbbb2905-570eadd1-6b5c-46af-a3a9-bddfbd01c242-000000@eu-west-1.amazonses.com","threadId":"47326","inReplyTo":null,"subject":"[PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Takuto Ikuta","fromEmail":"tikuta@google.com","sentAt":"2017-11-27T04:27:47Z","receivedAt":"2017-11-27T04:27:53Z","isPatch":true,"sender":{"key":"tikuta@google.com","avatar":null},"body":"Do not call prepare_packed_git for every refs.\nIn fetch-pack, git list ups entries in .git/objects/pack\ndirectory for each refs.\nSuch behavior makes git fetch-pack slow for the repository having the\nlarge number of refs, especially for windows.\n\nFor chromium repository, having more than 300000 remote refs, git fetch\ntakes more than 3 minutes if fetch-pack runs on my windows workstation.\nThis patch improves the time to around 17 seconds in the same machine.\n\nSigned-off-by: Takuto Ikuta \n---\n fetch-pack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 008b25d3db087..0184584e80599 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -716,7 +716,7 @@ static int everything_local(struct fetch_pack_args *args,\n \tfor (ref = *refs; ref; ref = ref->next) {\n \t\tstruct object *o;\n \n-\t\tif (!has_object_file(&ref->old_oid))\n+\t\tif (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK))\n \t\t\tcontinue;\n \n \t\to = parse_object(&ref->old_oid);\n\n--\nhttps://github.com/git/git/pull/437\n"},{"id":"333575","messageId":"xmqqo9noe3u0.fsf@gitster.mtv.corp.google.com","threadId":"47326","inReplyTo":"0102015ffbbb2905-570eadd1-6b5c-46af-a3a9-bddfbd01c242-000000@eu-west-1.amazonses.com","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-27T04:35:35Z","receivedAt":"2017-11-27T04:35:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Takuto Ikuta writes:\n\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 008b25d3db087..0184584e80599 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -716,7 +716,7 @@ static int everything_local(struct fetch_pack_args *args,\n> \tfor (ref = *refs; ref; ref = ref->next) {\n> \t\tstruct object *o;\n> \n> -\t\tif (!has_object_file(&ref->old_oid))\n> +\t\tif (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK))\n> \t\t\tcontinue;\n> \n\nIt appears that great minds think alike?\n\ncf. https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/\n\nThe 5-patch series that contains the same change as this one is\ncooking and will hopefully be in the released version before the end\nof the year.\n\n\n"},{"id":"333576","messageId":"20171127043740.GA5994@sigill","threadId":"47326","inReplyTo":"xmqqo9noe3u0.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-11-27T04:37:40Z","receivedAt":"2017-11-27T04:37:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 27, 2017 at 01:35:35PM +0900, Junio C Hamano wrote:\n\n> Takuto Ikuta writes:\n> \n> > diff --git a/fetch-pack.c b/fetch-pack.c\n> > index 008b25d3db087..0184584e80599 100644\n> > --- a/fetch-pack.c\n> > +++ b/fetch-pack.c\n> > @@ -716,7 +716,7 @@ static int everything_local(struct fetch_pack_args *args,\n> > \tfor (ref = *refs; ref; ref = ref->next) {\n> > \t\tstruct object *o;\n> > \n> > -\t\tif (!has_object_file(&ref->old_oid))\n> > +\t\tif (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK))\n> > \t\t\tcontinue;\n> > \n> \n> It appears that great minds think alike?\n> \n> cf. https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/\n\nIt's funny that we'd get two patches so close together. AFAIK the\nslowness here has been with us for years, and I just happened to\ninvestigate it recently.\n\n> The 5-patch series that contains the same change as this one is\n> cooking and will hopefully be in the released version before the end\n> of the year.\n\nI'd be curious if the 5th patch there provides an additional speedup for\nTakuto's case.\n\n-Peff\n"},{"id":"333578","messageId":"xmqqk1yce301.fsf@gitster.mtv.corp.google.com","threadId":"47326","inReplyTo":"20171127043740.GA5994@sigill","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-27T04:53:34Z","receivedAt":"2017-11-27T04:55:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> cf. https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/\n>\n> It's funny that we'd get two patches so close together. AFAIK the\n> slowness here has been with us for years, and I just happened to\n> investigate it recently.\n>\n>> The 5-patch series that contains the same change as this one is\n>> cooking and will hopefully be in the released version before the end\n>> of the year.\n>\n> I'd be curious if the 5th patch there provides an additional speedup for\n> Takuto's case.\n\nIndeed, it is a very good point.\n\nIIUC, the 5th one is about fetching tons of refs that you have never\nseen, right? If a repository that has trouble with everything-local\nis suffering because it right now has 300k remote-tracking branches,\nI'd imagine that these remote-tracking branches are being added at a\nconsiderable rate, so I'd not be surprised if these \"new\" refs\nbenefits from that patch. And it would be nice to know how much a\nreal life scenario actually does improve.\n\nThanks.\n"},{"id":"333587","messageId":"CALNjmMo3KH4QXY00ViVaiHdy=pSZ5j0BWRueO1T=sC_r0=3c3g@mail.gmail.com","threadId":"47326","inReplyTo":"xmqqk1yce301.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Takuto Ikuta","fromEmail":"tikuta@google.com","sentAt":"2017-11-27T05:01:36Z","receivedAt":"2017-11-27T05:02:23Z","isPatch":true,"sender":{"key":"tikuta@google.com","avatar":null},"body":"2017-11-27 13:53 GMT+09:00 Junio C Hamano :\n> Jeff King writes:\n>\n>>> cf. https://public-inbox.org/git/20171120202920.7ppcwmzkxifywtoj@sigill.intra.peff.net/\n>>\n>> It's funny that we'd get two patches so close together. AFAIK the\n>> slowness here has been with us for years, and I just happened to\n>> investigate it recently.\n\nYes, thank you for let me know.\nPlease ignore my patch, sorry.\n\n>>\n>>> The 5-patch series that contains the same change as this one is\n>>> cooking and will hopefully be in the released version before the end\n>>> of the year.\n>>\n>> I'd be curious if the 5th patch there provides an additional speedup for\n>> Takuto's case.\n>\n> Indeed, it is a very good point.\n>\n> IIUC, the 5th one is about fetching tons of refs that you have never\n> seen, right? If a repository that has trouble with everything-local\n> is suffering because it right now has 300k remote-tracking branches,\n> I'd imagine that these remote-tracking branches are being added at a\n> considerable rate, so I'd not be surprised if these \"new\" refs\n> benefits from that patch. And it would be nice to know how much a\n> real life scenario actually does improve.\n>\n> Thanks.\n\nIn chromium repository, your 5th patch does not improve performance,\ntook more than 5 minutes to run fetch on windows.\n4th patch is very important for the repository in daily fetch.\nI hope your 4th patch will be merged.\n\nThanks.\n-- \nTakuto Ikuta\nSoftware Engineer in Tokyo\nChrome Infrastructure (goma team)\n"},{"id":"333588","messageId":"20171127050420.GB6858@sigill","threadId":"47326","inReplyTo":"xmqqk1yce301.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-11-27T05:04:20Z","receivedAt":"2017-11-27T05:04:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 27, 2017 at 01:53:34PM +0900, Junio C Hamano wrote:\n\n> > I'd be curious if the 5th patch there provides an additional speedup for\n> > Takuto's case.\n> \n> Indeed, it is a very good point.\n> \n> IIUC, the 5th one is about fetching tons of refs that you have never\n> seen, right? If a repository that has trouble with everything-local\n> is suffering because it right now has 300k remote-tracking branches,\n> I'd imagine that these remote-tracking branches are being added at a\n> considerable rate, so I'd not be surprised if these \"new\" refs\n> benefits from that patch. And it would be nice to know how much a\n> real life scenario actually does improve.\n\nRight, I think it will only kick in for new refs (and maybe deleted\nones, but I didn't check).\n\nI actually did have a real life scenario that was improved, but it's not\nvery typical. As I've mentioned before, we keep a big shared-object\nrepository for forks of a single repo, so a fairly common operation is\n\n git -C network.git fetch ../$fork.git +refs/*:refs/remotes/$fork/*\n\nThe first time that's run on a fork, everything looks like a new ref.\n\nSo \"real life\", but thankfully not life for most git users. :)\n\n-Peff\n"},{"id":"333639","messageId":"alpine.DEB.2.21.1.1711272222100.6482@virtualbox","threadId":"47326","inReplyTo":"CALNjmMo3KH4QXY00ViVaiHdy=pSZ5j0BWRueO1T=sC_r0=3c3g@mail.gmail.com","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-11-27T21:24:46Z","receivedAt":"2017-11-27T21:25:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 27 Nov 2017, Takuto Ikuta wrote:\n\n> 2017-11-27 13:53 GMT+09:00 Junio C Hamano :\n> > Jeff King writes:\n> >\n> >>> The 5-patch series that contains the same change as this one is\n> >>> cooking and will hopefully be in the released version before the end\n> >>> of the year.\n> >>\n> >> I'd be curious if the 5th patch there provides an additional speedup\n> >> for Takuto's case.\n> >\n> > Indeed, it is a very good point.\n> >\n> > IIUC, the 5th one is about fetching tons of refs that you have never\n> > seen, right? If a repository that has trouble with everything-local\n> > is suffering because it right now has 300k remote-tracking branches,\n> > I'd imagine that these remote-tracking branches are being added at a\n> > considerable rate, so I'd not be surprised if these \"new\" refs\n> > benefits from that patch. And it would be nice to know how much a\n> > real life scenario actually does improve.\n> >\n> > Thanks.\n> \n> In chromium repository, your 5th patch does not improve performance,\n> took more than 5 minutes to run fetch on windows.\n> 4th patch is very important for the repository in daily fetch.\n> I hope your 4th patch will be merged.\n\nIf you want it in Git for Windows early (where performance is\ntraditionally worse than on Linux because Git is really optimized for\nLinux), I would not mind a Pull Request at\nhttps://github.com/git-for-windows/git.\n\nMy current plan is to release a new Git for Windows version on Wednesday,\nto allow for a new cURL version to be bundled.\n\nCiao,\nJohannes\n"},{"id":"333658","messageId":"xmqqbmjn9t48.fsf@gitster.mtv.corp.google.com","threadId":"47326","inReplyTo":"alpine.DEB.2.21.1.1711272222100.6482@virtualbox","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-27T23:52:55Z","receivedAt":"2017-11-27T23:53:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> My current plan is to release a new Git for Windows version on Wednesday,\n> to allow for a new cURL version to be bundled.\n\nIs this an updated 2.15.0 or are you planning to package 2.15.1? I\nam asking because you earlier asked me to hold 2.15.1 while you were\naway last week and there are accumulated changes on 'maint'.\n\nAs I won't be online later this week, unless I cut 2.15.1 today or\nearly tomorrow, it will have to be done early next week, and I'd\nprefer not to do 2.15.1 _immediately_ after a platform announces\nan updated release of 2.15.0 for obvious reasons.\n\n\n"},{"id":"333660","messageId":"alpine.DEB.2.21.1.1711280132480.6482@virtualbox","threadId":"47326","inReplyTo":"xmqqbmjn9t48.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-11-28T00:33:27Z","receivedAt":"2017-11-28T00:33:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 28 Nov 2017, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> > My current plan is to release a new Git for Windows version on Wednesday,\n> > to allow for a new cURL version to be bundled.\n> \n> Is this an updated 2.15.0 or are you planning to package 2.15.1?\n\nIf there is a 2.15.1 to pick up for me, I'll take it. Otherwise it'll be\nGit for Windows v2.15.0(2).\n\nCiao,\nDscho\n"},{"id":"333666","messageId":"CALNjmMpc2F4OSrQPBtRGwu6RM=V81PXJxOrP2w9dLdtL1=F+kw@mail.gmail.com","threadId":"47326","inReplyTo":"alpine.DEB.2.21.1.1711280132480.6482@virtualbox","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Takuto Ikuta","fromEmail":"tikuta@google.com","sentAt":"2017-11-28T01:58:40Z","receivedAt":"2017-11-28T01:59:28Z","isPatch":true,"sender":{"key":"tikuta@google.com","avatar":null},"body":"Hi Johannes,\n\nAs long as this PR is included in next Git for Windows release, I\nwon't suffer from slow git fetch.\nhttps://github.com/git-for-windows/git/pull/1372\n\nBut I sent you 2 PRs to follow right way.\nhttps://github.com/git-for-windows/git/pull/1379\nhttps://github.com/git-for-windows/git/pull/1380\nFeel free to merge these PRs.\n\nThanks.\n\n2017-11-28 9:33 GMT+09:00 Johannes Schindelin :\n> Hi Junio,\n>\n> On Tue, 28 Nov 2017, Junio C Hamano wrote:\n>\n>> Johannes Schindelin writes:\n>>\n>> > My current plan is to release a new Git for Windows version on Wednesday,\n>> > to allow for a new cURL version to be bundled.\n>>\n>> Is this an updated 2.15.0 or are you planning to package 2.15.1?\n>\n> If there is a 2.15.1 to pick up for me, I'll take it. Otherwise it'll be\n> Git for Windows v2.15.0(2).\n>\n> Ciao,\n> Dscho\n\n\n\n-- \nTakuto Ikuta\nSoftware Engineer in Tokyo\nChrome Infrastructure (goma team)\n"},{"id":"333683","messageId":"alpine.DEB.2.21.1.1711281226490.6482@virtualbox","threadId":"47326","inReplyTo":"CALNjmMpc2F4OSrQPBtRGwu6RM=V81PXJxOrP2w9dLdtL1=F+kw@mail.gmail.com","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-11-28T11:27:23Z","receivedAt":"2017-11-28T11:27:52Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 28 Nov 2017, Takuto Ikuta wrote:\n\n> As long as this PR is included in next Git for Windows release, I\n> won't suffer from slow git fetch.\n> https://github.com/git-for-windows/git/pull/1372\n> \n> But I sent you 2 PRs to follow right way.\n> https://github.com/git-for-windows/git/pull/1379\n> https://github.com/git-for-windows/git/pull/1380\n> Feel free to merge these PRs.\n\nI amended them slightly, and merged them.\n\nThank you,\nDscho\n"},{"id":"333684","messageId":"CALNjmMo2jVNXGGuP4t1LTgSLpSKRcYz=N_wiVwZotnnSvPFiTQ@mail.gmail.com","threadId":"47326","inReplyTo":"alpine.DEB.2.21.1.1711281226490.6482@virtualbox","subject":"Re: [PATCH] Use OBJECT_INFO_QUICK to speedup git fetch-pack","fromName":"Takuto Ikuta","fromEmail":"tikuta@google.com","sentAt":"2017-11-28T11:35:50Z","receivedAt":"2017-11-28T11:36:38Z","isPatch":true,"sender":{"key":"tikuta@google.com","avatar":null},"body":"Hi,\n\nThank you for merging!\n\nTakuto\n\n2017-11-28 20:27 GMT+09:00 Johannes Schindelin :\n> Hi,\n>\n> On Tue, 28 Nov 2017, Takuto Ikuta wrote:\n>\n>> As long as this PR is included in next Git for Windows release, I\n>> won't suffer from slow git fetch.\n>> https://github.com/git-for-windows/git/pull/1372\n>>\n>> But I sent you 2 PRs to follow right way.\n>> https://github.com/git-for-windows/git/pull/1379\n>> https://github.com/git-for-windows/git/pull/1380\n>> Feel free to merge these PRs.\n>\n> I amended them slightly, and merged them.\n>\n> Thank you,\n> Dscho\n\n\n\n-- \nTakuto Ikuta\nSoftware Engineer in Tokyo\nChrome Infrastructure (goma team)\n"}]}