{"thread":{"id":"60098","subject":"Fetching too many tags?","startedAt":"2023-08-10T06:16:21Z","lastAt":"2023-08-12T01:04:51Z","messageCount":5,"participants":["Ronan Pigott","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"480427","messageId":"274ec1a2152b0fd53b35c1591f5177e0b0713430@rjp.ie","threadId":"60098","inReplyTo":null,"subject":"Fetching too many tags?","fromName":"Ronan Pigott","fromEmail":"ronan@rjp.ie","sentAt":"2023-08-10T06:08:34Z","receivedAt":"2023-08-10T06:16:21Z","isPatch":false,"sender":{"key":"ronan@rjp.ie","avatar":"https://avatars.githubusercontent.com/u/12577529?v=4"},"body":"Hey git,\n\nI am interested in git performance today and can't figure out what's going on\nhere. I was wondering why my git-fetch might be slow in an up-to-date repo:\n\n  $ git pull\n  Already up to date.\n  $ time git fetch origin master\n  From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux\n   * branch                      master     -> FETCH_HEAD\n  git fetch origin master  0.13s user 0.06s system 10% cpu 1.705 total\n\nGIT_TRACE_CURL shows it spends most of the time transfering (all) tags from the\nremote. It's much faster with --no-tags:\n\n  $ time git fetch -n origin master\n  From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux\n   * branch                      master     -> FETCH_HEAD\n  git fetch -n origin master  0.11s user 0.03s system 36% cpu 0.383 total\n\nBut I don't have tagOpt set:\n\n  $ git config remote.origin.tagOpt || echo $?\n  1\n\nAnd the remote doesn't have to send me any commits, so I don't see why I should\nreceive any tags at all. Why might I be receiving so many tags?\n\nThanks,\nRonan\n"},{"id":"480569","messageId":"20230811180932.GB2816191@coredump.intra.peff.net","threadId":"60098","inReplyTo":"274ec1a2152b0fd53b35c1591f5177e0b0713430@rjp.ie","subject":"Re: Fetching too many tags?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-08-11T18:09:32Z","receivedAt":"2023-08-11T18:10:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 10, 2023 at 06:08:34AM +0000, Ronan Pigott wrote:\n\n> I am interested in git performance today and can't figure out what's going on\n> here. I was wondering why my git-fetch might be slow in an up-to-date repo:\n> \n>   $ git pull\n>   Already up to date.\n>   $ time git fetch origin master\n>   From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux\n>    * branch                      master     -> FETCH_HEAD\n>   git fetch origin master  0.13s user 0.06s system 10% cpu 1.705 total\n> \n> GIT_TRACE_CURL shows it spends most of the time transfering (all) tags from the\n> remote. It's much faster with --no-tags:\n> \n>   $ time git fetch -n origin master\n>   From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux\n>    * branch                      master     -> FETCH_HEAD\n>   git fetch -n origin master  0.11s user 0.03s system 36% cpu 0.383 total\n> \n> But I don't have tagOpt set:\n> \n>   $ git config remote.origin.tagOpt || echo $?\n>   1\n> \n> And the remote doesn't have to send me any commits, so I don't see why I should\n> receive any tags at all. Why might I be receiving so many tags?\n\nYou didn't define \"receiving tags\", but I assume you just mean that you\nsaw the tag names and object ids in the trace output. From the output\nabove, it looks like no actual tag objects were transferred.\n\nAnd the answer, then, is that this is how the Git protocol works. The\nserver says \"here are all the refs I know about\", then the client\ndecides what it wants from that list and asks the server to send the\nnecessary objects, after which it updates its local refs.\n\nSo the server will necessarily send all of the tags. Only the client\nknows what it already has and whether any of them are new. And in the\ndefault mode, which will fetch tags that point to commits we have, it is\nchecking each such new tag to see if it is worth fetching. Even if we\ndid not fetch new commits, we might see new tags that point to existing\ncommits.\n\nWhen you use \"--no-tags\", that explicitly says \"do not bother with tags\nat all\". Recent versions of Git have a protocol extension where the\nclient can say \"I am only interested in refs/heads/master; don't bother\ntelling me about other stuff\". Since the client knows we do not care\nabout tags, it can use that extension to get a much smaller ref\nadvertisement from the server.\n\n-Peff\n"},{"id":"480600","messageId":"5a0544e570fb962c95840d99994bf45aa638faa8@rjp.ie","threadId":"60098","inReplyTo":"20230811180932.GB2816191@coredump.intra.peff.net","subject":"Re: Fetching too many tags?","fromName":"Ronan Pigott","fromEmail":"ronan@rjp.ie","sentAt":"2023-08-11T22:06:43Z","receivedAt":"2023-08-11T22:07:03Z","isPatch":false,"sender":{"key":"ronan@rjp.ie","avatar":"https://avatars.githubusercontent.com/u/12577529?v=4"},"body":"> And the answer, then, is that this is how the Git protocol works. The\n> server says \"here are all the refs I know about\", then the client\n> decides what it wants from that list and asks the server to send the\n> necessary objects, after which it updates its local refs.\n\nThanks, this clears up some of my confusion. I had thought that the client sent\nthe server what we had and that the server would then decide what objects to\nsend over.\n\n> When you use \"--no-tags\", that explicitly says \"do not bother with tags\n> at all\". Recent versions of Git have a protocol extension where the\n> client can say \"I am only interested in refs/heads/master; don't bother\n> telling me about other stuff\". Since the client knows we do not care\n> about tags, it can use that extension to get a much smaller ref\n> advertisement from the server.\n\nDo you mean the --negotiation-tip fetch option? In my experience, it doesn't\nappear to have much of an effect in this case.\n\n  $ time git fetch origin master\n  From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux\n   * branch                      master     -> FETCH_HEAD\n  git fetch origin master  0.13s user 0.04s system 9% cpu 1.793 total\n  $ time git fetch --negotiation-tip=master origin master\n  From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux\n   * branch                      master     -> FETCH_HEAD\n  git fetch --negotiation-tip=master origin master  0.10s user 0.06s system 9% cpu 1.762 total\n\nIs that because (most) the tags point to commits reachable from master?\n\nMy prior (apparently incorrect) understanding of the fetch negotiation is based\non my interpretation of the description of this option in git-fetch(1):\n\n> By default, Git will report, to the server, commits reachable from all local\n> refs to find common commits in an attempt to reduce the size of the\n> to-be-received packfile. If specified, Git will only report commits reachable\n> from the given tips. This is useful to speed up fetches when the user knows\n> which local ref is likely to have commits in common with the upstream ref being\n> fetched.\n\nNow, if I understand correctly, the report does not include the tags that we\nalready have? \n\nCheers,\nRonan\n"},{"id":"480607","messageId":"20230811235820.GA3311880@coredump.intra.peff.net","threadId":"60098","inReplyTo":"5a0544e570fb962c95840d99994bf45aa638faa8@rjp.ie","subject":"Re: Fetching too many tags?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-08-11T23:58:20Z","receivedAt":"2023-08-11T23:58:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 11, 2023 at 10:06:43PM +0000, Ronan Pigott wrote:\n\n> > When you use \"--no-tags\", that explicitly says \"do not bother with tags\n> > at all\". Recent versions of Git have a protocol extension where the\n> > client can say \"I am only interested in refs/heads/master; don't bother\n> > telling me about other stuff\". Since the client knows we do not care\n> > about tags, it can use that extension to get a much smaller ref\n> > advertisement from the server.\n> \n> Do you mean the --negotiation-tip fetch option? In my experience, it doesn't\n> appear to have much of an effect in this case.\n\nNo, the \"negotiation\" phase only happens when there are objects to\nfetch, and the client and server have to agree on which ones. That's not\nhappening at all in your case (so --negotiation-tip won't have any\neffect).\n\nThe feature I was thinking of is that in Git's \"v2\" protocol, the client\ngets to speak first, and so it can say \"btw, I am only interested in\nthese refs\". v2 became the default in git v2.29 (of course both client\nand server have to support it, but kernel.org is definitely up to date\nthere).\n\nYou can see it in action with something like this:\n\n  GIT_TRACE_PACKET=1 git fetch --no-tags origin master\n\nThe \"ref-prefix\" lines are the client telling the server which prefixes\nit's interested in (we have to ask for several variants because \"master\"\nfrom the command line gets fully qualified based on what the other side\noffers). Try it without --no-tags and you'll see a wider ref-prefix\nrequest. If you try:\n\n  GIT_TRACE_PACKET=1 git -c protocol.version=0 fetch --no-tags origin master\n\nyou'll see the full advertisement, even with --no-tags. In v0, the\nserver speaks first and just dumps its complete list of refs.\n\n> > By default, Git will report, to the server, commits reachable from all local\n> > refs to find common commits in an attempt to reduce the size of the\n> > to-be-received packfile. If specified, Git will only report commits reachable\n> > from the given tips. This is useful to speed up fetches when the user knows\n> > which local ref is likely to have commits in common with the upstream ref being\n> > fetched.\n> \n> Now, if I understand correctly, the report does not include the tags that we\n> already have? \n\nSo there's no negotiation here at all, as I explained above. But when it\ndoes happen, Git should use all refs, including tags and branches, to\ntry to reach a common point in the history graph. If you run with\nGIT_TRACE_PACKET on a request that actually fetches objects, you'll see\n\"have\" and \"want\" lines from the client.\n\nFor a vanilla fetch from a server you regularly fetch from, the\nnegotiation is pretty boring and fast (the client tells the server about\nthe old commit at the tip of the branch, and the server immediately says\n\"OK, I know about that\").\n\nA more interesting one is if you fetch the kernel from Linus's repo, and\nthen fetch from the stable kernel repo after that. Or maybe vice versa.\nThere you have two histories that share significant chunks, but also\nhave each diverged. So you should see the client and server dumping\nsha1's at each other until they reach a common point. That's a case\nwhere --negotiation-tip can sometimes speed things up.\n\n-Peff\n"},{"id":"480619","messageId":"053992e6ab7c43d484c46901d31313019494daed@rjp.ie","threadId":"60098","inReplyTo":"20230811235820.GA3311880@coredump.intra.peff.net","subject":"Re: Fetching too many tags?","fromName":"Ronan Pigott","fromEmail":"ronan@rjp.ie","sentAt":"2023-08-12T01:04:42Z","receivedAt":"2023-08-12T01:04:51Z","isPatch":false,"sender":{"key":"ronan@rjp.ie","avatar":"https://avatars.githubusercontent.com/u/12577529?v=4"},"body":"> No, the \"negotiation\" phase only happens when there are objects to\n> fetch, and the client and server have to agree on which ones. That's not\n> happening at all in your case (so --negotiation-tip won't have any\n> effect).\n\nAh, I see.\n\n> The feature I was thinking of is that in Git's \"v2\" protocol, the client\n> gets to speak first, and so it can say \"btw, I am only interested in\n> these refs\". v2 became the default in git v2.29 (of course both client\n> and server have to support it, but kernel.org is definitely up to date\n> there).\n> \n> You can see it in action with something like this:\n> \n>  GIT_TRACE_PACKET=1 git fetch --no-tags origin master\n> \n> The \"ref-prefix\" lines are the client telling the server which prefixes\n> it's interested in (we have to ask for several variants because \"master\"\n> from the command line gets fully qualified based on what the other side\n> offers). Try it without --no-tags and you'll see a wider ref-prefix\n> request. If you try:\n\nThanks. I tried this and indeed without --no-tags there is an additional line\n\n> 17:41:29.163545 pkt-line.c:86   packet:   git< ref-prefix refs/tags/\n\nI understand now that this is why the server is telling me about all those tags.\nI had thought it would only need to tell me about tags that point to something\nreachable from master, and was confused why the server was advertising all the\ntags.\n\nThanks,\nRonan\n"}]}