{"thread":{"id":"57582","subject":"Noticed big performance issue when fetching multiple refs with depth 1 enabled","startedAt":"2022-03-18T14:45:26Z","lastAt":"2022-03-18T20:22:02Z","messageCount":2,"participants":["Ricard Valverde","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"451627","messageId":"CABkk9JoOYJ-tH9MrMkfdj5xMV+exXaJ2FxSjiSCMwCxsjaKqGg@mail.gmail.com","threadId":"57582","inReplyTo":null,"subject":"Noticed big performance issue when fetching multiple refs with depth 1 enabled","fromName":"Ricard Valverde","fromEmail":"ricard.valverde@socialpoint.es","sentAt":"2022-03-18T14:44:53Z","receivedAt":"2022-03-18T14:45:26Z","isPatch":false,"sender":{"key":"ricard.valverde@socialpoint.es","avatar":null},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\nDo a git fetch --depth=1 , with no remote changes, having a lot of\nremote branches makes the issue more visible\nWhat did you expect to happen? (Expected behavior)\nShallow fetch should be faster or as fast as a full fetch\nWhat happened instead? (Actual behavior)\nShallow fetch is about 10x slower than full fetch, consistently\nWhat's different between what you expected and what actually happened?\nI expected the shallow fetch to be faster than a normal fetch, or as\nfast. Did not expect it being 10x slower.\nAnything else you want to add:\nTested in a repository with ~10000 remote branches, a single Github\nhosted remote.\nAlso tested with a local remote, and Git 2.35.1 version, with\ncomparable results.\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.32.0 (Apple Git-132)\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Darwin 20.6.0 Darwin Kernel Version 20.6.0: Tue Feb 22 21:10:41\nPST 2022; root:xnu-7195.141.26~1/RELEASE_X86_64 x86_64\ncompiler info: clang: 13.0.0 (clang-1300.0.29.30)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /bin/bash\n\n\n[Enabled Hooks]\n"},{"id":"451643","messageId":"YjTp45Q7wmZTodP9@camp.crustytoothpaste.net","threadId":"57582","inReplyTo":"CABkk9JoOYJ-tH9MrMkfdj5xMV+exXaJ2FxSjiSCMwCxsjaKqGg@mail.gmail.com","subject":"Re: Noticed big performance issue when fetching multiple refs with depth 1 enabled","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2022-03-18T20:21:55Z","receivedAt":"2022-03-18T20:22:02Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2022-03-18 at 14:44:53, Ricard Valverde wrote:\n> Thank you for filling out a Git bug report!\n> Please answer the following questions to help us understand your issue.\n> \n> What did you do before the bug happened? (Steps to reproduce your issue)\n> Do a git fetch --depth=1 , with no remote changes, having a lot of\n> remote branches makes the issue more visible\n> What did you expect to happen? (Expected behavior)\n> Shallow fetch should be faster or as fast as a full fetch\n> What happened instead? (Actual behavior)\n> Shallow fetch is about 10x slower than full fetch, consistently\n> What's different between what you expected and what actually happened?\n> I expected the shallow fetch to be faster than a normal fetch, or as\n> fast. Did not expect it being 10x slower.\n> Anything else you want to add:\n> Tested in a repository with ~10000 remote branches, a single Github\n> hosted remote.\n> Also tested with a local remote, and Git 2.35.1 version, with\n> comparable results.\n\nThe reason you're seeing this is that fetching into an existing shallow\nremote is extremely expensive on the server side.  With a normal fetch,\nwe know that the user only needs objects which exist between the old and\nnew heads, and that they have every object which is reachable from the\nold heads.\n\nHowever, with a shallow fetch, we can't assume that, since by definition\nthe client side doesn't have all of those objects.  A substantial amount\nmore work must be done to determine what the client already has.  This\nis made worse by the fact that you have 10,000 branches.\n\nIf you're working in a CI system or such, you should use a fresh shallow\nclone each time, which will be much faster and easier on the server.\nOtherwise, you may find that reducing the number of refs significantly\ncan also help performance.  If you can guarantee that you'll be online\nwhen working with this project, a partial clone may also meet your\nneeds.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"}]}