{"thread":{"id":"54350","subject":"Optimizing for partial clone with '--filter=tree:0'","startedAt":"2020-10-05T23:02:53Z","lastAt":"2020-10-05T23:02:53Z","messageCount":1,"participants":["Alexandr Miloslavskiy"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"406971","messageId":"58274817-7ac6-b6ae-0d10-22485dfe5e0e@syntevo.com","threadId":"54350","inReplyTo":null,"subject":"Optimizing for partial clone with '--filter=tree:0'","fromName":"Alexandr Miloslavskiy","fromEmail":"alexandr.miloslavskiy@syntevo.com","sentAt":"2020-10-05T16:38:17Z","receivedAt":"2020-10-05T23:02:53Z","isPatch":false,"sender":{"key":"alexandr.miloslavskiy@syntevo.com","avatar":null},"body":"We are implementing a git UI. One interesting case is the repository\ncloned with '--filter=tree:0', because it makes it a lot harder to\nrun basic git operations such as file log and blame.\n\nEventually we arrived at a number of problems. We should be able to\nmake patches, at least for (2) and (4), if deemed worthy and the plan\nis clear enough. Note that optimal patches (as we see it) will involve\na protocol change.\n\n(1) Is it even considered a realistic use case?\n-----------------------------------------------\nI used Linux repository as an example of reasonably large repo:\n   https://github.com/torvalds/linux.git (951025 commits)\n\nI cloned Linux repository with various filters and got these stats:\n   git clone --bare <url>\n\t7'624'042 objects\n\t   2.86gb network\n\t   3.10gb disk\n   git clone --bare --filter=blob:none <url>\n\t5'484'714 (71.9%) objects\n\t   1.01gb (35.3%) network\n\t   1.16gb (37.4%) disk\n   git clone --bare --filter=tree:0 <url>\n\t  951'693 (12.5%) objects\n\t   0.47gb (16.4%) network\n\t   0.50gb (16.1%) disk\n   git clone --bare --depth 1 --branch master <url>\n\t   74'380 ( 0.9%) objects\n\t   0.19gb ( 6.6%) network\n\t   0.19gb ( 6.1%) disk\n\nMy conclusion is that '--filter=tree:0' could be desired because it\nreasonably saves disk space and network.\n\n(2) A command to enrich repo with trees\n---------------------------------------\nSince all filters currently include commit objects, it doesn't seem\npossible to append the trees alone to a repository that already has\ncommits. It seems that it could be possible to download trees+commits\nlike this:\n\n   git -c \"remote.origin.partialclonefilter=blob:none\" fetch\n   --deepen=999999 origin\n\n   Here, '--deepen' is a dirty hack to convince git to re-download\n   commits that are already present locally (without trees though).\n\n   Here, '-c' is a workaround for the problem where 'git fetch'\n   overwrites filter in config. This problem is probably solved in\n   cooking topic: 'fetch: do not override partial clone filter'.\n\nHowever, according to figures in (1), re-downloading commits should\ncost around the cost of 'clone --filter=tree:0', that is 0.5gb extra in\ncase of Linux repo. It would be nice to avoid that by having a filter\nlike \"trees only please\".\n\nIt would also be nice to get rid of '--deepen' hack.\n\n(3) Properly supporting 'git blame' and 'git log -- path'\n---------------------------------------------------------\nCurrently, promisor will download things one at a time, which is very\nslow. For example, 'git blame' will download trees for commits,\nprocessing one commit at a time. See (4) for a possible solution.\n\n(4) Command to download ALL trees for a subpath\n-----------------------------------------------\nE.g. for blamed path '/1/2/3/4.txt', only parent trees will be\ndownloaded:\n   '/1'\n   '/1/2'\n   '/1/2/3'\n\nSuch minimal approach should fall in line with user's intention for\nusing '--filter=tree:0' - user obviously wanted to minimize something,\nbe that disk or network used. It doesn't sound nice if the first\n'git blame' reverts to a repo with all trees, as if cloned with\n'--filter=blob:none'.\n\nCurrently '--filter=sparse:oid' is there to support that, but it is\nvery hard to use on client side, because it requires paths to be\nalready present in a commit on server.\n\nFor a possible solution, it sounds reasonable to have such filter:\n   --filter=sparse:pathlist=/1/2'\nPath list could be delimited with some special character, and paths\nthemselves could be escaped.\n\nOn top of helping with 'git blame' and 'git log', this feature should\nhelp a lot with sparse clones of large mono-repos, such as Google's\nsuper-mono-repo.\n"}]}