{"thread":{"id":"61426","subject":"Feature request: secondary index by path fragment","startedAt":"2024-05-06T23:11:38Z","lastAt":"2024-05-07T05:38:35Z","messageCount":4,"participants":["Robin H. Johnson","Junio C Hamano","Patrick Steinhardt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"494209","messageId":"robbat2-20240506T225759-090424131Z@orbis-terrarum.net","threadId":"61426","inReplyTo":null,"subject":"Feature request: secondary index by path fragment","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2024-05-06T23:11:37Z","receivedAt":"2024-05-06T23:11:38Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"Maybe this already happens in the code, but if not, please consider it\nas a feature request.\n\nGentoo has some tooling that boils down to repeated runs of 'git log -- somepath/'\nvia cgit as well as other shell tooling.\n\nIf the path is relatively deep for the tree (e.g. to a specific file or\nsub-directory), the size of history [1] makes that a very slow operation\nto go all the way back to the initial repo commit: ~12 seconds per\noperation on fast hardware, ~45 seconds on slower harder - even with the\npacks cached.\n\nI was wondering if Git could gain a secondary index of commits, based on\npath prefixes, that would speed up the 'git log' run.\n\nIt would need to be fast to append to the secondary index, because\nGentoo gets a steady flow of commits 24/7.\n\n[1] 825k+ commits based on GitHub stats.\n\n-- \nRobin Hugh Johnson\nGentoo Linux: Dev, Infra Lead, Foundation President & Treasurer\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85\nGnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136\n"},{"id":"494210","messageId":"xmqqttjawmos.fsf@gitster.g","threadId":"61426","inReplyTo":"robbat2-20240506T225759-090424131Z@orbis-terrarum.net","subject":"Re: Feature request: secondary index by path fragment","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-05-06T23:22:11Z","receivedAt":"2024-05-06T23:22:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Robin H. Johnson\" writes:\n\n> Gentoo has some tooling that boils down to repeated runs of 'git log -- somepath/'\n> via cgit as well as other shell tooling.\n> ...\n> I was wondering if Git could gain a secondary index of commits, based on\n> path prefixes, that would speed up the 'git log' run.\n\nPerhaps the bloom filters are good fit for the use case?\n"},{"id":"494220","messageId":"ZjmtJFF7rv7B8Nhj@tanuki","threadId":"61426","inReplyTo":"xmqqttjawmos.fsf@gitster.g","subject":"Re: Feature request: secondary index by path fragment","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-07T04:25:08Z","receivedAt":"2024-05-07T04:25:13Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, May 06, 2024 at 04:22:11PM -0700, Junio C Hamano wrote:\n> \"Robin H. Johnson\" writes:\n> \n> > Gentoo has some tooling that boils down to repeated runs of 'git log -- somepath/'\n> > via cgit as well as other shell tooling.\n> > ...\n> > I was wondering if Git could gain a secondary index of commits, based on\n> > path prefixes, that would speed up the 'git log' run.\n> \n> Perhaps the bloom filters are good fit for the use case?\n\nYes, Bloom filters are the first thing that pop into my mind here as\nthey are exactly designed to solve this problem. So if you rewrite your\ncommit graphs with `git commit-graph write --changed-paths --reachable`\nyou should hopefully see a significant speedup.\n\nIt does surface some a usability issues though:\n\n - There is no easy way to enable the computation of bloom filters via\n configuration, to the best of my knowledge.\n\n - How would a non-Git-expert know?\n\nIt makes me wonder whether we can maybe enable generation of Bloom\nfilters by default. The biggest downside is of course that writing\ncommit graphs becomes slower. But that should happen in the background\nfor normal users anyway, and most forges probably hand-roll maintenance\nand thus wouldn't care.\n\nIs there any other thing I'm missing why those are not written by\ndefault?\n\nPatrick\n"},{"id":"494236","messageId":"robbat2-20240507T053331-859497691Z@orbis-terrarum.net","threadId":"61426","inReplyTo":"ZjmtJFF7rv7B8Nhj@tanuki","subject":"Re: Feature request: secondary index by path fragment","fromName":"Robin H. Johnson","fromEmail":"robbat2@gentoo.org","sentAt":"2024-05-07T05:38:34Z","receivedAt":"2024-05-07T05:38:35Z","isPatch":false,"sender":{"key":"robbat2@gentoo.org","avatar":"https://avatars.githubusercontent.com/u/373898?v=4"},"body":"On Tue, May 07, 2024 at 06:25:08AM +0200, Patrick Steinhardt wrote:\n> On Mon, May 06, 2024 at 04:22:11PM -0700, Junio C Hamano wrote:\n> > \"Robin H. Johnson\" writes:\n> > \n> > > Gentoo has some tooling that boils down to repeated runs of 'git log -- somepath/'\n> > > via cgit as well as other shell tooling.\n> > > ...\n> > > I was wondering if Git could gain a secondary index of commits, based on\n> > > path prefixes, that would speed up the 'git log' run.\n> > \n> > Perhaps the bloom filters are good fit for the use case?\n> \n> Yes, Bloom filters are the first thing that pop into my mind here as\n> they are exactly designed to solve this problem. So if you rewrite your\n> commit graphs with `git commit-graph write --changed-paths --reachable`\n> you should hopefully see a significant speedup.\n\nGood news & bad news.\n\"git log -- sys-apps/pv >/dev/null\" as my testcase from before:\nThe fast system (2.45.0) went from 11 seconds to ~1 second!\nThe slow system (2.44.0) went from 45 seconds to 49 seconds :-(.\n\nI'll try to trace down why one system slowed down.\n\ncommit-graph command:\nfast: 1m10s\nslow: 3m43s\n\n> It makes me wonder whether we can maybe enable generation of Bloom\n> filters by default. The biggest downside is of course that writing\n> commit graphs becomes slower. But that should happen in the background\n> for normal users anyway, and most forges probably hand-roll maintenance\n> and thus wouldn't care.\nMost repos are also MUCH smaller than this, so it should be safe to\nenable.\n\n\n-- \nRobin Hugh Johnson\nGentoo Linux: Dev, Infra Lead, Foundation President & Treasurer\nE-Mail : robbat2@gentoo.org\nGnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85\nGnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136\n"}]}