{"thread":{"id":"62335","subject":"Should object repacking only update server-info for packs instead of doing it for refs?","startedAt":"2024-10-14T20:07:11Z","lastAt":"2024-10-15T13:56:44Z","messageCount":7,"participants":["Martin Fick","Luca Milanesio","Taylor Blau","Patrick Steinhardt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"505046","messageId":"LV8PR12MB9407CB282D1FD4998C659D29C2442@LV8PR12MB9407.namprd12.prod.outlook.com","threadId":"62335","inReplyTo":null,"subject":"Should object repacking only update server-info for packs instead of doing it for refs?","fromName":"Martin Fick","fromEmail":"mfick@nvidia.com","sentAt":"2024-10-14T20:07:08Z","receivedAt":"2024-10-14T20:07:11Z","isPatch":false,"sender":{"key":"mfick@nvidia.com","avatar":"https://gravatar.com/avatar/17127f5962cba0a9b2fe0c81d6d992c863dbae1f9b897fad48b1b59a09e1ac2e?d=mp&s=160"},"body":"I have been experimenting with trying to run git geometric repacking after every push on our servers and have noticed that even a NOOP geometric repack takes approximately 30s on one of my large repositories. Further investigation determined that it was the update-server-info call that was taking that 30s, so running repacking with the -n instead only takes 6ms in this case! I dug a bit deeper and found that it was the update_info_refs() call that takes all the time. This seemed a bit off to me. Yes, it makes sense that would be slow, I have almost 2Mrefs, but I had to ask myself, why is this being done from git repack? It seems counterintuitive that an operation designed to repack objects would be performing maintenance of any sort on refs? Should this be improved somehow? Should there be a -packs and a -refs switch to git update-server-info, and should git repack call it with -packs? Should git pack-refs be calling git update-server-info with -refs instead?\n\nThanks,\n\n-Martin\n\n"},{"id":"505050","messageId":"569E3AEA-AC68-4186-B38C-2E47DAB9890E@gmail.com","threadId":"62335","inReplyTo":"LV8PR12MB9407CB282D1FD4998C659D29C2442@LV8PR12MB9407.namprd12.prod.outlook.com","subject":"Re: Should object repacking only update server-info for packs instead of doing it for refs?","fromName":"Luca Milanesio","fromEmail":"luca.milanesio@gmail.com","sentAt":"2024-10-14T20:33:17Z","receivedAt":"2024-10-14T20:33:31Z","isPatch":false,"sender":{"key":"luca.milanesio@gmail.com","avatar":"https://gravatar.com/avatar/64e45570bb8baaba9a566592150e6c7341a300e314501a944cb95b0bc1380f49?d=mp&s=160"},"body":"\n\n> On 14 Oct 2024, at 21:07, Martin Fick <mfick@nvidia.com> wrote:\n> \n> I have been experimenting with trying to run git geometric repacking after every push on our servers\n\nI had the same sentiment when I studied how geometric repacking works.\nWhen we tried to do geometric repacking every 5 minutes on a busy repo, we ended up with a single pack file most of the time, which isn’t really what you would expect with a geometric progression.\n\nRunning it after *every* push would make more sense, as Martin is experimenting.\nPlease keep us posted on the results :-)\n\n> and have noticed that even a NOOP geometric repack takes approximately 30s on one of my large repositories. Further investigation determined that it was the update-server-info call that was taking that 30s, so running repacking with the -n instead only takes 6ms in this case! I dug a bit deeper and found that it was the update_info_refs() call that takes all the time. This seemed a bit off to me. Yes, it makes sense that would be slow, I have almost 2Mrefs, but I had to ask myself, why is this being done from git repack?\n\nI believe it should be done to update objects/info/packs\n\n> It seems counterintuitive that an operation designed to repack objects would be performing maintenance of any sort on refs?\n\nTrue, it should not touch info/refs IMHO, as you’re really not changing any refs.\n\nLuca."},{"id":"505069","messageId":"Zw2MVqi0FLcMuR8Z@nand.local","threadId":"62335","inReplyTo":"569E3AEA-AC68-4186-B38C-2E47DAB9890E@gmail.com","subject":"Re: Should object repacking only update server-info for packs instead of doing it for refs?","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-10-14T21:25:42Z","receivedAt":"2024-10-14T21:25:44Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 14, 2024 at 09:33:17PM +0100, Luca Milanesio wrote:\n> > It seems counterintuitive that an operation designed to repack\n> > objects would be performing maintenance of any sort on refs?\n>\n> True, it should not touch info/refs IMHO, as you’re really not\n> changing any refs.\n\nRight. I don't think that the current behavior is a bug, but just that\nit's doing unnecessary work from within 'git repack' to update the\ninfo/refs file when it's known ahead of time that the refs haven't\nchanged.\n\nI think it's reasonable to skip this step when repacking, but of course\nwe would still want to update info/packs (assuming that the repack\nwasn't a noop, of course).\n\nThanks,\nTaylor\n"},{"id":"505136","messageId":"Zw5LwB2ap1QnmG6L@pks.im","threadId":"62335","inReplyTo":"Zw2MVqi0FLcMuR8Z@nand.local","subject":"Re: Should object repacking only update server-info for packs instead of doing it for refs?","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-15T11:02:29Z","receivedAt":"2024-10-15T11:02:37Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 14, 2024 at 05:25:42PM -0400, Taylor Blau wrote:\n> On Mon, Oct 14, 2024 at 09:33:17PM +0100, Luca Milanesio wrote:\n> > > It seems counterintuitive that an operation designed to repack\n> > > objects would be performing maintenance of any sort on refs?\n> >\n> > True, it should not touch info/refs IMHO, as you’re really not\n> > changing any refs.\n> \n> Right. I don't think that the current behavior is a bug, but just that\n> it's doing unnecessary work from within 'git repack' to update the\n> info/refs file when it's known ahead of time that the refs haven't\n> changed.\n> \n> I think it's reasonable to skip this step when repacking, but of course\n> we would still want to update info/packs (assuming that the repack\n> wasn't a noop, of course).\n\nIt certainly may be reasonable. But in my opinion, it would be even more\nreasonable to not use the dumb HTTP transport at all. If you don't there\nis no reason to run git-update-server-info(1) in the first place, so\nyou'd neither generate info/refs nor info/packs.\n\nWe have been discussing in the past whether the dumb HTTP protocol\nshould be deprecated, and in the context of that discussion we were also\nwondering whether we should start disabling git-update-server-info(1) by\ndefault. Users don't generally need this, and most server operators\ndon't need this nowadays, either. So why generate data that is useless\nin almost all cases?\n\nPatrick\n"},{"id":"505158","messageId":"428B24FB-A648-43BA-9381-1B46D99E2F37@gmail.com","threadId":"62335","inReplyTo":"Zw5LwB2ap1QnmG6L@pks.im","subject":"Re: Should object repacking only update server-info for packs instead of doing it for refs?","fromName":"Luca Milanesio","fromEmail":"luca.milanesio@gmail.com","sentAt":"2024-10-15T11:52:12Z","receivedAt":"2024-10-15T11:52:25Z","isPatch":false,"sender":{"key":"luca.milanesio@gmail.com","avatar":"https://gravatar.com/avatar/64e45570bb8baaba9a566592150e6c7341a300e314501a944cb95b0bc1380f49?d=mp&s=160"},"body":"\n\n> On 15 Oct 2024, at 12:02, Patrick Steinhardt <ps@pks.im> wrote:\n> \n> On Mon, Oct 14, 2024 at 05:25:42PM -0400, Taylor Blau wrote:\n>> On Mon, Oct 14, 2024 at 09:33:17PM +0100, Luca Milanesio wrote:\n>>>> It seems counterintuitive that an operation designed to repack\n>>>> objects would be performing maintenance of any sort on refs?\n>>> \n>>> True, it should not touch info/refs IMHO, as you’re really not\n>>> changing any refs.\n>> \n>> Right. I don't think that the current behavior is a bug, but just that\n>> it's doing unnecessary work from within 'git repack' to update the\n>> info/refs file when it's known ahead of time that the refs haven't\n>> changed.\n>> \n>> I think it's reasonable to skip this step when repacking, but of course\n>> we would still want to update info/packs (assuming that the repack\n>> wasn't a noop, of course).\n> \n> It certainly may be reasonable. But in my opinion, it would be even more\n> reasonable to not use the dumb HTTP transport at all. If you don't there\n> is no reason to run git-update-server-info(1) in the first place, so\n> you'd neither generate info/refs nor info/packs.\n\nI don’t believe anyone uses it anymore, but someone *may* still use it, and therefore, Git should update the info/packs during geometric repacking.\nHowever, why not introduce a “kill switch” in git config to disable it so that all tools can automatically skip the expensive info/packs (and info/refs) updates?\n\n> We have been discussing in the past whether the dumb HTTP protocol\n> should be deprecated, and in the context of that discussion we were also\n> wondering whether we should start disabling git-update-server-info(1) by\n> default.\n\n+1 from me.\n\n> Users don't generally need this, and most server operators\n> don't need this nowadays, either. So why generate data that is useless\n> in almost all cases?\n\nCertainly not IMHO.\n\nLuca.\n\n"},{"id":"505160","messageId":"Zw5cvnkBQ1MzIS79@pks.im","threadId":"62335","inReplyTo":"428B24FB-A648-43BA-9381-1B46D99E2F37@gmail.com","subject":"Re: Should object repacking only update server-info for packs instead of doing it for refs?","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-15T12:14:59Z","receivedAt":"2024-10-15T12:15:04Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 15, 2024 at 12:52:12PM +0100, Luca Milanesio wrote:\n> \n> \n> > On 15 Oct 2024, at 12:02, Patrick Steinhardt <ps@pks.im> wrote:\n> > \n> > On Mon, Oct 14, 2024 at 05:25:42PM -0400, Taylor Blau wrote:\n> >> On Mon, Oct 14, 2024 at 09:33:17PM +0100, Luca Milanesio wrote:\n> >>>> It seems counterintuitive that an operation designed to repack\n> >>>> objects would be performing maintenance of any sort on refs?\n> >>> \n> >>> True, it should not touch info/refs IMHO, as you’re really not\n> >>> changing any refs.\n> >> \n> >> Right. I don't think that the current behavior is a bug, but just that\n> >> it's doing unnecessary work from within 'git repack' to update the\n> >> info/refs file when it's known ahead of time that the refs haven't\n> >> changed.\n> >> \n> >> I think it's reasonable to skip this step when repacking, but of course\n> >> we would still want to update info/packs (assuming that the repack\n> >> wasn't a noop, of course).\n> > \n> > It certainly may be reasonable. But in my opinion, it would be even more\n> > reasonable to not use the dumb HTTP transport at all. If you don't there\n> > is no reason to run git-update-server-info(1) in the first place, so\n> > you'd neither generate info/refs nor info/packs.\n> \n> I don’t believe anyone uses it anymore, but someone *may* still use\n> it, and therefore, Git should update the info/packs during geometric\n> repacking. However, why not introduce a “kill switch” in git config to\n> disable it so that all tools can automatically skip the expensive\n> info/packs (and info/refs) updates?\n\nWe already have those, see \"repack.updateServerInfo\" and\n\"receive.updateServerInfo\". Not quite a single kill switch, but it is\npossible :)\n\nPatrick\n"},{"id":"505164","messageId":"Zw50mSKuyvlMDTAQ@nand.local","threadId":"62335","inReplyTo":"Zw5cvnkBQ1MzIS79@pks.im","subject":"Re: Should object repacking only update server-info for packs instead of doing it for refs?","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-10-15T13:56:41Z","receivedAt":"2024-10-15T13:56:44Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Oct 15, 2024 at 02:14:59PM +0200, Patrick Steinhardt wrote:\n> > >> I think it's reasonable to skip this step when repacking, but of course\n> > >> we would still want to update info/packs (assuming that the repack\n> > >> wasn't a noop, of course).\n> > >\n> > > It certainly may be reasonable. But in my opinion, it would be even more\n> > > reasonable to not use the dumb HTTP transport at all. If you don't there\n> > > is no reason to run git-update-server-info(1) in the first place, so\n> > > you'd neither generate info/refs nor info/packs.\n> >\n> > I don’t believe anyone uses it anymore, but someone *may* still use\n> > it, and therefore, Git should update the info/packs during geometric\n> > repacking. However, why not introduce a “kill switch” in git config to\n> > disable it so that all tools can automatically skip the expensive\n> > info/packs (and info/refs) updates?\n>\n> We already have those, see \"repack.updateServerInfo\" and\n> \"receive.updateServerInfo\". Not quite a single kill switch, but it is\n> possible :)\n\nThere is also 'git repack -n' (which I know GitHub uses) to disable\nupdating info/refs and info/packs as a command-line flag (if for\nwhatever reason modifying the 'git repack' invocation is more\nconvenient than touching the repository configuration).\n\nThanks,\nTaylor\n"}]}