{"thread":{"id":"51305","subject":"git receive-pack deletes refs one at a time?","startedAt":"2019-06-13T17:33:43Z","lastAt":"2019-06-13T21:20:08Z","messageCount":4,"participants":["Nasser Grainawi","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"377158","messageId":"E203B539-3481-435F-8D08-9C405DF42923@codeaurora.org","threadId":"51305","inReplyTo":null,"subject":"git receive-pack deletes refs one at a time?","fromName":"Nasser Grainawi","fromEmail":"nasser@codeaurora.org","sentAt":"2019-06-13T17:33:40Z","receivedAt":"2019-06-13T17:33:43Z","isPatch":false,"sender":{"key":"nasser@codeaurora.org","avatar":"https://avatars.githubusercontent.com/u/757421?v=4"},"body":"I have a situation where I need to delete 100k+ refs on 15+ separate hosts/disks. This setup is using Gerrit replication, so I can trigger it all on one host and it will push the deletes to the rest (all running git-daemon v2.18.0 with receive-pack enabled). All the refs being deleted on the receiving ends are packed.\n\nWhat I see is the packed-refs file getting locked/updated over and over for each ref. I had assumed it would do something more like 'update-ref --stdin' and do a bulk removal of refs. Am I seeing the correct behavior? If yes, is there a specific reason it works this way or is \"bulk delete through push\" just a feature that hasn't been implemented yet?\n\nThanks,\nNasser\n\n-- \nQualcomm Innovation Center, Inc.\nThe Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, \na Linux Foundation Collaborative Project\n\n"},{"id":"377161","messageId":"20190613174331.GA19795@sigill.intra.peff.net","threadId":"51305","inReplyTo":"E203B539-3481-435F-8D08-9C405DF42923@codeaurora.org","subject":"Re: git receive-pack deletes refs one at a time?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-13T17:43:32Z","receivedAt":"2019-06-13T17:43:37Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 13, 2019 at 11:33:40AM -0600, Nasser Grainawi wrote:\n\n> I have a situation where I need to delete 100k+ refs on 15+ separate\n> hosts/disks. This setup is using Gerrit replication, so I can trigger\n> it all on one host and it will push the deletes to the rest (all\n> running git-daemon v2.18.0 with receive-pack enabled). All the refs\n> being deleted on the receiving ends are packed.\n> \n> What I see is the packed-refs file getting locked/updated over and\n> over for each ref. I had assumed it would do something more like\n> 'update-ref --stdin' and do a bulk removal of refs. Am I seeing the\n> correct behavior? If yes, is there a specific reason it works this way\n> or is \"bulk delete through push\" just a feature that hasn't been\n> implemented yet?\n\nThe underlying ref code is smart enough to coalesce all of the deletions\nin a single transaction into a single write of the packed-refs file.\n\nBut historically, pushes do not do a single ref transaction because we\nwould allow the push for one ref to succeed while others failed. Later,\nwe added an \"atomic\" mode that does it all in a single transaction.\n\nTry with \"git push --atomic\", which should be able to do it in a single\nwrite.\n\n-Peff\n"},{"id":"377184","messageId":"34FD9B2B-D53A-4CD2-AEF0-3AFAF1586EB7@codeaurora.org","threadId":"51305","inReplyTo":"20190613174331.GA19795@sigill.intra.peff.net","subject":"Re: git receive-pack deletes refs one at a time?","fromName":"Nasser Grainawi","fromEmail":"nasser@codeaurora.org","sentAt":"2019-06-13T19:46:05Z","receivedAt":"2019-06-13T19:46:08Z","isPatch":false,"sender":{"key":"nasser@codeaurora.org","avatar":"https://avatars.githubusercontent.com/u/757421?v=4"},"body":"\n> On Jun 13, 2019, at 11:43 AM, Jeff King <peff@peff.net> wrote:\n> \n> On Thu, Jun 13, 2019 at 11:33:40AM -0600, Nasser Grainawi wrote:\n> \n>> I have a situation where I need to delete 100k+ refs on 15+ separate\n>> hosts/disks. This setup is using Gerrit replication, so I can trigger\n>> it all on one host and it will push the deletes to the rest (all\n>> running git-daemon v2.18.0 with receive-pack enabled). All the refs\n>> being deleted on the receiving ends are packed.\n>> \n>> What I see is the packed-refs file getting locked/updated over and\n>> over for each ref. I had assumed it would do something more like\n>> 'update-ref --stdin' and do a bulk removal of refs. Am I seeing the\n>> correct behavior? If yes, is there a specific reason it works this way\n>> or is \"bulk delete through push\" just a feature that hasn't been\n>> implemented yet?\n> \n> The underlying ref code is smart enough to coalesce all of the deletions\n> in a single transaction into a single write of the packed-refs file.\n> \n> But historically, pushes do not do a single ref transaction because we\n> would allow the push for one ref to succeed while others failed. Later,\n> we added an \"atomic\" mode that does it all in a single transaction.\n> \n> Try with \"git push --atomic\", which should be able to do it in a single\n> write.\n\nThanks!\n\nIs there a way to get the bulk behavior without the all-or-nothing behavior?\n\n-- \nQualcomm Innovation Center, Inc.\nThe Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, \na Linux Foundation Collaborative Project\n\n"},{"id":"377188","messageId":"20190613212005.GA1970@sigill.intra.peff.net","threadId":"51305","inReplyTo":"34FD9B2B-D53A-4CD2-AEF0-3AFAF1586EB7@codeaurora.org","subject":"Re: git receive-pack deletes refs one at a time?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-13T21:20:05Z","receivedAt":"2019-06-13T21:20:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 13, 2019 at 01:46:05PM -0600, Nasser Grainawi wrote:\n\n> > The underlying ref code is smart enough to coalesce all of the deletions\n> > in a single transaction into a single write of the packed-refs file.\n> > \n> > But historically, pushes do not do a single ref transaction because we\n> > would allow the push for one ref to succeed while others failed. Later,\n> > we added an \"atomic\" mode that does it all in a single transaction.\n> > \n> > Try with \"git push --atomic\", which should be able to do it in a single\n> > write.\n> \n> Thanks!\n> \n> Is there a way to get the bulk behavior without the all-or-nothing behavior?\n\nNo, I don't think so. In theory the ref backend code could have a\n\"non-atomic\" flag that lets it do as much as possible of a single\ntransaction. But nobody has ever implemented that.\n\nIn practice, using \"--atomic\" shouldn't cause failures, unless you are\ntrying to delete refs that somebody else is modifying at the same time.\n\n-Peff\n"}]}