# git receive-pack deletes refs one at a time? 4 messages from 2019-06-13 to 2019-06-13. Participants: Nasser Grainawi, Jeff King. Thread: https://gitlist.dev/t/51305 ## Nasser Grainawi, 2019-06-13 17:33 Subject: git receive-pack deletes refs one at a time? Message-ID: URL: https://gitlist.dev/e/E203B539-3481-435F-8D08-9C405DF42923%40codeaurora.org ``` I have a situation where I need to delete 100k+ refs on 15+ separate hosts/disks. This setup is using Gerrit replication, so I can trigger it all on one host and it will push the deletes to the rest (all running git-daemon v2.18.0 with receive-pack enabled). All the refs being deleted on the receiving ends are packed. What I see is the packed-refs file getting locked/updated over and over for each ref. I had assumed it would do something more like 'update-ref --stdin' and do a bulk removal of refs. Am I seeing the correct behavior? If yes, is there a specific reason it works this way or is "bulk delete through push" just a feature that hasn't been implemented yet? Thanks, Nasser -- Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project ``` ## Jeff King, 2019-06-13 17:43 Subject: Re: git receive-pack deletes refs one at a time? Message-ID: <20190613174331.GA19795@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190613174331.GA19795%40sigill.intra.peff.net In-Reply-To: ``` On Thu, Jun 13, 2019 at 11:33:40AM -0600, Nasser Grainawi wrote: > I have a situation where I need to delete 100k+ refs on 15+ separate > hosts/disks. This setup is using Gerrit replication, so I can trigger > it all on one host and it will push the deletes to the rest (all > running git-daemon v2.18.0 with receive-pack enabled). All the refs > being deleted on the receiving ends are packed. > > What I see is the packed-refs file getting locked/updated over and > over for each ref. I had assumed it would do something more like > 'update-ref --stdin' and do a bulk removal of refs. Am I seeing the > correct behavior? If yes, is there a specific reason it works this way > or is "bulk delete through push" just a feature that hasn't been > implemented yet? The underlying ref code is smart enough to coalesce all of the deletions in a single transaction into a single write of the packed-refs file. But historically, pushes do not do a single ref transaction because we would allow the push for one ref to succeed while others failed. Later, we added an "atomic" mode that does it all in a single transaction. Try with "git push --atomic", which should be able to do it in a single write. -Peff ``` ## Nasser Grainawi, 2019-06-13 19:46 Subject: Re: git receive-pack deletes refs one at a time? Message-ID: <34FD9B2B-D53A-4CD2-AEF0-3AFAF1586EB7@codeaurora.org> URL: https://gitlist.dev/e/34FD9B2B-D53A-4CD2-AEF0-3AFAF1586EB7%40codeaurora.org In-Reply-To: <20190613174331.GA19795@sigill.intra.peff.net> ``` > On Jun 13, 2019, at 11:43 AM, Jeff King wrote: > > On Thu, Jun 13, 2019 at 11:33:40AM -0600, Nasser Grainawi wrote: > >> I have a situation where I need to delete 100k+ refs on 15+ separate >> hosts/disks. This setup is using Gerrit replication, so I can trigger >> it all on one host and it will push the deletes to the rest (all >> running git-daemon v2.18.0 with receive-pack enabled). All the refs >> being deleted on the receiving ends are packed. >> >> What I see is the packed-refs file getting locked/updated over and >> over for each ref. I had assumed it would do something more like >> 'update-ref --stdin' and do a bulk removal of refs. Am I seeing the >> correct behavior? If yes, is there a specific reason it works this way >> or is "bulk delete through push" just a feature that hasn't been >> implemented yet? > > The underlying ref code is smart enough to coalesce all of the deletions > in a single transaction into a single write of the packed-refs file. > > But historically, pushes do not do a single ref transaction because we > would allow the push for one ref to succeed while others failed. Later, > we added an "atomic" mode that does it all in a single transaction. > > Try with "git push --atomic", which should be able to do it in a single > write. Thanks! Is there a way to get the bulk behavior without the all-or-nothing behavior? -- Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project ``` ## Jeff King, 2019-06-13 21:20 Subject: Re: git receive-pack deletes refs one at a time? Message-ID: <20190613212005.GA1970@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190613212005.GA1970%40sigill.intra.peff.net In-Reply-To: <34FD9B2B-D53A-4CD2-AEF0-3AFAF1586EB7@codeaurora.org> ``` On Thu, Jun 13, 2019 at 01:46:05PM -0600, Nasser Grainawi wrote: > > The underlying ref code is smart enough to coalesce all of the deletions > > in a single transaction into a single write of the packed-refs file. > > > > But historically, pushes do not do a single ref transaction because we > > would allow the push for one ref to succeed while others failed. Later, > > we added an "atomic" mode that does it all in a single transaction. > > > > Try with "git push --atomic", which should be able to do it in a single > > write. > > Thanks! > > Is there a way to get the bulk behavior without the all-or-nothing behavior? No, I don't think so. In theory the ref backend code could have a "non-atomic" flag that lets it do as much as possible of a single transaction. But nobody has ever implemented that. In practice, using "--atomic" shouldn't cause failures, unless you are trying to delete refs that somebody else is modifying at the same time. -Peff ```