{"thread":{"id":"56421","subject":"[PATCH 0/2] pack-write,repack: prevent opening packs too early","startedAt":"2021-09-01T02:05:54Z","lastAt":"2021-09-10T01:35:39Z","messageCount":81,"participants":["Taylor Blau","Jeff King","Ævar Arnfjörð Bjarmason","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"434376","messageId":"cover.1630461918.git.me@ttaylorr.com","threadId":"56421","inReplyTo":null,"subject":"[PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T02:05:46Z","receivedAt":"2021-09-01T02:05:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This pair of patches fixes a race where the .idx is moved into place\nbefore the .rev file, allowing the pack to be in a state where it\nappears a .rev file wasn't generated.\n\nThis can cause Git to inadvertently take the slow \"generate the\nreverse index on-the-fly\", which does not impact correctness, but is\nunnecessarily slow when compared to reading the .rev file.\n\nThe race is fixed by moving the .idx into place only after all other\npack-related files have already been written. The first patch fixes\nthe direct `pack-objects` case, and the second patch fixes `repack`\n(which also renames pack files around).\n\nThanks in advance for your review.\n\nTaylor Blau (2):\n pack-write.c: rename `.idx` file into place last\n builtin/repack.c: move `.idx` files into place last\n\n builtin/repack.c | 2 +-\n pack-write.c | 12 ++++++------\n 2 files changed, 7 insertions(+), 7 deletions(-)\n\n-- \n2.33.0.96.g73915697e6\n"},{"id":"434377","messageId":"ea3b1a0d8edd7b6179f305349bb316be7ff92e18.1630461918.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1630461918.git.me@ttaylorr.com","subject":"[PATCH 1/2] pack-write.c: rename `.idx` file into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T02:06:26Z","receivedAt":"2021-09-01T02:06:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We treat the presence of an `.idx` file as the indicator of whether or\nnot it's safe to use a packfile. But `finish_tmp_packfile()` (which is\nresponsible for writing the pack and moving the temporary versions of\nall of its auxiliary files into place) is inconsistent about the write\norder.\n\nBut the `.rev` file is moved into place after the `.idx`, so it's\npossible for a process to open a pack in between the two (i.e., while\nthe `.idx` file is in place but the `.rev` file is not) and mistakenly\nfall back to generating the pack's reverse index in memory. Though racy,\nthis amounts to an unnecessary slow-down at worst, and doesn't affect\nthe correctness of the resulting reverse index.\n\nClose this race by moving the .rev file into place before moving the\n.idx file into place.\n\nWhile we're here, only call strbuf_setlen() if we actually modified the\nbuffer by bringing it inside of the same if-statement that calls\nrename().\n\nSigned-off-by: Taylor Blau \n---\nThe diff is a little inscrutable here, since it shows the .idx hunk\nmoving below the .rev hunk (instead of the .rev hunk moving above the\n.idx one), but the end-result is the same.\n\n pack-write.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex f1fc3ecafa..277c60165e 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -490,18 +490,18 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n\n \tstrbuf_setlen(name_buffer, basename_len);\n\n-\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n-\tif (rename(idx_tmp_name, name_buffer->buf))\n-\t\tdie_errno(\"unable to rename temporary index file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n-\n \tif (rev_tmp_name) {\n \t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n \t\tif (rename(rev_tmp_name, name_buffer->buf))\n \t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n+\n+\t\tstrbuf_setlen(name_buffer, basename_len);\n \t}\n\n+\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n+\tif (rename(idx_tmp_name, name_buffer->buf))\n+\t\tdie_errno(\"unable to rename temporary index file\");\n+\n \tstrbuf_setlen(name_buffer, basename_len);\n\n \tfree((void *)idx_tmp_name);\n--\n2.33.0.96.g73915697e6\n\n"},{"id":"434378","messageId":"a6a4d2154e83d41c10986c5f455279ab249a910c.1630461918.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1630461918.git.me@ttaylorr.com","subject":"[PATCH 2/2] builtin/repack.c: move `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T02:06:29Z","receivedAt":"2021-09-01T02:06:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar spirit as the previous patch, fix the identical problem\nfrom `git repack` (which invokes `pack-objects` with a temporary\nlocation for output, and then moves the files into their final locations\nitself).\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5f9bc74adc..c3e4771609 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -208,10 +208,10 @@ static struct {\n \tunsigned optional:1;\n } exts[] = {\n \t{\".pack\"},\n-\t{\".idx\"},\n \t{\".rev\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n+\t{\".idx\"},\n };\n \n static unsigned populate_pack_exts(char *name)\n-- \n2.33.0.96.g73915697e6\n"},{"id":"434382","messageId":"YS75P7r33NIBmFh2@coredump.intra.peff.net","threadId":"56421","inReplyTo":"cover.1630461918.git.me@ttaylorr.com","subject":"Re: [PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T03:53:35Z","receivedAt":"2021-09-01T03:53:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 31, 2021 at 10:05:46PM -0400, Taylor Blau wrote:\n\n> This pair of patches fixes a race where the .idx is moved into place\n> before the .rev file, allowing the pack to be in a state where it\n> appears a .rev file wasn't generated.\n> \n> This can cause Git to inadvertently take the slow \"generate the\n> reverse index on-the-fly\", which does not impact correctness, but is\n> unnecessarily slow when compared to reading the .rev file.\n> \n> The race is fixed by moving the .idx into place only after all other\n> pack-related files have already been written. The first patch fixes\n> the direct `pack-objects` case, and the second patch fixes `repack`\n> (which also renames pack files around).\n\nThese both look good to me, but I think we're missing one more spot.\n\nThe first patch covers git-pack-objects directly, like:\n\n git pack-objects .git/objects/pack/pack\n\nIn practice, though, we never do that. On-disk repacks happen via\ngit-repack, which always writes to temporary files. So I thought the\ncase in git-pack-objects wouldn't matter, but it does look like\nprepare_packed_git() will actually read .tmp-$$-pack-1234abcd files, and\nso might load our temporary pack. Unexpected, but we are covered by your\nfirst patch. And then the second patch covers us as we move those\ntemporary files into place.\n\nSo far so good. But the other obvious way to get a pack idx is via\nindex-pack (especially \"--stdin\").\n\nIt looks like we'd want the same re-ordering to happen in\nbuiltin/index-pack.c:final(), which is where we rename the temporary\nfiles into place.\n\nWe _might_ also want to re-order the calls to write_idx_file() and\nwrite_rev_file() in its caller, given that simultaneous readers are\nhappy to read our tmp_pack_* files. I guess the same might apply to the\nactual file write order pack-objects, too? I'm not sure if that's even\npossible, though; do we rely on side effects of generating the .idx when\ngenerating the other meta files?\n\nI think it might be more sensible if the reading side was taught to\nignore \".tmp-*\" and \"tmp_*\" (and possibly even \".*\", though it's\npossible somebody is relying on that).\n\n-Peff\n"},{"id":"434383","messageId":"YS8BwgfurPzhT4xh@nand.local","threadId":"56421","inReplyTo":"YS75P7r33NIBmFh2@coredump.intra.peff.net","subject":"Re: [PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T04:29:54Z","receivedAt":"2021-09-01T04:30:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 31, 2021 at 11:53:35PM -0400, Jeff King wrote:\n> So far so good. But the other obvious way to get a pack idx is via\n> index-pack (especially \"--stdin\").\n>\n> It looks like we'd want the same re-ordering to happen in\n> builtin/index-pack.c:final(), which is where we rename the temporary\n> files into place.\n\nSure, that's easy enough.\n\n> We _might_ also want to re-order the calls to write_idx_file() and\n> write_rev_file() in its caller, given that simultaneous readers are\n> happy to read our tmp_pack_* files. I guess the same might apply to the\n> actual file write order pack-objects, too? I'm not sure if that's even\n> possible, though; do we rely on side effects of generating the .idx when\n> generating the other meta files?\n\nThese are a little trickier. write_idx_file() is also responsible for\nrearranging the objects array into index (name) order on the way out,\nwhich write_rev_file() depends on in order to build up its mapping.\n\nSo you could sort the array at the call-site before calling either\nfunction, but it gets awkward since there are a handful of other callers\nof write_idx_file() besides the two we're discussing.\n\nIt could be done, but it feels like the wrong approach, because...\n\n> I think it might be more sensible if the reading side was taught to\n> ignore \".tmp-*\" and \"tmp_*\" (and possibly even \".*\", though it's\n> possible somebody is relying on that).\n\n...this seems like the much-better way to go. Git shouldn't have to\nworry about what order it writes the temporary files in, only what order\nthose temporary files are made non-temporary.\n\nBut I need to do some more investigation to make sure there aren't any\nother implications. So I'm happy to wait on that, or send a new version\nof this series with a patch to fix the race in\nbuiltin/index-pack.c:final(), too.\n\n(Unrelated to your suggestions above) another consideration for \"stuff\nto do later\" would be to flip the default of pack.writeReverseIndex. I\nhad intentions to do that in the 2.32 cycle, but I forgot about it.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"434386","messageId":"YS8IqxNbxy0YrXSe@coredump.intra.peff.net","threadId":"56421","inReplyTo":"YS8BwgfurPzhT4xh@nand.local","subject":"Re: [PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T04:59:23Z","receivedAt":"2021-09-01T04:59:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 01, 2021 at 12:29:54AM -0400, Taylor Blau wrote:\n\n> > We _might_ also want to re-order the calls to write_idx_file() and\n> > write_rev_file() in its caller, given that simultaneous readers are\n> > happy to read our tmp_pack_* files. I guess the same might apply to the\n> > actual file write order pack-objects, too? I'm not sure if that's even\n> > possible, though; do we rely on side effects of generating the .idx when\n> > generating the other meta files?\n> \n> These are a little trickier. write_idx_file() is also responsible for\n> rearranging the objects array into index (name) order on the way out,\n> which write_rev_file() depends on in order to build up its mapping.\n> \n> So you could sort the array at the call-site before calling either\n> function, but it gets awkward since there are a handful of other callers\n> of write_idx_file() besides the two we're discussing.\n\nYeah, I had in the back of my mind that there was some dependency there.\nI definitely prefer the \"readers should not pick up tmp-packs\" approach\nif that is workable.\n\n> > I think it might be more sensible if the reading side was taught to\n> > ignore \".tmp-*\" and \"tmp_*\" (and possibly even \".*\", though it's\n> > possible somebody is relying on that).\n> \n> ...this seems like the much-better way to go. Git shouldn't have to\n> worry about what order it writes the temporary files in, only what order\n> those temporary files are made non-temporary.\n> \n> But I need to do some more investigation to make sure there aren't any\n> other implications. So I'm happy to wait on that, or send a new version\n> of this series with a patch to fix the race in\n> builtin/index-pack.c:final(), too.\n\nI think if we kept it restricted to \".tmp-*\" and \"tmp_*\", it should be\npretty safe. The absolute worst case is that somebody trying to recover\na corrupted repository might have to rename the files manually, I would\nthink.\n\nBlocking \".*\" is a harder sell. If we were starting from scratch, I'd\nprobably do that, but now we don't know what weird things people might\nbe doing. So unless there's a huge gain, it's hard to justify. (If we\nwere starting from scratch, I'd actually probably insist they be named\npack-$checksum.pack, etc, but it's much too late for that now).\n\nSo anyway. I think we definitely want the index-pack.c change. We\n_could_ stop there and change the read side as a separate series, but I\nthink that until we do, the ordering changes on the write side aren't\nreally doing that much. They're shrinking the race a little, I guess,\nbut it's still very much there.\n\n> (Unrelated to your suggestions above) another consideration for \"stuff\n> to do later\" would be to flip the default of pack.writeReverseIndex. I\n> had intentions to do that in the 2.32 cycle, but I forgot about it.\n\nOh, yeah. We should definitely do that (in its own series). The .rev\nfiles have been a huge performance win, and I don't think there's any\nreason we wouldn't want to always use them.\n\n-Peff\n"},{"id":"434390","messageId":"YS8LyanAqro3PDlm@nand.local","threadId":"56421","inReplyTo":"YS8IqxNbxy0YrXSe@coredump.intra.peff.net","subject":"Re: [PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T05:12:41Z","receivedAt":"2021-09-01T05:12:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 01, 2021 at 12:59:23AM -0400, Jeff King wrote:\n> So anyway. I think we definitely want the index-pack.c change. We\n> _could_ stop there and change the read side as a separate series, but I\n> think that until we do, the ordering changes on the write side aren't\n> really doing that much. They're shrinking the race a little, I guess,\n> but it's still very much there.\n\nYeah, now that I've had a chance to look at it it seems pretty\nstraightforward. Probably limited to checks in prepare_pack(), and\nadd_pack_to_midx(), which are the lone two callbacks of\nfor_each_file_in_pack_dir().\n\nI'll make sure that I'm not missing anything and then send a reroll\ntomorrow.\n\nThanks,\nTaylor\n"},{"id":"434394","messageId":"YS8Y8Ag1Ya+Sn2zg@coredump.intra.peff.net","threadId":"56421","inReplyTo":"YS8LyanAqro3PDlm@nand.local","subject":"Re: [PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-01T06:08:48Z","receivedAt":"2021-09-01T06:08:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 01, 2021 at 01:12:41AM -0400, Taylor Blau wrote:\n\n> On Wed, Sep 01, 2021 at 12:59:23AM -0400, Jeff King wrote:\n> > So anyway. I think we definitely want the index-pack.c change. We\n> > _could_ stop there and change the read side as a separate series, but I\n> > think that until we do, the ordering changes on the write side aren't\n> > really doing that much. They're shrinking the race a little, I guess,\n> > but it's still very much there.\n> \n> Yeah, now that I've had a chance to look at it it seems pretty\n> straightforward. Probably limited to checks in prepare_pack(), and\n> add_pack_to_midx(), which are the lone two callbacks of\n> for_each_file_in_pack_dir().\n\nYes, that matches what I'd expect.\n\n> I'll make sure that I'm not missing anything and then send a reroll\n> tomorrow.\n\nGreat, thanks!\n\n-Peff\n"},{"id":"434474","messageId":"YS/zRlL40SkanszX@nand.local","threadId":"56421","inReplyTo":"YS8Y8Ag1Ya+Sn2zg@coredump.intra.peff.net","subject":"Re: [PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-01T21:40:22Z","receivedAt":"2021-09-01T21:40:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 01, 2021 at 02:08:48AM -0400, Jeff King wrote:\n> On Wed, Sep 01, 2021 at 01:12:41AM -0400, Taylor Blau wrote:\n>\n> > On Wed, Sep 01, 2021 at 12:59:23AM -0400, Jeff King wrote:\n> > > So anyway. I think we definitely want the index-pack.c change. We\n> > > _could_ stop there and change the read side as a separate series, but I\n> > > think that until we do, the ordering changes on the write side aren't\n> > > really doing that much. They're shrinking the race a little, I guess,\n> > > but it's still very much there.\n> >\n> > Yeah, now that I've had a chance to look at it it seems pretty\n> > straightforward. Probably limited to checks in prepare_pack(), and\n> > add_pack_to_midx(), which are the lone two callbacks of\n> > for_each_file_in_pack_dir().\n>\n> Yes, that matches what I'd expect.\n\nHmm. As I was wondering about about, this is more complicated than meets\nthe eye. Consider t5616.36, which tests that repacking does not loosen\npromisor objects.\n\nIn builtin/repack.c:repack_promisor_object(), the repack builtin tells\npack-objects about the pack that it just wrote with `--keep-pack` (and\nwe rely on that working in order to not loosen all of the objects that\nwe just wrote).\n\nExcept when we iterate through `get_all_packs()`, we don't see the keep\npack yet, because it is still prefixed with .tmp.\n\nSo, this does get kind of tricky. There are some internal callers that\ndo want to know about .tmp packs and a whole host of other callers that\ndon't or shouldn't. Maybe that should point us towards \"we should be\nmore careful about the order we write packs in, even temporary ones\".\n\nBut alternatively, perhaps it should point us to \"this must not actually\nbe happening very often, otherwise we would have heard more complaints.\"\nSo, ultimately I'm not sure of the best way forward. I'll have to think\non it some more...\n\nThanks,\nTaylor\n"},{"id":"434892","messageId":"YTeOJTlGpXBD8mns@coredump.intra.peff.net","threadId":"56421","inReplyTo":"YS/zRlL40SkanszX@nand.local","subject":"Re: [PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-07T16:07:01Z","receivedAt":"2021-09-07T16:07:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 01, 2021 at 05:40:22PM -0400, Taylor Blau wrote:\n\n> Hmm. As I was wondering about about, this is more complicated than meets\n> the eye. Consider t5616.36, which tests that repacking does not loosen\n> promisor objects.\n> \n> In builtin/repack.c:repack_promisor_object(), the repack builtin tells\n> pack-objects about the pack that it just wrote with `--keep-pack` (and\n> we rely on that working in order to not loosen all of the objects that\n> we just wrote).\n> \n> Except when we iterate through `get_all_packs()`, we don't see the keep\n> pack yet, because it is still prefixed with .tmp.\n> \n> So, this does get kind of tricky. There are some internal callers that\n> do want to know about .tmp packs and a whole host of other callers that\n> don't or shouldn't. Maybe that should point us towards \"we should be\n> more careful about the order we write packs in, even temporary ones\".\n\nSo I happened to be looking at some packing stuff again today, and I\nrealized the situation is much less dire than I made it out to be.\n\nOn the reading side, we _won't_ look at tmp_pack_*, nor tmp_idx_*,\nbecause they don't end in '.pack' or '.idx'. So really, the only\nconfusing case is the \".tmp-$$-etc.idx\" that is generated by\npack-objects (because it thinks of that as the final name to use\nitself), and then later rearranged by git-repack.\n\nSo I think we _are_ OK as long as the correct order is observed during\nthe rename-into-place steps.\n\nI.e., your original patch (plus the extra post in index-pack) would mean\nthat we're fully covered. The .tmp-*.idx ones would appear in the\ncorrect order due to pack-objects, and the final pack-*.idx ones due to\ngit-repack.\n\n-Peff\n"},{"id":"434912","messageId":"cover-0.3-00000000000-20210907T193600Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover.1630461918.git.me@ttaylorr.com","subject":"[PATCH 0/3] rename *.idx file into place last (also after *.bitmap)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T19:42:35Z","receivedAt":"2021-09-07T19:42:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I came up with this on top of Taylor's series which fixes the order in\nwhich we write files associated with pack files[1]. His series fixes a\nrace where we write *.idx before *.rev, but left the issue of writing\n*.bitmap after *.idx, this series fixes that. Now we'll really write\nthe *.idx last.\n\nThere's still a race in these combined serieses, but now it's to do\nwith our general lack of fsync(), i.e. there's no guarantee that just\nbecause we write files A, B and C in that order that without doing the\nproper fsync() dancing on the files and the fd for the directory that\nother processes will see it that way. In practice though there is, as\nfew/no OS's are so pedantic as to expose inconsistent ordering of\nwritten files in a directory, even though POSIX et al leave that\npossibily open.\n\n1. https://lore.kernel.org/git/cover.1630461918.git.me@ttaylorr.com/\n\nÆvar Arnfjörð Bjarmason (3):\n pack-write: use more idiomatic strbuf usage for packname construction\n pack-write: split up finish_tmp_packfile() function\n pack-write: rename *.idx file into place last (really!)\n\n builtin/pack-objects.c | 33 +++++++++++++++++------\n builtin/repack.c | 8 ++++++\n pack-write.c | 60 ++++++++++++++++++++++++++++--------------\n pack.h | 16 ++++++++++-\n 4 files changed, 88 insertions(+), 29 deletions(-)\n\n-- \n2.33.0.818.gd2ef2916285\n\n"},{"id":"434913","messageId":"patch-1.3-0e6ef07ce00-20210907T193600Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover-0.3-00000000000-20210907T193600Z-avarab@gmail.com","subject":"[PATCH 1/3] pack-write: use more idiomatic strbuf usage for packname construction","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T19:42:36Z","receivedAt":"2021-09-07T19:42:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change code added in 5889271114a (finish_tmp_packfile():use strbuf for\npathname construction, 2014-03-03) to do strbuf_reset() instead of\nnoting the length of the base template, and doing a strbuf_setlen() to\nreset it, also change the spacing in the finish_tmp_packfile() so that\neach setup of the template, rename, and strbuf_reset() is grouped\ntogether.\n\nSince the prototype of the previous \"name_buffer\" now has a \"const\"\nuse this chance to wrap the overly long definition of the\nfinish_tmp_packfile() function.\n\nThis doesn't really matter for now, but as we'll see makes the\nsubsequent change much easier, as we won't need to juggle the basename\ntemplate v.s. its current contents anymore when writing bitmaps.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/pack-objects.c | 17 +++++++++++------\n pack-write.c | 27 ++++++++++++---------------\n pack.h | 7 ++++++-\n 3 files changed, 29 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex df49f656b96..717003563db 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1216,7 +1216,7 @@ static void write_pack_file(void)\n \n \t\tif (!pack_to_stdout) {\n \t\t\tstruct stat st;\n-\t\t\tstruct strbuf tmpname = STRBUF_INIT;\n+\t\t\tstruct strbuf tmp_basename = STRBUF_INIT;\n \n \t\t\t/*\n \t\t\t * Packs are runtime accessed in their mtime\n@@ -1237,7 +1237,7 @@ static void write_pack_file(void)\n \t\t\t\t\twarning_errno(_(\"failed utime() on %s\"), pack_tmp_name);\n \t\t\t}\n \n-\t\t\tstrbuf_addf(&tmpname, \"%s-\", base_name);\n+\t\t\tstrbuf_addf(&tmp_basename, \"%s-\", base_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(hash);\n@@ -1245,12 +1245,16 @@ static void write_pack_file(void)\n \t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n-\t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\n+\t\t\tfinish_tmp_packfile(&tmp_basename, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n \t\t\t\t\t &pack_idx_opts, hash);\n \n \t\t\tif (write_bitmap_index) {\n-\t\t\t\tstrbuf_addf(&tmpname, \"%s.bitmap\", hash_to_hex(hash));\n+\t\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\t\t\tstrbuf_addf(&sb, \"%s%s.bitmap\",\n+\t\t\t\t\t tmp_basename.buf,\n+\t\t\t\t\t hash_to_hex(hash));\n \n \t\t\t\tstop_progress(&progress_state);\n \n@@ -1258,11 +1262,12 @@ static void write_pack_file(void)\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n \t\t\t\tbitmap_writer_build(&to_pack);\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n-\t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n+\t\t\t\t\t\t sb.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\n+\t\t\t\tstrbuf_release(&sb);\n \t\t\t}\n \n-\t\t\tstrbuf_release(&tmpname);\n+\t\t\tstrbuf_release(&tmp_basename);\n \t\t\tfree(pack_tmp_name);\n \t\t\tputs(hash_to_hex(hash));\n \t\t}\ndiff --git a/pack-write.c b/pack-write.c\nindex 277c60165e8..57b9fc11423 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -462,15 +462,15 @@ struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n \treturn hashfd(fd, *pack_tmp_name);\n }\n \n-void finish_tmp_packfile(struct strbuf *name_buffer,\n+void finish_tmp_packfile(const struct strbuf *tmp_basename,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n \t\t\t unsigned char hash[])\n {\n+\tstruct strbuf sb = STRBUF_INIT;\n \tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n-\tint basename_len = name_buffer->len;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n@@ -483,26 +483,23 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n \t\t\t\t pack_idx_opts->flags);\n \n-\tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n-\n-\tif (rename(pack_tmp_name, name_buffer->buf))\n+\tstrbuf_addf(&sb, \"%s%s.pack\", tmp_basename->buf, hash_to_hex(hash));\n+\tif (rename(pack_tmp_name, sb.buf))\n \t\tdie_errno(\"unable to rename temporary pack file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n+\tstrbuf_reset(&sb);\n \n \tif (rev_tmp_name) {\n-\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n-\t\tif (rename(rev_tmp_name, name_buffer->buf))\n+\t\tstrbuf_addf(&sb, \"%s%s.rev\", tmp_basename->buf,\n+\t\t\t hash_to_hex(hash));\n+\t\tif (rename(rev_tmp_name, sb.buf))\n \t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n-\n-\t\tstrbuf_setlen(name_buffer, basename_len);\n+\t\tstrbuf_reset(&sb);\n \t}\n \n-\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n-\tif (rename(idx_tmp_name, name_buffer->buf))\n+\tstrbuf_addf(&sb, \"%s%s.idx\", tmp_basename->buf, hash_to_hex(hash));\n+\tif (rename(idx_tmp_name, sb.buf))\n \t\tdie_errno(\"unable to rename temporary index file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n+\tstrbuf_reset(&sb);\n \n \tfree((void *)idx_tmp_name);\n }\ndiff --git a/pack.h b/pack.h\nindex fa139545262..ae0c9e04cd9 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,6 +110,11 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n-void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n+void finish_tmp_packfile(const struct strbuf *name_buffer,\n+\t\t\t const char *pack_tmp_name,\n+\t\t\t struct pack_idx_entry **written_list,\n+\t\t\t uint32_t nr_written,\n+\t\t\t struct pack_idx_option *pack_idx_opts,\n+\t\t\t unsigned char sha1[]);\n \n #endif\n-- \n2.33.0.818.gd2ef2916285\n\n"},{"id":"434914","messageId":"patch-2.3-42f83774fe8-20210907T193600Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover-0.3-00000000000-20210907T193600Z-avarab@gmail.com","subject":"[PATCH 2/3] pack-write: split up finish_tmp_packfile() function","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T19:42:37Z","receivedAt":"2021-09-07T19:42:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the finish_tmp_packfile() function and use the split-up\nversion in pack-objects.c. This change should not change any\nfunctionality, but sets up code flow for a bug fix where we'll be able\nto move the *.idx in-place after the *.bitmap is written.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/pack-objects.c | 9 ++++++---\n pack-write.c | 39 +++++++++++++++++++++++++++++++--------\n pack.h | 9 +++++++++\n 3 files changed, 46 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 717003563db..9fa4c6a9be8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1217,6 +1217,7 @@ static void write_pack_file(void)\n \t\tif (!pack_to_stdout) {\n \t\t\tstruct stat st;\n \t\t\tstruct strbuf tmp_basename = STRBUF_INIT;\n+\t\t\tchar *idx_tmp_name = NULL;\n \n \t\t\t/*\n \t\t\t * Packs are runtime accessed in their mtime\n@@ -1245,9 +1246,10 @@ static void write_pack_file(void)\n \t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n-\t\t\tfinish_tmp_packfile(&tmp_basename, pack_tmp_name,\n-\t\t\t\t\t written_list, nr_written,\n-\t\t\t\t\t &pack_idx_opts, hash);\n+\t\t\tstage_tmp_packfile(&tmp_basename, pack_tmp_name,\n+\t\t\t\t\t written_list, nr_written,\n+\t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n+\t\t\trename_tmp_packfile_idx(&tmp_basename, hash, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tstruct strbuf sb = STRBUF_INIT;\n@@ -1267,6 +1269,7 @@ static void write_pack_file(void)\n \t\t\t\tstrbuf_release(&sb);\n \t\t\t}\n \n+\t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmp_basename);\n \t\t\tfree(pack_tmp_name);\n \t\t\tputs(hash_to_hex(hash));\ndiff --git a/pack-write.c b/pack-write.c\nindex 57b9fc11423..ba5c4767dc8 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -468,16 +468,33 @@ void finish_tmp_packfile(const struct strbuf *tmp_basename,\n \t\t\t uint32_t nr_written,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n \t\t\t unsigned char hash[])\n+{\n+\tchar *idx_tmp_name = NULL;\n+\n+\tstage_tmp_packfile(tmp_basename, pack_tmp_name, written_list,\n+\t\t\t nr_written, pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(tmp_basename, hash, &idx_tmp_name);\n+\n+\tfree(idx_tmp_name);\n+}\n+\n+void stage_tmp_packfile(const struct strbuf *tmp_basename,\n+\t\t\tconst char *pack_tmp_name,\n+\t\t\tstruct pack_idx_entry **written_list,\n+\t\t\tuint32_t nr_written,\n+\t\t\tstruct pack_idx_option *pack_idx_opts,\n+\t\t\tunsigned char hash[],\n+\t\t\tchar **idx_tmp_name)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n+\tconst char *rev_tmp_name = NULL;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n \n-\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n-\t\t\t\t pack_idx_opts, hash);\n-\tif (adjust_shared_perm(idx_tmp_name))\n+\t*idx_tmp_name = (char *)write_idx_file(NULL, written_list, nr_written,\n+\t\t\t\t\t pack_idx_opts, hash);\n+\tif (adjust_shared_perm(*idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n@@ -496,12 +513,18 @@ void finish_tmp_packfile(const struct strbuf *tmp_basename,\n \t\tstrbuf_reset(&sb);\n \t}\n \n+\tstrbuf_release(&sb);\n+}\n+\n+void rename_tmp_packfile_idx(const struct strbuf *tmp_basename,\n+\t\t\t unsigned char hash[], char **idx_tmp_name)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n \tstrbuf_addf(&sb, \"%s%s.idx\", tmp_basename->buf, hash_to_hex(hash));\n-\tif (rename(idx_tmp_name, sb.buf))\n+\tif (rename(*idx_tmp_name, sb.buf))\n \t\tdie_errno(\"unable to rename temporary index file\");\n-\tstrbuf_reset(&sb);\n-\n-\tfree((void *)idx_tmp_name);\n+\tstrbuf_release(&sb);\n }\n \n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought)\ndiff --git a/pack.h b/pack.h\nindex ae0c9e04cd9..6c1508c3de7 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,6 +110,15 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n+void stage_tmp_packfile(const struct strbuf *tmp_basename,\n+\t\t\tconst char *pack_tmp_name,\n+\t\t\tstruct pack_idx_entry **written_list,\n+\t\t\tuint32_t nr_written,\n+\t\t\tstruct pack_idx_option *pack_idx_opts,\n+\t\t\tunsigned char hash[],\n+\t\t\tchar **idx_tmp_name);\n+void rename_tmp_packfile_idx(const struct strbuf *tmp_basename,\n+\t\t\t unsigned char hash[], char **idx_tmp_name);\n void finish_tmp_packfile(const struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n-- \n2.33.0.818.gd2ef2916285\n\n"},{"id":"434915","messageId":"patch-3.3-78976fcb7b2-20210907T193600Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover-0.3-00000000000-20210907T193600Z-avarab@gmail.com","subject":"[PATCH 3/3] pack-write: rename *.idx file into place last (really!)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T19:42:38Z","receivedAt":"2021-09-07T19:42:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Follow-up a preceding commit (pack-write.c: rename `.idx` file into\nplace last, 2021-08-16)[1] and rename the *.idx file in-place after we\nwrite the *.bitmap. The preceding commit fixed the issue of *.idx\nbeing written before *.rev files, but did not do so for *.idx files.\n\nSee 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21)\nfor commentary at the time when *.bitmap was implemented about how\nthose files are written out, nothing in that commit contradicts what's\nbeing done here.\n\nWhile we're at it let's add cross-commentary to both builtin/repack.c\nand builtin/pack-objects.c to point out the two places where we write\nout these sets of files in sequence.\n\n1. https://lore.kernel.org/git/a6a4d2154e83d41c10986c5f455279ab249a910c.1630461918.git.me@ttaylorr.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/pack-objects.c | 11 ++++++++++-\n builtin/repack.c | 8 ++++++++\n 2 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 9fa4c6a9be8..fad3fe5651d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1249,7 +1249,6 @@ static void write_pack_file(void)\n \t\t\tstage_tmp_packfile(&tmp_basename, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n \t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n-\t\t\trename_tmp_packfile_idx(&tmp_basename, hash, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tstruct strbuf sb = STRBUF_INIT;\n@@ -1269,6 +1268,16 @@ static void write_pack_file(void)\n \t\t\t\tstrbuf_release(&sb);\n \t\t\t}\n \n+\t\t\t/*\n+\t\t\t * We must write the *.idx last, so that anything that expects\n+\t\t\t * an accompanying *.rev, *.bitmap etc. can count on it being\n+\t\t\t * present.\n+\t\t\t *\n+\t\t\t * See also corresponding logic in the \"exts\"\n+\t\t\t * struct in builtin/repack.c\n+\t\t\t */\n+\t\t\trename_tmp_packfile_idx(&tmp_basename, hash, &idx_tmp_name);\n+\n \t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmp_basename);\n \t\t\tfree(pack_tmp_name);\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex c3e47716093..dd438ebb3ee 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -211,6 +211,14 @@ static struct {\n \t{\".rev\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n+\t/*\n+\t * We must write the *.idx last, so that anything that expects\n+\t * an accompanying *.rev, *.bitmap etc. can count on it being\n+\t * present.\n+\t *\n+\t * See also corresponding logic in write_pack_file() in\n+\t * builtin/pack-objects.c\n+\t */\n \t{\".idx\"},\n };\n \n-- \n2.33.0.818.gd2ef2916285\n\n"},{"id":"434920","messageId":"877dfskvxc.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"cover.1630461918.git.me@ttaylorr.com","subject":"Re: [PATCH 0/2] pack-write,repack: prevent opening packs too early","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T19:48:05Z","receivedAt":"2021-09-07T19:49:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 31 2021, Taylor Blau wrote:\n\n> This pair of patches fixes a race where the .idx is moved into place\n> before the .rev file, allowing the pack to be in a state where it\n> appears a .rev file wasn't generated.\n>\n> This can cause Git to inadvertently take the slow \"generate the\n> reverse index on-the-fly\", which does not impact correctness, but is\n> unnecessarily slow when compared to reading the .rev file.\n>\n> The race is fixed by moving the .idx into place only after all other\n> pack-related files have already been written. The first patch fixes\n> the direct `pack-objects` case, and the second patch fixes `repack`\n> (which also renames pack files around).\n>\n> Thanks in advance for your review.\n>\n> Taylor Blau (2):\n> pack-write.c: rename `.idx` file into place last\n> builtin/repack.c: move `.idx` files into place last\n>\n> builtin/repack.c | 2 +-\n> pack-write.c | 12 ++++++------\n> 2 files changed, 7 insertions(+), 7 deletions(-)\n\nI've reviewed this to the point of coming up with my own series on top\nof it, but it can be considered separately:\nhttps://lore.kernel.org/git/cover-0.3-00000000000-20210907T193600Z-avarab@gmail.com/\n\nThe only change (if any) would be to perhaps update the commit\nmessage(s) to note that the *.bitmap case is left, and that the promise\nof \"closing the race\" is still subject to the vagaries of our non-use of\nfsync() here, so pedantically speaking the race is still there.\n\nBut even without those suggested changes:\n\nReviewed-by: Ævar Arnfjörð Bjarmason \n"},{"id":"434963","messageId":"YTfl7U93ttUQu2jc@nand.local","threadId":"56421","inReplyTo":"patch-1.3-0e6ef07ce00-20210907T193600Z-avarab@gmail.com","subject":"Re: [PATCH 1/3] pack-write: use more idiomatic strbuf usage for packname construction","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-07T22:21:33Z","receivedAt":"2021-09-07T22:21:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 09:42:36PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Change code added in 5889271114a (finish_tmp_packfile():use strbuf for\n\ns/Change code/Code/ ?\n\n(I wondered also if the missing space in 5889271114a's subject line was\nintentional, but it does appear in the original commit.)\n\nReading this patch, I'm not sure I agree that this makes the later\nchanges any easier. To be honest, replacing things like\n\n> \tif (rev_tmp_name) {\n> -\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n> -\t\tif (rename(rev_tmp_name, name_buffer->buf))\n> +\t\tstrbuf_addf(&sb, \"%s%s.rev\", tmp_basename->buf,\n> +\t\t\t hash_to_hex(hash));\n> +\t\tif (rename(rev_tmp_name, sb.buf))\n> \t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n> -\n> -\t\tstrbuf_setlen(name_buffer, basename_len);\n> +\t\tstrbuf_reset(&sb);\n\nDoes not much help or hurt the readability, at least in my opinion. One\nadvantage of the pre-image is that we're doing less copying, but that's\nprobably splitting hairs at this point.\n\nSo, I would probably be just as happy without this patch. You mentioned\nthat it makes the later changes easier, but I couldn't come up with why.\nI may be missing something, in which case it may be helpful to know what\nthat is and how it makes this change necessary.\n\nThanks,\nTaylor\n"},{"id":"434964","messageId":"YTfncHIiljW/E5Lz@nand.local","threadId":"56421","inReplyTo":"patch-2.3-42f83774fe8-20210907T193600Z-avarab@gmail.com","subject":"Re: [PATCH 2/3] pack-write: split up finish_tmp_packfile() function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-07T22:28:00Z","receivedAt":"2021-09-07T22:28:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 09:42:37PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Split up the finish_tmp_packfile() function and use the split-up\n> version in pack-objects.c. This change should not change any\n> functionality, but sets up code flow for a bug fix where we'll be able\n> to move the *.idx in-place after the *.bitmap is written.\n\nSeems like a logical step to make, and all makes sense to me. One\nthought for future clean-up...\n\n> diff --git a/pack-write.c b/pack-write.c\n> index 57b9fc11423..ba5c4767dc8 100644\n> --- a/pack-write.c\n> +++ b/pack-write.c\n> @@ -468,16 +468,33 @@ void finish_tmp_packfile(const struct strbuf *tmp_basename,\n> \t\t\t uint32_t nr_written,\n> \t\t\t struct pack_idx_option *pack_idx_opts,\n> \t\t\t unsigned char hash[])\n> +{\n> +\tchar *idx_tmp_name = NULL;\n> +\n> +\tstage_tmp_packfile(tmp_basename, pack_tmp_name, written_list,\n> +\t\t\t nr_written, pack_idx_opts, hash, &idx_tmp_name);\n> +\trename_tmp_packfile_idx(tmp_basename, hash, &idx_tmp_name);\n> +\n> +\tfree(idx_tmp_name);\n> +}\n> +\n\nI was surprised that you did not replace the caller in write_pack_file\nwith this (and instead open-coded the implementation). But that makes\nsense, since the very next commit is going to move the bitmap into place\nbetween staging and renaming the .idx.\n\nThe only other caller of finish_tmp_packfile is in the bulk-checkin\ncode. So I might suggest that we get rid of this function altogether and\ninstead call stage_tmp_packfile() and rename_tmp_packfile_idx() directly\nin bulk-checkin.c:finish_bulk_checkin().\n\nThat would allow us to slim down the header, but more importantly would\nmake it clear that something like finish_tmp_packfile() shouldn't be\nused blindly in case there is other work to be down between staging and\nremaing into place.\n\n> -\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n> -\t\t\t\t pack_idx_opts, hash);\n> -\tif (adjust_shared_perm(idx_tmp_name))\n> +\t*idx_tmp_name = (char *)write_idx_file(NULL, written_list, nr_written,\n> +\t\t\t\t\t pack_idx_opts, hash);\n\nYuck (and unavoidable!)\n\nThanks,\nTaylor\n"},{"id":"434965","messageId":"YTfoQJf6M5BCIT1R@nand.local","threadId":"56421","inReplyTo":"patch-3.3-78976fcb7b2-20210907T193600Z-avarab@gmail.com","subject":"Re: [PATCH 3/3] pack-write: rename *.idx file into place last (really!)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-07T22:31:28Z","receivedAt":"2021-09-07T22:31:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 09:42:38PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> +\t\t\t/*\n> +\t\t\t * We must write the *.idx last, so that anything that expects\n> +\t\t\t * an accompanying *.rev, *.bitmap etc. can count on it being\n> +\t\t\t * present.\n> +\t\t\t *\n> +\t\t\t * See also corresponding logic in the \"exts\"\n> +\t\t\t * struct in builtin/repack.c\n> +\t\t\t */\n\nTBH, I'm not sure the cycle between comments helps much, and it would\nprobably suffice to say:\n\n /*\n * write the .idx last, so everything else can count on its existence\n */\n\nIn both places. But I don't think that it makes much of a difference,\nreally, so either outcome is fine with me.\n\nThanks,\nTaylor\n"},{"id":"434966","messageId":"YTfpYsJ0J5t+xf8v@nand.local","threadId":"56421","inReplyTo":"cover-0.3-00000000000-20210907T193600Z-avarab@gmail.com","subject":"Re: [PATCH 0/3] rename *.idx file into place last (also after *.bitmap)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-07T22:36:18Z","receivedAt":"2021-09-07T22:36:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 07, 2021 at 09:42:35PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> I came up with this on top of Taylor's series which fixes the order in\n> which we write files associated with pack files[1]. His series fixes a\n> race where we write *.idx before *.rev, but left the issue of writing\n> *.bitmap after *.idx, this series fixes that. Now we'll really write\n> the *.idx last.\n\nNice catch. And fixing this race (by moving the .bitmap into place\nbefore the .idx) doesn't create another race, because\nopen_pack_bitmap_1() returns early when the call to open_pack_index()\nfails.\n\n(And furthermore, open_pack_bitmap_1() is only called on the list of\npacks generated by calling prepare_pack() over each file in\nobjects/pack, which only adds a pack to the list if its .idx file\nexists, too).\n\nI reviewed this series and left a few notes which I\nwould like to see addressed/responded to before queuing, but this seems\nto be generally of the right approach to me.\n\nThanks,\nTaylor\n"},{"id":"434968","messageId":"87pmtkj7b8.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"YTfl7U93ttUQu2jc@nand.local","subject":"Re: [PATCH 1/3] pack-write: use more idiomatic strbuf usage for packname construction","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-07T23:22:05Z","receivedAt":"2021-09-07T23:26:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Sep 07 2021, Taylor Blau wrote:\n\n> On Tue, Sep 07, 2021 at 09:42:36PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Change code added in 5889271114a (finish_tmp_packfile():use strbuf for\n>\n> s/Change code/Code/ ?\n\nThat would make it:\n\n Change code added in X (...) to do strbuf_reset() instead...\n\nInstead of:\n\n Code added in X (...) to do strbuf_reset() instead...\n\n> (I wondered also if the missing space in 5889271114a's subject line was\n> intentional, but it does appear in the original commit.)\n\n*Nod*, it was automatically generated.\n\n> Reading this patch, I'm not sure I agree that this makes the later\n> changes any easier. To be honest, replacing things like\n>\n>> \tif (rev_tmp_name) {\n>> -\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n>> -\t\tif (rename(rev_tmp_name, name_buffer->buf))\n>> +\t\tstrbuf_addf(&sb, \"%s%s.rev\", tmp_basename->buf,\n>> +\t\t\t hash_to_hex(hash));\n>> +\t\tif (rename(rev_tmp_name, sb.buf))\n>> \t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n>> -\n>> -\t\tstrbuf_setlen(name_buffer, basename_len);\n>> +\t\tstrbuf_reset(&sb);\n>\n> Does not much help or hurt the readability, at least in my opinion. One\n> advantage of the pre-image is that we're doing less copying, but that's\n> probably splitting hairs at this point.\n\n> So, I would probably be just as happy without this patch. You mentioned\n> that it makes the later changes easier, but I couldn't come up with why.\n> I may be missing something, in which case it may be helpful to know what\n> that is and how it makes this change necessary.\n\nIt's not that continually appending/trimming the strbuf is per-se less\nreadable than having a \"prefix\" and copying/appending to it, and as you\npoint out this moves us towards more allocations, but in this case\nthat's not the bottleneck.\n\nIt's that if I retain the current pattern while splitting up these\nfunctions I'd need to pass a \"basename_len\" owned by the caller between\nthe two, and we'd end up juggling the \"tmpname\" as the \"if\n(write_bitmap_index)\" codepath is moved around.\n\nSo just having each site get the prefix and add its own suffix seemed\nmuch easier to deal with,.\n"},{"id":"434974","messageId":"patch-v2-1.4-29f5787651-20210908T003631Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover-v2-0.4-0000000000-20210908T003631Z-avarab@gmail.com","subject":"[PATCH v2 1/4] pack.h: line-wrap the definition of finish_tmp_packfile()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T00:38:25Z","receivedAt":"2021-09-08T00:38:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Line-wrap the definition of finish_tmp_packfile() to make subsequent\nchanges easier to read. See 0e990530ae (finish_tmp_packfile(): a\nhelper function, 2011-10-28) for the commit that introduced this\noverly long line.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n pack.h | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/pack.h b/pack.h\nindex fa13954526..1c17254c0a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,6 +110,11 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n-void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n+void finish_tmp_packfile(struct strbuf *name_buffer,\n+\t\t\t const char *pack_tmp_name,\n+\t\t\t struct pack_idx_entry **written_list,\n+\t\t\t uint32_t nr_written,\n+\t\t\t struct pack_idx_option *pack_idx_opts,\n+\t\t\t unsigned char sha1[]);\n \n #endif\n-- \n2.33.0.819.gea1b153a43c\n\n"},{"id":"434975","messageId":"cover-v2-0.4-0000000000-20210908T003631Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover.1630461918.git.me@ttaylorr.com","subject":"[PATCH v2 0/4] rename *.idx file into place last (also after *.bitmap)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T00:38:24Z","receivedAt":"2021-09-08T00:38:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I came up with this on top of Taylor's series which fixes the order in\nwhich we write files associated with pack files[1]. His series fixes a\nrace where we write *.idx before *.rev, but left the issue of writing\n*.bitmap after *.idx, this series fixes that. Now we'll really write\nthe *.idx last.\n\nThis v2 attempts to fix the comments Taylor had on v1, in particular I\nthink the borderline readability improvements in v1 are now pretty\nunambiguously more readable in 2/4 of this series. I also converted\nthe bulk-checkin.c caller, split up the wrapping change in pack.h into\nits own commit etc.\n\n1. https://lore.kernel.org/git/cover.1630461918.git.me@ttaylorr.com/ [1]\n\nÆvar Arnfjörð Bjarmason (4):\n pack.h: line-wrap the definition of finish_tmp_packfile()\n pack-write: refactor renaming in finish_tmp_packfile()\n pack-write: split up finish_tmp_packfile() function\n pack-write: rename *.idx file into place last (really!)\n\n builtin/pack-objects.c | 16 +++++++++---\n bulk-checkin.c | 16 ++++++++++++\n pack-write.c | 59 +++++++++++++++++++++---------------------\n pack.h | 10 ++++++-\n 4 files changed, 67 insertions(+), 34 deletions(-)\n\nRange-diff against v1:\n-: ---------- > 1: 29f5787651 pack.h: line-wrap the definition of finish_tmp_packfile()\n1: 0e6ef07ce0 ! 2: 7b39f4599b pack-write: use more idiomatic strbuf usage for packname construction\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - pack-write: use more idiomatic strbuf usage for packname construction\n + pack-write: refactor renaming in finish_tmp_packfile()\n \n - Change code added in 5889271114a (finish_tmp_packfile():use strbuf for\n - pathname construction, 2014-03-03) to do strbuf_reset() instead of\n - noting the length of the base template, and doing a strbuf_setlen() to\n - reset it, also change the spacing in the finish_tmp_packfile() so that\n - each setup of the template, rename, and strbuf_reset() is grouped\n - together.\n + Refactor the renaming in finish_tmp_packfile() so that it takes a\n + \"const struct strbuf *\" instead of a non-const, and refactor the\n + repetitive renaming pattern in finish_tmp_packfile() to use a new\n + static rename_tmp_packfile() helper function.\n \n - Since the prototype of the previous \"name_buffer\" now has a \"const\"\n - use this chance to wrap the overly long definition of the\n - finish_tmp_packfile() function.\n + The previous strbuf_reset() idiom originated with\n + 5889271114a (finish_tmp_packfile():use strbuf for pathname\n + construction, 2014-03-03), which in turn was a minimal adjustment of\n + pre-strbuf code added in 0e990530ae (finish_tmp_packfile(): a helper\n + function, 2011-10-28).\n \n - This doesn't really matter for now, but as we'll see makes the\n - subsequent change much easier, as we won't need to juggle the basename\n - template v.s. its current contents anymore when writing bitmaps.\n + Since the memory allocation is not a bottleneck here we can afford a\n + bit more readability at the cost of re-allocating this new \"struct\n + strbuf sb\".\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/pack-objects.c ##\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n - \n - \t\tif (!pack_to_stdout) {\n - \t\t\tstruct stat st;\n --\t\t\tstruct strbuf tmpname = STRBUF_INIT;\n -+\t\t\tstruct strbuf tmp_basename = STRBUF_INIT;\n - \n - \t\t\t/*\n - \t\t\t * Packs are runtime accessed in their mtime\n -@@ builtin/pack-objects.c: static void write_pack_file(void)\n - \t\t\t\t\twarning_errno(_(\"failed utime() on %s\"), pack_tmp_name);\n - \t\t\t}\n - \n --\t\t\tstrbuf_addf(&tmpname, \"%s-\", base_name);\n -+\t\t\tstrbuf_addf(&tmp_basename, \"%s-\", base_name);\n - \n - \t\t\tif (write_bitmap_index) {\n - \t\t\t\tbitmap_writer_set_checksum(hash);\n -@@ builtin/pack-objects.c: static void write_pack_file(void)\n - \t\t\t\t\t&to_pack, written_list, nr_written);\n - \t\t\t}\n - \n --\t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\n -+\t\t\tfinish_tmp_packfile(&tmp_basename, pack_tmp_name,\n - \t\t\t\t\t written_list, nr_written,\n \t\t\t\t\t &pack_idx_opts, hash);\n \n \t\t\tif (write_bitmap_index) {\n -\t\t\t\tstrbuf_addf(&tmpname, \"%s.bitmap\", hash_to_hex(hash));\n +\t\t\t\tstruct strbuf sb = STRBUF_INIT;\n +\n -+\t\t\t\tstrbuf_addf(&sb, \"%s%s.bitmap\",\n -+\t\t\t\t\t tmp_basename.buf,\n ++\t\t\t\tstrbuf_addf(&sb, \"%s%s.bitmap\", tmpname.buf,\n +\t\t\t\t\t hash_to_hex(hash));\n \n \t\t\t\tstop_progress(&progress_state);\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n +\t\t\t\tstrbuf_release(&sb);\n \t\t\t}\n \n --\t\t\tstrbuf_release(&tmpname);\n -+\t\t\tstrbuf_release(&tmp_basename);\n - \t\t\tfree(pack_tmp_name);\n - \t\t\tputs(hash_to_hex(hash));\n - \t\t}\n + \t\t\tstrbuf_release(&tmpname);\n \n ## pack-write.c ##\n @@ pack-write.c: struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n @@ pack-write.c: struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n }\n \n -void finish_tmp_packfile(struct strbuf *name_buffer,\n -+void finish_tmp_packfile(const struct strbuf *tmp_basename,\n ++static void rename_tmp_packfile(const char *source,\n ++\t\t\t\t const struct strbuf *basename,\n ++\t\t\t\t const unsigned char hash[],\n ++\t\t\t\t const char *ext)\n ++{\n ++\tstruct strbuf sb = STRBUF_INIT;\n ++\n ++\tstrbuf_addf(&sb, \"%s%s.%s\", basename->buf, hash_to_hex(hash), ext);\n ++\tif (rename(source, sb.buf))\n ++\t\tdie_errno(\"unable to rename temporary '*.%s' file to '%s'\",\n ++\t\t\t ext, sb.buf);\n ++\tstrbuf_release(&sb);\n ++}\n ++\n ++void finish_tmp_packfile(const struct strbuf *basename,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n - \t\t\t struct pack_idx_option *pack_idx_opts,\n +@@ pack-write.c: void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t unsigned char hash[])\n {\n -+\tstruct strbuf sb = STRBUF_INIT;\n \tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n -\tint basename_len = name_buffer->len;\n \n @@ pack-write.c: void finish_tmp_packfile(struct strbuf *name_buffer,\n -\tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n -\n -\tif (rename(pack_tmp_name, name_buffer->buf))\n -+\tstrbuf_addf(&sb, \"%s%s.pack\", tmp_basename->buf, hash_to_hex(hash));\n -+\tif (rename(pack_tmp_name, sb.buf))\n - \t\tdie_errno(\"unable to rename temporary pack file\");\n +-\t\tdie_errno(\"unable to rename temporary pack file\");\n -\n -\tstrbuf_setlen(name_buffer, basename_len);\n -+\tstrbuf_reset(&sb);\n - \n - \tif (rev_tmp_name) {\n +-\n +-\tif (rev_tmp_name) {\n -\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n -\t\tif (rename(rev_tmp_name, name_buffer->buf))\n -+\t\tstrbuf_addf(&sb, \"%s%s.rev\", tmp_basename->buf,\n -+\t\t\t hash_to_hex(hash));\n -+\t\tif (rename(rev_tmp_name, sb.buf))\n - \t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n +-\t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n -\n -\t\tstrbuf_setlen(name_buffer, basename_len);\n -+\t\tstrbuf_reset(&sb);\n - \t}\n - \n +-\t}\n +-\n -\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n -\tif (rename(idx_tmp_name, name_buffer->buf))\n -+\tstrbuf_addf(&sb, \"%s%s.idx\", tmp_basename->buf, hash_to_hex(hash));\n -+\tif (rename(idx_tmp_name, sb.buf))\n - \t\tdie_errno(\"unable to rename temporary index file\");\n +-\t\tdie_errno(\"unable to rename temporary index file\");\n -\n -\tstrbuf_setlen(name_buffer, basename_len);\n -+\tstrbuf_reset(&sb);\n ++\trename_tmp_packfile(pack_tmp_name, basename, hash, \"pack\");\n ++\tif (rev_tmp_name)\n ++\t\trename_tmp_packfile(rev_tmp_name, basename, hash, \"rev\");\n ++\trename_tmp_packfile(idx_tmp_name, basename, hash, \"idx\");\n \n \tfree((void *)idx_tmp_name);\n }\n @@ pack.h: int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n --void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n -+void finish_tmp_packfile(const struct strbuf *name_buffer,\n -+\t\t\t const char *pack_tmp_name,\n -+\t\t\t struct pack_idx_entry **written_list,\n -+\t\t\t uint32_t nr_written,\n -+\t\t\t struct pack_idx_option *pack_idx_opts,\n -+\t\t\t unsigned char sha1[]);\n - \n - #endif\n +-void finish_tmp_packfile(struct strbuf *name_buffer,\n ++void finish_tmp_packfile(const struct strbuf *basename,\n + \t\t\t const char *pack_tmp_name,\n + \t\t\t struct pack_idx_entry **written_list,\n + \t\t\t uint32_t nr_written,\n2: 42f83774fe ! 3: 1205f9d0c2 pack-write: split up finish_tmp_packfile() function\n @@ Commit message\n pack-write: split up finish_tmp_packfile() function\n \n Split up the finish_tmp_packfile() function and use the split-up\n - version in pack-objects.c. This change should not change any\n - functionality, but sets up code flow for a bug fix where we'll be able\n - to move the *.idx in-place after the *.bitmap is written.\n + version in pack-objects.c in preparation for moving the step of\n + renaming the *.idx file later as part of a function change.\n +\n + Since the only other caller of finish_tmp_packfile() was in\n + bulk-checkin.c, and it won't be needing a change to its *.idx\n + renaming, provide a thin wrapper for the old function as a static\n + function in that file. If other callers end up needing the simpler\n + version it could be moved back to \"pack-write.c\" and \"pack.h\".\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ builtin/pack-objects.c\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\tif (!pack_to_stdout) {\n \t\t\tstruct stat st;\n - \t\t\tstruct strbuf tmp_basename = STRBUF_INIT;\n + \t\t\tstruct strbuf tmpname = STRBUF_INIT;\n +\t\t\tchar *idx_tmp_name = NULL;\n \n \t\t\t/*\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n --\t\t\tfinish_tmp_packfile(&tmp_basename, pack_tmp_name,\n --\t\t\t\t\t written_list, nr_written,\n +-\t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\n ++\t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n + \t\t\t\t\t written_list, nr_written,\n -\t\t\t\t\t &pack_idx_opts, hash);\n -+\t\t\tstage_tmp_packfile(&tmp_basename, pack_tmp_name,\n -+\t\t\t\t\t written_list, nr_written,\n -+\t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n -+\t\t\trename_tmp_packfile_idx(&tmp_basename, hash, &idx_tmp_name);\n ++\t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n ++\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tstruct strbuf sb = STRBUF_INIT;\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\t\t}\n \n +\t\t\tfree(idx_tmp_name);\n - \t\t\tstrbuf_release(&tmp_basename);\n + \t\t\tstrbuf_release(&tmpname);\n \t\t\tfree(pack_tmp_name);\n \t\t\tputs(hash_to_hex(hash));\n \n - ## pack-write.c ##\n -@@ pack-write.c: void finish_tmp_packfile(const struct strbuf *tmp_basename,\n - \t\t\t uint32_t nr_written,\n - \t\t\t struct pack_idx_option *pack_idx_opts,\n - \t\t\t unsigned char hash[])\n + ## bulk-checkin.c ##\n +@@ bulk-checkin.c: static struct bulk_checkin_state {\n + \tuint32_t nr_written;\n + } state;\n + \n ++static void finish_tmp_packfile(const struct strbuf *basename,\n ++\t\t\t\tconst char *pack_tmp_name,\n ++\t\t\t\tstruct pack_idx_entry **written_list,\n ++\t\t\t\tuint32_t nr_written,\n ++\t\t\t\tstruct pack_idx_option *pack_idx_opts,\n ++\t\t\t\tunsigned char hash[])\n +{\n +\tchar *idx_tmp_name = NULL;\n +\n -+\tstage_tmp_packfile(tmp_basename, pack_tmp_name, written_list,\n -+\t\t\t nr_written, pack_idx_opts, hash, &idx_tmp_name);\n -+\trename_tmp_packfile_idx(tmp_basename, hash, &idx_tmp_name);\n ++\tstage_tmp_packfiles(basename, pack_tmp_name, written_list, nr_written,\n ++\t\t\t pack_idx_opts, hash, &idx_tmp_name);\n ++\trename_tmp_packfile_idx(basename, hash, &idx_tmp_name);\n +\n +\tfree(idx_tmp_name);\n +}\n +\n -+void stage_tmp_packfile(const struct strbuf *tmp_basename,\n -+\t\t\tconst char *pack_tmp_name,\n -+\t\t\tstruct pack_idx_entry **written_list,\n -+\t\t\tuint32_t nr_written,\n -+\t\t\tstruct pack_idx_option *pack_idx_opts,\n -+\t\t\tunsigned char hash[],\n -+\t\t\tchar **idx_tmp_name)\n + static void finish_bulk_checkin(struct bulk_checkin_state *state)\n + {\n + \tstruct object_id oid;\n +\n + ## pack-write.c ##\n +@@ pack-write.c: static void rename_tmp_packfile(const char *source,\n + \tstrbuf_release(&sb);\n + }\n + \n +-void finish_tmp_packfile(const struct strbuf *basename,\n ++void rename_tmp_packfile_idx(const struct strbuf *basename,\n ++\t\t\t unsigned char hash[], char **idx_tmp_name)\n ++{\n ++\trename_tmp_packfile(*idx_tmp_name, basename, hash, \"idx\");\n ++}\n ++\n ++void stage_tmp_packfiles(const struct strbuf *basename,\n + \t\t\t const char *pack_tmp_name,\n + \t\t\t struct pack_idx_entry **written_list,\n + \t\t\t uint32_t nr_written,\n + \t\t\t struct pack_idx_option *pack_idx_opts,\n +-\t\t\t unsigned char hash[])\n ++\t\t\t unsigned char hash[],\n ++\t\t\t char **idx_tmp_name)\n {\n - \tstruct strbuf sb = STRBUF_INIT;\n -\tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n +\tconst char *rev_tmp_name = NULL;\n \n @@ pack-write.c: void finish_tmp_packfile(const struct strbuf *tmp_basename,\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n -@@ pack-write.c: void finish_tmp_packfile(const struct strbuf *tmp_basename,\n - \t\tstrbuf_reset(&sb);\n - \t}\n - \n -+\tstrbuf_release(&sb);\n -+}\n -+\n -+void rename_tmp_packfile_idx(const struct strbuf *tmp_basename,\n -+\t\t\t unsigned char hash[], char **idx_tmp_name)\n -+{\n -+\tstruct strbuf sb = STRBUF_INIT;\n -+\n - \tstrbuf_addf(&sb, \"%s%s.idx\", tmp_basename->buf, hash_to_hex(hash));\n --\tif (rename(idx_tmp_name, sb.buf))\n -+\tif (rename(*idx_tmp_name, sb.buf))\n - \t\tdie_errno(\"unable to rename temporary index file\");\n --\tstrbuf_reset(&sb);\n +@@ pack-write.c: void finish_tmp_packfile(const struct strbuf *basename,\n + \trename_tmp_packfile(pack_tmp_name, basename, hash, \"pack\");\n + \tif (rev_tmp_name)\n + \t\trename_tmp_packfile(rev_tmp_name, basename, hash, \"rev\");\n +-\trename_tmp_packfile(idx_tmp_name, basename, hash, \"idx\");\n -\n -\tfree((void *)idx_tmp_name);\n -+\tstrbuf_release(&sb);\n }\n \n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought)\n @@ pack.h: int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n -+void stage_tmp_packfile(const struct strbuf *tmp_basename,\n -+\t\t\tconst char *pack_tmp_name,\n -+\t\t\tstruct pack_idx_entry **written_list,\n -+\t\t\tuint32_t nr_written,\n -+\t\t\tstruct pack_idx_option *pack_idx_opts,\n -+\t\t\tunsigned char hash[],\n -+\t\t\tchar **idx_tmp_name);\n -+void rename_tmp_packfile_idx(const struct strbuf *tmp_basename,\n -+\t\t\t unsigned char hash[], char **idx_tmp_name);\n - void finish_tmp_packfile(const struct strbuf *name_buffer,\n +-void finish_tmp_packfile(const struct strbuf *basename,\n ++void stage_tmp_packfiles(const struct strbuf *basename,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n + \t\t\t uint32_t nr_written,\n + \t\t\t struct pack_idx_option *pack_idx_opts,\n +-\t\t\t unsigned char sha1[]);\n ++\t\t\t unsigned char hash[],\n ++\t\t\t char **idx_tmp_name);\n ++void rename_tmp_packfile_idx(const struct strbuf *tmp_basename,\n ++\t\t\t unsigned char hash[], char **idx_tmp_name);\n + \n + #endif\n3: 78976fcb7b ! 4: 70f4a9767d pack-write: rename *.idx file into place last (really!)\n @@ Commit message\n those files are written out, nothing in that commit contradicts what's\n being done here.\n \n - While we're at it let's add cross-commentary to both builtin/repack.c\n - and builtin/pack-objects.c to point out the two places where we write\n - out these sets of files in sequence.\n + Note that the referenced earlier commit[1] is overly optimistic about\n + \"clos[ing the] race\", i.e. yes we'll now write the files in the right\n + order, but we might still race due to our sloppy use of fsync(). See\n + the thread at [2] for a rabbit hole of various discussions about\n + filesystem races in the face of doing and not doing fsync() (and if\n + doing fsync(), not doing it properly).\n \n 1. https://lore.kernel.org/git/a6a4d2154e83d41c10986c5f455279ab249a910c.1630461918.git.me@ttaylorr.com/\n + 2. https://lore.kernel.org/git/8735qgkvv1.fsf@evledraar.gmail.com/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/pack-objects.c ##\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n - \t\t\tstage_tmp_packfile(&tmp_basename, pack_tmp_name,\n - \t\t\t\t\t written_list, nr_written,\n - \t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n --\t\t\trename_tmp_packfile_idx(&tmp_basename, hash, &idx_tmp_name);\n + \t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n + \t\t\t\t\t written_list, nr_written,\n + \t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n +-\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tstruct strbuf sb = STRBUF_INIT;\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\t\t\tstrbuf_release(&sb);\n \t\t\t}\n \n -+\t\t\t/*\n -+\t\t\t * We must write the *.idx last, so that anything that expects\n -+\t\t\t * an accompanying *.rev, *.bitmap etc. can count on it being\n -+\t\t\t * present.\n -+\t\t\t *\n -+\t\t\t * See also corresponding logic in the \"exts\"\n -+\t\t\t * struct in builtin/repack.c\n -+\t\t\t */\n -+\t\t\trename_tmp_packfile_idx(&tmp_basename, hash, &idx_tmp_name);\n ++\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n +\n \t\t\tfree(idx_tmp_name);\n - \t\t\tstrbuf_release(&tmp_basename);\n + \t\t\tstrbuf_release(&tmpname);\n \t\t\tfree(pack_tmp_name);\n -\n - ## builtin/repack.c ##\n -@@ builtin/repack.c: static struct {\n - \t{\".rev\", 1},\n - \t{\".bitmap\", 1},\n - \t{\".promisor\", 1},\n -+\t/*\n -+\t * We must write the *.idx last, so that anything that expects\n -+\t * an accompanying *.rev, *.bitmap etc. can count on it being\n -+\t * present.\n -+\t *\n -+\t * See also corresponding logic in write_pack_file() in\n -+\t * builtin/pack-objects.c\n -+\t */\n - \t{\".idx\"},\n - };\n - \n-- \n2.33.0.819.gea1b153a43c\n\n"},{"id":"434976","messageId":"patch-v2-2.4-7b39f4599b-20210908T003631Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover-v2-0.4-0000000000-20210908T003631Z-avarab@gmail.com","subject":"[PATCH v2 2/4] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T00:38:26Z","receivedAt":"2021-09-08T00:38:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Refactor the renaming in finish_tmp_packfile() so that it takes a\n\"const struct strbuf *\" instead of a non-const, and refactor the\nrepetitive renaming pattern in finish_tmp_packfile() to use a new\nstatic rename_tmp_packfile() helper function.\n\nThe previous strbuf_reset() idiom originated with\n5889271114a (finish_tmp_packfile():use strbuf for pathname\nconstruction, 2014-03-03), which in turn was a minimal adjustment of\npre-strbuf code added in 0e990530ae (finish_tmp_packfile(): a helper\nfunction, 2011-10-28).\n\nSince the memory allocation is not a bottleneck here we can afford a\nbit more readability at the cost of re-allocating this new \"struct\nstrbuf sb\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/pack-objects.c | 8 ++++++--\n pack-write.c | 41 +++++++++++++++++++----------------------\n pack.h | 2 +-\n 3 files changed, 26 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex df49f656b9..f2569b5ca2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1250,7 +1250,10 @@ static void write_pack_file(void)\n \t\t\t\t\t &pack_idx_opts, hash);\n \n \t\t\tif (write_bitmap_index) {\n-\t\t\t\tstrbuf_addf(&tmpname, \"%s.bitmap\", hash_to_hex(hash));\n+\t\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\t\t\tstrbuf_addf(&sb, \"%s%s.bitmap\", tmpname.buf,\n+\t\t\t\t\t hash_to_hex(hash));\n \n \t\t\t\tstop_progress(&progress_state);\n \n@@ -1258,8 +1261,9 @@ static void write_pack_file(void)\n \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n \t\t\t\tbitmap_writer_build(&to_pack);\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n-\t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n+\t\t\t\t\t\t sb.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\n+\t\t\t\tstrbuf_release(&sb);\n \t\t\t}\n \n \t\t\tstrbuf_release(&tmpname);\ndiff --git a/pack-write.c b/pack-write.c\nindex 277c60165e..363b395d67 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -462,7 +462,21 @@ struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n \treturn hashfd(fd, *pack_tmp_name);\n }\n \n-void finish_tmp_packfile(struct strbuf *name_buffer,\n+static void rename_tmp_packfile(const char *source,\n+\t\t\t\t const struct strbuf *basename,\n+\t\t\t\t const unsigned char hash[],\n+\t\t\t\t const char *ext)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tstrbuf_addf(&sb, \"%s%s.%s\", basename->buf, hash_to_hex(hash), ext);\n+\tif (rename(source, sb.buf))\n+\t\tdie_errno(\"unable to rename temporary '*.%s' file to '%s'\",\n+\t\t\t ext, sb.buf);\n+\tstrbuf_release(&sb);\n+}\n+\n+void finish_tmp_packfile(const struct strbuf *basename,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n@@ -470,7 +484,6 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t unsigned char hash[])\n {\n \tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n-\tint basename_len = name_buffer->len;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n@@ -483,26 +496,10 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n \t\t\t\t pack_idx_opts->flags);\n \n-\tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n-\n-\tif (rename(pack_tmp_name, name_buffer->buf))\n-\t\tdie_errno(\"unable to rename temporary pack file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n-\n-\tif (rev_tmp_name) {\n-\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n-\t\tif (rename(rev_tmp_name, name_buffer->buf))\n-\t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n-\n-\t\tstrbuf_setlen(name_buffer, basename_len);\n-\t}\n-\n-\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n-\tif (rename(idx_tmp_name, name_buffer->buf))\n-\t\tdie_errno(\"unable to rename temporary index file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n+\trename_tmp_packfile(pack_tmp_name, basename, hash, \"pack\");\n+\tif (rev_tmp_name)\n+\t\trename_tmp_packfile(rev_tmp_name, basename, hash, \"rev\");\n+\trename_tmp_packfile(idx_tmp_name, basename, hash, \"idx\");\n \n \tfree((void *)idx_tmp_name);\n }\ndiff --git a/pack.h b/pack.h\nindex 1c17254c0a..594d5176aa 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,7 +110,7 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n-void finish_tmp_packfile(struct strbuf *name_buffer,\n+void finish_tmp_packfile(const struct strbuf *basename,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n-- \n2.33.0.819.gea1b153a43c\n\n"},{"id":"434977","messageId":"patch-v2-3.4-1205f9d0c2-20210908T003631Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover-v2-0.4-0000000000-20210908T003631Z-avarab@gmail.com","subject":"[PATCH v2 3/4] pack-write: split up finish_tmp_packfile() function","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T00:38:27Z","receivedAt":"2021-09-08T00:38:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Split up the finish_tmp_packfile() function and use the split-up\nversion in pack-objects.c in preparation for moving the step of\nrenaming the *.idx file later as part of a function change.\n\nSince the only other caller of finish_tmp_packfile() was in\nbulk-checkin.c, and it won't be needing a change to its *.idx\nrenaming, provide a thin wrapper for the old function as a static\nfunction in that file. If other callers end up needing the simpler\nversion it could be moved back to \"pack-write.c\" and \"pack.h\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/pack-objects.c | 7 +++++--\n bulk-checkin.c | 16 ++++++++++++++++\n pack-write.c | 22 +++++++++++++---------\n pack.h | 7 +++++--\n 4 files changed, 39 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f2569b5ca2..33567aaa74 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1217,6 +1217,7 @@ static void write_pack_file(void)\n \t\tif (!pack_to_stdout) {\n \t\t\tstruct stat st;\n \t\t\tstruct strbuf tmpname = STRBUF_INIT;\n+\t\t\tchar *idx_tmp_name = NULL;\n \n \t\t\t/*\n \t\t\t * Packs are runtime accessed in their mtime\n@@ -1245,9 +1246,10 @@ static void write_pack_file(void)\n \t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n-\t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\n+\t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n-\t\t\t\t\t &pack_idx_opts, hash);\n+\t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n+\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tstruct strbuf sb = STRBUF_INIT;\n@@ -1266,6 +1268,7 @@ static void write_pack_file(void)\n \t\t\t\tstrbuf_release(&sb);\n \t\t\t}\n \n+\t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmpname);\n \t\t\tfree(pack_tmp_name);\n \t\t\tputs(hash_to_hex(hash));\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b023d9959a..ca7697e9f7 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -23,6 +23,22 @@ static struct bulk_checkin_state {\n \tuint32_t nr_written;\n } state;\n \n+static void finish_tmp_packfile(const struct strbuf *basename,\n+\t\t\t\tconst char *pack_tmp_name,\n+\t\t\t\tstruct pack_idx_entry **written_list,\n+\t\t\t\tuint32_t nr_written,\n+\t\t\t\tstruct pack_idx_option *pack_idx_opts,\n+\t\t\t\tunsigned char hash[])\n+{\n+\tchar *idx_tmp_name = NULL;\n+\n+\tstage_tmp_packfiles(basename, pack_tmp_name, written_list, nr_written,\n+\t\t\t pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(basename, hash, &idx_tmp_name);\n+\n+\tfree(idx_tmp_name);\n+}\n+\n static void finish_bulk_checkin(struct bulk_checkin_state *state)\n {\n \tstruct object_id oid;\ndiff --git a/pack-write.c b/pack-write.c\nindex 363b395d67..09af8fccae 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -476,21 +476,28 @@ static void rename_tmp_packfile(const char *source,\n \tstrbuf_release(&sb);\n }\n \n-void finish_tmp_packfile(const struct strbuf *basename,\n+void rename_tmp_packfile_idx(const struct strbuf *basename,\n+\t\t\t unsigned char hash[], char **idx_tmp_name)\n+{\n+\trename_tmp_packfile(*idx_tmp_name, basename, hash, \"idx\");\n+}\n+\n+void stage_tmp_packfiles(const struct strbuf *basename,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n-\t\t\t unsigned char hash[])\n+\t\t\t unsigned char hash[],\n+\t\t\t char **idx_tmp_name)\n {\n-\tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n+\tconst char *rev_tmp_name = NULL;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n \n-\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n-\t\t\t\t pack_idx_opts, hash);\n-\tif (adjust_shared_perm(idx_tmp_name))\n+\t*idx_tmp_name = (char *)write_idx_file(NULL, written_list, nr_written,\n+\t\t\t\t\t pack_idx_opts, hash);\n+\tif (adjust_shared_perm(*idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n@@ -499,9 +506,6 @@ void finish_tmp_packfile(const struct strbuf *basename,\n \trename_tmp_packfile(pack_tmp_name, basename, hash, \"pack\");\n \tif (rev_tmp_name)\n \t\trename_tmp_packfile(rev_tmp_name, basename, hash, \"rev\");\n-\trename_tmp_packfile(idx_tmp_name, basename, hash, \"idx\");\n-\n-\tfree((void *)idx_tmp_name);\n }\n \n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought)\ndiff --git a/pack.h b/pack.h\nindex 594d5176aa..d5814f3158 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,11 +110,14 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n-void finish_tmp_packfile(const struct strbuf *basename,\n+void stage_tmp_packfiles(const struct strbuf *basename,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n-\t\t\t unsigned char sha1[]);\n+\t\t\t unsigned char hash[],\n+\t\t\t char **idx_tmp_name);\n+void rename_tmp_packfile_idx(const struct strbuf *tmp_basename,\n+\t\t\t unsigned char hash[], char **idx_tmp_name);\n \n #endif\n-- \n2.33.0.819.gea1b153a43c\n\n"},{"id":"434978","messageId":"patch-v2-4.4-70f4a9767d-20210908T003631Z-avarab@gmail.com","threadId":"56421","inReplyTo":"cover-v2-0.4-0000000000-20210908T003631Z-avarab@gmail.com","subject":"[PATCH v2 4/4] pack-write: rename *.idx file into place last (really!)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T00:38:28Z","receivedAt":"2021-09-08T00:38:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Follow-up a preceding commit (pack-write.c: rename `.idx` file into\nplace last, 2021-08-16)[1] and rename the *.idx file in-place after we\nwrite the *.bitmap. The preceding commit fixed the issue of *.idx\nbeing written before *.rev files, but did not do so for *.idx files.\n\nSee 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21)\nfor commentary at the time when *.bitmap was implemented about how\nthose files are written out, nothing in that commit contradicts what's\nbeing done here.\n\nNote that the referenced earlier commit[1] is overly optimistic about\n\"clos[ing the] race\", i.e. yes we'll now write the files in the right\norder, but we might still race due to our sloppy use of fsync(). See\nthe thread at [2] for a rabbit hole of various discussions about\nfilesystem races in the face of doing and not doing fsync() (and if\ndoing fsync(), not doing it properly).\n\n1. https://lore.kernel.org/git/a6a4d2154e83d41c10986c5f455279ab249a910c.1630461918.git.me@ttaylorr.com/\n2. https://lore.kernel.org/git/8735qgkvv1.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/pack-objects.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 33567aaa74..396240c25a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1249,7 +1249,6 @@ static void write_pack_file(void)\n \t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n \t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n-\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tstruct strbuf sb = STRBUF_INIT;\n@@ -1268,6 +1267,8 @@ static void write_pack_file(void)\n \t\t\t\tstrbuf_release(&sb);\n \t\t\t}\n \n+\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n+\n \t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmpname);\n \t\t\tfree(pack_tmp_name);\n-- \n2.33.0.819.gea1b153a43c\n\n"},{"id":"434985","messageId":"87lf47kgkh.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"patch-v2-4.4-70f4a9767d-20210908T003631Z-avarab@gmail.com","subject":"Re: [PATCH v2 4/4] pack-write: rename *.idx file into place last (really!)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T01:14:12Z","receivedAt":"2021-09-08T01:21:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> Follow-up a preceding commit (pack-write.c: rename `.idx` file into\n> place last, 2021-08-16)[1] and rename the *.idx file in-place after we\n> write the *.bitmap. The preceding commit fixed the issue of *.idx\n> being written before *.rev files, but did not do so for *.idx files.\n>\n> See 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21)\n> for commentary at the time when *.bitmap was implemented about how\n> those files are written out, nothing in that commit contradicts what's\n> being done here.\n>\n> Note that the referenced earlier commit[1] is overly optimistic about\n> \"clos[ing the] race\", i.e. yes we'll now write the files in the right\n> order, but we might still race due to our sloppy use of fsync(). See\n> the thread at [2] for a rabbit hole of various discussions about\n> filesystem races in the face of doing and not doing fsync() (and if\n> doing fsync(), not doing it properly).\n\nActually I think it's a bit worse than that, we will unconditionally\nfsync() the *.pack we write out, but in stage_tmp_packfiles() (the\nbehavior pre-dates both this series and its parent, I just think my\nstage_tmp_packfiles() is easier to follow) we'll not write the *.idx\nfile with fsync() since we won't pass WRITE_IDX_VERIFY.\n\nThe same goes for *.rev (which oddly makes its fsync() conditional on\nWRITE_IDX_VERIFY), but not *.bitmap, which fsyncs unconditionally just\nlike *.pack does.\n\nAnd then of course we'll do all these in-place renames but nothing\nfsyncs the fd of the directory, so the metadata and new names being\ncommitted to disk & visible to other processes is anyone's guess.\n\nBut not only is that metadata commit not made, but due to the\ninconsistent fsync() we might end up with an *.idx that's partial and\nrenamed in-place.\n\nIn any case, any such issues pre-date this series and the series by\nTaylor it depends on, just adding some #leftoverbits for future fsync()\nfixes since I spent time looking into it.\n"},{"id":"435016","messageId":"YTg6dH6qQKFH1lnw@nand.local","threadId":"56421","inReplyTo":"patch-v2-2.4-7b39f4599b-20210908T003631Z-avarab@gmail.com","subject":"Re: [PATCH v2 2/4] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T04:22:12Z","receivedAt":"2021-09-08T04:24:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 08, 2021 at 02:38:26AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> -void finish_tmp_packfile(struct strbuf *name_buffer,\n> +static void rename_tmp_packfile(const char *source,\n> +\t\t\t\t const struct strbuf *basename,\n> +\t\t\t\t const unsigned char hash[],\n> +\t\t\t\t const char *ext)\n> +{\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\n> +\tstrbuf_addf(&sb, \"%s%s.%s\", basename->buf, hash_to_hex(hash), ext);\n> +\tif (rename(source, sb.buf))\n> +\t\tdie_errno(\"unable to rename temporary '*.%s' file to '%s'\",\n> +\t\t\t ext, sb.buf);\n> +\tstrbuf_release(&sb);\n> +}\n\nAt the end of the day, I really do not mind the extra copying of\nbasename. But if you were interested in a potential alternative, I'd\nsuggest making basename non-const and doing instead:\n\n static void rename_tmp_packfile(struct strbuf *basename, ...)\n {\n size_t basename_len = basename->len;\n strbuf_addf(basename, \".%s\", ext);\n if (rename(source, basename.buf))\n die_errno(...);\n\n strbuf_setlen(basename, basename_len);\n }\n\nwhere the contract of this function is \"I will modify the buffer you\ngave me (so do not read or write to it concurrently) but will return it\nto you in the same state as it was provided\".\n\nThat would be an improvement in readability (because of your idea to\nextract rename_tmp_packfile()) but would result in no new copying, which\nwould be nice to avoid if we can.\n\nBut I do not feel that strongly, it just seems like an unnecessary\nintroduction of copying where we don't otherwise need it.\n\nThanks,\nTaylor\n"},{"id":"435017","messageId":"YTg69VUlpJvSNvAm@nand.local","threadId":"56421","inReplyTo":"patch-v2-4.4-70f4a9767d-20210908T003631Z-avarab@gmail.com","subject":"Re: [PATCH v2 4/4] pack-write: rename *.idx file into place last (really!)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T04:24:21Z","receivedAt":"2021-09-08T04:24:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 08, 2021 at 02:38:28AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Follow-up a preceding commit (pack-write.c: rename `.idx` file into\n> place last, 2021-08-16)[1] and rename the *.idx file in-place after we\n> write the *.bitmap. The preceding commit fixed the issue of *.idx\n> being written before *.rev files, but did not do so for *.idx files.\n\ns/idx/bitmap/ in the last line, I believe.\n\nOtherwise this patch looks good to me.\n\nThanks,\nTaylor\n"},{"id":"435059","messageId":"87v93bidhn.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"87lf47kgkh.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 4/4] pack-write: rename *.idx file into place last (really!)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T09:18:19Z","receivedAt":"2021-09-08T10:11:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Wed, Sep 08 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Follow-up a preceding commit (pack-write.c: rename `.idx` file into\n>> place last, 2021-08-16)[1] and rename the *.idx file in-place after we\n>> write the *.bitmap. The preceding commit fixed the issue of *.idx\n>> being written before *.rev files, but did not do so for *.idx files.\n>>\n>> See 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21)\n>> for commentary at the time when *.bitmap was implemented about how\n>> those files are written out, nothing in that commit contradicts what's\n>> being done here.\n>>\n>> Note that the referenced earlier commit[1] is overly optimistic about\n>> \"clos[ing the] race\", i.e. yes we'll now write the files in the right\n>> order, but we might still race due to our sloppy use of fsync(). See\n>> the thread at [2] for a rabbit hole of various discussions about\n>> filesystem races in the face of doing and not doing fsync() (and if\n>> doing fsync(), not doing it properly).\n>\n> Actually I think it's a bit worse than that, we will unconditionally\n> fsync() the *.pack we write out, but in stage_tmp_packfiles() (the\n> behavior pre-dates both this series and its parent, I just think my\n> stage_tmp_packfiles() is easier to follow) we'll not write the *.idx\n> file with fsync() since we won't pass WRITE_IDX_VERIFY.\n>\n> The same goes for *.rev (which oddly makes its fsync() conditional on\n> WRITE_IDX_VERIFY), but not *.bitmap, which fsyncs unconditionally just\n> like *.pack does.\n\nI was confused here, we use fsync() with *.bitmap because there's no\n--verify mode for it, i.e. we'll always fsync() the *.rev files if we're\nactually writing them \"for real\", the non-fsync() path is when we're\ndoing the \"noop-write\" to /dev/null with \"index-pack --verify\",\ni.e. writing to an FD without an associated \"real\" filename.\n\nThat we use WRITE_IDX_VERIFY to decide on that fsync() for the *.rev\nfiles, as opposed to WRITE_REV_VERIFY is a bug in 8ef50d9958\n(pack-write.c: prepare to write 'pack-*.rev' files, 2021-01-25).\n\nIn practice if WRITE_REV_VERIFY is true then WRITE_IDX_VERIFY is true as\nwell. See e37d0b8730 (builtin/index-pack.c: write reverse indexes,\n2021-01-25) and its interaction with 68be2fea50 (receive-pack,\nfetch-pack: reject bogus pack that records objects twice,\n2011-11-16). I.e. both flags are set if --verify is there.\n\nSo it's not a bug as far as the behavior of the program goes, it just\nmakes for some very confusing code. Why should *.rev fsyncing be\ncontingent on whether or not we want to fsync *.idx files on writes? It\nshouldn't, but it turns out those two flags always go hand in hand if\nwe're writing *.rev at all.\n\nBut re the thread started at\n about how\nexactly we inspect these flags I think we should just take that change\nas-is for now, and refactor & fix this properly later.\n\nOnce write_rev_file{_order}() don't pretend to take the trie-state\nbitflags of WRITE_REV|WRITE_REV_VERIFY|WRITE_IDX_VERIFY but just take a\nboolean \"int write_file\" the whole callchain & confusion around \"do we\nhave more than 1 flag?\" goes away.\n\nI.e. something like the end-state shown by this patch, which I'm not\nsubmitting now because it conflicts with the other in-flight topic (and\nit's on top of some trivial but not-included whitespace changes in\npack.h):\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 20406f6775..69301fc044 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -778,7 +778,7 @@ static const char *create_index(void)\n \t\tdie(\"internal consistency error creating the index\");\n \n \ttmpfile = write_idx_file(NULL, idx, object_count, &pack_idx_opts,\n-\t\t\t\t pack_data->hash);\n+\t\t\t\t pack_data->hash, 0);\n \tfree(idx);\n \treturn tmpfile;\n }\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 8336466865..b9c2a21216 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1590,12 +1590,9 @@ static int git_index_pack_config(const char *k, const char *v, void *cb)\n \t\t}\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"pack.writereverseindex\")) {\n-\t\tif (git_config_bool(k, v))\n-\t\t\topts->flags |= WRITE_REV;\n-\t\telse\n-\t\t\topts->flags &= ~WRITE_REV;\n-\t}\n+\tif (!strcmp(k, \"pack.writereverseindex\"))\n+\t\topts->write_rev = git_config_bool(k, v);\n+\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -1713,7 +1710,7 @@ static void show_pack_info(int stat_only)\n \n int cmd_index_pack(int argc, const char **argv, const char *prefix)\n {\n-\tint i, fix_thin_pack = 0, verify = 0, stat_only = 0, rev_index;\n+\tint i, fix_thin_pack = 0, verify = 0, stat_only = 0;\n \tconst char *curr_index;\n \tconst char *curr_rev_index = NULL;\n \tconst char *index_name = NULL, *pack_name = NULL, *rev_index_name = NULL;\n@@ -1747,10 +1744,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (prefix && chdir(prefix))\n \t\tdie(_(\"Cannot come back to cwd\"));\n \n-\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n-\t\trev_index = 1;\n-\telse\n-\t\trev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV));\n+\topts.write_rev = git_env_bool(GIT_TEST_WRITE_REV_INDEX,\n+\t\t\t\t opts.write_rev);\n \n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\n@@ -1831,9 +1826,9 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t\tdie(_(\"unknown hash algorithm '%s'\"), arg);\n \t\t\t\trepo_set_hash_algo(the_repository, hash_algo);\n \t\t\t} else if (!strcmp(arg, \"--rev-index\")) {\n-\t\t\t\trev_index = 1;\n+\t\t\t\topts.write_rev = 1;\n \t\t\t} else if (!strcmp(arg, \"--no-rev-index\")) {\n-\t\t\t\trev_index = 0;\n+\t\t\t\topts.write_rev = 0;\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n@@ -1855,9 +1850,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (!index_name && pack_name)\n \t\tindex_name = derive_filename(pack_name, \"pack\", \"idx\", &index_name_buf);\n \n-\topts.flags &= ~(WRITE_REV | WRITE_REV_VERIFY);\n-\tif (rev_index) {\n-\t\topts.flags |= verify ? WRITE_REV_VERIFY : WRITE_REV;\n+\tif (opts.write_rev) {\n \t\tif (index_name)\n \t\t\trev_index_name = derive_filename(index_name,\n \t\t\t\t\t\t\t \"idx\", \"rev\",\n@@ -1868,10 +1861,9 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\tif (!index_name)\n \t\t\tdie(_(\"--verify with no packfile name given\"));\n \t\tread_idx_option(&opts, index_name);\n-\t\topts.flags |= WRITE_IDX_VERIFY | WRITE_IDX_STRICT;\n \t}\n \tif (strict)\n-\t\topts.flags |= WRITE_IDX_STRICT;\n+\t\topts.write_idx_strict = 1;\n \n \tif (HAVE_THREADS && !nr_threads) {\n \t\tnr_threads = online_cpus();\n@@ -1915,11 +1907,12 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tALLOC_ARRAY(idx_objects, nr_objects);\n \tfor (i = 0; i < nr_objects; i++)\n \t\tidx_objects[i] = &objects[i].idx;\n-\tcurr_index = write_idx_file(index_name, idx_objects, nr_objects, &opts, pack_hash);\n-\tif (rev_index)\n+\tcurr_index = write_idx_file(index_name, idx_objects, nr_objects, &opts,\n+\t\t\t\t pack_hash, verify);\n+\tif (opts.write_rev)\n \t\tcurr_rev_index = write_rev_file(rev_index_name, idx_objects,\n \t\t\t\t\t\tnr_objects, pack_hash,\n-\t\t\t\t\t\topts.flags);\n+\t\t\t\t\t\tverify);\n \tfree(idx_objects);\n \n \tif (!verify)\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex df49f656b9..1dae01800f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3139,10 +3139,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.writereverseindex\")) {\n-\t\tif (git_config_bool(k, v))\n-\t\t\tpack_idx_opts.flags |= WRITE_REV;\n-\t\telse\n-\t\t\tpack_idx_opts.flags &= ~WRITE_REV;\n+\t\tpack_idx_opts.write_rev = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"uploadpack.blobpackfileuri\")) {\n@@ -4030,8 +4027,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n-\tif (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0))\n-\t\tpack_idx_opts.flags |= WRITE_REV;\n+\tpack_idx_opts.write_rev = git_env_bool(GIT_TEST_WRITE_REV_INDEX,\n+\t\t\t\t\t pack_idx_opts.write_rev);\n \n \tprogress = isatty(2);\n \targc = parse_options(argc, argv, prefix, pack_objects_options,\ndiff --git a/midx.c b/midx.c\nindex 321c6fdd2f..63da740cd4 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -874,7 +874,7 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,\n \tstrbuf_addf(&buf, \"%s-%s.rev\", midx_name, hash_to_hex(midx_hash));\n \n \ttmp_file = write_rev_file_order(NULL, ctx->pack_order, ctx->entries_nr,\n-\t\t\t\t\tmidx_hash, WRITE_REV);\n+\t\t\t\t\tmidx_hash, 0);\n \n \tif (finalize_object_file(tmp_file, buf.buf))\n \t\tdie(_(\"cannot store reverse index file\"));\ndiff --git a/pack-write.c b/pack-write.c\nindex 1883848e7c..4f54191499 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -44,7 +44,7 @@ static int need_large_offset(off_t offset, const struct pack_idx_option *opts)\n */\n const char *write_idx_file(const char *index_name, struct pack_idx_entry **objects,\n \t\t\t int nr_objects, const struct pack_idx_option *opts,\n-\t\t\t const unsigned char *sha1)\n+\t\t\t const unsigned char *sha1, int verify)\n {\n \tstruct hashfile *f;\n \tstruct pack_idx_entry **sorted_by_sha, **list, **last;\n@@ -65,7 +65,7 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec\n \telse\n \t\tsorted_by_sha = list = last = NULL;\n \n-\tif (opts->flags & WRITE_IDX_VERIFY) {\n+\tif (verify) {\n \t\tassert(index_name);\n \t\tf = hashfd_check(index_name);\n \t} else {\n@@ -119,7 +119,7 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec\n \t\tif (index_version < 2)\n \t\t\thashwrite_be32(f, obj->offset);\n \t\thashwrite(f, obj->oid.hash, the_hash_algo->rawsz);\n-\t\tif ((opts->flags & WRITE_IDX_STRICT) &&\n+\t\tif (opts->write_idx_strict &&\n \t\t (i && oideq(&list[-2]->oid, &obj->oid)))\n \t\t\tdie(\"The same object %s appears twice in the pack\",\n \t\t\t oid_to_hex(&obj->oid));\n@@ -162,8 +162,7 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec\n \n \thashwrite(f, sha1, the_hash_algo->rawsz);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |\n-\t\t\t\t ((opts->flags & WRITE_IDX_VERIFY)\n-\t\t\t\t ? 0 : CSUM_FSYNC));\n+\t\t\t (verify ? 0 : CSUM_FSYNC));\n \treturn index_name;\n }\n \n@@ -218,22 +217,19 @@ const char *write_rev_file(const char *rev_name,\n \t\t\t struct pack_idx_entry **objects,\n \t\t\t uint32_t nr_objects,\n \t\t\t const unsigned char *hash,\n-\t\t\t unsigned flags)\n+\t\t\t int verify)\n {\n \tuint32_t *pack_order;\n \tuint32_t i;\n \tconst char *ret;\n \n-\tif (!(flags & WRITE_REV) && !(flags & WRITE_REV_VERIFY))\n-\t\treturn NULL;\n-\n \tALLOC_ARRAY(pack_order, nr_objects);\n \tfor (i = 0; i < nr_objects; i++)\n \t\tpack_order[i] = i;\n \tQSORT_S(pack_order, nr_objects, pack_order_cmp, objects);\n \n \tret = write_rev_file_order(rev_name, pack_order, nr_objects, hash,\n-\t\t\t\t flags);\n+\t\t\t\t verify);\n \n \tfree(pack_order);\n \n@@ -244,15 +240,12 @@ const char *write_rev_file_order(const char *rev_name,\n \t\t\t\t uint32_t *pack_order,\n \t\t\t\t uint32_t nr_objects,\n \t\t\t\t const unsigned char *hash,\n-\t\t\t\t unsigned flags)\n+\t\t\t\t int verify)\n {\n \tstruct hashfile *f;\n \tint fd;\n \n-\tif ((flags & WRITE_REV) && (flags & WRITE_REV_VERIFY))\n-\t\tdie(_(\"cannot both write and verify reverse index\"));\n-\n-\tif (flags & WRITE_REV) {\n+\tif (!verify) {\n \t\tif (!rev_name) {\n \t\t\tstruct strbuf tmp_file = STRBUF_INIT;\n \t\t\tfd = odb_mkstemp(&tmp_file, \"pack/tmp_rev_XXXXXX\");\n@@ -264,7 +257,7 @@ const char *write_rev_file_order(const char *rev_name,\n \t\t\t\tdie_errno(\"unable to create '%s'\", rev_name);\n \t\t}\n \t\tf = hashfd(fd, rev_name);\n-\t} else if (flags & WRITE_REV_VERIFY) {\n+\t} else {\n \t\tstruct stat statbuf;\n \t\tif (stat(rev_name, &statbuf)) {\n \t\t\tif (errno == ENOENT) {\n@@ -274,8 +267,7 @@ const char *write_rev_file_order(const char *rev_name,\n \t\t\t\tdie_errno(_(\"could not stat: %s\"), rev_name);\n \t\t}\n \t\tf = hashfd_check(rev_name);\n-\t} else\n-\t\treturn NULL;\n+\t}\n \n \twrite_rev_header(f);\n \n@@ -286,7 +278,7 @@ const char *write_rev_file_order(const char *rev_name,\n \t\tdie(_(\"failed to make %s readable\"), rev_name);\n \n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE |\n-\t\t\t\t ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));\n+\t\t\t (verify ? 0 : CSUM_FSYNC));\n \n \treturn rev_name;\n }\n@@ -479,12 +471,13 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n \n \tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n-\t\t\t\t pack_idx_opts, hash);\n+\t\t\t\t pack_idx_opts, hash, 0);\n \tif (adjust_shared_perm(idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n-\trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n-\t\t\t\t pack_idx_opts->flags);\n+\tif (pack_idx_opts->write_rev)\n+\t\trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n+\t\t\t\t\t 0);\n \n \tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n \ndiff --git a/pack.h b/pack.h\nindex aa23be74df..e096dea4c9 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -38,12 +38,8 @@ struct pack_header {\n #define PACK_IDX_SIGNATURE 0xff744f63\t/* \"\\377tOc\" */\n \n struct pack_idx_option {\n-\tunsigned flags;\n-\t/* flag bits */\n-#define WRITE_IDX_VERIFY 01 /* verify only, do not write the idx file */\n-#define WRITE_IDX_STRICT 02\n-#define WRITE_REV 04\n-#define WRITE_REV_VERIFY 010\n+\tunsigned int write_idx_strict:1,\n+\t\t write_rev:1;\n \n \tuint32_t version;\n \tuint32_t off32_limit;\n@@ -84,7 +80,8 @@ typedef int (*verify_fn)(const struct object_id *, enum object_type, unsigned lo\n const char *write_idx_file(const char *index_name,\n \t\t\t struct pack_idx_entry **objects, int nr_objects,\n \t\t\t const struct pack_idx_option *opts,\n-\t\t\t const unsigned char *sha1);\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int verify);\n int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, off_t offset, off_t len, unsigned int nr);\n int verify_pack_index(struct packed_git *);\n int verify_pack(struct repository *, struct packed_git *, verify_fn fn, struct progress *, uint32_t);\n@@ -99,11 +96,11 @@ void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_\n const char *write_rev_file(const char *rev_name,\n \t\t\t struct pack_idx_entry **objects,\n \t\t\t uint32_t nr_objects,\n-\t\t\t const unsigned char *hash, unsigned flags);\n+\t\t\t const unsigned char *hash, int verify);\n const char *write_rev_file_order(const char *rev_name, uint32_t *pack_order,\n \t\t\t\t uint32_t nr_objects,\n \t\t\t\t const unsigned char *hash,\n-\t\t\t\t unsigned flags);\n+\t\t\t\t int verify);\n \n /*\n * The \"hdr\" output buffer should be at least this big, which will handle sizes\n"},{"id":"435184","messageId":"cover.1631139433.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1630461918.git.me@ttaylorr.com","subject":"[PATCH v2 0/3] prevent opening packs too early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T22:17:14Z","receivedAt":"2021-09-08T22:17:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a very small reroll of a couple of patches to make sure that packs are\nnot read before all of their auxiliary files exist and are moved into place, by\nrenaming the .idx file into place last.\n\nNew since the original version is a patch to apply similar treatment to\nindex-pack, as noticed by Peff in [1]. This should be queued before Ævar's\nseries on top.\n\n[1]: https://lore.kernel.org/git/YS75P7r33NIBmFh2@coredump.intra.peff.net/\n\nTaylor Blau (3):\n pack-write.c: rename `.idx` files into place last\n builtin/repack.c: move `.idx` files into place last\n builtin/index-pack.c: move `.idx` files into place last\n\n builtin/index-pack.c | 16 ++++++++--------\n builtin/repack.c | 2 +-\n pack-write.c | 12 ++++++------\n 3 files changed, 15 insertions(+), 15 deletions(-)\n\nRange-diff against v1:\n1: ea3b1a0d8e ! 1: cb3a843994 pack-write.c: rename `.idx` file into place last\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - pack-write.c: rename `.idx` file into place last\n + pack-write.c: rename `.idx` files into place last\n \n We treat the presence of an `.idx` file as the indicator of whether or\n not it's safe to use a packfile. But `finish_tmp_packfile()` (which is\n2: a6a4d2154e = 2: 925f9ada2a builtin/repack.c: move `.idx` files into place last\n-: ---------- > 3: 460b7b9151 builtin/index-pack.c: move `.idx` files into place last\n-- \n2.33.0.96.g73915697e6\n"},{"id":"435185","messageId":"cb3a8439943635d52bc15693d71c355f96b6219a.1631139433.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631139433.git.me@ttaylorr.com","subject":"[PATCH v2 1/3] pack-write.c: rename `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T22:17:17Z","receivedAt":"2021-09-08T22:17:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We treat the presence of an `.idx` file as the indicator of whether or\nnot it's safe to use a packfile. But `finish_tmp_packfile()` (which is\nresponsible for writing the pack and moving the temporary versions of\nall of its auxiliary files into place) is inconsistent about the write\norder.\n\nBut the `.rev` file is moved into place after the `.idx`, so it's\npossible for a process to open a pack in between the two (i.e., while\nthe `.idx` file is in place but the `.rev` file is not) and mistakenly\nfall back to generating the pack's reverse index in memory. Though racy,\nthis amounts to an unnecessary slow-down at worst, and doesn't affect\nthe correctness of the resulting reverse index.\n\nClose this race by moving the .rev file into place before moving the\n.idx file into place.\n\nWhile we're here, only call strbuf_setlen() if we actually modified the\nbuffer by bringing it inside of the same if-statement that calls\nrename().\n\nSigned-off-by: Taylor Blau \n---\n pack-write.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex f1fc3ecafa..277c60165e 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -490,18 +490,18 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \n \tstrbuf_setlen(name_buffer, basename_len);\n \n-\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n-\tif (rename(idx_tmp_name, name_buffer->buf))\n-\t\tdie_errno(\"unable to rename temporary index file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n-\n \tif (rev_tmp_name) {\n \t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n \t\tif (rename(rev_tmp_name, name_buffer->buf))\n \t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n+\n+\t\tstrbuf_setlen(name_buffer, basename_len);\n \t}\n \n+\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n+\tif (rename(idx_tmp_name, name_buffer->buf))\n+\t\tdie_errno(\"unable to rename temporary index file\");\n+\n \tstrbuf_setlen(name_buffer, basename_len);\n \n \tfree((void *)idx_tmp_name);\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435186","messageId":"925f9ada2adf4712f273140ce4da0ce49e5e6c06.1631139433.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631139433.git.me@ttaylorr.com","subject":"[PATCH v2 2/3] builtin/repack.c: move `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T22:17:21Z","receivedAt":"2021-09-08T22:17:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar spirit as the previous patch, fix the identical problem\nfrom `git repack` (which invokes `pack-objects` with a temporary\nlocation for output, and then moves the files into their final locations\nitself).\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5f9bc74adc..c3e4771609 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -208,10 +208,10 @@ static struct {\n \tunsigned optional:1;\n } exts[] = {\n \t{\".pack\"},\n-\t{\".idx\"},\n \t{\".rev\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n+\t{\".idx\"},\n };\n \n static unsigned populate_pack_exts(char *name)\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435187","messageId":"460b7b915157503df2ebef61f6863ed259586385.1631139433.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631139433.git.me@ttaylorr.com","subject":"[PATCH v2 3/3] builtin/index-pack.c: move `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-08T22:17:24Z","receivedAt":"2021-09-08T22:17:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar spirit as the previous patch, fix the identical problem\nin `git index-pack`.\n\nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 8336466865..6ca47c5c03 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1517,14 +1517,6 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t} else if (from_stdin)\n \t\tchmod(final_pack_name, 0444);\n \n-\tif (final_index_name != curr_index_name) {\n-\t\tif (!final_index_name)\n-\t\t\tfinal_index_name = odb_pack_name(&index_name, hash, \"idx\");\n-\t\tif (finalize_object_file(curr_index_name, final_index_name))\n-\t\t\tdie(_(\"cannot store index file\"));\n-\t} else\n-\t\tchmod(final_index_name, 0444);\n-\n \tif (curr_rev_index_name) {\n \t\tif (final_rev_index_name != curr_rev_index_name) {\n \t\t\tif (!final_rev_index_name)\n@@ -1535,6 +1527,14 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t\tchmod(final_rev_index_name, 0444);\n \t}\n \n+\tif (final_index_name != curr_index_name) {\n+\t\tif (!final_index_name)\n+\t\t\tfinal_index_name = odb_pack_name(&index_name, hash, \"idx\");\n+\t\tif (finalize_object_file(curr_index_name, final_index_name))\n+\t\t\tdie(_(\"cannot store index file\"));\n+\t} else\n+\t\tchmod(final_index_name, 0444);\n+\n \tif (do_fsck_object) {\n \t\tstruct packed_git *p;\n \t\tp = add_packed_git(final_index_name, strlen(final_index_name), 0);\n-- \n2.33.0.96.g73915697e6\n"},{"id":"435189","messageId":"877dfqhb8n.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"cover.1631139433.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/3] prevent opening packs too early","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-08T23:52:01Z","receivedAt":"2021-09-08T23:57:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Taylor Blau wrote:\n\n> Here is a very small reroll of a couple of patches to make sure that packs are\n> not read before all of their auxiliary files exist and are moved into place, by\n> renaming the .idx file into place last.\n>\n> New since the original version is a patch to apply similar treatment to\n> index-pack, as noticed by Peff in [1]. This should be queued before Ævar's\n> series on top.\n\nI read Junio's earlier [1] as a suggestion\nthat we combine the two into a singe series. I do think that yields a\nbetter end-result, in particular your 1/3 is much more readable if the\nrefactoring in my 2/4.\n\nI'm mot of the way done with such a rewrite, which also incorporates\nyour suggestion for how to manage memory in rename_tmp_packfile(), but\nI'll hold of on what you & Junio have to say about next steps before\nadding to the re-roll competition Junio mentioned...\n\n1. https://lore.kernel.org/git/xmqq8s063m7m.fsf@gitster.g\n"},{"id":"435190","messageId":"8735qeh8h5.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"877dfqhb8n.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/3] prevent opening packs too early","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T00:50:59Z","receivedAt":"2021-09-09T00:56:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Wed, Sep 08 2021, Taylor Blau wrote:\n>\n>> Here is a very small reroll of a couple of patches to make sure that packs are\n>> not read before all of their auxiliary files exist and are moved into place, by\n>> renaming the .idx file into place last.\n>>\n>> New since the original version is a patch to apply similar treatment to\n>> index-pack, as noticed by Peff in [1]. This should be queued before Ævar's\n>> series on top.\n>\n> I read Junio's earlier [1] as a suggestion\n> that we combine the two into a singe series. I do think that yields a\n> better end-result, in particular your 1/3 is much more readable if the\n> refactoring in my 2/4.\n>\n> I'm mot of the way done with such a rewrite, which also incorporates\n> your suggestion for how to manage memory in rename_tmp_packfile(), but\n> I'll hold of on what you & Junio have to say about next steps before\n> adding to the re-roll competition Junio mentioned...\n>\n> 1. https://lore.kernel.org/git/xmqq8s063m7m.fsf@gitster.g\n\nI've got that at\nhttps://github.com/git/git/compare/master...avar:avar-tb/idx-rename-race-3\n\nRange-diff to my own v2 + your v1 follows (i.e. not my v2 + your v2\nhere). I can submit this as-is to the list, or something else. Junio\nnominated you \"team leader\", so I'll go with your plan :)\n\nI think the functional changes on top of the function refactoring make\nit much easier to see what's going on, e.g.:\n\nhttps://github.com/git/git/commit/832d9fa083d20abab35695037152dfeb1f9fe50a\n\nI took the liberty of similarly creating a helper function in\nindex-pack.c, which turns your commit there into this:\nhttps://github.com/git/git/commit/8c2dcbf6011c466a806491672c0d7308e8105a1d\n\nThe relevant commit messages were also adjusted to note that in the\nearlier *.idx v.s. *.rev we're leaving the similar *.bitmap problem for\na later change.\n\n3: 29f57876515 = 1: 4869f97408b pack.h: line-wrap the definition of finish_tmp_packfile()\n4: 7b39f4599b1 ! 2: c0b1ec90d43 pack-write: refactor renaming in finish_tmp_packfile()\n @@ Metadata\n ## Commit message ##\n pack-write: refactor renaming in finish_tmp_packfile()\n \n - Refactor the renaming in finish_tmp_packfile() so that it takes a\n - \"const struct strbuf *\" instead of a non-const, and refactor the\n - repetitive renaming pattern in finish_tmp_packfile() to use a new\n - static rename_tmp_packfile() helper function.\n + Refactor the renaming in finish_tmp_packfile() into a helper\n + function. The callers are now expected to pass a \"name_buffer\" ending\n + in \"pack-OID.\" instead of the previous \"pack-\", we then append \"pack\",\n + \"idx\" or \"rev\" to it.\n +\n + By doing the strbuf_setlen() in rename_tmp_packfile() we re-use the\n + buffer and avoid the repeated allocations we'd get if that function\n + had its own temporary \"struct strbuf\".\n +\n + This approach of re-using the buffer does make the last user in\n + pack-object.c's write_pack_file() slightly awkward, we needlessly do a\n + strbuf_setlen() there just before the strbuf_release() for\n + consistency. In subsequent changes we'll move that bitmap writing code\n + around, so let's not skip the strbuf_setlen() now.\n \n The previous strbuf_reset() idiom originated with\n 5889271114a (finish_tmp_packfile():use strbuf for pathname\n @@ Commit message\n pre-strbuf code added in 0e990530ae (finish_tmp_packfile(): a helper\n function, 2011-10-28).\n \n - Since the memory allocation is not a bottleneck here we can afford a\n - bit more readability at the cost of re-allocating this new \"struct\n - strbuf sb\".\n -\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/pack-objects.c ##\n +@@ builtin/pack-objects.c: static void write_pack_file(void)\n + \t\t\t\t\twarning_errno(_(\"failed utime() on %s\"), pack_tmp_name);\n + \t\t\t}\n + \n +-\t\t\tstrbuf_addf(&tmpname, \"%s-\", base_name);\n ++\t\t\tstrbuf_addf(&tmpname, \"%s-%s.\", base_name,\n ++\t\t\t\t hash_to_hex(hash));\n + \n + \t\t\tif (write_bitmap_index) {\n + \t\t\t\tbitmap_writer_set_checksum(hash);\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\t\t\t\t &pack_idx_opts, hash);\n \n \t\t\tif (write_bitmap_index) {\n -\t\t\t\tstrbuf_addf(&tmpname, \"%s.bitmap\", hash_to_hex(hash));\n -+\t\t\t\tstruct strbuf sb = STRBUF_INIT;\n -+\n -+\t\t\t\tstrbuf_addf(&sb, \"%s%s.bitmap\", tmpname.buf,\n -+\t\t\t\t\t hash_to_hex(hash));\n ++\t\t\t\tsize_t tmpname_len = tmpname.len;\n \n ++\t\t\t\tstrbuf_addstr(&tmpname, \"bitmap\");\n \t\t\t\tstop_progress(&progress_state);\n \n + \t\t\t\tbitmap_writer_show_progress(progress);\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n - \t\t\t\tbitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1);\n - \t\t\t\tbitmap_writer_build(&to_pack);\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n --\t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n -+\t\t\t\t\t\t sb.buf, write_bitmap_options);\n + \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\n -+\t\t\t\tstrbuf_release(&sb);\n ++\t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n \t\t\tstrbuf_release(&tmpname);\n @@ pack-write.c: struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n \treturn hashfd(fd, *pack_tmp_name);\n }\n \n --void finish_tmp_packfile(struct strbuf *name_buffer,\n -+static void rename_tmp_packfile(const char *source,\n -+\t\t\t\t const struct strbuf *basename,\n -+\t\t\t\t const unsigned char hash[],\n -+\t\t\t\t const char *ext)\n ++static void rename_tmp_packfile(struct strbuf *nb, const char *source,\n ++\t\t\t\tconst char *ext)\n +{\n -+\tstruct strbuf sb = STRBUF_INIT;\n ++\tsize_t nb_len = nb->len;\n +\n -+\tstrbuf_addf(&sb, \"%s%s.%s\", basename->buf, hash_to_hex(hash), ext);\n -+\tif (rename(source, sb.buf))\n ++\tstrbuf_addstr(nb, ext);\n ++\tif (rename(source, nb->buf))\n +\t\tdie_errno(\"unable to rename temporary '*.%s' file to '%s'\",\n -+\t\t\t ext, sb.buf);\n -+\tstrbuf_release(&sb);\n ++\t\t\t ext, nb->buf);\n ++\tstrbuf_setlen(nb, nb_len);\n +}\n +\n -+void finish_tmp_packfile(const struct strbuf *basename,\n + void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n - \t\t\t uint32_t nr_written,\n @@ pack-write.c: void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t unsigned char hash[])\n {\n @@ pack-write.c: void finish_tmp_packfile(struct strbuf *name_buffer,\n -\n -\tstrbuf_setlen(name_buffer, basename_len);\n -\n +-\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n +-\tif (rename(idx_tmp_name, name_buffer->buf))\n +-\t\tdie_errno(\"unable to rename temporary index file\");\n +-\n +-\tstrbuf_setlen(name_buffer, basename_len);\n +-\n -\tif (rev_tmp_name) {\n -\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n -\t\tif (rename(rev_tmp_name, name_buffer->buf))\n -\t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n --\n --\t\tstrbuf_setlen(name_buffer, basename_len);\n -\t}\n -\n --\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n --\tif (rename(idx_tmp_name, name_buffer->buf))\n --\t\tdie_errno(\"unable to rename temporary index file\");\n --\n -\tstrbuf_setlen(name_buffer, basename_len);\n -+\trename_tmp_packfile(pack_tmp_name, basename, hash, \"pack\");\n ++\trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n ++\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n +\tif (rev_tmp_name)\n -+\t\trename_tmp_packfile(rev_tmp_name, basename, hash, \"rev\");\n -+\trename_tmp_packfile(idx_tmp_name, basename, hash, \"idx\");\n ++\t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n \n \tfree((void *)idx_tmp_name);\n }\n -\n - ## pack.h ##\n -@@ pack.h: int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n - int read_pack_header(int fd, struct pack_header *);\n - \n - struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n --void finish_tmp_packfile(struct strbuf *name_buffer,\n -+void finish_tmp_packfile(const struct strbuf *basename,\n - \t\t\t const char *pack_tmp_name,\n - \t\t\t struct pack_idx_entry **written_list,\n - \t\t\t uint32_t nr_written,\n-: ----------- > 3: 832d9fa083d pack-write.c: rename `.idx` files after `*.rev'\n2: a6a4d2154e8 ! 4: 9f7e005ba03 builtin/repack.c: move `.idx` files into place last\n @@ Commit message\n itself).\n \n Signed-off-by: Taylor Blau \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n ## builtin/repack.c ##\n @@ builtin/repack.c: static struct {\n1: ea3b1a0d8ed ! 5: 457d4b9787c pack-write.c: rename `.idx` file into place last\n @@\n ## Metadata ##\n -Author: Taylor Blau \n +Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - pack-write.c: rename `.idx` file into place last\n + index-pack: refactor renaming in final()\n \n - We treat the presence of an `.idx` file as the indicator of whether or\n - not it's safe to use a packfile. But `finish_tmp_packfile()` (which is\n - responsible for writing the pack and moving the temporary versions of\n - all of its auxiliary files into place) is inconsistent about the write\n - order.\n + Refactor the renaming in final() into a helper function, this is\n + similar in spirit to a preceding refactoring of finish_tmp_packfile()\n + in pack-write.c.\n \n - But the `.rev` file is moved into place after the `.idx`, so it's\n - possible for a process to open a pack in between the two (i.e., while\n - the `.idx` file is in place but the `.rev` file is not) and mistakenly\n - fall back to generating the pack's reverse index in memory. Though racy,\n - this amounts to an unnecessary slow-down at worst, and doesn't affect\n - the correctness of the resulting reverse index.\n + Before e37d0b8730b (builtin/index-pack.c: write reverse indexes,\n + 2021-01-25) it probably wasn't worth it to have this sort of helper,\n + due to the differing \"else if\" case for \"pack\" files v.s. \"idx\" files.\n \n - Close this race by moving the .rev file into place before moving the\n - .idx file into place.\n + But since we've got \"rev\" as well now, let's do the renaming via a\n + helper, this is both a net decrease in lines, and improves the\n + readability, since we can easily see at a glance that the logic for\n + writing these three types of files is exactly the same, aside from the\n + obviously differing cases of \"*final_xyz_name\" being NULL, and\n + \"else_chmod_if\" being different.\n \n - While we're here, only call strbuf_setlen() if we actually modified the\n - buffer by bringing it inside of the same if-statement that calls\n - rename().\n + Signed-off-by: Ævar Arnfjörð Bjarmason \n \n - Signed-off-by: Taylor Blau \n -\n - ## pack-write.c ##\n -@@ pack-write.c: void finish_tmp_packfile(struct strbuf *name_buffer,\n + ## builtin/index-pack.c ##\n +@@ builtin/index-pack.c: static void write_special_file(const char *suffix, const char *msg,\n + \tstrbuf_release(&name_buf);\n + }\n \n - \tstrbuf_setlen(name_buffer, basename_len);\n ++static void rename_tmp_packfile(const char **final_xyz_name,\n ++\t\t\t\tconst char *curr_xyz_name,\n ++\t\t\t\tstruct strbuf *xyz_name, unsigned char *hash,\n ++\t\t\t\tconst char *ext, int else_chmod_if)\n ++{\n ++\tif (*final_xyz_name != curr_xyz_name) {\n ++\t\tif (!*final_xyz_name)\n ++\t\t\t*final_xyz_name = odb_pack_name(xyz_name, hash, ext);\n ++\t\tif (finalize_object_file(curr_xyz_name, *final_xyz_name))\n ++\t\t\tdie(_(\"unable to rename temporary '*.%s' file to '%s\"),\n ++\t\t\t ext, *final_xyz_name);\n ++\t} else if (else_chmod_if) {\n ++\t\tchmod(*final_xyz_name, 0444);\n ++\t}\n ++}\n ++\n + static void final(const char *final_pack_name, const char *curr_pack_name,\n + \t\t const char *final_index_name, const char *curr_index_name,\n + \t\t const char *final_rev_index_name, const char *curr_rev_index_name,\n +@@ builtin/index-pack.c: static void final(const char *final_pack_name, const char *curr_pack_name,\n + \t\twrite_special_file(\"promisor\", promisor_msg, final_pack_name,\n + \t\t\t\t hash, NULL);\n \n --\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n --\tif (rename(idx_tmp_name, name_buffer->buf))\n --\t\tdie_errno(\"unable to rename temporary index file\");\n +-\tif (final_pack_name != curr_pack_name) {\n +-\t\tif (!final_pack_name)\n +-\t\t\tfinal_pack_name = odb_pack_name(&pack_name, hash, \"pack\");\n +-\t\tif (finalize_object_file(curr_pack_name, final_pack_name))\n +-\t\t\tdie(_(\"cannot store pack file\"));\n +-\t} else if (from_stdin)\n +-\t\tchmod(final_pack_name, 0444);\n -\n --\tstrbuf_setlen(name_buffer, basename_len);\n +-\tif (final_index_name != curr_index_name) {\n +-\t\tif (!final_index_name)\n +-\t\t\tfinal_index_name = odb_pack_name(&index_name, hash, \"idx\");\n +-\t\tif (finalize_object_file(curr_index_name, final_index_name))\n +-\t\t\tdie(_(\"cannot store index file\"));\n +-\t} else\n +-\t\tchmod(final_index_name, 0444);\n -\n - \tif (rev_tmp_name) {\n - \t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n - \t\tif (rename(rev_tmp_name, name_buffer->buf))\n - \t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n -+\n -+\t\tstrbuf_setlen(name_buffer, basename_len);\n - \t}\n - \n -+\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n -+\tif (rename(idx_tmp_name, name_buffer->buf))\n -+\t\tdie_errno(\"unable to rename temporary index file\");\n -+\n - \tstrbuf_setlen(name_buffer, basename_len);\n +-\tif (curr_rev_index_name) {\n +-\t\tif (final_rev_index_name != curr_rev_index_name) {\n +-\t\t\tif (!final_rev_index_name)\n +-\t\t\t\tfinal_rev_index_name = odb_pack_name(&rev_index_name, hash, \"rev\");\n +-\t\t\tif (finalize_object_file(curr_rev_index_name, final_rev_index_name))\n +-\t\t\t\tdie(_(\"cannot store reverse index file\"));\n +-\t\t} else\n +-\t\t\tchmod(final_rev_index_name, 0444);\n +-\t}\n ++\trename_tmp_packfile(&final_pack_name, curr_pack_name, &pack_name,\n ++\t\t\t hash, \"pack\", from_stdin);\n ++\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n ++\t\t\t hash, \"idx\", 1);\n ++\tif (curr_rev_index_name)\n ++\t\trename_tmp_packfile(&final_rev_index_name, curr_rev_index_name,\n ++\t\t\t\t &rev_index_name, hash, \"rev\", 1);\n \n - \tfree((void *)idx_tmp_name);\n + \tif (do_fsck_object) {\n + \t\tstruct packed_git *p;\n-: ----------- > 6: 8c2dcbf6011 builtin/index-pack.c: move `.idx` files into place last\n5: 1205f9d0c25 ! 7: 3a0ee7e4a99 pack-write: split up finish_tmp_packfile() function\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\t\t\t\t written_list, nr_written,\n -\t\t\t\t\t &pack_idx_opts, hash);\n +\t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n -+\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n ++\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n - \t\t\t\tstruct strbuf sb = STRBUF_INIT;\n + \t\t\t\tsize_t tmpname_len = tmpname.len;\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n - \t\t\t\tstrbuf_release(&sb);\n + \t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n +\t\t\tfree(idx_tmp_name);\n @@ bulk-checkin.c: static struct bulk_checkin_state {\n \tuint32_t nr_written;\n } state;\n \n -+static void finish_tmp_packfile(const struct strbuf *basename,\n ++static void finish_tmp_packfile(struct strbuf *basename,\n +\t\t\t\tconst char *pack_tmp_name,\n +\t\t\t\tstruct pack_idx_entry **written_list,\n +\t\t\t\tuint32_t nr_written,\n @@ bulk-checkin.c: static struct bulk_checkin_state {\n +\n +\tstage_tmp_packfiles(basename, pack_tmp_name, written_list, nr_written,\n +\t\t\t pack_idx_opts, hash, &idx_tmp_name);\n -+\trename_tmp_packfile_idx(basename, hash, &idx_tmp_name);\n ++\trename_tmp_packfile_idx(basename, &idx_tmp_name);\n +\n +\tfree(idx_tmp_name);\n +}\n @@ bulk-checkin.c: static struct bulk_checkin_state {\n static void finish_bulk_checkin(struct bulk_checkin_state *state)\n {\n \tstruct object_id oid;\n +@@ bulk-checkin.c: static void finish_bulk_checkin(struct bulk_checkin_state *state)\n + \t\tclose(fd);\n + \t}\n + \n +-\tstrbuf_addf(&packname, \"%s/pack/pack-\", get_object_directory());\n ++\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\", get_object_directory(),\n ++\t\t oid_to_hex(&oid));\n + \tfinish_tmp_packfile(&packname, state->pack_tmp_name,\n + \t\t\t state->written, state->nr_written,\n + \t\t\t &state->pack_idx_opts, oid.hash);\n \n ## pack-write.c ##\n -@@ pack-write.c: static void rename_tmp_packfile(const char *source,\n - \tstrbuf_release(&sb);\n +@@ pack-write.c: static void rename_tmp_packfile(struct strbuf *nb, const char *source,\n + \tstrbuf_setlen(nb, nb_len);\n }\n \n --void finish_tmp_packfile(const struct strbuf *basename,\n -+void rename_tmp_packfile_idx(const struct strbuf *basename,\n -+\t\t\t unsigned char hash[], char **idx_tmp_name)\n +-void finish_tmp_packfile(struct strbuf *name_buffer,\n ++void rename_tmp_packfile_idx(struct strbuf *name_buffer,\n ++\t\t\t char **idx_tmp_name)\n +{\n -+\trename_tmp_packfile(*idx_tmp_name, basename, hash, \"idx\");\n ++\trename_tmp_packfile(name_buffer, *idx_tmp_name, \"idx\");\n +}\n +\n -+void stage_tmp_packfiles(const struct strbuf *basename,\n ++void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n @@ pack-write.c: static void rename_tmp_packfile(const char *source,\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n -@@ pack-write.c: void finish_tmp_packfile(const struct strbuf *basename,\n - \trename_tmp_packfile(pack_tmp_name, basename, hash, \"pack\");\n +@@ pack-write.c: void finish_tmp_packfile(struct strbuf *name_buffer,\n + \trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n \tif (rev_tmp_name)\n - \t\trename_tmp_packfile(rev_tmp_name, basename, hash, \"rev\");\n --\trename_tmp_packfile(idx_tmp_name, basename, hash, \"idx\");\n + \t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n +-\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n -\n -\tfree((void *)idx_tmp_name);\n }\n @@ pack.h: int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n --void finish_tmp_packfile(const struct strbuf *basename,\n -+void stage_tmp_packfiles(const struct strbuf *basename,\n +-void finish_tmp_packfile(struct strbuf *name_buffer,\n ++void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n @@ pack.h: int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n -\t\t\t unsigned char sha1[]);\n +\t\t\t unsigned char hash[],\n +\t\t\t char **idx_tmp_name);\n -+void rename_tmp_packfile_idx(const struct strbuf *tmp_basename,\n -+\t\t\t unsigned char hash[], char **idx_tmp_name);\n ++void rename_tmp_packfile_idx(struct strbuf *basename,\n ++\t\t\t char **idx_tmp_name);\n \n #endif\n6: 70f4a9767d3 ! 8: b1b232b80de pack-write: rename *.idx file into place last (really!)\n @@ Metadata\n Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - pack-write: rename *.idx file into place last (really!)\n + pack-objects: rename .idx files into place after .bitmap files\n \n - Follow-up a preceding commit (pack-write.c: rename `.idx` file into\n - place last, 2021-08-16)[1] and rename the *.idx file in-place after we\n - write the *.bitmap. The preceding commit fixed the issue of *.idx\n - being written before *.rev files, but did not do so for *.idx files.\n + In preceding commits the race of renaming .idx files in place before\n + .rev files and other auxiliary files was fixed in pack-write.c's\n + finish_tmp_packfile(), builtin/repack.c's \"struct exts\", and\n + builtin/index-pack.c's final(). As noted in the change to pack-write.c\n + we left in place the issue of writing *.bitmap files after the *.idx,\n + let's fix that issue.\n \n See 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21)\n for commentary at the time when *.bitmap was implemented about how\n those files are written out, nothing in that commit contradicts what's\n being done here.\n \n - Note that the referenced earlier commit[1] is overly optimistic about\n - \"clos[ing the] race\", i.e. yes we'll now write the files in the right\n - order, but we might still race due to our sloppy use of fsync(). See\n - the thread at [2] for a rabbit hole of various discussions about\n - filesystem races in the face of doing and not doing fsync() (and if\n - doing fsync(), not doing it properly).\n + Note that this commit and preceding ones only close any race condition\n + with *.idx files being written before their auxiliary files if we're\n + optimistic about our lack of fsync()-ing in this are not tripping us\n + over. See the thread at [1] for a rabbit hole of various discussions\n + about filesystem races in the face of doing and not doing fsync() (and\n + if doing fsync(), not doing it properly).\n \n - 1. https://lore.kernel.org/git/a6a4d2154e83d41c10986c5f455279ab249a910c.1630461918.git.me@ttaylorr.com/\n - 2. https://lore.kernel.org/git/8735qgkvv1.fsf@evledraar.gmail.com/\n + In particular, in this case of writing to \".git/objects/pack\" we only\n + write and fsync() the individual files, but if we wanted to guarantee\n + that the metadata update was seen in that way by concurrent processes\n + we'd need to fsync() on the \"fd\" of the containing directory. That\n + concern is probably more theoretical than not, modern OS's tend to be\n + more on the forgiving side than the overly pedantic side of\n + implementing POSIX FS semantics.\n +\n + 1. https://lore.kernel.org/git/8735qgkvv1.fsf@evledraar.gmail.com/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ builtin/pack-objects.c: static void write_pack_file(void)\n \t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n \t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n --\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n +-\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n - \t\t\t\tstruct strbuf sb = STRBUF_INIT;\n + \t\t\t\tsize_t tmpname_len = tmpname.len;\n @@ builtin/pack-objects.c: static void write_pack_file(void)\n - \t\t\t\tstrbuf_release(&sb);\n + \t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n -+\t\t\trename_tmp_packfile_idx(&tmpname, hash, &idx_tmp_name);\n ++\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n +\n \t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmpname);\n"},{"id":"435195","messageId":"YTlfp/tRmNy0kt+b@nand.local","threadId":"56421","inReplyTo":"8735qeh8h5.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/3] prevent opening packs too early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T01:13:11Z","receivedAt":"2021-09-09T01:13:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 02:50:59AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n> > On Wed, Sep 08 2021, Taylor Blau wrote:\n> >\n> >> Here is a very small reroll of a couple of patches to make sure that packs are\n> >> not read before all of their auxiliary files exist and are moved into place, by\n> >> renaming the .idx file into place last.\n> >>\n> >> New since the original version is a patch to apply similar treatment to\n> >> index-pack, as noticed by Peff in [1]. This should be queued before Ævar's\n> >> series on top.\n> >\n> > I read Junio's earlier [1] as a suggestion\n> > that we combine the two into a singe series. I do think that yields a\n> > better end-result, in particular your 1/3 is much more readable if the\n> > refactoring in my 2/4.\n> >\n> > I'm mot of the way done with such a rewrite, which also incorporates\n> > your suggestion for how to manage memory in rename_tmp_packfile(), but\n> > I'll hold of on what you & Junio have to say about next steps before\n> > adding to the re-roll competition Junio mentioned...\n> >\n> > 1. https://lore.kernel.org/git/xmqq8s063m7m.fsf@gitster.g\n>\n> I've got that at\n> https://github.com/git/git/compare/master...avar:avar-tb/idx-rename-race-3\n\nBeautiful :-).\n\nI mentioned in my response to [1] that I missed that message when\nsending v2 of my series to fix a couple of these races. And I was even\nhappy to unify our two series, but you did all of the work for me while\nI was eating dinner. Thank you!\n\nI fetched your branch, reviewed, and am happy with the result. So I\nwould be content to apply my s-o-b to your patches and resubmit them as\na unified series.\n\nBut I did wonder if you wanted to include [2] in this series as well.\nIt's kind of different, but also related enough that it probably makes\nsense to just pull it in. So I'm inclined to just do that, unless you\nhave any objections.\n\nThanks,\nTaylor\n\n[2]: https://lore.kernel.org/git/patch-1.1-366ba928bd-20210908T010743Z-avarab@gmail.com/\n"},{"id":"435198","messageId":"87y286fs3v.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"YTlfp/tRmNy0kt+b@nand.local","subject":"Re: [PATCH v2 0/3] prevent opening packs too early","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T01:33:02Z","receivedAt":"2021-09-09T01:35:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Taylor Blau wrote:\n\n> On Thu, Sep 09, 2021 at 02:50:59AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> > On Wed, Sep 08 2021, Taylor Blau wrote:\n>> >\n>> >> Here is a very small reroll of a couple of patches to make sure that packs are\n>> >> not read before all of their auxiliary files exist and are moved into place, by\n>> >> renaming the .idx file into place last.\n>> >>\n>> >> New since the original version is a patch to apply similar treatment to\n>> >> index-pack, as noticed by Peff in [1]. This should be queued before Ævar's\n>> >> series on top.\n>> >\n>> > I read Junio's earlier [1] as a suggestion\n>> > that we combine the two into a singe series. I do think that yields a\n>> > better end-result, in particular your 1/3 is much more readable if the\n>> > refactoring in my 2/4.\n>> >\n>> > I'm mot of the way done with such a rewrite, which also incorporates\n>> > your suggestion for how to manage memory in rename_tmp_packfile(), but\n>> > I'll hold of on what you & Junio have to say about next steps before\n>> > adding to the re-roll competition Junio mentioned...\n>> >\n>> > 1. https://lore.kernel.org/git/xmqq8s063m7m.fsf@gitster.g\n>>\n>> I've got that at\n>> https://github.com/git/git/compare/master...avar:avar-tb/idx-rename-race-3\n>\n> Beautiful :-).\n>\n> I mentioned in my response to [1] that I missed that message when\n> sending v2 of my series to fix a couple of these races. And I was even\n> happy to unify our two series, but you did all of the work for me while\n> I was eating dinner. Thank you!\n>\n> I fetched your branch, reviewed, and am happy with the result. So I\n> would be content to apply my s-o-b to your patches and resubmit them as\n> a unified series.\n\nSounds good to me, also in particular any typo/grammar etc. fixes to my\nchanges are most welcome, I tend to sneak those in, and any code changes\nyou see fit of course.\n\n> But I did wonder if you wanted to include [2] in this series as well.\n> It's kind of different, but also related enough that it probably makes\n> sense to just pull it in. So I'm inclined to just do that, unless you\n> have any objections.\n\nIn this latest push-out of \"next\" Junio's merged that one down already,\nsee 1972c5931bb (Merge branch 'ab/reverse-midx-optim' into next,\n2021-09-08).\n\nSo I think at this point it could be built on top of that, but given\nthat the two don't conflict in any way (textually or semantically) it's\nprobably better to base this larger topic on \"master\" and proceed with\nthem independently.\n\n> [2]: https://lore.kernel.org/git/patch-1.1-366ba928bd-20210908T010743Z-avarab@gmail.com/\n\n"},{"id":"435200","messageId":"87pmtifp2e.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"87y286fs3v.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/3] prevent opening packs too early","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T02:36:17Z","receivedAt":"2021-09-09T02:41:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Wed, Sep 08 2021, Taylor Blau wrote:\n>\n>> On Thu, Sep 09, 2021 at 02:50:59AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> On Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> > On Wed, Sep 08 2021, Taylor Blau wrote:\n>>> >\n>>> >> Here is a very small reroll of a couple of patches to make sure that packs are\n>>> >> not read before all of their auxiliary files exist and are moved into place, by\n>>> >> renaming the .idx file into place last.\n>>> >>\n>>> >> New since the original version is a patch to apply similar treatment to\n>>> >> index-pack, as noticed by Peff in [1]. This should be queued before Ævar's\n>>> >> series on top.\n>>> >\n>>> > I read Junio's earlier [1] as a suggestion\n>>> > that we combine the two into a singe series. I do think that yields a\n>>> > better end-result, in particular your 1/3 is much more readable if the\n>>> > refactoring in my 2/4.\n>>> >\n>>> > I'm mot of the way done with such a rewrite, which also incorporates\n>>> > your suggestion for how to manage memory in rename_tmp_packfile(), but\n>>> > I'll hold of on what you & Junio have to say about next steps before\n>>> > adding to the re-roll competition Junio mentioned...\n>>> >\n>>> > 1. https://lore.kernel.org/git/xmqq8s063m7m.fsf@gitster.g\n>>>\n>>> I've got that at\n>>> https://github.com/git/git/compare/master...avar:avar-tb/idx-rename-race-3\n>>\n>> Beautiful :-).\n>>\n>> I mentioned in my response to [1] that I missed that message when\n>> sending v2 of my series to fix a couple of these races. And I was even\n>> happy to unify our two series, but you did all of the work for me while\n>> I was eating dinner. Thank you!\n>>\n>> I fetched your branch, reviewed, and am happy with the result. So I\n>> would be content to apply my s-o-b to your patches and resubmit them as\n>> a unified series.\n>\n> Sounds good to me, also in particular any typo/grammar etc. fixes to my\n> changes are most welcome, I tend to sneak those in, and any code changes\n> you see fit of course.\n>\n>> But I did wonder if you wanted to include [2] in this series as well.\n>> It's kind of different, but also related enough that it probably makes\n>> sense to just pull it in. So I'm inclined to just do that, unless you\n>> have any objections.\n>\n> In this latest push-out of \"next\" Junio's merged that one down already,\n> see 1972c5931bb (Merge branch 'ab/reverse-midx-optim' into next,\n> 2021-09-08).\n>\n> So I think at this point it could be built on top of that, but given\n> that the two don't conflict in any way (textually or semantically) it's\n> probably better to base this larger topic on \"master\" and proceed with\n> them independently.\n>\n>> [2]: https://lore.kernel.org/git/patch-1.1-366ba928bd-20210908T010743Z-avarab@gmail.com/\n\nAlso: That avar-tb/idx-rename-race-3 has a bug where I didn't update the\nbulk-checkin.c code accordingly, missed it because I was running a glob\nof tests that included \"*pack*,*rev*\", but missed \"*large*\". I've got a\navar-tb/idx-rename-race-4 which fixes it. Sorry about that.\n\nhttps://github.com/git/git/compare/master...avar:avar-tb/idx-rename-race-4\n"},{"id":"435201","messageId":"YTl2Vn6eB2MyXehM@nand.local","threadId":"56421","inReplyTo":"87pmtifp2e.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/3] prevent opening packs too early","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T02:49:58Z","receivedAt":"2021-09-09T02:50:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 04:36:17AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Also: That avar-tb/idx-rename-race-3 has a bug where I didn't update the\n> bulk-checkin.c code accordingly, missed it because I was running a glob\n> of tests that included \"*pack*,*rev*\", but missed \"*large*\". I've got a\n> avar-tb/idx-rename-race-4 which fixes it. Sorry about that.\n>\n> https://github.com/git/git/compare/master...avar:avar-tb/idx-rename-race-4\n\nYep, I noticed the same thing while running make test on the series\nbefore sending it out. Your fix matches mine (which is to move the\nchange from 7/8 to 2/8).\n\nWhile I was fixing the second patch, I inserted a new one just before it\nto store the hash in a `unsigned char hash[GIT_MAX_RAWSZ]` instead of a\n`struct object_id`.\n\nThanks,\nTaylor\n"},{"id":"435203","messageId":"cover.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1630461918.git.me@ttaylorr.com","subject":"[PATCH 0/9] packfile: avoid .idx rename races","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:24:43Z","receivedAt":"2021-09-09T03:24:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series is a unification of [1] and [2] which together prevent a handful of\nraces when code that writes packfiles moves the `.idx` into place before all\nother auxiliary files are properly renamed.\n\nThis can lead to races like not reading the `.rev` file even if one was\ngenerated and is about to be moved into place.\n\nCredit goes to Ævar for preparing what is more-or-less sent here. I polished a\nfew of the commit messages, and added the second patch on top of his result. It\nisn't necessary, but felt like good hygiene when I was reading the surrounding\ncode.\n\nThanks in advance for reviewing.\n\n[1]: https://lore.kernel.org/git/cover.1631139433.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/cover-0.3-00000000000-20210907T193600Z-avarab@gmail.com/\n\nTaylor Blau (4):\n bulk-checkin.c: store checksum directly\n pack-write.c: rename `.idx` files after `*.rev`\n builtin/repack.c: move `.idx` files into place last\n builtin/index-pack.c: move `.idx` files into place last\n\nÆvar Arnfjörð Bjarmason (5):\n pack.h: line-wrap the definition of finish_tmp_packfile()\n pack-write: refactor renaming in finish_tmp_packfile()\n index-pack: refactor renaming in final()\n pack-write: split up finish_tmp_packfile() function\n pack-objects: rename .idx files into place after .bitmap files\n\n builtin/index-pack.c | 48 +++++++++++++++++------------------\n builtin/pack-objects.c | 15 ++++++++---\n builtin/repack.c | 2 +-\n bulk-checkin.c | 31 +++++++++++++++++------\n pack-write.c | 57 +++++++++++++++++++++---------------------\n pack.h | 10 +++++++-\n 6 files changed, 96 insertions(+), 67 deletions(-)\n\n-- \n2.33.0.96.g73915697e6\n"},{"id":"435204","messageId":"0b07aa4947caff53878a9000e2866f98146e2ccd.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 1/9] pack.h: line-wrap the definition of finish_tmp_packfile()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:24:47Z","receivedAt":"2021-09-09T03:24:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nLine-wrap the definition of finish_tmp_packfile() to make subsequent\nchanges easier to read. See 0e990530ae (finish_tmp_packfile(): a\nhelper function, 2011-10-28) for the commit that introduced this\noverly long line.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n pack.h | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/pack.h b/pack.h\nindex fa13954526..1c17254c0a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,6 +110,11 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n-void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n+void finish_tmp_packfile(struct strbuf *name_buffer,\n+\t\t\t const char *pack_tmp_name,\n+\t\t\t struct pack_idx_entry **written_list,\n+\t\t\t uint32_t nr_written,\n+\t\t\t struct pack_idx_option *pack_idx_opts,\n+\t\t\t unsigned char sha1[]);\n \n #endif\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435205","messageId":"20b35ce0505d7bcc84b6fd68a33e0b6b1afa31e6.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 2/9] bulk-checkin.c: store checksum directly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:24:51Z","receivedAt":"2021-09-09T03:25:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"finish_bulk_checkin() stores the checksum from finalize_hashfile() by\nwriting to the `hash` member of `struct object_id`, but that hash has\nnothing to do with an object id (it's just a convenient location that\nhappens to be sized correctly).\n\nStore the hash directly in an unsigned char array. This behaves the same\nas writing to the `hash` member, but makes the intent clearer (and\navoids allocating an extra four bytes for the `algo` member of `struct\nobject_id`).\n\nSigned-off-by: Taylor Blau \n---\n bulk-checkin.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b023d9959a..6283bc8bd9 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -25,7 +25,7 @@ static struct bulk_checkin_state {\n \n static void finish_bulk_checkin(struct bulk_checkin_state *state)\n {\n-\tstruct object_id oid;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct strbuf packname = STRBUF_INIT;\n \tint i;\n \n@@ -37,11 +37,11 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \t\tunlink(state->pack_tmp_name);\n \t\tgoto clear_exit;\n \t} else if (state->nr_written == 1) {\n-\t\tfinalize_hashfile(state->f, oid.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n+\t\tfinalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n \t} else {\n-\t\tint fd = finalize_hashfile(state->f, oid.hash, 0);\n-\t\tfixup_pack_header_footer(fd, oid.hash, state->pack_tmp_name,\n-\t\t\t\t\t state->nr_written, oid.hash,\n+\t\tint fd = finalize_hashfile(state->f, hash, 0);\n+\t\tfixup_pack_header_footer(fd, hash, state->pack_tmp_name,\n+\t\t\t\t\t state->nr_written, hash,\n \t\t\t\t\t state->offset);\n \t\tclose(fd);\n \t}\n@@ -49,7 +49,7 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \tstrbuf_addf(&packname, \"%s/pack/pack-\", get_object_directory());\n \tfinish_tmp_packfile(&packname, state->pack_tmp_name,\n \t\t\t state->written, state->nr_written,\n-\t\t\t &state->pack_idx_opts, oid.hash);\n+\t\t\t &state->pack_idx_opts, hash);\n \tfor (i = 0; i < state->nr_written; i++)\n \t\tfree(state->written[i]);\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435206","messageId":"35052ef494dbc55119614f3e22742d8d814b21b1.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 3/9] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:24:55Z","receivedAt":"2021-09-09T03:25:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nRefactor the renaming in finish_tmp_packfile() into a helper function.\nThe callers are now expected to pass a \"name_buffer\" ending in\n\"pack-OID.\" instead of the previous \"pack-\", we then append \"pack\",\n\"idx\" or \"rev\" to it.\n\nBy doing the strbuf_setlen() in rename_tmp_packfile() we reuse the\nbuffer and avoid the repeated allocations we'd get if that function had\nits own temporary \"struct strbuf\".\n\nThis approach of reusing the buffer does make the last user in\npack-object.c's write_pack_file() slightly awkward, since we needlessly\ndo a strbuf_setlen() before calling strbuf_release() for consistency. In\nsubsequent changes we'll move that bitmap writing code around, so let's\nnot skip the strbuf_setlen() now.\n\nThe previous strbuf_reset() idiom originated with 5889271114a\n(finish_tmp_packfile():use strbuf for pathname construction,\n2014-03-03), which in turn was a minimal adjustment of pre-strbuf code\nadded in 0e990530ae (finish_tmp_packfile(): a helper function,\n2011-10-28).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++--\n bulk-checkin.c | 3 ++-\n pack-write.c | 37 ++++++++++++++++---------------------\n 3 files changed, 23 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex df49f656b9..2a105c8d6e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1237,7 +1237,8 @@ static void write_pack_file(void)\n \t\t\t\t\twarning_errno(_(\"failed utime() on %s\"), pack_tmp_name);\n \t\t\t}\n \n-\t\t\tstrbuf_addf(&tmpname, \"%s-\", base_name);\n+\t\t\tstrbuf_addf(&tmpname, \"%s-%s.\", base_name,\n+\t\t\t\t hash_to_hex(hash));\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(hash);\n@@ -1250,8 +1251,9 @@ static void write_pack_file(void)\n \t\t\t\t\t &pack_idx_opts, hash);\n \n \t\t\tif (write_bitmap_index) {\n-\t\t\t\tstrbuf_addf(&tmpname, \"%s.bitmap\", hash_to_hex(hash));\n+\t\t\t\tsize_t tmpname_len = tmpname.len;\n \n+\t\t\t\tstrbuf_addstr(&tmpname, \"bitmap\");\n \t\t\t\tstop_progress(&progress_state);\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n@@ -1260,6 +1262,7 @@ static void write_pack_file(void)\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\n+\t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n \t\t\tstrbuf_release(&tmpname);\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 6283bc8bd9..c19d471f0b 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -46,7 +46,8 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \t\tclose(fd);\n \t}\n \n-\tstrbuf_addf(&packname, \"%s/pack/pack-\", get_object_directory());\n+\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\", get_object_directory(),\n+\t\t hash_to_hex(hash));\n \tfinish_tmp_packfile(&packname, state->pack_tmp_name,\n \t\t\t state->written, state->nr_written,\n \t\t\t &state->pack_idx_opts, hash);\ndiff --git a/pack-write.c b/pack-write.c\nindex 2767b78619..95b063be94 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -458,6 +458,18 @@ struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n \treturn hashfd(fd, *pack_tmp_name);\n }\n \n+static void rename_tmp_packfile(struct strbuf *nb, const char *source,\n+\t\t\t\tconst char *ext)\n+{\n+\tsize_t nb_len = nb->len;\n+\n+\tstrbuf_addstr(nb, ext);\n+\tif (rename(source, nb->buf))\n+\t\tdie_errno(\"unable to rename temporary '*.%s' file to '%s'\",\n+\t\t\t ext, nb->buf);\n+\tstrbuf_setlen(nb, nb_len);\n+}\n+\n void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n@@ -466,7 +478,6 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t unsigned char hash[])\n {\n \tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n-\tint basename_len = name_buffer->len;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n@@ -479,26 +490,10 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n \t\t\t\t pack_idx_opts->flags);\n \n-\tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n-\n-\tif (rename(pack_tmp_name, name_buffer->buf))\n-\t\tdie_errno(\"unable to rename temporary pack file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n-\n-\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n-\tif (rename(idx_tmp_name, name_buffer->buf))\n-\t\tdie_errno(\"unable to rename temporary index file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n-\n-\tif (rev_tmp_name) {\n-\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n-\t\tif (rename(rev_tmp_name, name_buffer->buf))\n-\t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n-\t}\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n+\trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n+\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n+\tif (rev_tmp_name)\n+\t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n \n \tfree((void *)idx_tmp_name);\n }\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435207","messageId":"0fb2c25f5ad8bfdccd653f760b1c4beeb05273e7.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 4/9] pack-write.c: rename `.idx` files after `*.rev`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:24:59Z","receivedAt":"2021-09-09T03:25:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We treat the presence of an `.idx` file as the indicator of whether or\nnot it's safe to use a packfile. But `finish_tmp_packfile()` (which is\nresponsible for writing the pack and moving the temporary versions of\nall of its auxiliary files into place) is inconsistent about the write\norder.\n\nSpecifically, it moves the `.rev` file into place after the `.idx`,\nleaving open the possibility to open a pack which looks \"ready\" (because\nthe `.idx` file exists and is readable) but appears momentarily to not\nhave a `.rev` file. This causes Git to fall back to generating the\npack's reverse index in memory.\n\nThough racy, this amounts to an unnecessary slow-down at worst, and\ndoesn't affect the correctness of the resulting reverse index.\n\nClose this race by moving the .rev file into place before moving the\n.idx file into place.\n\nThis still leaves the issue of `.idx` files being renamed into place\nbefore the auxiliary `.bitmap` file is renamed when in pack-object.c's\nwrite_pack_file() \"write_bitmap_index\" is true. That race will be\naddressed in subsequent commits.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n pack-write.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 95b063be94..077710090e 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -491,9 +491,9 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t\t pack_idx_opts->flags);\n \n \trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n-\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n \tif (rev_tmp_name)\n \t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n+\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n \n \tfree((void *)idx_tmp_name);\n }\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435208","messageId":"3b10a97ec0e7c9e672904e6415909a1b8cea872e.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 5/9] builtin/repack.c: move `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:25:03Z","receivedAt":"2021-09-09T03:25:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar spirit as the previous patch, fix the identical problem\nfrom `git repack` (which invokes `pack-objects` with a temporary\nlocation for output, and then moves the files into their final locations\nitself).\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5f9bc74adc..c3e4771609 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -208,10 +208,10 @@ static struct {\n \tunsigned optional:1;\n } exts[] = {\n \t{\".pack\"},\n-\t{\".idx\"},\n \t{\".rev\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n+\t{\".idx\"},\n };\n \n static unsigned populate_pack_exts(char *name)\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435209","messageId":"3c9b515907ecb632faf73ce8db83efed1493d1f1.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 6/9] index-pack: refactor renaming in final()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:25:09Z","receivedAt":"2021-09-09T03:25:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nRefactor the renaming in final() into a helper function, this is\nsimilar in spirit to a preceding refactoring of finish_tmp_packfile()\nin pack-write.c.\n\nBefore e37d0b8730b (builtin/index-pack.c: write reverse indexes,\n2021-01-25) it probably wasn't worth it to have this sort of helper,\ndue to the differing \"else if\" case for \"pack\" files v.s. \"idx\" files.\n\nBut since we've got \"rev\" as well now, let's do the renaming via a\nhelper, this is both a net decrease in lines, and improves the\nreadability, since we can easily see at a glance that the logic for\nwriting these three types of files is exactly the same, aside from the\nobviously differing cases of \"*final_xyz_name\" being NULL, and\n\"else_chmod_if\" being different.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 48 +++++++++++++++++++++-----------------------\n 1 file changed, 23 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 6cc4890217..cd4e85f5bb 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1477,6 +1477,22 @@ static void write_special_file(const char *suffix, const char *msg,\n \tstrbuf_release(&name_buf);\n }\n \n+static void rename_tmp_packfile(const char **final_xyz_name,\n+\t\t\t\tconst char *curr_xyz_name,\n+\t\t\t\tstruct strbuf *xyz_name, unsigned char *hash,\n+\t\t\t\tconst char *ext, int else_chmod_if)\n+{\n+\tif (*final_xyz_name != curr_xyz_name) {\n+\t\tif (!*final_xyz_name)\n+\t\t\t*final_xyz_name = odb_pack_name(xyz_name, hash, ext);\n+\t\tif (finalize_object_file(curr_xyz_name, *final_xyz_name))\n+\t\t\tdie(_(\"unable to rename temporary '*.%s' file to '%s\"),\n+\t\t\t ext, *final_xyz_name);\n+\t} else if (else_chmod_if) {\n+\t\tchmod(*final_xyz_name, 0444);\n+\t}\n+}\n+\n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n \t\t const char *final_rev_index_name, const char *curr_rev_index_name,\n@@ -1505,31 +1521,13 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\twrite_special_file(\"promisor\", promisor_msg, final_pack_name,\n \t\t\t\t hash, NULL);\n \n-\tif (final_pack_name != curr_pack_name) {\n-\t\tif (!final_pack_name)\n-\t\t\tfinal_pack_name = odb_pack_name(&pack_name, hash, \"pack\");\n-\t\tif (finalize_object_file(curr_pack_name, final_pack_name))\n-\t\t\tdie(_(\"cannot store pack file\"));\n-\t} else if (from_stdin)\n-\t\tchmod(final_pack_name, 0444);\n-\n-\tif (final_index_name != curr_index_name) {\n-\t\tif (!final_index_name)\n-\t\t\tfinal_index_name = odb_pack_name(&index_name, hash, \"idx\");\n-\t\tif (finalize_object_file(curr_index_name, final_index_name))\n-\t\t\tdie(_(\"cannot store index file\"));\n-\t} else\n-\t\tchmod(final_index_name, 0444);\n-\n-\tif (curr_rev_index_name) {\n-\t\tif (final_rev_index_name != curr_rev_index_name) {\n-\t\t\tif (!final_rev_index_name)\n-\t\t\t\tfinal_rev_index_name = odb_pack_name(&rev_index_name, hash, \"rev\");\n-\t\t\tif (finalize_object_file(curr_rev_index_name, final_rev_index_name))\n-\t\t\t\tdie(_(\"cannot store reverse index file\"));\n-\t\t} else\n-\t\t\tchmod(final_rev_index_name, 0444);\n-\t}\n+\trename_tmp_packfile(&final_pack_name, curr_pack_name, &pack_name,\n+\t\t\t hash, \"pack\", from_stdin);\n+\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n+\t\t\t hash, \"idx\", 1);\n+\tif (curr_rev_index_name)\n+\t\trename_tmp_packfile(&final_rev_index_name, curr_rev_index_name,\n+\t\t\t\t &rev_index_name, hash, \"rev\", 1);\n \n \tif (do_fsck_object) {\n \t\tstruct packed_git *p;\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435210","messageId":"5c553229b00c5d09c74a0947d1cbc9bd5cc880e4.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 8/9] pack-write: split up finish_tmp_packfile() function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:25:17Z","receivedAt":"2021-09-09T03:26:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nSplit up the finish_tmp_packfile() function and use the split-up version\nin pack-objects.c in preparation for moving the step of renaming the\n*.idx file later as part of a function change.\n\nSince the only other caller of finish_tmp_packfile() was in\nbulk-checkin.c, and it won't be needing a change to its *.idx renaming,\nprovide a thin wrapper for the old function as a static function in that\nfile. If other callers end up needing the simpler version it could be\nmoved back to \"pack-write.c\" and \"pack.h\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++--\n bulk-checkin.c | 16 ++++++++++++++++\n pack-write.c | 22 +++++++++++++---------\n pack.h | 7 +++++--\n 4 files changed, 39 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2a105c8d6e..944134b6f2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1217,6 +1217,7 @@ static void write_pack_file(void)\n \t\tif (!pack_to_stdout) {\n \t\t\tstruct stat st;\n \t\t\tstruct strbuf tmpname = STRBUF_INIT;\n+\t\t\tchar *idx_tmp_name = NULL;\n \n \t\t\t/*\n \t\t\t * Packs are runtime accessed in their mtime\n@@ -1246,9 +1247,10 @@ static void write_pack_file(void)\n \t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n-\t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\n+\t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n-\t\t\t\t\t &pack_idx_opts, hash);\n+\t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n+\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tsize_t tmpname_len = tmpname.len;\n@@ -1265,6 +1267,7 @@ static void write_pack_file(void)\n \t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n+\t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmpname);\n \t\t\tfree(pack_tmp_name);\n \t\t\tputs(hash_to_hex(hash));\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex c19d471f0b..8785b2ac80 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -23,6 +23,22 @@ static struct bulk_checkin_state {\n \tuint32_t nr_written;\n } state;\n \n+static void finish_tmp_packfile(struct strbuf *basename,\n+\t\t\t\tconst char *pack_tmp_name,\n+\t\t\t\tstruct pack_idx_entry **written_list,\n+\t\t\t\tuint32_t nr_written,\n+\t\t\t\tstruct pack_idx_option *pack_idx_opts,\n+\t\t\t\tunsigned char hash[])\n+{\n+\tchar *idx_tmp_name = NULL;\n+\n+\tstage_tmp_packfiles(basename, pack_tmp_name, written_list, nr_written,\n+\t\t\t pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(basename, &idx_tmp_name);\n+\n+\tfree(idx_tmp_name);\n+}\n+\n static void finish_bulk_checkin(struct bulk_checkin_state *state)\n {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/pack-write.c b/pack-write.c\nindex 077710090e..d40d2ddd01 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -470,21 +470,28 @@ static void rename_tmp_packfile(struct strbuf *nb, const char *source,\n \tstrbuf_setlen(nb, nb_len);\n }\n \n-void finish_tmp_packfile(struct strbuf *name_buffer,\n+void rename_tmp_packfile_idx(struct strbuf *name_buffer,\n+\t\t\t char **idx_tmp_name)\n+{\n+\trename_tmp_packfile(name_buffer, *idx_tmp_name, \"idx\");\n+}\n+\n+void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n-\t\t\t unsigned char hash[])\n+\t\t\t unsigned char hash[],\n+\t\t\t char **idx_tmp_name)\n {\n-\tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n+\tconst char *rev_tmp_name = NULL;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n \n-\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n-\t\t\t\t pack_idx_opts, hash);\n-\tif (adjust_shared_perm(idx_tmp_name))\n+\t*idx_tmp_name = (char *)write_idx_file(NULL, written_list, nr_written,\n+\t\t\t\t\t pack_idx_opts, hash);\n+\tif (adjust_shared_perm(*idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n@@ -493,9 +500,6 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n \tif (rev_tmp_name)\n \t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n-\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n-\n-\tfree((void *)idx_tmp_name);\n }\n \n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought)\ndiff --git a/pack.h b/pack.h\nindex 1c17254c0a..b22bfc4a18 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,11 +110,14 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n-void finish_tmp_packfile(struct strbuf *name_buffer,\n+void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n-\t\t\t unsigned char sha1[]);\n+\t\t\t unsigned char hash[],\n+\t\t\t char **idx_tmp_name);\n+void rename_tmp_packfile_idx(struct strbuf *basename,\n+\t\t\t char **idx_tmp_name);\n \n #endif\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435211","messageId":"8d67a715014b9daa9a6ca3a77be647e733324522.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 7/9] builtin/index-pack.c: move `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:25:13Z","receivedAt":"2021-09-09T03:26:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar spirit as preceding patches to `git repack` and `git\npack-objects`, fix the identical problem in `git index-pack`.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex cd4e85f5bb..bf294d9083 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1523,11 +1523,11 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \n \trename_tmp_packfile(&final_pack_name, curr_pack_name, &pack_name,\n \t\t\t hash, \"pack\", from_stdin);\n-\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n-\t\t\t hash, \"idx\", 1);\n \tif (curr_rev_index_name)\n \t\trename_tmp_packfile(&final_rev_index_name, curr_rev_index_name,\n \t\t\t\t &rev_index_name, hash, \"rev\", 1);\n+\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n+\t\t\t hash, \"idx\", 1);\n \n \tif (do_fsck_object) {\n \t\tstruct packed_git *p;\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435212","messageId":"d8286cf1075dc85231128145c5abb0db3881032b.1631157880.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH 9/9] pack-objects: rename .idx files into place after .bitmap files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T03:25:22Z","receivedAt":"2021-09-09T03:26:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nIn preceding commits the race of renaming .idx files in place before\n.rev files and other auxiliary files was fixed in pack-write.c's\nfinish_tmp_packfile(), builtin/repack.c's \"struct exts\", and\nbuiltin/index-pack.c's final(). As noted in the change to pack-write.c\nwe left in place the issue of writing *.bitmap files after the *.idx,\nlet's fix that issue.\n\nSee 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21)\nfor commentary at the time when *.bitmap was implemented about how\nthose files are written out, nothing in that commit contradicts what's\nbeing done here.\n\nNote that this commit and preceding ones only close any race condition\nwith *.idx files being written before their auxiliary files if we're\noptimistic about our lack of fsync()-ing in this are not tripping us\nover. See the thread at [1] for a rabbit hole of various discussions\nabout filesystem races in the face of doing and not doing fsync() (and\nif doing fsync(), not doing it properly).\n\nIn particular, in this case of writing to \".git/objects/pack\" we only\nwrite and fsync() the individual files, but if we wanted to guarantee\nthat the metadata update was seen in that way by concurrent processes\nwe'd need to fsync() on the \"fd\" of the containing directory. That\nconcern is probably more theoretical than not, modern OS's tend to be\nmore on the forgiving side than the overly pedantic side of\nimplementing POSIX FS semantics.\n\n1. https://lore.kernel.org/git/8735qgkvv1.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 944134b6f2..a01767a384 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1250,7 +1250,6 @@ static void write_pack_file(void)\n \t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n \t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n-\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tsize_t tmpname_len = tmpname.len;\n@@ -1267,6 +1266,8 @@ static void write_pack_file(void)\n \t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n+\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n+\n \t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmpname);\n \t\t\tfree(pack_tmp_name);\n-- \n2.33.0.96.g73915697e6\n"},{"id":"435215","messageId":"87lf46faz9.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"20b35ce0505d7bcc84b6fd68a33e0b6b1afa31e6.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 2/9] bulk-checkin.c: store checksum directly","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T07:38:48Z","receivedAt":"2021-09-09T07:46:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Taylor Blau wrote:\n\n> finish_bulk_checkin() stores the checksum from finalize_hashfile() by\n> writing to the `hash` member of `struct object_id`, but that hash has\n> nothing to do with an object id (it's just a convenient location that\n> happens to be sized correctly).\n>\n> Store the hash directly in an unsigned char array. This behaves the same\n> as writing to the `hash` member, but makes the intent clearer (and\n> avoids allocating an extra four bytes for the `algo` member of `struct\n> object_id`).\n>\n> Signed-off-by: Taylor Blau \n> ---\n> bulk-checkin.c | 12 ++++++------\n> 1 file changed, 6 insertions(+), 6 deletions(-)\n>\n> diff --git a/bulk-checkin.c b/bulk-checkin.c\n> index b023d9959a..6283bc8bd9 100644\n> --- a/bulk-checkin.c\n> +++ b/bulk-checkin.c\n> @@ -25,7 +25,7 @@ static struct bulk_checkin_state {\n> \n> static void finish_bulk_checkin(struct bulk_checkin_state *state)\n> {\n> -\tstruct object_id oid;\n> +\tunsigned char hash[GIT_MAX_RAWSZ];\n> \tstruct strbuf packname = STRBUF_INIT;\n> \tint i;\n> \n> @@ -37,11 +37,11 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n> \t\tunlink(state->pack_tmp_name);\n> \t\tgoto clear_exit;\n> \t} else if (state->nr_written == 1) {\n> -\t\tfinalize_hashfile(state->f, oid.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n> +\t\tfinalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n> \t} else {\n> -\t\tint fd = finalize_hashfile(state->f, oid.hash, 0);\n> -\t\tfixup_pack_header_footer(fd, oid.hash, state->pack_tmp_name,\n> -\t\t\t\t\t state->nr_written, oid.hash,\n> +\t\tint fd = finalize_hashfile(state->f, hash, 0);\n> +\t\tfixup_pack_header_footer(fd, hash, state->pack_tmp_name,\n> +\t\t\t\t\t state->nr_written, hash,\n> \t\t\t\t\t state->offset);\n> \t\tclose(fd);\n> \t}\n> @@ -49,7 +49,7 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n> \tstrbuf_addf(&packname, \"%s/pack/pack-\", get_object_directory());\n> \tfinish_tmp_packfile(&packname, state->pack_tmp_name,\n> \t\t\t state->written, state->nr_written,\n> -\t\t\t &state->pack_idx_opts, oid.hash);\n> +\t\t\t &state->pack_idx_opts, hash);\n> \tfor (i = 0; i < state->nr_written; i++)\n> \t\tfree(state->written[i]);\n\n[This commit looks good, nothing needs to be fixed here, just a\ndigression]:\n\nThis is a good change, FWIW I came up with in my version, and then ended\nup ejecting it to resist the urge to go on some general cleanup spree, I\nguess we can just fix *this* one :)\n\nAnyway, I agree that this should be in & having it is good.\n\nThe code pattern being fixed here is more insidious than your commit\nmessage describes though, since fa33c3aae23 (bulk-checkin.c: convert to\nuse struct object_id, 2015-03-13) when this was converted from \"unsigned\nchar sha1[20]\" to \"struct object_id\" we've had a landmine waiting to be\nstepped on here.\n\n\nThe \"oid\" automatic variable will get initialized to some garbage, we\nthen write to just the \"hash\" member, but leave the \"algo\" member in\nsome undefined state.\n\nThus when you e.g. call oid_to_hex(&oid) here you might get a segfault,\ne.g. on my box because in hex.c we'll try to access\n&hash_algos[oid->algo], where oid->algo happens to be set to the garbage\n-9872.\n\nIf somebody's interested in some follow-up cleanup renaming the \"hash\"\nmember to say \"hash2\" in hash.h, and compiling with \"make -k\" will find\nall the sites that are using it directly,\n\nI looked in detail at a few, and many can either be converted to just\nuse \"struct object_id\" without playing around with \"oid.hash\", and some\nshare the bug/landmine being fixed here.\n\n\n"},{"id":"435216","messageId":"87h7eufaw5.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"0fb2c25f5ad8bfdccd653f760b1c4beeb05273e7.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 4/9] pack-write.c: rename `.idx` files after `*.rev`","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T07:46:57Z","receivedAt":"2021-09-09T07:47:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Taylor Blau wrote:\n\n> [...]\n>\n> Signed-off-by: Taylor Blau \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Taylor Blau \n\ngit commit --amend --signed-off-by-harder ? :)\n\nI.e. the duplicate header entry can go, maybe something Junio will fix\nup while queuing...\n"},{"id":"435218","messageId":"87czpifamz.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"8d67a715014b9daa9a6ca3a77be647e733324522.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 7/9] builtin/index-pack.c: move `.idx` files into place last","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T07:52:47Z","receivedAt":"2021-09-09T07:53:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Taylor Blau wrote:\n\n> Signed-off-by: Taylor Blau \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Taylor Blau \n\nDitto the comment on 4/9 about duplicate Signed-off-by headers, looks\ngood otherwise!\n"},{"id":"435219","messageId":"878s06fa14.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"d8286cf1075dc85231128145c5abb0db3881032b.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 9/9] pack-objects: rename .idx files into place after .bitmap files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T07:54:44Z","receivedAt":"2021-09-09T08:06:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Taylor Blau wrote:\n\n> From: Ævar Arnfjörð Bjarmason \n>\n> In preceding commits the race of renaming .idx files in place before\n> .rev files and other auxiliary files was fixed in pack-write.c's\n> finish_tmp_packfile(), builtin/repack.c's \"struct exts\", and\n> builtin/index-pack.c's final(). As noted in the change to pack-write.c\n> we left in place the issue of writing *.bitmap files after the *.idx,\n> let's fix that issue.\n>\n> See 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21)\n> for commentary at the time when *.bitmap was implemented about how\n> those files are written out, nothing in that commit contradicts what's\n> being done here.\n>\n> Note that this commit and preceding ones only close any race condition\n> with *.idx files being written before their auxiliary files if we're\n> optimistic about our lack of fsync()-ing in this are not tripping us\n> over. See the thread at [1] for a rabbit hole of various discussions\n> about filesystem races in the face of doing and not doing fsync() (and\n> if doing fsync(), not doing it properly).\n>\n> In particular, in this case of writing to \".git/objects/pack\" we only\n> write and fsync() the individual files, but if we wanted to guarantee\n> that the metadata update was seen in that way by concurrent processes\n> we'd need to fsync() on the \"fd\" of the containing directory. That\n> concern is probably more theoretical than not, modern OS's tend to be\n> more on the forgiving side than the overly pedantic side of\n> implementing POSIX FS semantics.\n\nSome weird gramma/phrasing of mine left over here, i.e. the \"[...] in\nthis are not tripping us over.\". On reflection perhaps it's better to\nreplace these last two paragraphs with say:\n\n With this and preceding commits we've covered all the cases where we\n wrote another auxiliary file before the *.idx file, and should thus\n never have another concurrent process try to use an incomplete set\n of pack-OID.* files.\n\n This assumes that the renames we make here and elsewhere appear on\n the filesystem in the order we performed them. This assumption is\n known to be false in the face of pedantic POSIX FS semantics. See\n the thread around [1] for discussions on that point.\n\n We fsync() the file descriptors we're writing out for all the\n auxiliary files, but we don't yet fsync() the file descriptor for\n the containing directory. Therefore our data might have been written\n out, but it's anyone's guess what the state of the directory\n containing the file is after we write the *.idx.\n\n In practice modern OS's are known to be forgiving on that point, so\n this will probably solve races in practice for most users. It will\n almost certainly make them better than they were before when we\n didn't write *.idx files last. We should more generally improve our\n use of fsync() to cover containing directories, but that'll\n hopefully be addressed by some follow-up series.\n"},{"id":"435220","messageId":"874kauf9wq.fsf@evledraar.gmail.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 0/9] packfile: avoid .idx rename races","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T08:06:24Z","receivedAt":"2021-09-09T08:08:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 08 2021, Taylor Blau wrote:\n\n> This series is a unification of [1] and [2] which together prevent a handful of\n> races when code that writes packfiles moves the `.idx` into place before all\n> other auxiliary files are properly renamed.\n>\n> This can lead to races like not reading the `.rev` file even if one was\n> generated and is about to be moved into place.\n>\n> Credit goes to Ævar for preparing what is more-or-less sent here. I polished a\n> few of the commit messages, and added the second patch on top of his result. It\n> isn't necessary, but felt like good hygiene when I was reading the surrounding\n> code.\n>\n> Thanks in advance for reviewing.\n\nThanks a lot!\n\nI think it's probably redundant to note it at this point but I've given\nthis a thorough review and it all looks good to me. I left some\ncomments/musings on minor points in the series, but none of those IMO\nrequire a re-roll except perhaps the duplicate Signed-off-by headers,\ndepending on what Junio thinks & if he's going to fix those while\nqueuing.\n"},{"id":"435268","messageId":"YTocEy7PZqL6nvuY@nand.local","threadId":"56421","inReplyTo":"87h7eufaw5.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/9] pack-write.c: rename `.idx` files after `*.rev`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T14:37:07Z","receivedAt":"2021-09-09T14:46:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 09:46:57AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Wed, Sep 08 2021, Taylor Blau wrote:\n>\n> > [...]\n> >\n> > Signed-off-by: Taylor Blau \n> > Signed-off-by: Ævar Arnfjörð Bjarmason \n> > Signed-off-by: Taylor Blau \n>\n> git commit --amend --signed-off-by-harder ? :)\n\nI don't think so, though I had to check myself before sending.\nDocumentation/SubmittingPatches says:\n\n Notice that you can place your own `Signed-off-by` trailer when\n forwarding somebody else's patch with the above rules for\n D-C-O. Indeed you are encouraged to do so. Do not forget to\n place an in-body \"From: \" line at the beginning to properly attribute\n the change to its true author (see (2) above).\n\nHere it's awkward because you modified my original patch and then I sent\nyour modified result out. But I think the s-o-b chain still makes sense:\nI wrote the original patch, and signed it off. Then you modified it,\nsigning that off. Finally, I sent it to the list, signing off on your\nmodification.\n\n> I.e. the duplicate header entry can go, maybe something Junio will fix\n> up while queuing...\n\nIf I misread SubmittingPatches then I'm happy to fix it myself, but this\nwas the intended outcome.\n\nThanks,\nTaylor\n"},{"id":"435269","messageId":"YToc+W7q1F8IXYK+@nand.local","threadId":"56421","inReplyTo":"874kauf9wq.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/9] packfile: avoid .idx rename races","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T14:40:57Z","receivedAt":"2021-09-09T14:47:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 10:06:24AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> I think it's probably redundant to note it at this point but I've given\n> this a thorough review and it all looks good to me. I left some\n> comments/musings on minor points in the series, but none of those IMO\n> require a re-roll except perhaps the duplicate Signed-off-by headers,\n> depending on what Junio thinks & if he's going to fix those while\n> queuing.\n\nThanks, and likewise your parts of this series look good to me\n(obviously, or I wouldn't have sent it otherwise ;)).\n\nThe duplicate s-o-b's are intentional, but see my response in 2/9 (I'd\nlink to it, but vger seems to be a little sluggish for me this morning)\nfor why. If I made an error there, I'm happy to fix it up for Junio.\n\nThanks,\nTaylor\n"},{"id":"435315","messageId":"xmqqk0jpzgxu.fsf@gitster.g","threadId":"56421","inReplyTo":"35052ef494dbc55119614f3e22742d8d814b21b1.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 3/9] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T19:29:01Z","receivedAt":"2021-09-09T19:29:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> @@ -1250,8 +1251,9 @@ static void write_pack_file(void)\n> \t\t\t\t\t &pack_idx_opts, hash);\n> \n> \t\t\tif (write_bitmap_index) {\n> -\t\t\t\tstrbuf_addf(&tmpname, \"%s.bitmap\", hash_to_hex(hash));\n> +\t\t\t\tsize_t tmpname_len = tmpname.len;\n> \n> +\t\t\t\tstrbuf_addstr(&tmpname, \"bitmap\");\n> \t\t\t\tstop_progress(&progress_state);\n> \n> \t\t\t\tbitmap_writer_show_progress(progress);\n> @@ -1260,6 +1262,7 @@ static void write_pack_file(void)\n> \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n> \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n> \t\t\t\twrite_bitmap_index = 0;\n> +\t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n> \t\t\t}\n> \n> \t\t\tstrbuf_release(&tmpname);\n\nThis runs setlen on tmpname strbuf and immediately releases (the\nclose brace before release closes the \"if (write_bitmap_index)\", not\nany loop. If we plan to insert more code to use tmpname where the\nblank line we see above is in the future, it may make sense, but\neven in such a case, adding setlen() only when it starts to matter\nmay make it easier to follow.\n\nIn any case, the above correctly adjusts tmpname to have a \nplus '.' at the end of tmpname to call finish_tmp_packfile().\n\n> diff --git a/bulk-checkin.c b/bulk-checkin.c\n> index 6283bc8bd9..c19d471f0b 100644\n> --- a/bulk-checkin.c\n> +++ b/bulk-checkin.c\n> @@ -46,7 +46,8 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n> \t\tclose(fd);\n> \t}\n> \n> -\tstrbuf_addf(&packname, \"%s/pack/pack-\", get_object_directory());\n> +\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\", get_object_directory(),\n> +\t\t hash_to_hex(hash));\n> \tfinish_tmp_packfile(&packname, state->pack_tmp_name,\n> \t\t\t state->written, state->nr_written,\n> \t\t\t &state->pack_idx_opts, hash);\n\nOK.\n\n> diff --git a/pack-write.c b/pack-write.c\n> index 2767b78619..95b063be94 100644\n> --- a/pack-write.c\n> +++ b/pack-write.c\n> @@ -458,6 +458,18 @@ struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n> \treturn hashfd(fd, *pack_tmp_name);\n> }\n> \n> +static void rename_tmp_packfile(struct strbuf *nb, const char *source,\n> +\t\t\t\tconst char *ext)\n> +{\n> +\tsize_t nb_len = nb->len;\n> +\n> +\tstrbuf_addstr(nb, ext);\n> +\tif (rename(source, nb->buf))\n> +\t\tdie_errno(\"unable to rename temporary '*.%s' file to '%s'\",\n> +\t\t\t ext, nb->buf);\n\nI do not like '*.%s' here. Without '*' it is clear enough, and\nbecause nb->buf has already the same ext information, \n\n\tunable to rename temporary file to '%s'.\n\nwould be even simpler without losing any information than this\nrewrite does.\n\nThe original explicitly used the more human-facing terms like \"pack\nfile\", so we are losing information by this refactoring, but the\nloss is not too bad. In one case, namely \".idx\" files, it is even\nbetter, as the original says \"temporary index file\" to refer to the\nnew .idx file, which makes it ambiguous with _the_ index file.\n\n> +\tstrbuf_setlen(nb, nb_len);\n> +}\n\nIn the longer term if we were to add more auxiliary files next to\neach of the .pack file, it makes perfect sense that the common\nprefix is fed to rename_tmp_packfile() and the function reverts\ncontents of the prefix buffer back when it is done with it. But it\nwould be more friendly to those adding more calls to this function\nin the future to document the convention in a comment before the\nfunction.\n\nAlso, the name \"nb\" would need rethinking. As far as the callers\nare concerned, that is not a full name, but they are supplying the\ncommon prefix to the function. Perhaps \"struct strbuf *name_prefix\"\nor soemthing might be better? I dunno.\n\n> void finish_tmp_packfile(struct strbuf *name_buffer,\n> \t\t\t const char *pack_tmp_name,\n> \t\t\t struct pack_idx_entry **written_list,\n> @@ -466,7 +478,6 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n> \t\t\t unsigned char hash[])\n> {\n> \tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n> -\tint basename_len = name_buffer->len;\n> \n> \tif (adjust_shared_perm(pack_tmp_name))\n> \t\tdie_errno(\"unable to make temporary pack file readable\");\n> @@ -479,26 +490,10 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n> \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n> \t\t\t\t pack_idx_opts->flags);\n\nIt is unfortunate that write_idx_file() and write_rev_file() take\nhash and come up with the temporary filename on their own (which\nmeans their resulting filenames may not share the same prefix as\n.pack and .idx files), but it is just the naming of temporaries and\ninconsistencies among them is, eh, temporary, so it is OK.\n\n> +\trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n> +\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n> +\tif (rev_tmp_name)\n> +\t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n> \n> \tfree((void *)idx_tmp_name);\n> }\n"},{"id":"435317","messageId":"xmqqee9xzgrh.fsf@gitster.g","threadId":"56421","inReplyTo":"0fb2c25f5ad8bfdccd653f760b1c4beeb05273e7.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 4/9] pack-write.c: rename `.idx` files after `*.rev`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T19:32:50Z","receivedAt":"2021-09-09T19:32:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> This still leaves the issue of `.idx` files being renamed into place\n> before the auxiliary `.bitmap` file is renamed when in pack-object.c's\n> write_pack_file() \"write_bitmap_index\" is true. That race will be\n> addressed in subsequent commits.\n\nOK. I was about to suggest s/after .*rev/last/ on the title, but\nthe above makes it clear that we are not ready for such a title.\n\n>\n> Signed-off-by: Taylor Blau \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Taylor Blau \n> ---\n> pack-write.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/pack-write.c b/pack-write.c\n> index 95b063be94..077710090e 100644\n> --- a/pack-write.c\n> +++ b/pack-write.c\n> @@ -491,9 +491,9 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n> \t\t\t\t pack_idx_opts->flags);\n> \n> \trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n> -\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n> \tif (rev_tmp_name)\n> \t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n> +\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n> \n> \tfree((void *)idx_tmp_name);\n> }\n"},{"id":"435318","messageId":"xmqqa6klzghf.fsf@gitster.g","threadId":"56421","inReplyTo":"3b10a97ec0e7c9e672904e6415909a1b8cea872e.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 5/9] builtin/repack.c: move `.idx` files into place last","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T19:38:52Z","receivedAt":"2021-09-09T19:38:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> In a similar spirit as the previous patch, fix the identical problem\n> from `git repack` (which invokes `pack-objects` with a temporary\n> location for output, and then moves the files into their final locations\n> itself).\n\nOK. This array is used in cmd_repack() to call rename() in the\norder in which its elements appear, so moving the entry to the last\nmeans it will be renamed the last.\n\n> Signed-off-by: Taylor Blau \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/repack.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 5f9bc74adc..c3e4771609 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -208,10 +208,10 @@ static struct {\n> \tunsigned optional:1;\n> } exts[] = {\n> \t{\".pack\"},\n> -\t{\".idx\"},\n> \t{\".rev\", 1},\n> \t{\".bitmap\", 1},\n> \t{\".promisor\", 1},\n> +\t{\".idx\"},\n> };\n\nAnd the .idx entry MUST stay the last. I wonder if dropping the\ntrailing comma and replace it with a \"/* must be at the end */\"\ncomment would be an effective way to ensure that.\n"},{"id":"435319","messageId":"xmqq35qdzg73.fsf@gitster.g","threadId":"56421","inReplyTo":"3c9b515907ecb632faf73ce8db83efed1493d1f1.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 6/9] index-pack: refactor renaming in final()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T19:45:04Z","receivedAt":"2021-09-09T19:45:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> But since we've got \"rev\" as well now, let's do the renaming via a\n> helper, this is both a net decrease in lines, and improves the\n> readability,...\n\nxyz_ may be cute, but is distracting. I do not think it loses any\ninformation if we used final_name, current_name, etc.; it may make\nthe result even easier to read.\n\n> +static void rename_tmp_packfile(const char **final_xyz_name,\n> +\t\t\t\tconst char *curr_xyz_name,\n> +\t\t\t\tstruct strbuf *xyz_name, unsigned char *hash,\n> +\t\t\t\tconst char *ext, int else_chmod_if)\n> +{\n> +\tif (*final_xyz_name != curr_xyz_name) {\n> +\t\tif (!*final_xyz_name)\n> +\t\t\t*final_xyz_name = odb_pack_name(xyz_name, hash, ext);\n> +\t\tif (finalize_object_file(curr_xyz_name, *final_xyz_name))\n> +\t\t\tdie(_(\"unable to rename temporary '*.%s' file to '%s\"),\n> +\t\t\t ext, *final_xyz_name);\n> +\t} else if (else_chmod_if) {\n> +\t\tchmod(*final_xyz_name, 0444);\n> +\t}\n> +}\n\n\"else_chmod_if\" is unclear. The caller must be aware of what 'else'\nrefers to and anybody adding a new caller is forced to go look at\nthe body of this helper. I think chmod (or \"make_read_only\")\nhappens only when the current one already has the final name, so\nperhaps that is what the name should highlight? \n\nmake-read-only-if-same or somesuch?\n\nThanks.\n"},{"id":"435320","messageId":"xmqqy285y1la.fsf@gitster.g","threadId":"56421","inReplyTo":"8d67a715014b9daa9a6ca3a77be647e733324522.1631157880.git.me@ttaylorr.com","subject":"Re: [PATCH 7/9] builtin/index-pack.c: move `.idx` files into place last","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T19:45:53Z","receivedAt":"2021-09-09T19:45:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> In a similar spirit as preceding patches to `git repack` and `git\n> pack-objects`, fix the identical problem in `git index-pack`.\n>\n> Signed-off-by: Taylor Blau \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/index-pack.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index cd4e85f5bb..bf294d9083 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -1523,11 +1523,11 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n> \n> \trename_tmp_packfile(&final_pack_name, curr_pack_name, &pack_name,\n> \t\t\t hash, \"pack\", from_stdin);\n> -\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n> -\t\t\t hash, \"idx\", 1);\n> \tif (curr_rev_index_name)\n> \t\trename_tmp_packfile(&final_rev_index_name, curr_rev_index_name,\n> \t\t\t\t &rev_index_name, hash, \"rev\", 1);\n> +\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n> +\t\t\t hash, \"idx\", 1);\n\nGood.\n"},{"id":"435321","messageId":"xmqqr1dxy1b0.fsf@gitster.g","threadId":"56421","inReplyTo":"878s06fa14.fsf@evledraar.gmail.com","subject":"Re: [PATCH 9/9] pack-objects: rename .idx files into place after .bitmap files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T19:52:03Z","receivedAt":"2021-09-09T19:52:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> We fsync() the file descriptors we're writing out for all the\n> auxiliary files, but we don't yet fsync() the file descriptor for\n> the containing directory. Therefore our data might have been written\n> out, but it's anyone's guess what the state of the directory\n> containing the file is after we write the *.idx.\n>\n> In practice modern OS's are known to be forgiving on that point, so\n> this will probably solve races in practice for most users. It will\n> almost certainly make them better than they were before when we\n> didn't write *.idx files last. We should more generally improve our\n> use of fsync() to cover containing directories, but that'll\n> hopefully be addressed by some follow-up series.\n\nI'd probably drop the last paragraph, and replace it with a single\nsentence \"we may want to fsync the containing directory once after\nplacing *.idx file in place, but it is outside of the scope of this\nseries\", if I were doing this series.\n\nOther than that (and I made a few comments on other patches), these\nwere pleasant read.\n\nThanks.\n"},{"id":"435322","messageId":"xmqqmtoly1a9.fsf@gitster.g","threadId":"56421","inReplyTo":"YToc+W7q1F8IXYK+@nand.local","subject":"Re: [PATCH 0/9] packfile: avoid .idx rename races","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T19:52:30Z","receivedAt":"2021-09-09T19:52:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> The duplicate s-o-b's are intentional, but see my response in 2/9 (I'd\n> link to it, but vger seems to be a little sluggish for me this morning)\n\nLooks good to me. Thanks.\n"},{"id":"435333","messageId":"YTp3rYqxLE5/X521@nand.local","threadId":"56421","inReplyTo":"xmqqk0jpzgxu.fsf@gitster.g","subject":"Re: [PATCH 3/9] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T21:07:57Z","receivedAt":"2021-09-09T21:08:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 12:29:01PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > @@ -1250,8 +1251,9 @@ static void write_pack_file(void)\n> > \t\t\t\t\t &pack_idx_opts, hash);\n> >\n> > \t\t\tif (write_bitmap_index) {\n> > -\t\t\t\tstrbuf_addf(&tmpname, \"%s.bitmap\", hash_to_hex(hash));\n> > +\t\t\t\tsize_t tmpname_len = tmpname.len;\n> >\n> > +\t\t\t\tstrbuf_addstr(&tmpname, \"bitmap\");\n> > \t\t\t\tstop_progress(&progress_state);\n> >\n> > \t\t\t\tbitmap_writer_show_progress(progress);\n> > @@ -1260,6 +1262,7 @@ static void write_pack_file(void)\n> > \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n> > \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n> > \t\t\t\twrite_bitmap_index = 0;\n> > +\t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n> > \t\t\t}\n> >\n> > \t\t\tstrbuf_release(&tmpname);\n>\n> This runs setlen on tmpname strbuf and immediately releases (the\n> close brace before release closes the \"if (write_bitmap_index)\", not\n> any loop. If we plan to insert more code to use tmpname where the\n> blank line we see above is in the future, it may make sense, but\n> even in such a case, adding setlen() only when it starts to matter\n> may make it easier to follow.\n>\n> In any case, the above correctly adjusts tmpname to have a \n> plus '.' at the end of tmpname to call finish_tmp_packfile().\n\nÆvar makes a slight mention of this in their commit message:\n\n This approach of reusing the buffer does make the last user in\n pack-object.c's write_pack_file() slightly awkward, since we\n needlessly do a strbuf_setlen() before calling strbuf_release() for\n consistency. In subsequent changes we'll move that bitmap writing\n code around, so let's not skip the strbuf_setlen() now.\n\nAnd this is to set us up for the final patch which moves the call to\nrename_tmp_packfile_idx(&tmpname, &idx_tmp_name) after the closing brace\nin the quoted portion of your message and the strbuf_release(). There,\nwe'll want the strbuf reset to the appropriate contents and length, and\nnot released.\n\nBut in the meantime it is awkward.\n\n> I do not like '*.%s' here. Without '*' it is clear enough, and\n> because nb->buf has already the same ext information,\n>\n> [...] But it would be more friendly to those adding more calls to this\n> function in the future to document the convention in a comment before\n> the function.\n>\n> Also, the name \"nb\" would need rethinking. As far as the callers\n> are concerned, that is not a full name, but they are supplying the\n> common prefix to the function. Perhaps \"struct strbuf *name_prefix\"\n> or soemthing might be better? I dunno.\n\nIn each of these three, I wasn't able to decide if you wanted these\naddressed in a newer version of this series, or if you were happy enough\nwith the result to pick it up. I'm happy to send you a new version, but\ndon't want to clog your inbox if you were already planning on picking\nthis up.\n\nThanks,\nTaylor\n"},{"id":"435334","messageId":"YTp365kOEpRenuml@nand.local","threadId":"56421","inReplyTo":"xmqqa6klzghf.fsf@gitster.g","subject":"Re: [PATCH 5/9] builtin/repack.c: move `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T21:08:59Z","receivedAt":"2021-09-09T21:09:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 12:38:52PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n> > diff --git a/builtin/repack.c b/builtin/repack.c\n> > index 5f9bc74adc..c3e4771609 100644\n> > --- a/builtin/repack.c\n> > +++ b/builtin/repack.c\n> > @@ -208,10 +208,10 @@ static struct {\n> > \tunsigned optional:1;\n> > } exts[] = {\n> > \t{\".pack\"},\n> > -\t{\".idx\"},\n> > \t{\".rev\", 1},\n> > \t{\".bitmap\", 1},\n> > \t{\".promisor\", 1},\n> > +\t{\".idx\"},\n> > };\n>\n> And the .idx entry MUST stay the last. I wonder if dropping the\n> trailing comma and replace it with a \"/* must be at the end */\"\n> comment would be an effective way to ensure that.\n\nHeh, Ævar even wrote that up, and I responded that I did not think it\nhelped much. Again, I'm happy to add it back if anybody feels strongly.\n\nThanks,\nTaylor\n"},{"id":"435335","messageId":"YTp4foh7gyIl3h1k@nand.local","threadId":"56421","inReplyTo":"xmqq35qdzg73.fsf@gitster.g","subject":"Re: [PATCH 6/9] index-pack: refactor renaming in final()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T21:11:26Z","receivedAt":"2021-09-09T21:11:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 12:45:04PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > But since we've got \"rev\" as well now, let's do the renaming via a\n> > helper, this is both a net decrease in lines, and improves the\n> > readability,...\n>\n> xyz_ may be cute, but is distracting. I do not think it loses any\n> information if we used final_name, current_name, etc.; it may make\n> the result even easier to read.\n>\n> [...]\n>\n> make-read-only-if-same or somesuch?\n\nBoth of these suggestions (dropping the \"xyz\"s and renaming the last\nparameter to \"make_read_only_if_same\") make sense. Will apply.\n\nThanks,\nTaylor\n"},{"id":"435336","messageId":"YTp44A+kPOf8U+7w@nand.local","threadId":"56421","inReplyTo":"xmqqr1dxy1b0.fsf@gitster.g","subject":"Re: [PATCH 9/9] pack-objects: rename .idx files into place after .bitmap files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T21:13:04Z","receivedAt":"2021-09-09T21:13:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 12:52:03PM -0700, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason writes:\n>\n> > We fsync() the file descriptors we're writing out for all the\n> > auxiliary files, but we don't yet fsync() the file descriptor for\n> > the containing directory. Therefore our data might have been written\n> > out, but it's anyone's guess what the state of the directory\n> > containing the file is after we write the *.idx.\n> >\n> > In practice modern OS's are known to be forgiving on that point, so\n> > this will probably solve races in practice for most users. It will\n> > almost certainly make them better than they were before when we\n> > didn't write *.idx files last. We should more generally improve our\n> > use of fsync() to cover containing directories, but that'll\n> > hopefully be addressed by some follow-up series.\n>\n> I'd probably drop the last paragraph, and replace it with a single\n> sentence \"we may want to fsync the containing directory once after\n> placing *.idx file in place, but it is outside of the scope of this\n> series\", if I were doing this series.\n\nYep, I think that this makes things clearer, and I prefer it to Ævar's\nsuggestion (which conveys more information than I think is necessary\nhere).\n\nThanks,\nTaylor\n"},{"id":"435348","messageId":"cover.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631157880.git.me@ttaylorr.com","subject":"[PATCH v2 0/9] packfile: avoid .idx rename races","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:21Z","receivedAt":"2021-09-09T23:24:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a relatively small reroll of mine and Ævar's series to prevent\npackfile-related races when moving the `.idx` into place before other auxiliary\nfiles are renamed.\n\nAll changes are cosmetic, but all of the feedback on the previous round was a\nstrict improvement in the overall quality, so it seemed pertinent to send an\nimproved version. Range-diff is below, and thanks in advance for your review.\n\nTaylor Blau (4):\n bulk-checkin.c: store checksum directly\n pack-write.c: rename `.idx` files after `*.rev`\n builtin/repack.c: move `.idx` files into place last\n builtin/index-pack.c: move `.idx` files into place last\n\nÆvar Arnfjörð Bjarmason (5):\n pack.h: line-wrap the definition of finish_tmp_packfile()\n pack-write: refactor renaming in finish_tmp_packfile()\n index-pack: refactor renaming in final()\n pack-write: split up finish_tmp_packfile() function\n pack-objects: rename .idx files into place after .bitmap files\n\n builtin/index-pack.c | 48 +++++++++++++++++------------------\n builtin/pack-objects.c | 15 ++++++++---\n builtin/repack.c | 2 +-\n bulk-checkin.c | 31 +++++++++++++++++------\n pack-write.c | 57 +++++++++++++++++++++---------------------\n pack.h | 10 +++++++-\n 6 files changed, 96 insertions(+), 67 deletions(-)\n\nRange-diff against v1:\n -: ---------- > 1: 0b07aa4947 pack.h: line-wrap the definition of finish_tmp_packfile()\n 1: 20b35ce050 ! 2: c46d3c29b4 bulk-checkin.c: store checksum directly\n @@ Commit message\n Store the hash directly in an unsigned char array. This behaves the same\n as writing to the `hash` member, but makes the intent clearer (and\n avoids allocating an extra four bytes for the `algo` member of `struct\n - object_id`).\n + object_id`). It likewise prevents the possibility of a segfault when\n + reading `algo` (e.g., by calling `oid_to_hex()`) if it is uninitialized.\n \n Signed-off-by: Taylor Blau \n \n 2: 35052ef494 ! 3: 2e907e309d pack-write: refactor renaming in finish_tmp_packfile()\n @@ pack-write.c: struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n \treturn hashfd(fd, *pack_tmp_name);\n }\n \n -+static void rename_tmp_packfile(struct strbuf *nb, const char *source,\n ++static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source,\n +\t\t\t\tconst char *ext)\n +{\n -+\tsize_t nb_len = nb->len;\n ++\tsize_t name_prefix_len = name_prefix->len;\n +\n -+\tstrbuf_addstr(nb, ext);\n -+\tif (rename(source, nb->buf))\n -+\t\tdie_errno(\"unable to rename temporary '*.%s' file to '%s'\",\n -+\t\t\t ext, nb->buf);\n -+\tstrbuf_setlen(nb, nb_len);\n ++\tstrbuf_addstr(name_prefix, ext);\n ++\tif (rename(source, name_prefix->buf))\n ++\t\tdie_errno(\"unable to rename temporary file to '%s'\",\n ++\t\t\t name_prefix->buf);\n ++\tstrbuf_setlen(name_prefix, name_prefix_len);\n +}\n +\n void finish_tmp_packfile(struct strbuf *name_buffer,\n 3: 0fb2c25f5a = 4: 41d34b1f18 pack-write.c: rename `.idx` files after `*.rev`\n 4: 3b10a97ec0 = 5: 6b340b7eba builtin/repack.c: move `.idx` files into place last\n 5: 3c9b515907 ! 6: 1e4d0ea0f3 index-pack: refactor renaming in final()\n @@ Commit message\n helper, this is both a net decrease in lines, and improves the\n readability, since we can easily see at a glance that the logic for\n writing these three types of files is exactly the same, aside from the\n - obviously differing cases of \"*final_xyz_name\" being NULL, and\n - \"else_chmod_if\" being different.\n + obviously differing cases of \"*final_name\" being NULL, and\n + \"make_read_only_if_same\" being different.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Taylor Blau \n @@ builtin/index-pack.c: static void write_special_file(const char *suffix, const c\n \tstrbuf_release(&name_buf);\n }\n \n -+static void rename_tmp_packfile(const char **final_xyz_name,\n -+\t\t\t\tconst char *curr_xyz_name,\n -+\t\t\t\tstruct strbuf *xyz_name, unsigned char *hash,\n -+\t\t\t\tconst char *ext, int else_chmod_if)\n ++static void rename_tmp_packfile(const char **final_name,\n ++\t\t\t\tconst char *curr_name,\n ++\t\t\t\tstruct strbuf *name, unsigned char *hash,\n ++\t\t\t\tconst char *ext, int make_read_only_if_same)\n +{\n -+\tif (*final_xyz_name != curr_xyz_name) {\n -+\t\tif (!*final_xyz_name)\n -+\t\t\t*final_xyz_name = odb_pack_name(xyz_name, hash, ext);\n -+\t\tif (finalize_object_file(curr_xyz_name, *final_xyz_name))\n ++\tif (*final_name != curr_name) {\n ++\t\tif (!*final_name)\n ++\t\t\t*final_name = odb_pack_name(name, hash, ext);\n ++\t\tif (finalize_object_file(curr_name, *final_name))\n +\t\t\tdie(_(\"unable to rename temporary '*.%s' file to '%s\"),\n -+\t\t\t ext, *final_xyz_name);\n -+\t} else if (else_chmod_if) {\n -+\t\tchmod(*final_xyz_name, 0444);\n ++\t\t\t ext, *final_name);\n ++\t} else if (make_read_only_if_same) {\n ++\t\tchmod(*final_name, 0444);\n +\t}\n +}\n +\n 6: 8d67a71501 = 7: 906e75d707 builtin/index-pack.c: move `.idx` files into place last\n 7: 5c553229b0 ! 8: 90bebe4e51 pack-write: split up finish_tmp_packfile() function\n @@ bulk-checkin.c: static struct bulk_checkin_state {\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \n ## pack-write.c ##\n -@@ pack-write.c: static void rename_tmp_packfile(struct strbuf *nb, const char *source,\n - \tstrbuf_setlen(nb, nb_len);\n +@@ pack-write.c: static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source,\n + \tstrbuf_setlen(name_prefix, name_prefix_len);\n }\n \n -void finish_tmp_packfile(struct strbuf *name_buffer,\n 8: d8286cf107 ! 9: 1409725509 pack-objects: rename .idx files into place after .bitmap files\n @@ Commit message\n about filesystem races in the face of doing and not doing fsync() (and\n if doing fsync(), not doing it properly).\n \n - In particular, in this case of writing to \".git/objects/pack\" we only\n - write and fsync() the individual files, but if we wanted to guarantee\n - that the metadata update was seen in that way by concurrent processes\n - we'd need to fsync() on the \"fd\" of the containing directory. That\n - concern is probably more theoretical than not, modern OS's tend to be\n - more on the forgiving side than the overly pedantic side of\n - implementing POSIX FS semantics.\n + We may want to fsync the containing directory once after renaming the\n + *.idx file into place, but that is outside the scope of this series.\n \n 1. https://lore.kernel.org/git/8735qgkvv1.fsf@evledraar.gmail.com/\n \n-- \n2.33.0.96.g73915697e6\n"},{"id":"435349","messageId":"0b07aa4947caff53878a9000e2866f98146e2ccd.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 1/9] pack.h: line-wrap the definition of finish_tmp_packfile()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:30Z","receivedAt":"2021-09-09T23:24:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nLine-wrap the definition of finish_tmp_packfile() to make subsequent\nchanges easier to read. See 0e990530ae (finish_tmp_packfile(): a\nhelper function, 2011-10-28) for the commit that introduced this\noverly long line.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n pack.h | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/pack.h b/pack.h\nindex fa13954526..1c17254c0a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,6 +110,11 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n-void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n+void finish_tmp_packfile(struct strbuf *name_buffer,\n+\t\t\t const char *pack_tmp_name,\n+\t\t\t struct pack_idx_entry **written_list,\n+\t\t\t uint32_t nr_written,\n+\t\t\t struct pack_idx_option *pack_idx_opts,\n+\t\t\t unsigned char sha1[]);\n \n #endif\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435350","messageId":"c46d3c29b44969beafc2bd27d33aa088eda0d4d6.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 2/9] bulk-checkin.c: store checksum directly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:32Z","receivedAt":"2021-09-09T23:24:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"finish_bulk_checkin() stores the checksum from finalize_hashfile() by\nwriting to the `hash` member of `struct object_id`, but that hash has\nnothing to do with an object id (it's just a convenient location that\nhappens to be sized correctly).\n\nStore the hash directly in an unsigned char array. This behaves the same\nas writing to the `hash` member, but makes the intent clearer (and\navoids allocating an extra four bytes for the `algo` member of `struct\nobject_id`). It likewise prevents the possibility of a segfault when\nreading `algo` (e.g., by calling `oid_to_hex()`) if it is uninitialized.\n\nSigned-off-by: Taylor Blau \n---\n bulk-checkin.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex b023d9959a..6283bc8bd9 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -25,7 +25,7 @@ static struct bulk_checkin_state {\n \n static void finish_bulk_checkin(struct bulk_checkin_state *state)\n {\n-\tstruct object_id oid;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct strbuf packname = STRBUF_INIT;\n \tint i;\n \n@@ -37,11 +37,11 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \t\tunlink(state->pack_tmp_name);\n \t\tgoto clear_exit;\n \t} else if (state->nr_written == 1) {\n-\t\tfinalize_hashfile(state->f, oid.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n+\t\tfinalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE);\n \t} else {\n-\t\tint fd = finalize_hashfile(state->f, oid.hash, 0);\n-\t\tfixup_pack_header_footer(fd, oid.hash, state->pack_tmp_name,\n-\t\t\t\t\t state->nr_written, oid.hash,\n+\t\tint fd = finalize_hashfile(state->f, hash, 0);\n+\t\tfixup_pack_header_footer(fd, hash, state->pack_tmp_name,\n+\t\t\t\t\t state->nr_written, hash,\n \t\t\t\t\t state->offset);\n \t\tclose(fd);\n \t}\n@@ -49,7 +49,7 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \tstrbuf_addf(&packname, \"%s/pack/pack-\", get_object_directory());\n \tfinish_tmp_packfile(&packname, state->pack_tmp_name,\n \t\t\t state->written, state->nr_written,\n-\t\t\t &state->pack_idx_opts, oid.hash);\n+\t\t\t &state->pack_idx_opts, hash);\n \tfor (i = 0; i < state->nr_written; i++)\n \t\tfree(state->written[i]);\n \n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435351","messageId":"2e907e309de6804c329a9efb6a6d89fb318683f3.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 3/9] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:37Z","receivedAt":"2021-09-09T23:24:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nRefactor the renaming in finish_tmp_packfile() into a helper function.\nThe callers are now expected to pass a \"name_buffer\" ending in\n\"pack-OID.\" instead of the previous \"pack-\", we then append \"pack\",\n\"idx\" or \"rev\" to it.\n\nBy doing the strbuf_setlen() in rename_tmp_packfile() we reuse the\nbuffer and avoid the repeated allocations we'd get if that function had\nits own temporary \"struct strbuf\".\n\nThis approach of reusing the buffer does make the last user in\npack-object.c's write_pack_file() slightly awkward, since we needlessly\ndo a strbuf_setlen() before calling strbuf_release() for consistency. In\nsubsequent changes we'll move that bitmap writing code around, so let's\nnot skip the strbuf_setlen() now.\n\nThe previous strbuf_reset() idiom originated with 5889271114a\n(finish_tmp_packfile():use strbuf for pathname construction,\n2014-03-03), which in turn was a minimal adjustment of pre-strbuf code\nadded in 0e990530ae (finish_tmp_packfile(): a helper function,\n2011-10-28).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++--\n bulk-checkin.c | 3 ++-\n pack-write.c | 37 ++++++++++++++++---------------------\n 3 files changed, 23 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex df49f656b9..2a105c8d6e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1237,7 +1237,8 @@ static void write_pack_file(void)\n \t\t\t\t\twarning_errno(_(\"failed utime() on %s\"), pack_tmp_name);\n \t\t\t}\n \n-\t\t\tstrbuf_addf(&tmpname, \"%s-\", base_name);\n+\t\t\tstrbuf_addf(&tmpname, \"%s-%s.\", base_name,\n+\t\t\t\t hash_to_hex(hash));\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tbitmap_writer_set_checksum(hash);\n@@ -1250,8 +1251,9 @@ static void write_pack_file(void)\n \t\t\t\t\t &pack_idx_opts, hash);\n \n \t\t\tif (write_bitmap_index) {\n-\t\t\t\tstrbuf_addf(&tmpname, \"%s.bitmap\", hash_to_hex(hash));\n+\t\t\t\tsize_t tmpname_len = tmpname.len;\n \n+\t\t\t\tstrbuf_addstr(&tmpname, \"bitmap\");\n \t\t\t\tstop_progress(&progress_state);\n \n \t\t\t\tbitmap_writer_show_progress(progress);\n@@ -1260,6 +1262,7 @@ static void write_pack_file(void)\n \t\t\t\tbitmap_writer_finish(written_list, nr_written,\n \t\t\t\t\t\t tmpname.buf, write_bitmap_options);\n \t\t\t\twrite_bitmap_index = 0;\n+\t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n \t\t\tstrbuf_release(&tmpname);\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 6283bc8bd9..c19d471f0b 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -46,7 +46,8 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \t\tclose(fd);\n \t}\n \n-\tstrbuf_addf(&packname, \"%s/pack/pack-\", get_object_directory());\n+\tstrbuf_addf(&packname, \"%s/pack/pack-%s.\", get_object_directory(),\n+\t\t hash_to_hex(hash));\n \tfinish_tmp_packfile(&packname, state->pack_tmp_name,\n \t\t\t state->written, state->nr_written,\n \t\t\t &state->pack_idx_opts, hash);\ndiff --git a/pack-write.c b/pack-write.c\nindex 2767b78619..58bc5fbcdf 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -458,6 +458,18 @@ struct hashfile *create_tmp_packfile(char **pack_tmp_name)\n \treturn hashfd(fd, *pack_tmp_name);\n }\n \n+static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source,\n+\t\t\t\tconst char *ext)\n+{\n+\tsize_t name_prefix_len = name_prefix->len;\n+\n+\tstrbuf_addstr(name_prefix, ext);\n+\tif (rename(source, name_prefix->buf))\n+\t\tdie_errno(\"unable to rename temporary file to '%s'\",\n+\t\t\t name_prefix->buf);\n+\tstrbuf_setlen(name_prefix, name_prefix_len);\n+}\n+\n void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n@@ -466,7 +478,6 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t unsigned char hash[])\n {\n \tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n-\tint basename_len = name_buffer->len;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n@@ -479,26 +490,10 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n \t\t\t\t pack_idx_opts->flags);\n \n-\tstrbuf_addf(name_buffer, \"%s.pack\", hash_to_hex(hash));\n-\n-\tif (rename(pack_tmp_name, name_buffer->buf))\n-\t\tdie_errno(\"unable to rename temporary pack file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n-\n-\tstrbuf_addf(name_buffer, \"%s.idx\", hash_to_hex(hash));\n-\tif (rename(idx_tmp_name, name_buffer->buf))\n-\t\tdie_errno(\"unable to rename temporary index file\");\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n-\n-\tif (rev_tmp_name) {\n-\t\tstrbuf_addf(name_buffer, \"%s.rev\", hash_to_hex(hash));\n-\t\tif (rename(rev_tmp_name, name_buffer->buf))\n-\t\t\tdie_errno(\"unable to rename temporary reverse-index file\");\n-\t}\n-\n-\tstrbuf_setlen(name_buffer, basename_len);\n+\trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n+\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n+\tif (rev_tmp_name)\n+\t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n \n \tfree((void *)idx_tmp_name);\n }\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435352","messageId":"41d34b1f180350c466243ed5aa7f6a48a2ff2aeb.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 4/9] pack-write.c: rename `.idx` files after `*.rev`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:41Z","receivedAt":"2021-09-09T23:24:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"We treat the presence of an `.idx` file as the indicator of whether or\nnot it's safe to use a packfile. But `finish_tmp_packfile()` (which is\nresponsible for writing the pack and moving the temporary versions of\nall of its auxiliary files into place) is inconsistent about the write\norder.\n\nSpecifically, it moves the `.rev` file into place after the `.idx`,\nleaving open the possibility to open a pack which looks \"ready\" (because\nthe `.idx` file exists and is readable) but appears momentarily to not\nhave a `.rev` file. This causes Git to fall back to generating the\npack's reverse index in memory.\n\nThough racy, this amounts to an unnecessary slow-down at worst, and\ndoesn't affect the correctness of the resulting reverse index.\n\nClose this race by moving the .rev file into place before moving the\n.idx file into place.\n\nThis still leaves the issue of `.idx` files being renamed into place\nbefore the auxiliary `.bitmap` file is renamed when in pack-object.c's\nwrite_pack_file() \"write_bitmap_index\" is true. That race will be\naddressed in subsequent commits.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n pack-write.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/pack-write.c b/pack-write.c\nindex 58bc5fbcdf..b9f9cd5c14 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -491,9 +491,9 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \t\t\t\t pack_idx_opts->flags);\n \n \trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n-\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n \tif (rev_tmp_name)\n \t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n+\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n \n \tfree((void *)idx_tmp_name);\n }\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435353","messageId":"6b340b7eba1509ac12291d2d1a20c146a2800f80.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 5/9] builtin/repack.c: move `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:44Z","receivedAt":"2021-09-09T23:24:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar spirit as the previous patch, fix the identical problem\nfrom `git repack` (which invokes `pack-objects` with a temporary\nlocation for output, and then moves the files into their final locations\nitself).\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5f9bc74adc..c3e4771609 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -208,10 +208,10 @@ static struct {\n \tunsigned optional:1;\n } exts[] = {\n \t{\".pack\"},\n-\t{\".idx\"},\n \t{\".rev\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n+\t{\".idx\"},\n };\n \n static unsigned populate_pack_exts(char *name)\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435354","messageId":"1e4d0ea0f398ba48cf1b1f87122ab343bdb296e2.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 6/9] index-pack: refactor renaming in final()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:49Z","receivedAt":"2021-09-09T23:25:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nRefactor the renaming in final() into a helper function, this is\nsimilar in spirit to a preceding refactoring of finish_tmp_packfile()\nin pack-write.c.\n\nBefore e37d0b8730b (builtin/index-pack.c: write reverse indexes,\n2021-01-25) it probably wasn't worth it to have this sort of helper,\ndue to the differing \"else if\" case for \"pack\" files v.s. \"idx\" files.\n\nBut since we've got \"rev\" as well now, let's do the renaming via a\nhelper, this is both a net decrease in lines, and improves the\nreadability, since we can easily see at a glance that the logic for\nwriting these three types of files is exactly the same, aside from the\nobviously differing cases of \"*final_name\" being NULL, and\n\"make_read_only_if_same\" being different.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 48 +++++++++++++++++++++-----------------------\n 1 file changed, 23 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 6cc4890217..3e3736cd95 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1477,6 +1477,22 @@ static void write_special_file(const char *suffix, const char *msg,\n \tstrbuf_release(&name_buf);\n }\n \n+static void rename_tmp_packfile(const char **final_name,\n+\t\t\t\tconst char *curr_name,\n+\t\t\t\tstruct strbuf *name, unsigned char *hash,\n+\t\t\t\tconst char *ext, int make_read_only_if_same)\n+{\n+\tif (*final_name != curr_name) {\n+\t\tif (!*final_name)\n+\t\t\t*final_name = odb_pack_name(name, hash, ext);\n+\t\tif (finalize_object_file(curr_name, *final_name))\n+\t\t\tdie(_(\"unable to rename temporary '*.%s' file to '%s\"),\n+\t\t\t ext, *final_name);\n+\t} else if (make_read_only_if_same) {\n+\t\tchmod(*final_name, 0444);\n+\t}\n+}\n+\n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n \t\t const char *final_rev_index_name, const char *curr_rev_index_name,\n@@ -1505,31 +1521,13 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\twrite_special_file(\"promisor\", promisor_msg, final_pack_name,\n \t\t\t\t hash, NULL);\n \n-\tif (final_pack_name != curr_pack_name) {\n-\t\tif (!final_pack_name)\n-\t\t\tfinal_pack_name = odb_pack_name(&pack_name, hash, \"pack\");\n-\t\tif (finalize_object_file(curr_pack_name, final_pack_name))\n-\t\t\tdie(_(\"cannot store pack file\"));\n-\t} else if (from_stdin)\n-\t\tchmod(final_pack_name, 0444);\n-\n-\tif (final_index_name != curr_index_name) {\n-\t\tif (!final_index_name)\n-\t\t\tfinal_index_name = odb_pack_name(&index_name, hash, \"idx\");\n-\t\tif (finalize_object_file(curr_index_name, final_index_name))\n-\t\t\tdie(_(\"cannot store index file\"));\n-\t} else\n-\t\tchmod(final_index_name, 0444);\n-\n-\tif (curr_rev_index_name) {\n-\t\tif (final_rev_index_name != curr_rev_index_name) {\n-\t\t\tif (!final_rev_index_name)\n-\t\t\t\tfinal_rev_index_name = odb_pack_name(&rev_index_name, hash, \"rev\");\n-\t\t\tif (finalize_object_file(curr_rev_index_name, final_rev_index_name))\n-\t\t\t\tdie(_(\"cannot store reverse index file\"));\n-\t\t} else\n-\t\t\tchmod(final_rev_index_name, 0444);\n-\t}\n+\trename_tmp_packfile(&final_pack_name, curr_pack_name, &pack_name,\n+\t\t\t hash, \"pack\", from_stdin);\n+\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n+\t\t\t hash, \"idx\", 1);\n+\tif (curr_rev_index_name)\n+\t\trename_tmp_packfile(&final_rev_index_name, curr_rev_index_name,\n+\t\t\t\t &rev_index_name, hash, \"rev\", 1);\n \n \tif (do_fsck_object) {\n \t\tstruct packed_git *p;\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435355","messageId":"90bebe4e51c3bc84d156912db84e0d2774f56354.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 8/9] pack-write: split up finish_tmp_packfile() function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:56Z","receivedAt":"2021-09-09T23:25:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nSplit up the finish_tmp_packfile() function and use the split-up version\nin pack-objects.c in preparation for moving the step of renaming the\n*.idx file later as part of a function change.\n\nSince the only other caller of finish_tmp_packfile() was in\nbulk-checkin.c, and it won't be needing a change to its *.idx renaming,\nprovide a thin wrapper for the old function as a static function in that\nfile. If other callers end up needing the simpler version it could be\nmoved back to \"pack-write.c\" and \"pack.h\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 7 +++++--\n bulk-checkin.c | 16 ++++++++++++++++\n pack-write.c | 22 +++++++++++++---------\n pack.h | 7 +++++--\n 4 files changed, 39 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2a105c8d6e..944134b6f2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1217,6 +1217,7 @@ static void write_pack_file(void)\n \t\tif (!pack_to_stdout) {\n \t\t\tstruct stat st;\n \t\t\tstruct strbuf tmpname = STRBUF_INIT;\n+\t\t\tchar *idx_tmp_name = NULL;\n \n \t\t\t/*\n \t\t\t * Packs are runtime accessed in their mtime\n@@ -1246,9 +1247,10 @@ static void write_pack_file(void)\n \t\t\t\t\t&to_pack, written_list, nr_written);\n \t\t\t}\n \n-\t\t\tfinish_tmp_packfile(&tmpname, pack_tmp_name,\n+\t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n-\t\t\t\t\t &pack_idx_opts, hash);\n+\t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n+\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tsize_t tmpname_len = tmpname.len;\n@@ -1265,6 +1267,7 @@ static void write_pack_file(void)\n \t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n+\t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmpname);\n \t\t\tfree(pack_tmp_name);\n \t\t\tputs(hash_to_hex(hash));\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex c19d471f0b..8785b2ac80 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -23,6 +23,22 @@ static struct bulk_checkin_state {\n \tuint32_t nr_written;\n } state;\n \n+static void finish_tmp_packfile(struct strbuf *basename,\n+\t\t\t\tconst char *pack_tmp_name,\n+\t\t\t\tstruct pack_idx_entry **written_list,\n+\t\t\t\tuint32_t nr_written,\n+\t\t\t\tstruct pack_idx_option *pack_idx_opts,\n+\t\t\t\tunsigned char hash[])\n+{\n+\tchar *idx_tmp_name = NULL;\n+\n+\tstage_tmp_packfiles(basename, pack_tmp_name, written_list, nr_written,\n+\t\t\t pack_idx_opts, hash, &idx_tmp_name);\n+\trename_tmp_packfile_idx(basename, &idx_tmp_name);\n+\n+\tfree(idx_tmp_name);\n+}\n+\n static void finish_bulk_checkin(struct bulk_checkin_state *state)\n {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/pack-write.c b/pack-write.c\nindex b9f9cd5c14..c21aed64b3 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -470,21 +470,28 @@ static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source,\n \tstrbuf_setlen(name_prefix, name_prefix_len);\n }\n \n-void finish_tmp_packfile(struct strbuf *name_buffer,\n+void rename_tmp_packfile_idx(struct strbuf *name_buffer,\n+\t\t\t char **idx_tmp_name)\n+{\n+\trename_tmp_packfile(name_buffer, *idx_tmp_name, \"idx\");\n+}\n+\n+void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n-\t\t\t unsigned char hash[])\n+\t\t\t unsigned char hash[],\n+\t\t\t char **idx_tmp_name)\n {\n-\tconst char *idx_tmp_name, *rev_tmp_name = NULL;\n+\tconst char *rev_tmp_name = NULL;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n \n-\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n-\t\t\t\t pack_idx_opts, hash);\n-\tif (adjust_shared_perm(idx_tmp_name))\n+\t*idx_tmp_name = (char *)write_idx_file(NULL, written_list, nr_written,\n+\t\t\t\t\t pack_idx_opts, hash);\n+\tif (adjust_shared_perm(*idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n \n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n@@ -493,9 +500,6 @@ void finish_tmp_packfile(struct strbuf *name_buffer,\n \trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n \tif (rev_tmp_name)\n \t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n-\trename_tmp_packfile(name_buffer, idx_tmp_name, \"idx\");\n-\n-\tfree((void *)idx_tmp_name);\n }\n \n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought)\ndiff --git a/pack.h b/pack.h\nindex 1c17254c0a..b22bfc4a18 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -110,11 +110,14 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n int read_pack_header(int fd, struct pack_header *);\n \n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n-void finish_tmp_packfile(struct strbuf *name_buffer,\n+void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n-\t\t\t unsigned char sha1[]);\n+\t\t\t unsigned char hash[],\n+\t\t\t char **idx_tmp_name);\n+void rename_tmp_packfile_idx(struct strbuf *basename,\n+\t\t\t char **idx_tmp_name);\n \n #endif\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435356","messageId":"906e75d7076e547fe2947dc701054e554f29dedb.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 7/9] builtin/index-pack.c: move `.idx` files into place last","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:24:53Z","receivedAt":"2021-09-09T23:25:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a similar spirit as preceding patches to `git repack` and `git\npack-objects`, fix the identical problem in `git index-pack`.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/index-pack.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 3e3736cd95..f267dce49e 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1523,11 +1523,11 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \n \trename_tmp_packfile(&final_pack_name, curr_pack_name, &pack_name,\n \t\t\t hash, \"pack\", from_stdin);\n-\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n-\t\t\t hash, \"idx\", 1);\n \tif (curr_rev_index_name)\n \t\trename_tmp_packfile(&final_rev_index_name, curr_rev_index_name,\n \t\t\t\t &rev_index_name, hash, \"rev\", 1);\n+\trename_tmp_packfile(&final_index_name, curr_index_name, &index_name,\n+\t\t\t hash, \"idx\", 1);\n \n \tif (do_fsck_object) {\n \t\tstruct packed_git *p;\n-- \n2.33.0.96.g73915697e6\n\n"},{"id":"435357","messageId":"1409725509fad5708abc2586c5679887561c5ed6.1631228928.git.me@ttaylorr.com","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"[PATCH v2 9/9] pack-objects: rename .idx files into place after .bitmap files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:25:00Z","receivedAt":"2021-09-09T23:25:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Ævar Arnfjörð Bjarmason \n\nIn preceding commits the race of renaming .idx files in place before\n.rev files and other auxiliary files was fixed in pack-write.c's\nfinish_tmp_packfile(), builtin/repack.c's \"struct exts\", and\nbuiltin/index-pack.c's final(). As noted in the change to pack-write.c\nwe left in place the issue of writing *.bitmap files after the *.idx,\nlet's fix that issue.\n\nSee 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21)\nfor commentary at the time when *.bitmap was implemented about how\nthose files are written out, nothing in that commit contradicts what's\nbeing done here.\n\nNote that this commit and preceding ones only close any race condition\nwith *.idx files being written before their auxiliary files if we're\noptimistic about our lack of fsync()-ing in this are not tripping us\nover. See the thread at [1] for a rabbit hole of various discussions\nabout filesystem races in the face of doing and not doing fsync() (and\nif doing fsync(), not doing it properly).\n\nWe may want to fsync the containing directory once after renaming the\n*.idx file into place, but that is outside the scope of this series.\n\n1. https://lore.kernel.org/git/8735qgkvv1.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 944134b6f2..a01767a384 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1250,7 +1250,6 @@ static void write_pack_file(void)\n \t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n \t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n-\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tsize_t tmpname_len = tmpname.len;\n@@ -1267,6 +1266,8 @@ static void write_pack_file(void)\n \t\t\t\tstrbuf_setlen(&tmpname, tmpname_len);\n \t\t\t}\n \n+\t\t\trename_tmp_packfile_idx(&tmpname, &idx_tmp_name);\n+\n \t\t\tfree(idx_tmp_name);\n \t\t\tstrbuf_release(&tmpname);\n \t\t\tfree(pack_tmp_name);\n-- \n2.33.0.96.g73915697e6\n"},{"id":"435358","messageId":"xmqqilz9wcm9.fsf@gitster.g","threadId":"56421","inReplyTo":"YTp3rYqxLE5/X521@nand.local","subject":"Re: [PATCH 3/9] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T23:30:38Z","receivedAt":"2021-09-09T23:30:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> In each of these three, I wasn't able to decide if you wanted these\n> addressed in a newer version of this series, or if you were happy enough\n> with the result to pick it up. I'm happy to send you a new version, but\n> don't want to clog your inbox if you were already planning on picking\n> this up.\n\nWell, it is often a no-cost operation to replace a topic that has\nbeen in 'seen' with a newer round, so you do not have to worry about\nmy inbox. As I often say, if it turns out to be a bad idea, I can\njust drop it from 'seen' as if I didn't see it ;-)\n\nAnyway.\n\nIf a newer version will come, I'd love to see the review comments at\nleast considered (be it from me or from anybody else) --- after all,\nif the original were good enough, reviewer(s) wouldn't have raised\nthem as potential issues.\n\nOf course, \"considered\" is the key word. No need to blindly follow;\nas long as there is a solid reason to justify why changing would be\nworsening the well-written original.\n\nThanks.\n"},{"id":"435359","messageId":"YTqZXN8pVRtODQjb@nand.local","threadId":"56421","inReplyTo":"xmqqilz9wcm9.fsf@gitster.g","subject":"Re: [PATCH 3/9] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-09T23:31:40Z","receivedAt":"2021-09-09T23:31:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 09, 2021 at 04:30:38PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > In each of these three, I wasn't able to decide if you wanted these\n> > addressed in a newer version of this series, or if you were happy enough\n> > with the result to pick it up. I'm happy to send you a new version, but\n> > don't want to clog your inbox if you were already planning on picking\n> > this up.\n>\n> Well, it is often a no-cost operation to replace a topic that has\n> been in 'seen' with a newer round, so you do not have to worry about\n> my inbox. As I often say, if it turns out to be a bad idea, I can\n> just drop it from 'seen' as if I didn't see it ;-)\n>\n> Anyway.\n>\n> If a newer version will come, I'd love to see the review comments at\n> least considered (be it from me or from anybody else) --- after all,\n> if the original were good enough, reviewer(s) wouldn't have raised\n> them as potential issues.\n>\n> Of course, \"considered\" is the key word. No need to blindly follow;\n> as long as there is a solid reason to justify why changing would be\n> worsening the well-written original.\n\nAgreed strongly :-). After reading through your review again, I felt\nlike the individual components were greater than the sum of their parts,\nand that the v2 of this combined series is on the whole better than v1.\n\nThanks for taking the time to review it.\n\nThanks,\nTaylor\n"},{"id":"435362","messageId":"xmqqa6klw74m.fsf@gitster.g","threadId":"56421","inReplyTo":"YTp3rYqxLE5/X521@nand.local","subject":"Re: [PATCH 3/9] pack-write: refactor renaming in finish_tmp_packfile()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-10T01:29:13Z","receivedAt":"2021-09-10T01:29:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Ævar makes a slight mention of this in their commit message:\n>\n> This approach of reusing the buffer does make the last user in\n> pack-object.c's write_pack_file() slightly awkward, since we\n> needlessly do a strbuf_setlen() before calling strbuf_release() for\n> consistency. In subsequent changes we'll move that bitmap writing\n> code around, so let's not skip the strbuf_setlen() now.\n>\n> And this is to set us up for the final patch which moves the call to\n> rename_tmp_packfile_idx(&tmpname, &idx_tmp_name) after the closing brace\n> in the quoted portion of your message and the strbuf_release(). There,\n> we'll want the strbuf reset to the appropriate contents and length, and\n> not released.\n>\n> But in the meantime it is awkward.\n\nThat is why I said \"adding setlen only when t starts to matter may\nmake it easier to follow\". It won't make it awkward at this step,\nand it will make it stand out why it is needed when it is added,\npresumably at the very end, when rename_tmp_packfile_idx() call is\nadded after this if() block.\n\n\n\n"},{"id":"435363","messageId":"xmqq5yv9w6u0.fsf@gitster.g","threadId":"56421","inReplyTo":"cover.1631228928.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/9] packfile: avoid .idx rename races","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-10T01:35:35Z","receivedAt":"2021-09-10T01:35:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Here is a relatively small reroll of mine and Ævar's series to prevent\n> packfile-related races when moving the `.idx` into place before other auxiliary\n> files are renamed.\n\nWill replace. I didn't see anything questionable in range-diff (of\ncourse, lack of anything is hard to spot but at least there is no\nnew iffy stuff and there wasn't anything iffy in the original, so\n... ;-)\n"}]}