{"thread":{"id":"44577","subject":"'git repack' and repack.writeBitmaps=true with kept packs","startedAt":"2016-12-01T00:15:41Z","lastAt":"2016-12-01T05:16:49Z","messageCount":2,"participants":["Steven Noonan","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"306650","messageId":"CAKbGBLjZ2WLVRM9f=by337xLhPgKCy10T8ra6Qz7OWA=QF-5yA@mail.gmail.com","threadId":"44577","inReplyTo":null,"subject":"'git repack' and repack.writeBitmaps=true with kept packs","fromName":"Steven Noonan","fromEmail":"steven@uplinklabs.net","sentAt":"2016-12-01T00:15:33Z","receivedAt":"2016-12-01T00:15:41Z","isPatch":false,"sender":{"key":"steven@uplinklabs.net","avatar":"https://gravatar.com/avatar/b0cd397a10638433f76e084531aa0af3bef85f8fdb59b1ebe2ddaf168cd100e9?d=mp&s=160"},"body":"I have some unexpected behavior with 'git repack' on git 2.10.2 and 2.11.0.\n\n$ cat /etc/gitconfig\n[pack]\n        writeBitmapHashCache = true\n[repack]\n        writeBitmaps = true\n$ touch objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.keep\n$ find objects\nobjects\nobjects/pack\nobjects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.keep\nobjects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.idx\nobjects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.bitmap\nobjects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.pack\nobjects/info\nobjects/info/packs\n$ git repack -Adfl\nCounting objects: 16321, done.\nDelta compression using up to 24 threads.\nCompressing objects: 100% (16118/16118), done.\nWriting objects: 100% (16321/16321), done.\nReusing bitmaps: 110, done.\nSelecting bitmap commits: 3257, done.\nBuilding bitmaps: 100% (137/137), done.\nTotal 16321 (delta 11568), reused 4507 (delta 0)\n$ du -sh objects/pack/pack*\n100K    objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.bitmap\n524K    objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.idx\n512     objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.keep\n4.3M    objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.pack\n100K    objects/pack/pack-6043151bee7bd61bdae6e3a2ba0f13cd1b0277af.bitmap\n524K    objects/pack/pack-6043151bee7bd61bdae6e3a2ba0f13cd1b0277af.idx\n5.8M    objects/pack/pack-6043151bee7bd61bdae6e3a2ba0f13cd1b0277af.pack\n\nIt seems like it's behaving as though I've provided\n--pack-kept-objects. In order to ensure the .bitmap is created, it\nrepacks everything, including everything in existing .pack files (not\nrespecting .keep). But then it's not deleting the old .pack file\n(oddly, respecting .keep).\n\nWhat I'd expect it to do here is ignore the 'repack.writeBitmaps =\ntrue' value if there's a .keep that needs to be respected. Is this not\na correct assumption?\n"},{"id":"306674","messageId":"20161201051640.gsavaexw55mwycza@sigill.intra.peff.net","threadId":"44577","inReplyTo":"CAKbGBLjZ2WLVRM9f=by337xLhPgKCy10T8ra6Qz7OWA=QF-5yA@mail.gmail.com","subject":"Re: 'git repack' and repack.writeBitmaps=true with kept packs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-12-01T05:16:41Z","receivedAt":"2016-12-01T05:16:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 30, 2016 at 04:15:33PM -0800, Steven Noonan wrote:\n\n> It seems like it's behaving as though I've provided\n> --pack-kept-objects. In order to ensure the .bitmap is created, it\n> repacks everything, including everything in existing .pack files (not\n> respecting .keep). But then it's not deleting the old .pack file\n> (oddly, respecting .keep).\n\nRight, that's exactly what's happening.\n\nThe bitmaps require a completely reachable set inside the pack, so if\nyou omit some objects that are in .keep packs, we cannot generate the\nbitmap. So we have to either disable bitmaps, or pack the kept objects.\nBy default, we do the latter (and I'll explain why in a minute).\n\nWe can't delete the .keep packfiles because we don't know for sure that\nwe've included all of their contents in the new pack (not to mention\nthat somebody asked to keep them, and we don't know why; we should\nrespect that).\n\n> What I'd expect it to do here is ignore the 'repack.writeBitmaps =\n> true' value if there's a .keep that needs to be respected. Is this not\n> a correct assumption?\n\nIn practice, I think that ends up worse. The .keep files are used by\nreceive-pack as lockfiles for incoming pushes. So imagine you kick off a\nfull repack just as somebody is pushing, and repack sees the temporary\n.keep file. Your options are:\n\n  1. Disable bitmaps, leaving the repository with no bitmaps at all\n     until the next repack (because you're deleting the old bitmaps\n     along with the old, non-kept pack).\n\n  2. Duplicate the newly pushed objects in the pack (if they're even\n     reachable; you're also racing to see the ref updates). Now you have\n     bitmaps, but you're wasting a bit of space to store the racy push\n     twice (and it goes away next time you repack).\n\nIf you're running a Git server which depends on bitmaps for good\nperformance, then (2) is much better. And that's the default.\n\nIf you want to override it, you can pass --no-pack-kept-objects, or set\nrepack.packKeptObjects to false.\n\nI think the documentation for --pack-kept-objects could be a bit more\nclear for this case. It doesn't mention the default value, nor that what\nyou really want with \"-b\" is probably \"--no-pack-kept-objects\".\n\n-Peff\n"}]}