{"thread":{"id":"42795","subject":"[PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","startedAt":"2016-07-07T19:47:26Z","lastAt":"2016-09-13T07:50:13Z","messageCount":62,"participants":["Kirill Smelkov","Jeff King","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"290973","messageId":"20160707190917.20011-1-kirr@nexedi.com","threadId":"42795","inReplyTo":null,"subject":"[PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-07T19:09:17Z","receivedAt":"2016-07-07T19:47:26Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nWe can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\n (at least that's my understanding after briefly looking at the code)\n\nWe also need to care and teach add_object_entry_from_bitmap() to respect\n--local via not adding nonlocal loose object to resultant pack (this\nis bitmap-codepath counterpart of daae0625 (pack-objects: extend --local\nto mean ignore non-local loose objects too) -- not to break 'loose\nobjects in alternate ODB are not repacked' in t7700-repack.sh .\n\nOtherwise all git tests pass, and for pack-objects -> file we get nice\nspeedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nCc: Vicent Marti \nCc: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n builtin/pack-objects.c | 7 +++++--\n t/t5310-pack-bitmaps.sh | 9 +++++++++\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a2f8cfd..be0ebe8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1052,6 +1052,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n {\n \tuint32_t index_pos;\n \n+\tif (local && has_loose_object_nonlocal(sha1))\n+\t\treturn 0;\n+\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n@@ -2488,7 +2491,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (prepare_bitmap_walk(revs) < 0)\n \t\treturn -1;\n \n-\tif (pack_options_allow_reuse() &&\n+\tif (pack_options_allow_reuse() && pack_to_stdout &&\n \t !reuse_partial_packfile_from_bitmap(\n \t\t\t&reuse_packfile,\n \t\t\t&reuse_packfile_objects,\n@@ -2773,7 +2776,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..533fc31 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -118,6 +118,15 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# pack-objects uses bitmap index by default, when it is available\n+\tpacksha1=$(git pack-objects --all mypack output &&\n-- \n2.9.0.431.gb11dac7.dirty\n"},{"id":"290985","messageId":"20160707205223.GA11753@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160707190917.20011-1-kirr@nexedi.com","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-07T20:52:23Z","receivedAt":"2016-07-07T20:52:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 07, 2016 at 10:09:17PM +0300, Kirill Smelkov wrote:\n\n> Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\n> if a repository has bitmap index, pack-objects can nicely speedup\n> \"Counting objects\" graph traversal phase. That however was done only for\n> case when resultant pack is sent to stdout, not written into a file.\n> \n> We can teach pack-objects to use bitmap index for initial object\n> counting phase when generating resultant pack file too:\n\nI'm not sure this is a good idea in general. When bitmaps are in use, we\ncannot fill out the details in the object-packing list as thoroughly. In\nparticular:\n\n - we will not compute the same write order (which is based on\n traversal order), leading to packs that have less efficient cache\n characteristics\n\n - we don't learn about the filename of trees and blobs, which is going\n to make the delta step much less efficient. This might be mitigated\n by turning on the bitmap name-hash cache; I don't recall how much\n detail pack-objects needs on the name (i.e., the full name versus\n just the hash).\n\nThere may be other subtle things, too. The general idea of tying the\nbitmap use to pack_to_stdout is that you _do_ want to use it for\nserving fetches and pushes, but for a full on-disk repack via gc, it's\nmore important to generate a good pack.\n\nYour use case:\n\n> git-backup extracts many packs on repositories restoration. That was my\n> initial motivation for the patch.\n\nSeems to be somewhere in between. I'm not sure I understand how you're\ninvoking pack-objects here, but I wonder if you should be using\n\"pack-objects --stdout\" yourself.\n\nBut even if it is the right thing for your use case to be using bitmaps\nto generate an on-disk bitmap, I think we should be making sure it\n_doesn't_ trigger when doing a normal repack.\n\n-Peff\n"},{"id":"291028","messageId":"20160708103855.GD5462@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160707205223.GA11753@sigill.intra.peff.net","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-08T10:38:55Z","receivedAt":"2016-07-08T10:39:06Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Peff first of all thanks for feedback,\n\nOn Thu, Jul 07, 2016 at 04:52:23PM -0400, Jeff King wrote:\n> On Thu, Jul 07, 2016 at 10:09:17PM +0300, Kirill Smelkov wrote:\n> \n> > Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\n> > if a repository has bitmap index, pack-objects can nicely speedup\n> > \"Counting objects\" graph traversal phase. That however was done only for\n> > case when resultant pack is sent to stdout, not written into a file.\n> > \n> > We can teach pack-objects to use bitmap index for initial object\n> > counting phase when generating resultant pack file too:\n> \n> I'm not sure this is a good idea in general. When bitmaps are in use, we\n> cannot fill out the details in the object-packing list as thoroughly. In\n> particular:\n> \n> - we will not compute the same write order (which is based on\n> traversal order), leading to packs that have less efficient cache\n> characteristics\n\nI agree the order can be not exactly the same. Still if original pack is\npacked well (with good recency order), while using bitmap we will tend\nto traverse it in close to original order.\n\nMaybe I'm not completely right on this, but to me it looks to be the\ncase because if objects in original pack are put there linearly sorted\nby recency order, and we use bitmap index to set of all reachable\nobjects from a root, and then just _linearly_ gather all those objects\nfrom original pack by 1s in bitmap and put them in the same order into\ndestination pack, the recency order won't be broken.\n\nOr am I maybe misunderstanding something?\n\nPlease also see below:\n\n> - we don't learn about the filename of trees and blobs, which is going\n> to make the delta step much less efficient. This might be mitigated\n> by turning on the bitmap name-hash cache; I don't recall how much\n> detail pack-objects needs on the name (i.e., the full name versus\n> just the hash).\n\nIf I understand it right, it uses only uint32_t name hash while searching. From\npack-objects.{h,c} :\n\n---- 8< ----\nstruct object_entry {\n\t...\n\tuint32_t hash; /* name hint hash */\n\n\n/*\n * We search for deltas in a list sorted by type, by filename hash, and then\n * by size, so that we see progressively smaller and smaller files.\n * That's because we prefer deltas to be from the bigger file\n * to the smaller -- deletes are potentially cheaper, but perhaps\n * more importantly, the bigger file is likely the more recent\n * one. The deepest deltas are therefore the oldest objects which are\n * less susceptible to be accessed often.\n */\nstatic int type_size_sort(const void *_a, const void *_b)\n{\n const struct object_entry *a = *(struct object_entry **)_a;\n const struct object_entry *b = *(struct object_entry **)_b;\n\n if (a->type > b->type)\n return -1;\n if (a->type < b->type) \n return 1;\n if (a->hash > b->hash)\n return -1;\n if (a->hash < b->hash)\n return 1;\n\t...\n---- 8< ----\n\nDocumentation/technical/pack-heuristics.txt also confirms this:\n\n---- 8< ----\n ...\n The quote from the above linus should be rewritten a\n bit (wait for it):\n - first sort by type. Different objects never delta with\n each other.\n - then sort by filename/dirname. hash of the basename\n occupies the top BITS_PER_INT-DIR_BITS bits, and bottom\n DIR_BITS are for the hash of leading path elements.\n\n ...\n\n If I might add, the trick is to make files that _might_ be similar be\n located close to each other in the hash buckets based on their file\n names. It used to be that \"foo/Makefile\", \"bar/baz/quux/Makefile\" and\n \"Makefile\" all landed in the same bucket due to their common basename,\n \"Makefile\". However, now they land in \"close\" buckets.\n \n The algorithm allows not just for the _same_ bucket, but for _close_\n buckets to be considered delta candidates. The rationale is\n essentially that files, like Makefiles, often have very similar\n content no matter what directory they live in.\n---- 8< ----\n\n\nSo yes, exactly as you say with pack.writeBitmapHashCache=true (ae4f07fb) the\ndelta-search heuristics is almost as efficient as with just raw filenames.\n\nI can confirm this also via e.g. (with my patch applied) :\n\n---- 8< ----\n$ time echo 0186ac99 | git pack-objects --no-use-bitmap-index --revs erp5pack-plain\nCounting objects: 627171, done.\nCompressing objects: 100% (176949/176949), done.\n50570987560d481742af4a8083028c2322a0534a\nWriting objects: 100% (627171/627171), done.\nTotal 627171 (delta 439404), reused 594820 (delta 410210)\n\nreal 0m37.272s\nuser 0m33.648s\nsys 0m1.580s\n\n$ time echo 0186ac99 | git pack-objects --revs erp5pack-bitmap\nCounting objects: 627171, done.\nCompressing objects: 100% (176914/176914), done.\n7c15a9b1eca1326e679297b217c5a48954625ca2\nWriting objects: 100% (627171/627171), done.\nTotal 627171 (delta 439484), reused 594855 (delta 410245)\n\nreal 0m27.020s\nuser 0m23.364s\nsys 0m0.992s\n\n$ ll erp5pack-{plain,bitmap}*\n\t 17561860 erp5pack-bitmap-7c15a9b1eca1326e679297b217c5a48954625ca2.idx\n\t 238760161 erp5pack-bitmap-7c15a9b1eca1326e679297b217c5a48954625ca2.pack\n\t 17561860 erp5pack-plain-50570987560d481742af4a8083028c2322a0534a.idx\n\t 238634201 erp5pack-plain-50570987560d481742af4a8083028c2322a0534a.pack\n---- 8< ----\n\n( By the way about pack generated with bitmap retaining close recency\n order:\n\n ---- 8< ----\n $ git verify-pack -v erp5pack-plain-50570987560d481742af4a8083028c2322a0534a.pack >1\n $ git verify-pack -v erp5pack-bitmap-7c15a9b1eca1326e679297b217c5a48954625ca2.pack >2\n $ grep commit 1 |awk '{print $1}' >1.commit\n $ grep commit 2 |awk '{print $1}' >2.commit\n $ wc -l 1.commit\n 46136 1.commit\n $ wc -l 2.commit\n 46136 2.commit\n $ diff -u0 1.commit 2.commit |wc -l\n 55\n ---- 8< ----\n \n so 55/46136 shows it is very almost the same. )\n\n\n> There may be other subtle things, too. The general idea of tying the\n> bitmap use to pack_to_stdout is that you _do_ want to use it for\n> serving fetches and pushes, but for a full on-disk repack via gc, it's\n> more important to generate a good pack.\n\nIt is better we send good packs to clients too, right? And with\npack.writeBitmapHashCache=true and retaining recency order (please see\nabove, but again maybe I'm not completely right) to me we should be still\ngenerating a good pack while using bitmap reachability index for object\ngraph traversal.\n\n> Your use case:\n> \n> > git-backup extracts many packs on repositories restoration. That was my\n> > initial motivation for the patch.\n> \n> Seems to be somewhere in between. I'm not sure I understand how you're\n> invoking pack-objects here,\n\nIt is just\n\n pack-objects --revs --reuse-object --reuse-delta --delta-base-offset extractedrepo/objects/pack/pack < SHA1-HEADS\n\n https://lab.nexedi.com/kirr/git-backup/blob/7fcb8c67/git-backup.go#L829\n\n> but I wonder if you should be using \"pack-objects --stdout\" yourself.\n\nI already tried --stdout. The problem is on repository extraction we\nneed to both extract the pack and index it. While `pack-object file`\ndoes both, for --stdout case we need to additionally index extracted\npack with `git index-pack`, and standalone `git index-pack` is very slow\n- in my experience much slower than generating the pack itself:\n\n---- 8< ----\n$ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\nCounting objects: 627171, done.\nCompressing objects: 100% (176914/176914), done.\nTotal 627171 (delta 439484), reused 594855 (delta 410245)\n\nreal 0m22.309s\nuser 0m21.148s\nsys 0m0.932s\n\n$ ll erp5pack-stdout*\n 238760161 erp5pack-stdout.pack\n\n$ time git index-pack erp5pack-stdout.pack\n7c15a9b1eca1326e679297b217c5a48954625ca2\n\nreal 0m50.873s <-- more than 2 times slower than time to generate pack itself!\nuser 0m49.300s\nsys 0m1.360s\n\n$ ll erp5pack-stdout*\n 17561860 erp5pack-stdout.idx\n 238760161 erp5pack-stdout.pack\n---- 8< ----\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n \nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n \nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\n\nI've tried to briefly see why index-pack is so slow and offhand I can\nsee that it needs to load all objects, decompresses them etc (maybe I'm\nnot so right here - I looked only briefly), while pack-objects while\ngenerating the pack has all needed information directly at hand and thus\ncan emit index much more easily.\n\nFor sever - clients scenario, index-pack load is put onto clients thus\noffloading server, but for my use case where extracted repository is on\nthe same machine the load does not go away.\n\nThat's why for me it makes more sense to emit both pack and its index in\none go.\n\nStill it would be interesting to eventually see why index-pack is so\nanomaly slow.\n\n> But even if it is the right thing for your use case to be using bitmaps\n> to generate an on-disk bitmap, I think we should be making sure it\n> _doesn't_ trigger when doing a normal repack.\n\nSo seems the way forward here is to teach pack-objects not to silently\ndrop explicit --use-pack-bitmap for cases when it can handle it?\n(currently even if this option was given, for !stdout cases pack-objects\nsimply drop use_bitmap_index to 0).\n\nAnd to make sure default for use_bitmap_index is 0 for !stdout cases?\n\nOr are we fine with my arguments about recency order staying the same\nwhen using bitmap reachability index for object graph traversal, and this\nway the patch is fine to go in as it is?\n\nThanks again,\nKirill\n"},{"id":"291295","messageId":"20160712190807.GB4809@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160708103855.GD5462@teco.navytux.spb.ru","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-12T19:08:08Z","receivedAt":"2016-07-12T19:08:23Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote:\n> Peff first of all thanks for feedback,\n> \n> On Thu, Jul 07, 2016 at 04:52:23PM -0400, Jeff King wrote:\n> > On Thu, Jul 07, 2016 at 10:09:17PM +0300, Kirill Smelkov wrote:\n> > \n> > > Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\n> > > if a repository has bitmap index, pack-objects can nicely speedup\n> > > \"Counting objects\" graph traversal phase. That however was done only for\n> > > case when resultant pack is sent to stdout, not written into a file.\n> > > \n> > > We can teach pack-objects to use bitmap index for initial object\n> > > counting phase when generating resultant pack file too:\n> > \n> > I'm not sure this is a good idea in general. When bitmaps are in use, we\n> > cannot fill out the details in the object-packing list as thoroughly. In\n> > particular:\n> > \n> > - we will not compute the same write order (which is based on\n> > traversal order), leading to packs that have less efficient cache\n> > characteristics\n> \n> I agree the order can be not exactly the same. Still if original pack is\n> packed well (with good recency order), while using bitmap we will tend\n> to traverse it in close to original order.\n> \n> Maybe I'm not completely right on this, but to me it looks to be the\n> case because if objects in original pack are put there linearly sorted\n> by recency order, and we use bitmap index to set of all reachable\n> objects from a root, and then just _linearly_ gather all those objects\n> from original pack by 1s in bitmap and put them in the same order into\n> destination pack, the recency order won't be broken.\n> \n> Or am I maybe misunderstanding something?\n> \n> Please also see below:\n> \n> > - we don't learn about the filename of trees and blobs, which is going\n> > to make the delta step much less efficient. This might be mitigated\n> > by turning on the bitmap name-hash cache; I don't recall how much\n> > detail pack-objects needs on the name (i.e., the full name versus\n> > just the hash).\n> \n> If I understand it right, it uses only uint32_t name hash while searching. From\n> pack-objects.{h,c} :\n> \n> ---- 8< ----\n> struct object_entry {\n> \t...\n> \tuint32_t hash; /* name hint hash */\n> \n> \n> /*\n> * We search for deltas in a list sorted by type, by filename hash, and then\n> * by size, so that we see progressively smaller and smaller files.\n> * That's because we prefer deltas to be from the bigger file\n> * to the smaller -- deletes are potentially cheaper, but perhaps\n> * more importantly, the bigger file is likely the more recent\n> * one. The deepest deltas are therefore the oldest objects which are\n> * less susceptible to be accessed often.\n> */\n> static int type_size_sort(const void *_a, const void *_b)\n> {\n> const struct object_entry *a = *(struct object_entry **)_a;\n> const struct object_entry *b = *(struct object_entry **)_b;\n> \n> if (a->type > b->type)\n> return -1;\n> if (a->type < b->type) \n> return 1;\n> if (a->hash > b->hash)\n> return -1;\n> if (a->hash < b->hash)\n> return 1;\n> \t...\n> ---- 8< ----\n> \n> Documentation/technical/pack-heuristics.txt also confirms this:\n> \n> ---- 8< ----\n> ...\n> The quote from the above linus should be rewritten a\n> bit (wait for it):\n> - first sort by type. Different objects never delta with\n> each other.\n> - then sort by filename/dirname. hash of the basename\n> occupies the top BITS_PER_INT-DIR_BITS bits, and bottom\n> DIR_BITS are for the hash of leading path elements.\n> \n> ...\n> \n> If I might add, the trick is to make files that _might_ be similar be\n> located close to each other in the hash buckets based on their file\n> names. It used to be that \"foo/Makefile\", \"bar/baz/quux/Makefile\" and\n> \"Makefile\" all landed in the same bucket due to their common basename,\n> \"Makefile\". However, now they land in \"close\" buckets.\n> \n> The algorithm allows not just for the _same_ bucket, but for _close_\n> buckets to be considered delta candidates. The rationale is\n> essentially that files, like Makefiles, often have very similar\n> content no matter what directory they live in.\n> ---- 8< ----\n> \n> \n> So yes, exactly as you say with pack.writeBitmapHashCache=true (ae4f07fb) the\n> delta-search heuristics is almost as efficient as with just raw filenames.\n> \n> I can confirm this also via e.g. (with my patch applied) :\n> \n> ---- 8< ----\n> $ time echo 0186ac99 | git pack-objects --no-use-bitmap-index --revs erp5pack-plain\n> Counting objects: 627171, done.\n> Compressing objects: 100% (176949/176949), done.\n> 50570987560d481742af4a8083028c2322a0534a\n> Writing objects: 100% (627171/627171), done.\n> Total 627171 (delta 439404), reused 594820 (delta 410210)\n> \n> real 0m37.272s\n> user 0m33.648s\n> sys 0m1.580s\n> \n> $ time echo 0186ac99 | git pack-objects --revs erp5pack-bitmap\n> Counting objects: 627171, done.\n> Compressing objects: 100% (176914/176914), done.\n> 7c15a9b1eca1326e679297b217c5a48954625ca2\n> Writing objects: 100% (627171/627171), done.\n> Total 627171 (delta 439484), reused 594855 (delta 410245)\n> \n> real 0m27.020s\n> user 0m23.364s\n> sys 0m0.992s\n> \n> $ ll erp5pack-{plain,bitmap}*\n> \t 17561860 erp5pack-bitmap-7c15a9b1eca1326e679297b217c5a48954625ca2.idx\n> \t 238760161 erp5pack-bitmap-7c15a9b1eca1326e679297b217c5a48954625ca2.pack\n> \t 17561860 erp5pack-plain-50570987560d481742af4a8083028c2322a0534a.idx\n> \t 238634201 erp5pack-plain-50570987560d481742af4a8083028c2322a0534a.pack\n> ---- 8< ----\n> \n> ( By the way about pack generated with bitmap retaining close recency\n> order:\n> \n> ---- 8< ----\n> $ git verify-pack -v erp5pack-plain-50570987560d481742af4a8083028c2322a0534a.pack >1\n> $ git verify-pack -v erp5pack-bitmap-7c15a9b1eca1326e679297b217c5a48954625ca2.pack >2\n> $ grep commit 1 |awk '{print $1}' >1.commit\n> $ grep commit 2 |awk '{print $1}' >2.commit\n> $ wc -l 1.commit\n> 46136 1.commit\n> $ wc -l 2.commit\n> 46136 2.commit\n> $ diff -u0 1.commit 2.commit |wc -l\n> 55\n> ---- 8< ----\n> \n> so 55/46136 shows it is very almost the same. )\n> \n> \n> > There may be other subtle things, too. The general idea of tying the\n> > bitmap use to pack_to_stdout is that you _do_ want to use it for\n> > serving fetches and pushes, but for a full on-disk repack via gc, it's\n> > more important to generate a good pack.\n> \n> It is better we send good packs to clients too, right? And with\n> pack.writeBitmapHashCache=true and retaining recency order (please see\n> above, but again maybe I'm not completely right) to me we should be still\n> generating a good pack while using bitmap reachability index for object\n> graph traversal.\n> \n> > Your use case:\n> > \n> > > git-backup extracts many packs on repositories restoration. That was my\n> > > initial motivation for the patch.\n> > \n> > Seems to be somewhere in between. I'm not sure I understand how you're\n> > invoking pack-objects here,\n> \n> It is just\n> \n> pack-objects --revs --reuse-object --reuse-delta --delta-base-offset extractedrepo/objects/pack/pack < SHA1-HEADS\n> \n> https://lab.nexedi.com/kirr/git-backup/blob/7fcb8c67/git-backup.go#L829\n> \n> > but I wonder if you should be using \"pack-objects --stdout\" yourself.\n> \n> I already tried --stdout. The problem is on repository extraction we\n> need to both extract the pack and index it. While `pack-object file`\n> does both, for --stdout case we need to additionally index extracted\n> pack with `git index-pack`, and standalone `git index-pack` is very slow\n> - in my experience much slower than generating the pack itself:\n> \n> ---- 8< ----\n> $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n> Counting objects: 627171, done.\n> Compressing objects: 100% (176914/176914), done.\n> Total 627171 (delta 439484), reused 594855 (delta 410245)\n> \n> real 0m22.309s\n> user 0m21.148s\n> sys 0m0.932s\n> \n> $ ll erp5pack-stdout*\n> 238760161 erp5pack-stdout.pack\n> \n> $ time git index-pack erp5pack-stdout.pack\n> 7c15a9b1eca1326e679297b217c5a48954625ca2\n> \n> real 0m50.873s <-- more than 2 times slower than time to generate pack itself!\n> user 0m49.300s\n> sys 0m1.360s\n> \n> $ ll erp5pack-stdout*\n> 17561860 erp5pack-stdout.idx\n> 238760161 erp5pack-stdout.pack\n> ---- 8< ----\n> \n> So the time for\n> \n> `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n> \n> while\n> \n> `pack-objects file.pack` which does both pack and index is 27s.\n> \n> And even\n> \n> `pack-objects --no-use-bitmap-index file.pack` is 37s.\n> \n> \n> I've tried to briefly see why index-pack is so slow and offhand I can\n> see that it needs to load all objects, decompresses them etc (maybe I'm\n> not so right here - I looked only briefly), while pack-objects while\n> generating the pack has all needed information directly at hand and thus\n> can emit index much more easily.\n> \n> For sever - clients scenario, index-pack load is put onto clients thus\n> offloading server, but for my use case where extracted repository is on\n> the same machine the load does not go away.\n> \n> That's why for me it makes more sense to emit both pack and its index in\n> one go.\n> \n> Still it would be interesting to eventually see why index-pack is so\n> anomaly slow.\n> \n> > But even if it is the right thing for your use case to be using bitmaps\n> > to generate an on-disk bitmap, I think we should be making sure it\n> > _doesn't_ trigger when doing a normal repack.\n> \n> So seems the way forward here is to teach pack-objects not to silently\n> drop explicit --use-pack-bitmap for cases when it can handle it?\n> (currently even if this option was given, for !stdout cases pack-objects\n> simply drop use_bitmap_index to 0).\n> \n> And to make sure default for use_bitmap_index is 0 for !stdout cases?\n> \n> Or are we fine with my arguments about recency order staying the same\n> when using bitmap reachability index for object graph traversal, and this\n> way the patch is fine to go in as it is?\n\nSince there is no reply I assume the safe way to go is to let default\nfor pack-to-file case to be \"not using bitmap index\". Please find updated\npatch and interdiff below. I would still be grateful for feedback on\nmy above use-bitmap-for-pack-to-file arguments.\n\nThanks,\nKirill\n\n(interdiff)\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e455fae..1888f42 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2241,12 +2241,20 @@ pack.packSizeLimit::\n \tCommon unit suffixes of 'k', 'm', or 'g' are\n \tsupported.\n \n-pack.useBitmaps::\n+pack.useBitmaps (deprecated)::\n+\tThis is a deprecated synonym for `pack.useBitmaps.stdout`.\n+\n+pack.useBitmaps.stdout::\n \tWhen true, git will use pack bitmaps (if available) when packing\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n \n+pack.useBitmaps.file::\n+\tWhen true, git will use pack bitmaps (if available) when packing\n+\tto file (e.g., on repack). Defaults to false. You should not\n+\tgenerally need to turn this on unless you know what you are doing.\n+\n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex be0ebe8..7aaa1af 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_stdout = 1, use_bitmap_file = 0;\n+static int use_bitmap_index;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2227,8 +2228,12 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n-\tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\tif (!strcmp(k, \"pack.usebitmaps\") || !strcmp(k, \"pack.usebitmaps.stdout\")) {\n+\t\tuse_bitmap_stdout = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(k, \"pack.usebitmaps.file\")) {\n+\t\tuse_bitmap_file = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2705,6 +2710,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n+\tuse_bitmap_index = pack_to_stdout ? use_bitmap_stdout : use_bitmap_file;\n \tif (!pack_compression_seen && core_compression_seen)\n \t\tpack_compression_level = core_compression_level;\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 533fc31..9fab2bb 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -122,9 +122,14 @@ test_expect_success 'pack-objects to file can use bitmap' '\n \t# make sure we still have 1 bitmap index from previous tests\n \tls .git/objects/pack/ | grep bitmap >output &&\n \ttest_line_count = 1 output &&\n-\t# pack-objects uses bitmap index by default, when it is available\n-\tpacksha1=$(git pack-objects --all mypack packa.verify &&\n+\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n+\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n+\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n '\n \n test_expect_success 'full repack, reusing previous bitmaps' '\n\n\n---- 8< ----\nFrom: Kirill Smelkov \nDate: Thu, 7 Jul 2016 20:12:00 +0300\nSubject: [PATCH v2] pack-objects: Teach it to use reachability bitmap index when\n generating non-stdout pack too\n\nStarting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nWe can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\n (at least that's my understanding after briefly looking at the code)\n\nWe also need to care and teach add_object_entry_from_bitmap() to respect\n--local via not adding nonlocal loose object to resultant pack (this\nis bitmap-codepath counterpart of daae0625 (pack-objects: extend --local\nto mean ignore non-local loose objects too) -- not to break 'loose\nobjects in alternate ODB are not repacked' in t7700-repack.sh .\n\nOtherwise all git tests pass, and for pack-objects -> file we get nice\nspeedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff King suggested that it might be not generally a good idea to\nuse bitmap reachability index when repacking a repository. For this\nreason when packing to a file the default is not to use bitmap, while\nfor packing-to-stdout case the default stays to be \"bitmap is used\".\n\nThe defaults can be configured with\n\n pack.useBitmaps.stdout (renamed from pack.useBitmaps), and\n pack.useBitmaps.file\n\nMore context:\n\n http://article.gmane.org/gmane.comp.version-control.git/299063\n http://article.gmane.org/gmane.comp.version-control.git/299107\n\nCc: Vicent Marti \nCc: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n Documentation/config.txt | 10 +++++++++-\n builtin/pack-objects.c | 19 ++++++++++++++-----\n t/t5310-pack-bitmaps.sh | 14 ++++++++++++++\n 3 files changed, 37 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e455fae..1888f42 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2241,12 +2241,20 @@ pack.packSizeLimit::\n \tCommon unit suffixes of 'k', 'm', or 'g' are\n \tsupported.\n \n-pack.useBitmaps::\n+pack.useBitmaps (deprecated)::\n+\tThis is a deprecated synonym for `pack.useBitmaps.stdout`.\n+\n+pack.useBitmaps.stdout::\n \tWhen true, git will use pack bitmaps (if available) when packing\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n \n+pack.useBitmaps.file::\n+\tWhen true, git will use pack bitmaps (if available) when packing\n+\tto file (e.g., on repack). Defaults to false. You should not\n+\tgenerally need to turn this on unless you know what you are doing.\n+\n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a2f8cfd..7aaa1af 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_stdout = 1, use_bitmap_file = 0;\n+static int use_bitmap_index;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -1052,6 +1053,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n {\n \tuint32_t index_pos;\n \n+\tif (local && has_loose_object_nonlocal(sha1))\n+\t\treturn 0;\n+\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n@@ -2224,8 +2228,12 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n-\tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\tif (!strcmp(k, \"pack.usebitmaps\") || !strcmp(k, \"pack.usebitmaps.stdout\")) {\n+\t\tuse_bitmap_stdout = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(k, \"pack.usebitmaps.file\")) {\n+\t\tuse_bitmap_file = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2488,7 +2496,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (prepare_bitmap_walk(revs) < 0)\n \t\treturn -1;\n \n-\tif (pack_options_allow_reuse() &&\n+\tif (pack_options_allow_reuse() && pack_to_stdout &&\n \t !reuse_partial_packfile_from_bitmap(\n \t\t\t&reuse_packfile,\n \t\t\t&reuse_packfile_objects,\n@@ -2702,6 +2710,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n+\tuse_bitmap_index = pack_to_stdout ? use_bitmap_stdout : use_bitmap_file;\n \tif (!pack_compression_seen && core_compression_seen)\n \t\tpack_compression_level = core_compression_level;\n \n@@ -2773,7 +2782,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..9fab2bb 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -118,6 +118,20 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.verify &&\n+\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n+\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n+\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.0.431.g3cb5c84\n"},{"id":"291334","messageId":"20160713082653.GA18144@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160708103855.GD5462@teco.navytux.spb.ru","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-13T08:26:53Z","receivedAt":"2016-07-13T08:27:05Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote:\n\n> > - we will not compute the same write order (which is based on\n> > traversal order), leading to packs that have less efficient cache\n> > characteristics\n> \n> I agree the order can be not exactly the same. Still if original pack is\n> packed well (with good recency order), while using bitmap we will tend\n> to traverse it in close to original order.\n> \n> Maybe I'm not completely right on this, but to me it looks to be the\n> case because if objects in original pack are put there linearly sorted\n> by recency order, and we use bitmap index to set of all reachable\n> objects from a root, and then just _linearly_ gather all those objects\n> from original pack by 1s in bitmap and put them in the same order into\n> destination pack, the recency order won't be broken.\n> \n> Or am I maybe misunderstanding something?\n\nYeah, I think you can go some of the way by reusing the order from the\nold pack. But keep in mind that the bitmap result may also contain\nobjects that are not yet packed. Those will just come in a big lump at\nthe end of the bitmap (these are the \"extended entries\" in the bitmap\ncode).\n\nSo I think if you were to repeatedly \"git repack -adb\" over time, you\nwould get worse and worse ordering as objects are added to the\nrepository.\n\nAs an aside, two other things that pack order matters for: it makes the\nbitmaps themselves compress better (because it increases locality of\nreachability, so you get nice runs of \"1\" or \"0\" bits). It also makes\nthe pack-reuse code more efficient (since in an ideal case, you can just\ndump a big block of data from the front of the pack). Note that the\npack-reuse code that's in upstream git isn't that great; I have a better\nsystem on my big pile of patches to send upstream (that never seems to\nget smaller; ).\n\n> > - we don't learn about the filename of trees and blobs, which is going\n> > to make the delta step much less efficient. This might be mitigated\n> > by turning on the bitmap name-hash cache; I don't recall how much\n> > detail pack-objects needs on the name (i.e., the full name versus\n> > just the hash).\n> \n> If I understand it right, it uses only uint32_t name hash while searching. From\n> pack-objects.{h,c} :\n\nYeah, I think you are right. Not having the real names is a problem for\ndoing rev-list output, but I think pack-objects doesn't care (though do\nnote that the name-hash cache is not enabled by default).\n\n> > There may be other subtle things, too. The general idea of tying the\n> > bitmap use to pack_to_stdout is that you _do_ want to use it for\n> > serving fetches and pushes, but for a full on-disk repack via gc, it's\n> > more important to generate a good pack.\n> \n> It is better we send good packs to clients too, right? And with\n> pack.writeBitmapHashCache=true and retaining recency order (please see\n> above, but again maybe I'm not completely right) to me we should be still\n> generating a good pack while using bitmap reachability index for object\n> graph traversal.\n\nWe do want to send the client a good pack, but it's always a tradeoff.\nWe could spend much more time searching for the perfect delta, but at\nsome point we have to decide on how much CPU to spend serving them.\nLikewise, even if the bitmapped packs we send are in slightly worse\norder, saving a minute of CPU time off of every clone of the kernel is a\nbig deal.\n\nWe also take robustness shortcuts when sending to clients. For example,\nwhen doing an on-disk repack we re-crc32 all of the delta data we are\nreusing, even if we don't actually inflate it (because we would want to\nstop immediately if we see even a single bit flipped on disk). But we\ndon't check them when sending to a client, because we know they are\ngoing to actually `index-pack` it and get a stronger consistency check\nanyway, and don't want to waste server CPU.\n\nThe bitmaps are sort of the same. If there is a bug or corruption in the\nbitmap, the worst case is that we send a broken pack to the client, who\nwill complain that we did not give them all of the objects. It's a\nmomentary problem that can be fixed. If you use them for an on-disk\nrepack, then the next step is usually to delete all of the old packs. So\na corruption there carries forward, and is irreversible.\n\nAs I understand your use case, it is OK to do the less careful things.\nIt's just that pack-objects until now has been split into two modes:\npacking to a file is careful, and packing to stdout is less so. And you\nwant to pack to a file in the non-careful mode.\n\n> > but I wonder if you should be using \"pack-objects --stdout\" yourself.\n> \n> I already tried --stdout. The problem is on repository extraction we\n> need to both extract the pack and index it. While `pack-object file`\n> does both, for --stdout case we need to additionally index extracted\n> pack with `git index-pack`, and standalone `git index-pack` is very slow\n> - in my experience much slower than generating the pack itself:\n\nAh, right, that makes sense. The packfile does not carry the sha1 of the\nobjects. A receiving index-pack has to compute them itself, including\ninflating and applying all of the deltas! By contrast, a pack to stdout\ncan be quite quick, because in most cases it can avoid even inflating\nmost of the data; where possible it just sends the zlib data straight\nfrom disk to the client.\n\nSo I do agree \"--stdout\" is not ideal for you (or at the very least, you\nreally want pack-objects to generate the index from its internal table\nrather than having to reconstruct it just from the pack stream).\n\n> > But even if it is the right thing for your use case to be using bitmaps\n> > to generate an on-disk bitmap, I think we should be making sure it\n> > _doesn't_ trigger when doing a normal repack.\n> \n> So seems the way forward here is to teach pack-objects not to silently\n> drop explicit --use-pack-bitmap for cases when it can handle it?\n> (currently even if this option was given, for !stdout cases pack-objects\n> simply drop use_bitmap_index to 0).\n> \n> And to make sure default for use_bitmap_index is 0 for !stdout cases?\n\nI think it would be reasonable to accept \"--use-bitmap-index\" on the\ncommand line as an override for \"yes, really, this is what I want\". So\nthe logic would be something like:\n\n static int use_bitmap_index_default = 1;\n static int use_bitmap_index = -1;\n\n ... parse config; if we see pack.usebitmaps, set\n use_bitmap_index_default ...\n\n ... parse command line, setting use_bitmap_index ...\n\n /* \"soft\" reasons not to use bitmaps */\n if (!pack_to_stdout)\n\tuse_bitmap_index_default = 0;\n\n /* now install our default if the user didn't otherwise specify */\n if (use_bitmap_index < 0)\n\tuse_bitmap_index = use_bitmap_index_default;\n\n /* \"hard\" reasons not to use bitmaps; these just won't work at all */\n if (!use_internal_rev_list || is_repository_shallow())\n\tuse_bitmap_index = 0;\n\n-Peff\n"},{"id":"291335","messageId":"20160713083044.GB18144@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160712190807.GB4809@teco.navytux.spb.ru","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-13T08:30:44Z","receivedAt":"2016-07-13T08:31:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 12, 2016 at 10:08:08PM +0300, Kirill Smelkov wrote:\n\n> > Or are we fine with my arguments about recency order staying the same\n> > when using bitmap reachability index for object graph traversal, and this\n> > way the patch is fine to go in as it is?\n> \n> Since there is no reply I assume the safe way to go is to let default\n> for pack-to-file case to be \"not using bitmap index\". Please find updated\n> patch and interdiff below. I would still be grateful for feedback on\n> my above use-bitmap-for-pack-to-file arguments.\n\nYeah, I think that is a reasonable approach. I see here you've added new\nconfig, though, and I don't think we want that.\n\nFor your purposes, where you're driving pack-objects individually, I\nthink a command-line option makes more sense.\n\nIf we did want to have a flag for \"use bitmaps when repacking via\nrepack\", I think it should be \"repack.useBitmaps\", and git-repack should\npass the command-line option to pack-objects. pack-objects is porcelain\nand should not really be reading config at all. You'll note that\npack.writeBitmaps was a mistake and got deprecated in favor of\nrepack.writeBitmaps. I think pack.useBitmaps is a mistake, too, but\nnobody has really noticed or cared because there's no good reason to set\nit (the more interesting question is: are there bitmaps available? and\nif so, we try to use them).\n\n-Peff\n"},{"id":"291336","messageId":"20160713105216.GB16000@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160713083044.GB18144@sigill.intra.peff.net","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-13T10:52:17Z","receivedAt":"2016-07-13T10:54:55Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Wed, Jul 13, 2016 at 04:26:53AM -0400, Jeff King wrote:\n> On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote:\n> \n> > > - we will not compute the same write order (which is based on\n> > > traversal order), leading to packs that have less efficient cache\n> > > characteristics\n> > \n> > I agree the order can be not exactly the same. Still if original pack is\n> > packed well (with good recency order), while using bitmap we will tend\n> > to traverse it in close to original order.\n> > \n> > Maybe I'm not completely right on this, but to me it looks to be the\n> > case because if objects in original pack are put there linearly sorted\n> > by recency order, and we use bitmap index to set of all reachable\n> > objects from a root, and then just _linearly_ gather all those objects\n> > from original pack by 1s in bitmap and put them in the same order into\n> > destination pack, the recency order won't be broken.\n> > \n> > Or am I maybe misunderstanding something?\n> \n> Yeah, I think you can go some of the way by reusing the order from the\n> old pack. But keep in mind that the bitmap result may also contain\n> objects that are not yet packed. Those will just come in a big lump at\n> the end of the bitmap (these are the \"extended entries\" in the bitmap\n> code).\n> \n> So I think if you were to repeatedly \"git repack -adb\" over time, you\n> would get worse and worse ordering as objects are added to the\n> repository.\n\nJeff, first of all thanks for clarifying.\n\nSo it is not-yet-packed-objects which make packing with bitmap less\nefficient. I was originally keeping in mind fresh repacked repository\nwith just built bitmap index and for that case extracting pack with\nbitmap index seems to be just ok, but the more not-yet-packed objects we\nhave the worse the result can be.\n\n> As an aside, two other things that pack order matters for: it makes the\n> bitmaps themselves compress better (because it increases locality of\n> reachability, so you get nice runs of \"1\" or \"0\" bits).\n\nYes I agree and thanks for bringing this up - putting objects in recency\norder in pack also makes bitmap index to have larger runs of same 1 or 0.\n\n> It also makes\n> the pack-reuse code more efficient (since in an ideal case, you can just\n> dump a big block of data from the front of the pack). Note that the\n> pack-reuse code that's in upstream git isn't that great; I have a better\n> system on my big pile of patches to send upstream (that never seems to\n> get smaller; ).\n\nYes, it also make sense. I saw write_reused_pack() in upstream git just\ncopy raw bytes from original to destination pack. You mentioned you have\nsomething better for pack reuse - in your patch queue, in two words, is\nit now reusing pack based on object, not raw bytes, or is it something\nelse?\n\nIn other words in which way it works better? (I'm just curious here as\nit is interesting to know)\n\n\n> > > - we don't learn about the filename of trees and blobs, which is going\n> > > to make the delta step much less efficient. This might be mitigated\n> > > by turning on the bitmap name-hash cache; I don't recall how much\n> > > detail pack-objects needs on the name (i.e., the full name versus\n> > > just the hash).\n> > \n> > If I understand it right, it uses only uint32_t name hash while searching. From\n> > pack-objects.{h,c} :\n> \n> Yeah, I think you are right. Not having the real names is a problem for\n> doing rev-list output, but I think pack-objects doesn't care (though do\n> note that the name-hash cache is not enabled by default).\n\nYes, for packing it is only hash which is used. And I assume name-hash\nfor bitmap is not enabled by default for compatibility with JGit code.\n\nIt would make sense to me to eventually enable name-hash bitmap\nextension by default, as packing result is much better with it. And\nthose who care about compatibility with JGit can just turn it off in\ntheir git config.\n\nJust my thoughts.\n\n> > > There may be other subtle things, too. The general idea of tying the\n> > > bitmap use to pack_to_stdout is that you _do_ want to use it for\n> > > serving fetches and pushes, but for a full on-disk repack via gc, it's\n> > > more important to generate a good pack.\n> > \n> > It is better we send good packs to clients too, right? And with\n> > pack.writeBitmapHashCache=true and retaining recency order (please see\n> > above, but again maybe I'm not completely right) to me we should be still\n> > generating a good pack while using bitmap reachability index for object\n> > graph traversal.\n> \n> We do want to send the client a good pack, but it's always a tradeoff.\n> We could spend much more time searching for the perfect delta, but at\n> some point we have to decide on how much CPU to spend serving them.\n> Likewise, even if the bitmapped packs we send are in slightly worse\n> order, saving a minute of CPU time off of every clone of the kernel is a\n> big deal.\n\nYes, this I understand and agree. Like I said above I was imagining\nfreshly repacked repo with recently rebuilt bitmap index and for that\ncase we send a good pack with bitmaps out-of-the-box.\n\n> We also take robustness shortcuts when sending to clients. For example,\n> when doing an on-disk repack we re-crc32 all of the delta data we are\n> reusing, even if we don't actually inflate it (because we would want to\n> stop immediately if we see even a single bit flipped on disk). But we\n> don't check them when sending to a client, because we know they are\n> going to actually `index-pack` it and get a stronger consistency check\n> anyway, and don't want to waste server CPU.\n> \n> The bitmaps are sort of the same. If there is a bug or corruption in the\n> bitmap, the worst case is that we send a broken pack to the client, who\n> will complain that we did not give them all of the objects. It's a\n> momentary problem that can be fixed. If you use them for an on-disk\n> repack, then the next step is usually to delete all of the old packs. So\n> a corruption there carries forward, and is irreversible.\n\nThanks for clarifying here. I did not knew pack-to-file is assumed to be\nrobust and pack-to-stdout is assumed to be allowed to be less so. Or at\nleast I did not thought about it this way before.\n\n> As I understand your use case, it is OK to do the less careful things.\n> It's just that pack-objects until now has been split into two modes:\n> packing to a file is careful, and packing to stdout is less so. And you\n> want to pack to a file in the non-careful mode.\n\nYes, it should be ok, as after repository extraction git-backup\nverifies rev-list for all refs\n\n https://lab.nexedi.com/kirr/git-backup/blob/7fcb8c67/git-backup.go#L855\n\nAnd if an object is missing - e.g. a blob - rev-list complains:\n\n fatal: missing blob object '980a0d5f19a64b4b30a87d4206aade58726b60e3'\n\nthough it does not catch blob corruptions.\n\nAs with when using bitmap index (due to bug in bitmap code or bitmap\nindex corruprtion) the worst that can happen is not all objects are\nextracted, this should be effective measure to catch it.\n\nThe original whole-backup repository is also not removed, so we can\nre-extract objects anytime.\n\nSo yes, using bitmap reachability index for faster extraction from\nfreshly repacked and bitmap indexed backup repository should be ok and\nmake sense to me.\n\n\n> > > but I wonder if you should be using \"pack-objects --stdout\" yourself.\n> > \n> > I already tried --stdout. The problem is on repository extraction we\n> > need to both extract the pack and index it. While `pack-object file`\n> > does both, for --stdout case we need to additionally index extracted\n> > pack with `git index-pack`, and standalone `git index-pack` is very slow\n> > - in my experience much slower than generating the pack itself:\n> \n> Ah, right, that makes sense. The packfile does not carry the sha1 of the\n> objects. A receiving index-pack has to compute them itself, including\n> inflating and applying all of the deltas! By contrast, a pack to stdout\n> can be quite quick, because in most cases it can avoid even inflating\n> most of the data; where possible it just sends the zlib data straight\n> from disk to the client.\n> \n> So I do agree \"--stdout\" is not ideal for you (or at the very least, you\n> really want pack-objects to generate the index from its internal table\n> rather than having to reconstruct it just from the pack stream).\n\nYes, and thanks for clarifying a bit why standalone index-pack can be\nslow.\n\n> > > But even if it is the right thing for your use case to be using bitmaps\n> > > to generate an on-disk bitmap, I think we should be making sure it\n> > > _doesn't_ trigger when doing a normal repack.\n> > \n> > So seems the way forward here is to teach pack-objects not to silently\n> > drop explicit --use-pack-bitmap for cases when it can handle it?\n> > (currently even if this option was given, for !stdout cases pack-objects\n> > simply drop use_bitmap_index to 0).\n> > \n> > And to make sure default for use_bitmap_index is 0 for !stdout cases?\n> \n> I think it would be reasonable to accept \"--use-bitmap-index\" on the\n> command line as an override for \"yes, really, this is what I want\". So\n> the logic would be something like:\n> \n> static int use_bitmap_index_default = 1;\n> static int use_bitmap_index = -1;\n> \n> ... parse config; if we see pack.usebitmaps, set\n> use_bitmap_index_default ...\n> \n> ... parse command line, setting use_bitmap_index ...\n> \n> /* \"soft\" reasons not to use bitmaps */\n> if (!pack_to_stdout)\n> \tuse_bitmap_index_default = 0;\n> \n> /* now install our default if the user didn't otherwise specify */\n> if (use_bitmap_index < 0)\n> \tuse_bitmap_index = use_bitmap_index_default;\n> \n> /* \"hard\" reasons not to use bitmaps; these just won't work at all */\n> if (!use_internal_rev_list || is_repository_shallow())\n> \tuse_bitmap_index = 0;\n\n\nOn Wed, Jul 13, 2016 at 04:30:44AM -0400, Jeff King wrote:\n> On Tue, Jul 12, 2016 at 10:08:08PM +0300, Kirill Smelkov wrote:\n> \n> > > Or are we fine with my arguments about recency order staying the same\n> > > when using bitmap reachability index for object graph traversal, and this\n> > > way the patch is fine to go in as it is?\n> > \n> > Since there is no reply I assume the safe way to go is to let default\n> > for pack-to-file case to be \"not using bitmap index\". Please find updated\n> > patch and interdiff below. I would still be grateful for feedback on\n> > my above use-bitmap-for-pack-to-file arguments.\n> \n> Yeah, I think that is a reasonable approach. I see here you've added new\n> config, though, and I don't think we want that.\n> \n> For your purposes, where you're driving pack-objects individually, I\n> think a command-line option makes more sense.\n\nYes, I was going to use --use-bitmap-index explicitly, but I thought\nsince we already have pack.useBitmaps for consistency it is better to\nintroduce controlling to-file config point.\n\n\n> If we did want to have a flag for \"use bitmaps when repacking via\n> repack\", I think it should be \"repack.useBitmaps\", and git-repack should\n> pass the command-line option to pack-objects. pack-objects is porcelain\n> and should not really be reading config at all. You'll note that\n> pack.writeBitmaps was a mistake and got deprecated in favor of\n> repack.writeBitmaps. I think pack.useBitmaps is a mistake, too, but\n> nobody has really noticed or cared because there's no good reason to set\n> it (the more interesting question is: are there bitmaps available? and\n> if so, we try to use them).\n\nProbably pack.useBitmaps is of no use in normal situation, but for\ndebugging problems related to bitmaps it can be handy. Though when\nsomeone debugs he/she can just adjust pack-objects.c . So should we\ndeprecate and eventually remove pack.useBitmaps ?\n\nAnyway, please find below updated patch according to your suggestion.\nHope it is ok now.\n\nThanks again,\nKirill\n\n\n(interdiff)\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 8027951..4b14806 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2229,19 +2229,14 @@ pack.packSizeLimit::\n \tCommon unit suffixes of 'k', 'm', or 'g' are\n \tsupported.\n \n-pack.useBitmaps (deprecated)::\n-\tThis is a deprecated synonym for `pack.useBitmaps.stdout`.\n-\n-pack.useBitmaps.stdout::\n+pack.useBitmaps::\n \tWhen true, git will use pack bitmaps (if available) when packing\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n-\n-pack.useBitmaps.file::\n-\tWhen true, git will use pack bitmaps (if available) when packing\n-\tto file (e.g., on repack). Defaults to false. You should not\n-\tgenerally need to turn this on unless you know what you are doing.\n++\n+*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n+\tpack bitmaps.\n \n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7aaa1af..ffe8da6 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,8 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_stdout = 1, use_bitmap_file = 0;\n-static int use_bitmap_index;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2228,12 +2228,8 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\telse\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n-\tif (!strcmp(k, \"pack.usebitmaps\") || !strcmp(k, \"pack.usebitmaps.stdout\")) {\n-\t\tuse_bitmap_stdout = git_config_bool(k, v);\n-\t\treturn 0;\n-\t}\n-\tif (!strcmp(k, \"pack.usebitmaps.file\")) {\n-\t\tuse_bitmap_file = git_config_bool(k, v);\n+\tif (!strcmp(k, \"pack.usebitmaps\")) {\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2710,7 +2706,6 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n-\tuse_bitmap_index = pack_to_stdout ? use_bitmap_stdout : use_bitmap_file;\n \tif (!pack_compression_seen && core_compression_seen)\n \t\tpack_compression_level = core_compression_level;\n \n@@ -2782,6 +2777,22 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n \tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n\n---- 8< ----\nFrom: Kirill Smelkov \nSubject: [PATCH v3] pack-objects: Teach it to use reachability bitmap index when\n generating non-stdout pack too\n\nStarting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nWe can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\n (at least that's my understanding after briefly looking at the code)\n\nWe also need to care and teach add_object_entry_from_bitmap() to respect\n--local via not adding nonlocal loose object to resultant pack (this\nis bitmap-codepath counterpart of daae0625 (pack-objects: extend --local\nto mean ignore non-local loose objects too) -- not to break 'loose\nobjects in alternate ODB are not repacked' in t7700-repack.sh .\n\nOtherwise all git tests pass, and for pack-objects -> file we get nice\nspeedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff King suggested that it might be not generally a good idea to\nuse bitmap reachability index when repacking a repository. The reason\nhere is for on-disk repack by default we want\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly.\n\nMore context:\n\n http://article.gmane.org/gmane.comp.version-control.git/299063\n http://article.gmane.org/gmane.comp.version-control.git/299107\n http://article.gmane.org/gmane.comp.version-control.git/299420\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n Documentation/config.txt | 3 +++\n builtin/pack-objects.c | 28 ++++++++++++++++++++++++----\n t/t5310-pack-bitmaps.sh | 14 ++++++++++++++\n 3 files changed, 41 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex db05dec..4b14806 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2234,6 +2234,9 @@ pack.useBitmaps::\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n++\n+*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n+\tpack bitmaps.\n \n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a2f8cfd..ffe8da6 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -1052,6 +1053,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n {\n \tuint32_t index_pos;\n \n+\tif (local && has_loose_object_nonlocal(sha1))\n+\t\treturn 0;\n+\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n@@ -2225,7 +2229,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2488,7 +2492,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (prepare_bitmap_walk(revs) < 0)\n \t\treturn -1;\n \n-\tif (pack_options_allow_reuse() &&\n+\tif (pack_options_allow_reuse() && pack_to_stdout &&\n \t !reuse_partial_packfile_from_bitmap(\n \t\t\t&reuse_packfile,\n \t\t\t&reuse_packfile_objects,\n@@ -2773,7 +2777,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..9fab2bb 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -118,6 +118,20 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.verify &&\n+\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n+\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n+\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.0.431.g3cb5c84\n"},{"id":"291632","messageId":"20160717170649.GA3830@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160713105216.GB16000@teco.navytux.spb.ru","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-17T17:06:49Z","receivedAt":"2016-07-17T17:07:15Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Wed, Jul 13, 2016 at 01:52:16PM +0300, Kirill Smelkov wrote:\n> On Wed, Jul 13, 2016 at 04:26:53AM -0400, Jeff King wrote:\n> > On Fri, Jul 08, 2016 at 01:38:55PM +0300, Kirill Smelkov wrote:\n> > \n> > > > - we will not compute the same write order (which is based on\n> > > > traversal order), leading to packs that have less efficient cache\n> > > > characteristics\n> > > \n> > > I agree the order can be not exactly the same. Still if original pack is\n> > > packed well (with good recency order), while using bitmap we will tend\n> > > to traverse it in close to original order.\n> > > \n> > > Maybe I'm not completely right on this, but to me it looks to be the\n> > > case because if objects in original pack are put there linearly sorted\n> > > by recency order, and we use bitmap index to set of all reachable\n> > > objects from a root, and then just _linearly_ gather all those objects\n> > > from original pack by 1s in bitmap and put them in the same order into\n> > > destination pack, the recency order won't be broken.\n> > > \n> > > Or am I maybe misunderstanding something?\n> > \n> > Yeah, I think you can go some of the way by reusing the order from the\n> > old pack. But keep in mind that the bitmap result may also contain\n> > objects that are not yet packed. Those will just come in a big lump at\n> > the end of the bitmap (these are the \"extended entries\" in the bitmap\n> > code).\n> > \n> > So I think if you were to repeatedly \"git repack -adb\" over time, you\n> > would get worse and worse ordering as objects are added to the\n> > repository.\n> \n> Jeff, first of all thanks for clarifying.\n> \n> So it is not-yet-packed-objects which make packing with bitmap less\n> efficient. I was originally keeping in mind fresh repacked repository\n> with just built bitmap index and for that case extracting pack with\n> bitmap index seems to be just ok, but the more not-yet-packed objects we\n> have the worse the result can be.\n> \n> > As an aside, two other things that pack order matters for: it makes the\n> > bitmaps themselves compress better (because it increases locality of\n> > reachability, so you get nice runs of \"1\" or \"0\" bits).\n> \n> Yes I agree and thanks for bringing this up - putting objects in recency\n> order in pack also makes bitmap index to have larger runs of same 1 or 0.\n> \n> > It also makes\n> > the pack-reuse code more efficient (since in an ideal case, you can just\n> > dump a big block of data from the front of the pack). Note that the\n> > pack-reuse code that's in upstream git isn't that great; I have a better\n> > system on my big pile of patches to send upstream (that never seems to\n> > get smaller; ).\n> \n> Yes, it also make sense. I saw write_reused_pack() in upstream git just\n> copy raw bytes from original to destination pack. You mentioned you have\n> something better for pack reuse - in your patch queue, in two words, is\n> it now reusing pack based on object, not raw bytes, or is it something\n> else?\n> \n> In other words in which way it works better? (I'm just curious here as\n> it is interesting to know)\n> \n> \n> > > > - we don't learn about the filename of trees and blobs, which is going\n> > > > to make the delta step much less efficient. This might be mitigated\n> > > > by turning on the bitmap name-hash cache; I don't recall how much\n> > > > detail pack-objects needs on the name (i.e., the full name versus\n> > > > just the hash).\n> > > \n> > > If I understand it right, it uses only uint32_t name hash while searching. From\n> > > pack-objects.{h,c} :\n> > \n> > Yeah, I think you are right. Not having the real names is a problem for\n> > doing rev-list output, but I think pack-objects doesn't care (though do\n> > note that the name-hash cache is not enabled by default).\n> \n> Yes, for packing it is only hash which is used. And I assume name-hash\n> for bitmap is not enabled by default for compatibility with JGit code.\n> \n> It would make sense to me to eventually enable name-hash bitmap\n> extension by default, as packing result is much better with it. And\n> those who care about compatibility with JGit can just turn it off in\n> their git config.\n> \n> Just my thoughts.\n> \n> > > > There may be other subtle things, too. The general idea of tying the\n> > > > bitmap use to pack_to_stdout is that you _do_ want to use it for\n> > > > serving fetches and pushes, but for a full on-disk repack via gc, it's\n> > > > more important to generate a good pack.\n> > > \n> > > It is better we send good packs to clients too, right? And with\n> > > pack.writeBitmapHashCache=true and retaining recency order (please see\n> > > above, but again maybe I'm not completely right) to me we should be still\n> > > generating a good pack while using bitmap reachability index for object\n> > > graph traversal.\n> > \n> > We do want to send the client a good pack, but it's always a tradeoff.\n> > We could spend much more time searching for the perfect delta, but at\n> > some point we have to decide on how much CPU to spend serving them.\n> > Likewise, even if the bitmapped packs we send are in slightly worse\n> > order, saving a minute of CPU time off of every clone of the kernel is a\n> > big deal.\n> \n> Yes, this I understand and agree. Like I said above I was imagining\n> freshly repacked repo with recently rebuilt bitmap index and for that\n> case we send a good pack with bitmaps out-of-the-box.\n> \n> > We also take robustness shortcuts when sending to clients. For example,\n> > when doing an on-disk repack we re-crc32 all of the delta data we are\n> > reusing, even if we don't actually inflate it (because we would want to\n> > stop immediately if we see even a single bit flipped on disk). But we\n> > don't check them when sending to a client, because we know they are\n> > going to actually `index-pack` it and get a stronger consistency check\n> > anyway, and don't want to waste server CPU.\n> > \n> > The bitmaps are sort of the same. If there is a bug or corruption in the\n> > bitmap, the worst case is that we send a broken pack to the client, who\n> > will complain that we did not give them all of the objects. It's a\n> > momentary problem that can be fixed. If you use them for an on-disk\n> > repack, then the next step is usually to delete all of the old packs. So\n> > a corruption there carries forward, and is irreversible.\n> \n> Thanks for clarifying here. I did not knew pack-to-file is assumed to be\n> robust and pack-to-stdout is assumed to be allowed to be less so. Or at\n> least I did not thought about it this way before.\n> \n> > As I understand your use case, it is OK to do the less careful things.\n> > It's just that pack-objects until now has been split into two modes:\n> > packing to a file is careful, and packing to stdout is less so. And you\n> > want to pack to a file in the non-careful mode.\n> \n> Yes, it should be ok, as after repository extraction git-backup\n> verifies rev-list for all refs\n> \n> https://lab.nexedi.com/kirr/git-backup/blob/7fcb8c67/git-backup.go#L855\n> \n> And if an object is missing - e.g. a blob - rev-list complains:\n> \n> fatal: missing blob object '980a0d5f19a64b4b30a87d4206aade58726b60e3'\n> \n> though it does not catch blob corruptions.\n> \n> As with when using bitmap index (due to bug in bitmap code or bitmap\n> index corruprtion) the worst that can happen is not all objects are\n> extracted, this should be effective measure to catch it.\n> \n> The original whole-backup repository is also not removed, so we can\n> re-extract objects anytime.\n> \n> So yes, using bitmap reachability index for faster extraction from\n> freshly repacked and bitmap indexed backup repository should be ok and\n> make sense to me.\n> \n> \n> > > > but I wonder if you should be using \"pack-objects --stdout\" yourself.\n> > > \n> > > I already tried --stdout. The problem is on repository extraction we\n> > > need to both extract the pack and index it. While `pack-object file`\n> > > does both, for --stdout case we need to additionally index extracted\n> > > pack with `git index-pack`, and standalone `git index-pack` is very slow\n> > > - in my experience much slower than generating the pack itself:\n> > \n> > Ah, right, that makes sense. The packfile does not carry the sha1 of the\n> > objects. A receiving index-pack has to compute them itself, including\n> > inflating and applying all of the deltas! By contrast, a pack to stdout\n> > can be quite quick, because in most cases it can avoid even inflating\n> > most of the data; where possible it just sends the zlib data straight\n> > from disk to the client.\n> > \n> > So I do agree \"--stdout\" is not ideal for you (or at the very least, you\n> > really want pack-objects to generate the index from its internal table\n> > rather than having to reconstruct it just from the pack stream).\n> \n> Yes, and thanks for clarifying a bit why standalone index-pack can be\n> slow.\n> \n> > > > But even if it is the right thing for your use case to be using bitmaps\n> > > > to generate an on-disk bitmap, I think we should be making sure it\n> > > > _doesn't_ trigger when doing a normal repack.\n> > > \n> > > So seems the way forward here is to teach pack-objects not to silently\n> > > drop explicit --use-pack-bitmap for cases when it can handle it?\n> > > (currently even if this option was given, for !stdout cases pack-objects\n> > > simply drop use_bitmap_index to 0).\n> > > \n> > > And to make sure default for use_bitmap_index is 0 for !stdout cases?\n> > \n> > I think it would be reasonable to accept \"--use-bitmap-index\" on the\n> > command line as an override for \"yes, really, this is what I want\". So\n> > the logic would be something like:\n> > \n> > static int use_bitmap_index_default = 1;\n> > static int use_bitmap_index = -1;\n> > \n> > ... parse config; if we see pack.usebitmaps, set\n> > use_bitmap_index_default ...\n> > \n> > ... parse command line, setting use_bitmap_index ...\n> > \n> > /* \"soft\" reasons not to use bitmaps */\n> > if (!pack_to_stdout)\n> > \tuse_bitmap_index_default = 0;\n> > \n> > /* now install our default if the user didn't otherwise specify */\n> > if (use_bitmap_index < 0)\n> > \tuse_bitmap_index = use_bitmap_index_default;\n> > \n> > /* \"hard\" reasons not to use bitmaps; these just won't work at all */\n> > if (!use_internal_rev_list || is_repository_shallow())\n> > \tuse_bitmap_index = 0;\n> \n> \n> On Wed, Jul 13, 2016 at 04:30:44AM -0400, Jeff King wrote:\n> > On Tue, Jul 12, 2016 at 10:08:08PM +0300, Kirill Smelkov wrote:\n> > \n> > > > Or are we fine with my arguments about recency order staying the same\n> > > > when using bitmap reachability index for object graph traversal, and this\n> > > > way the patch is fine to go in as it is?\n> > > \n> > > Since there is no reply I assume the safe way to go is to let default\n> > > for pack-to-file case to be \"not using bitmap index\". Please find updated\n> > > patch and interdiff below. I would still be grateful for feedback on\n> > > my above use-bitmap-for-pack-to-file arguments.\n> > \n> > Yeah, I think that is a reasonable approach. I see here you've added new\n> > config, though, and I don't think we want that.\n> > \n> > For your purposes, where you're driving pack-objects individually, I\n> > think a command-line option makes more sense.\n> \n> Yes, I was going to use --use-bitmap-index explicitly, but I thought\n> since we already have pack.useBitmaps for consistency it is better to\n> introduce controlling to-file config point.\n> \n> \n> > If we did want to have a flag for \"use bitmaps when repacking via\n> > repack\", I think it should be \"repack.useBitmaps\", and git-repack should\n> > pass the command-line option to pack-objects. pack-objects is porcelain\n> > and should not really be reading config at all. You'll note that\n> > pack.writeBitmaps was a mistake and got deprecated in favor of\n> > repack.writeBitmaps. I think pack.useBitmaps is a mistake, too, but\n> > nobody has really noticed or cared because there's no good reason to set\n> > it (the more interesting question is: are there bitmaps available? and\n> > if so, we try to use them).\n> \n> Probably pack.useBitmaps is of no use in normal situation, but for\n> debugging problems related to bitmaps it can be handy. Though when\n> someone debugs he/she can just adjust pack-objects.c . So should we\n> deprecate and eventually remove pack.useBitmaps ?\n> \n> Anyway, please find below updated patch according to your suggestion.\n> Hope it is ok now.\n\nPing. Is the patch ok or something needs to be improved still?\n\nThanks beforehand for feedback,\nKirill\n\n\n> (interdiff)\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 8027951..4b14806 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -2229,19 +2229,14 @@ pack.packSizeLimit::\n> \tCommon unit suffixes of 'k', 'm', or 'g' are\n> \tsupported.\n> \n> -pack.useBitmaps (deprecated)::\n> -\tThis is a deprecated synonym for `pack.useBitmaps.stdout`.\n> -\n> -pack.useBitmaps.stdout::\n> +pack.useBitmaps::\n> \tWhen true, git will use pack bitmaps (if available) when packing\n> \tto stdout (e.g., during the server side of a fetch). Defaults to\n> \ttrue. You should not generally need to turn this off unless\n> \tyou are debugging pack bitmaps.\n> -\n> -pack.useBitmaps.file::\n> -\tWhen true, git will use pack bitmaps (if available) when packing\n> -\tto file (e.g., on repack). Defaults to false. You should not\n> -\tgenerally need to turn this on unless you know what you are doing.\n> ++\n> +*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n> +\tpack bitmaps.\n> \n> pack.writeBitmaps (deprecated)::\n> \tThis is a deprecated synonym for `repack.writeBitmaps`.\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 7aaa1af..ffe8da6 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -66,8 +66,8 @@ static struct packed_git *reuse_packfile;\n> static uint32_t reuse_packfile_objects;\n> static off_t reuse_packfile_offset;\n> \n> -static int use_bitmap_stdout = 1, use_bitmap_file = 0;\n> -static int use_bitmap_index;\n> +static int use_bitmap_index_default = 1;\n> +static int use_bitmap_index = -1;\n> static int write_bitmap_index;\n> static uint16_t write_bitmap_options;\n> \n> @@ -2228,12 +2228,8 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\telse\n> \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n> \t}\n> -\tif (!strcmp(k, \"pack.usebitmaps\") || !strcmp(k, \"pack.usebitmaps.stdout\")) {\n> -\t\tuse_bitmap_stdout = git_config_bool(k, v);\n> -\t\treturn 0;\n> -\t}\n> -\tif (!strcmp(k, \"pack.usebitmaps.file\")) {\n> -\t\tuse_bitmap_file = git_config_bool(k, v);\n> +\tif (!strcmp(k, \"pack.usebitmaps\")) {\n> +\t\tuse_bitmap_index_default = git_config_bool(k, v);\n> \t\treturn 0;\n> \t}\n> \tif (!strcmp(k, \"pack.threads\")) {\n> @@ -2710,7 +2706,6 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \n> \treset_pack_idx_option(&pack_idx_opts);\n> \tgit_config(git_pack_config, NULL);\n> -\tuse_bitmap_index = pack_to_stdout ? use_bitmap_stdout : use_bitmap_file;\n> \tif (!pack_compression_seen && core_compression_seen)\n> \t\tpack_compression_level = core_compression_level;\n> \n> @@ -2782,6 +2777,22 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n> \t\tunpack_unreachable_expiration = 0;\n> \n> +\t/*\n> +\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n> +\t *\n> +\t * - to produce good pack (with bitmap index not-yet-packed objects are\n> +\t * packed in suboptimal order).\n> +\t *\n> +\t * - to use more robust pack-generation codepath (avoiding possible\n> +\t * bugs in bitmap code and possible bitmap index corruption).\n> +\t */\n> +\tif (!pack_to_stdout)\n> +\t\tuse_bitmap_index_default = 0;\n> +\n> +\tif (use_bitmap_index < 0)\n> +\t\tuse_bitmap_index = use_bitmap_index_default;\n> +\n> +\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n> \tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n> \t\tuse_bitmap_index = 0;\n> \n> \n> ---- 8< ----\n> From: Kirill Smelkov \n> Subject: [PATCH v3] pack-objects: Teach it to use reachability bitmap index when\n> generating non-stdout pack too\n> \n> Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\n> if a repository has bitmap index, pack-objects can nicely speedup\n> \"Counting objects\" graph traversal phase. That however was done only for\n> case when resultant pack is sent to stdout, not written into a file.\n> \n> We can teach pack-objects to use bitmap index for initial object\n> counting phase when generating resultant pack file too:\n> \n> - if we know bitmap index generation is not enabled for resultant pack:\n> \n> Current code has singleton bitmap_git so cannot work simultaneously\n> with two bitmap indices.\n> \n> - if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n> \n> Because on pack reuse raw entries are directly written out to destination\n> pack by write_reused_pack() bypassing needed for pack index generation\n> bookkeeping done by regular codepath in write_one() and friends.\n> \n> (at least that's my understanding after briefly looking at the code)\n> \n> We also need to care and teach add_object_entry_from_bitmap() to respect\n> --local via not adding nonlocal loose object to resultant pack (this\n> is bitmap-codepath counterpart of daae0625 (pack-objects: extend --local\n> to mean ignore non-local loose objects too) -- not to break 'loose\n> objects in alternate ODB are not repacked' in t7700-repack.sh .\n> \n> Otherwise all git tests pass, and for pack-objects -> file we get nice\n> speedup:\n> \n> erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n> repository managed by git-backup[2] via\n> \n> time echo 0186ac99 | git pack-objects --revs erp5pack\n> \n> before: 37.2s\n> after: 26.2s\n> \n> And for `git repack -adb` packed git.git\n> \n> time echo 5c589a73 | git pack-objects --revs gitpack\n> \n> before: 7.1s\n> after: 3.6s\n> \n> i.e. it can be 30% - 50% speedup for pack extraction.\n> \n> git-backup extracts many packs on repositories restoration. That was my\n> initial motivation for the patch.\n> \n> [1] https://lab.nexedi.com/nexedi/erp5\n> [2] https://lab.nexedi.com/kirr/git-backup\n> \n> NOTE\n> \n> Jeff King suggested that it might be not generally a good idea to\n> use bitmap reachability index when repacking a repository. The reason\n> here is for on-disk repack by default we want\n> \n> - to produce good pack (with bitmap index not-yet-packed objects are\n> emitted to pack in suboptimal order).\n> \n> - to use more robust pack-generation codepath (avoiding possible\n> bugs in bitmap code and possible bitmap index corruption).\n> \n> Jeff also suggests that pack.useBitmaps was probably a mistake to\n> introduce originally. This way we are not adding another config point,\n> but instead just always default to-file pack-objects not to use bitmap\n> index: Tools which need to generate on-disk packs with using bitmap, can\n> pass --use-bitmap-index explicitly.\n> \n> More context:\n> \n> http://article.gmane.org/gmane.comp.version-control.git/299063\n> http://article.gmane.org/gmane.comp.version-control.git/299107\n> http://article.gmane.org/gmane.comp.version-control.git/299420\n> \n> Cc: Vicent Marti \n> Helped-by: Jeff King \n> Signed-off-by: Kirill Smelkov \n> ---\n> Documentation/config.txt | 3 +++\n> builtin/pack-objects.c | 28 ++++++++++++++++++++++++----\n> t/t5310-pack-bitmaps.sh | 14 ++++++++++++++\n> 3 files changed, 41 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index db05dec..4b14806 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -2234,6 +2234,9 @@ pack.useBitmaps::\n> \tto stdout (e.g., during the server side of a fetch). Defaults to\n> \ttrue. You should not generally need to turn this off unless\n> \tyou are debugging pack bitmaps.\n> ++\n> +*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n> +\tpack bitmaps.\n> \n> pack.writeBitmaps (deprecated)::\n> \tThis is a deprecated synonym for `repack.writeBitmaps`.\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index a2f8cfd..ffe8da6 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n> static uint32_t reuse_packfile_objects;\n> static off_t reuse_packfile_offset;\n> \n> -static int use_bitmap_index = 1;\n> +static int use_bitmap_index_default = 1;\n> +static int use_bitmap_index = -1;\n> static int write_bitmap_index;\n> static uint16_t write_bitmap_options;\n> \n> @@ -1052,6 +1053,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> {\n> \tuint32_t index_pos;\n> \n> +\tif (local && has_loose_object_nonlocal(sha1))\n> +\t\treturn 0;\n> +\n> \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> \t\treturn 0;\n> \n> @@ -2225,7 +2229,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n> \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n> \t}\n> \tif (!strcmp(k, \"pack.usebitmaps\")) {\n> -\t\tuse_bitmap_index = git_config_bool(k, v);\n> +\t\tuse_bitmap_index_default = git_config_bool(k, v);\n> \t\treturn 0;\n> \t}\n> \tif (!strcmp(k, \"pack.threads\")) {\n> @@ -2488,7 +2492,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n> \tif (prepare_bitmap_walk(revs) < 0)\n> \t\treturn -1;\n> \n> -\tif (pack_options_allow_reuse() &&\n> +\tif (pack_options_allow_reuse() && pack_to_stdout &&\n> \t !reuse_partial_packfile_from_bitmap(\n> \t\t\t&reuse_packfile,\n> \t\t\t&reuse_packfile_objects,\n> @@ -2773,7 +2777,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n> \t\tunpack_unreachable_expiration = 0;\n> \n> -\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n> +\t/*\n> +\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n> +\t *\n> +\t * - to produce good pack (with bitmap index not-yet-packed objects are\n> +\t * packed in suboptimal order).\n> +\t *\n> +\t * - to use more robust pack-generation codepath (avoiding possible\n> +\t * bugs in bitmap code and possible bitmap index corruption).\n> +\t */\n> +\tif (!pack_to_stdout)\n> +\t\tuse_bitmap_index_default = 0;\n> +\n> +\tif (use_bitmap_index < 0)\n> +\t\tuse_bitmap_index = use_bitmap_index_default;\n> +\n> +\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n> +\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n> \t\tuse_bitmap_index = 0;\n> \n> \tif (pack_to_stdout || !rev_list_all)\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index 3893afd..9fab2bb 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -118,6 +118,20 @@ test_expect_success 'incremental repack can disable bitmaps' '\n> \tgit repack -d --no-write-bitmap-index\n> '\n> \n> +test_expect_success 'pack-objects to file can use bitmap' '\n> +\t# make sure we still have 1 bitmap index from previous tests\n> +\tls .git/objects/pack/ | grep bitmap >output &&\n> +\ttest_line_count = 1 output &&\n> +\t# verify equivalent packs are generated with/without using bitmap index\n> +\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa +\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb +\tgit verify-pack -v packa-$packasha1.pack >packa.verify &&\n> +\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n> +\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n> +\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n> +\ttest_cmp packa.objects packb.objects\n> +'\n> +\n> test_expect_success 'full repack, reusing previous bitmaps' '\n> \tgit repack -ad &&\n> \tls .git/objects/pack/ | grep bitmap >output &&\n> -- \n> 2.9.0.431.g3cb5c84\n"},{"id":"291716","messageId":"20160719112907.GA12916@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160717170649.GA3830@teco.navytux.spb.ru","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-19T11:29:07Z","receivedAt":"2016-07-19T11:29:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jul 17, 2016 at 08:06:49PM +0300, Kirill Smelkov wrote:\n\n> > Anyway, please find below updated patch according to your suggestion.\n> > Hope it is ok now.\n> \n> Ping. Is the patch ok or something needs to be improved still?\n\nSorry, I'm traveling and haven't carefully reviewed it yet. It's still\non my list, but it may be a few days.\n\n-Peff\n"},{"id":"291717","messageId":"20160719121431.GB3947@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160719112907.GA12916@sigill.intra.peff.net","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-19T12:14:31Z","receivedAt":"2016-07-19T12:14:39Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Tue, Jul 19, 2016 at 05:29:07AM -0600, Jeff King wrote:\n> On Sun, Jul 17, 2016 at 08:06:49PM +0300, Kirill Smelkov wrote:\n> \n> > > Anyway, please find below updated patch according to your suggestion.\n> > > Hope it is ok now.\n> > \n> > Ping. Is the patch ok or something needs to be improved still?\n> \n> Sorry, I'm traveling and haven't carefully reviewed it yet. It's still\n> on my list, but it may be a few days.\n\nJeff thanks for feedback. Have a good traveling and good to know the patch\nwas not forgotten. I will be waiting for the time while you are on trip.\n\nThanks again for feedback,\nKirill\n"},{"id":"292116","messageId":"20160725184025.GA12297@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160713105216.GB16000@teco.navytux.spb.ru","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-25T18:40:25Z","receivedAt":"2016-07-25T18:40:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 13, 2016 at 01:52:17PM +0300, Kirill Smelkov wrote:\n\n> > So I think if you were to repeatedly \"git repack -adb\" over time, you\n> > would get worse and worse ordering as objects are added to the\n> > repository.\n> \n> Jeff, first of all thanks for clarifying.\n> \n> So it is not-yet-packed-objects which make packing with bitmap less\n> efficient. I was originally keeping in mind fresh repacked repository\n> with just built bitmap index and for that case extracting pack with\n> bitmap index seems to be just ok, but the more not-yet-packed objects we\n> have the worse the result can be.\n\nRight. So I think your scheme is fine as long as you are doing your\nregular \"pack all into one\" repacks with a real walk, and then\n\"branching\" off of that with one-off bitmap-computed packs into files\n(even if you later take a bunch of those files and pull them into a\nsingle bitmapped, as long as that final \"all into one\" does the walk).\n\nOr I guess another way to think about it would be that if you're\ncomputing bitmaps, you'd want to do the actual traversal.\n\n> Yes, it also make sense. I saw write_reused_pack() in upstream git just\n> copy raw bytes from original to destination pack. You mentioned you have\n> something better for pack reuse - in your patch queue, in two words, is\n> it now reusing pack based on object, not raw bytes, or is it something\n> else?\n> \n> In other words in which way it works better? (I'm just curious here as\n> it is interesting to know)\n\nThe problem with the existing pack-reuse code is that it doesn't kick in\noften enough. I think it looks to see that the client wants some\npercentage of the pack (e.g., 90%), and then just sends the whole\nbeginning. This works especially badly if you have a bunch of related\nrepositories packed together (e.g., all of the forks of torvalds/linux\non GitHub), because you'll never hit 90% of that big pack; it has too\nmuch unrelated cruft, even if most of the stuff you want _is_ at the\nbeginning. And \"percent of pack\" is not really a useful metric anyway.\n\nSo the better scheme is more like:\n\n 1. Generate the bitmap of objects to send using reachability bitmaps.\n\n 2. Do a quick scan of their content in the packfile to see which can\n be reused verbatim. If they're base objects, we can send them\n as-is. If they're deltas, we can send them if their base is going\n to be sent. This fills in another bitmap of \"reusable\" objects.\n\n After a long string of unusable objects, you can give up and set\n the rest of the bitmap to zeroes.\n\n 3. Walk the \"reuse\" bitmap and send out the objects more-or-less\n verbatim. You do have make adjustments to delta-base-offsets for\n any \"holes\" (so if an object's entry says \"my base is 500 bytes\n back\", but you omitted some objects in between, you have to adjust\n that offset).\n\nThe upside is that you can send out those objects without even making a\n\"struct object_entry\" for them, which drastically reduces the memory\nrequirements for serving a clone. Any objects which didn't get marked\nfor reuse just get handled in the usual way (so stuff that was not close\nby in the pack, or stuff that was pushed since your last big repack).\n\nThe downside is that because those objects aren't in our normal packing\nlist, they're not available as delta bases for the new objects we _do_\nsend. So it can make the resulting pack a little bit bigger.\n\n> Yes, for packing it is only hash which is used. And I assume name-hash\n> for bitmap is not enabled by default for compatibility with JGit code.\n>\n> It would make sense to me to eventually enable name-hash bitmap\n> extension by default, as packing result is much better with it. And\n> those who care about compatibility with JGit can just turn it off in\n> their git config.\n\nCorrect, the defaults are for JGit compatibility. If you are not using\nJGit, you should have it on all the time. We went with the conservative\ndefault, but as more people using regular Git bitmaps, it would probably\nbe good to make them less arcane and confusing to use.\n\n> > As I understand your use case, it is OK to do the less careful things.\n> > It's just that pack-objects until now has been split into two modes:\n> > packing to a file is careful, and packing to stdout is less so. And you\n> > want to pack to a file in the non-careful mode.\n> \n> Yes, it should be ok, as after repository extraction git-backup\n> verifies rev-list for all refs\n> \n> https://lab.nexedi.com/kirr/git-backup/blob/7fcb8c67/git-backup.go#L855\n> \n> And if an object is missing - e.g. a blob - rev-list complains:\n> \n> fatal: missing blob object '980a0d5f19a64b4b30a87d4206aade58726b60e3'\n> \n> though it does not catch blob corruptions.\n\nRight, that makes sense. Even the pack-to-disk code invoked by\ngit-repack is not foolproof for blob corruptions. It is only checking a\ncrc, not the full sha1. So it's better than nothing, but not as careful\nas a full index-pack.\n\n> Probably pack.useBitmaps is of no use in normal situation, but for\n> debugging problems related to bitmaps it can be handy. Though when\n> someone debugs he/she can just adjust pack-objects.c . So should we\n> deprecate and eventually remove pack.useBitmaps ?\n\nIn my opinion, yes. If we had any debugging option, it should be\nsomething like \"core.usebitmaps\", to tell _all_ of git to pretend that\nbitmaps don't exist (right now only pack-objects respects it, but we\ncould be using them to optimize more traversals).\n\n> ---- 8< ----\n> From: Kirill Smelkov \n> Subject: [PATCH v3] pack-objects: Teach it to use reachability bitmap index when\n> generating non-stdout pack too\n> \n> Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\n> if a repository has bitmap index, pack-objects can nicely speedup\n> \"Counting objects\" graph traversal phase. That however was done only for\n> case when resultant pack is sent to stdout, not written into a file.\n\nI think we can give more motivation and context here with some of the\nbits that have come out in our discussion. Like:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\n> We can teach pack-objects to use bitmap index for initial object\n> counting phase when generating resultant pack file too:\n> \n> - if we know bitmap index generation is not enabled for resultant pack:\n> \n> Current code has singleton bitmap_git so cannot work simultaneously\n> with two bitmap indices.\n\nSo one reason is that it is not currently possible with the\nimplementation. But I think it also gets to the above bit about\n\"optimal\" packs. We do not want to generate bitmaps off of bitmaps,\nbecause we lose information about the write order. That's probably worth\nmentioning here.\n\n> - if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n> \n> Because on pack reuse raw entries are directly written out to destination\n> pack by write_reused_pack() bypassing needed for pack index generation\n> bookkeeping done by regular codepath in write_one() and friends.\n> \n> (at least that's my understanding after briefly looking at the code)\n\nYes, that's right. We definitely want pack-reuse off for this case.\n\n> NOTE\n> \n> Jeff King suggested that it might be not generally a good idea to\n> use bitmap reachability index when repacking a repository. The reason\n> here is for on-disk repack by default we want\n> \n> - to produce good pack (with bitmap index not-yet-packed objects are\n> emitted to pack in suboptimal order).\n> \n> - to use more robust pack-generation codepath (avoiding possible\n> bugs in bitmap code and possible bitmap index corruption).\n\nAh, this kind of covers the bits I talked about above. I think it makes\nmore sense to introduce them as part of the motivation, though, rather\nthan as a note here.\n\n> Jeff also suggests that pack.useBitmaps was probably a mistake to\n> introduce originally. This way we are not adding another config point,\n> but instead just always default to-file pack-objects not to use bitmap\n> index: Tools which need to generate on-disk packs with using bitmap, can\n> pass --use-bitmap-index explicitly.\n\nThis part is important, though. Basically the reason we respect the\ncommand-line option is that we know that git-repack would never set it\nexplicitly, so it is the hint that pack-objects can use to know which\ncase we are serving: a careful repack of our data, or just extraction of\nsome objects.\n\n> @@ -1052,6 +1053,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> {\n> \tuint32_t index_pos;\n> \n> +\tif (local && has_loose_object_nonlocal(sha1))\n> +\t\treturn 0;\n> +\n> \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> \t\treturn 0;\n\nHrm. Adding entries from the bitmap should ideally be very fast, but\nhere we're introducing extra lookups in the object database. I guess it\nonly kicks in when --local is given, though, which most bitmap-using\npaths would not do.\n\nBut is this check enough? The non-bitmap code path calls\nwant_object_in_pack, which checks not only loose objects, but also\nnon-local packs, and .keep.\n\nThose don't kick in for your use case. I wonder if we should simply have\nsomething like:\n\n if (local || ignore_packed_keep)\n\tuse_bitmap_index = 0;\n\nand just skip bitmaps for those cases. That's easy to reason about, and\nI don't think anybody would care (your use case does not, and the repack\nuse case is already not going to use bitmaps).\n\n> @@ -2773,7 +2777,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n> \t\tunpack_unreachable_expiration = 0;\n> \n> -\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n> +\t/*\n> +\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n> +\t *\n> +\t * - to produce good pack (with bitmap index not-yet-packed objects are\n> +\t * packed in suboptimal order).\n> +\t *\n> +\t * - to use more robust pack-generation codepath (avoiding possible\n> +\t * bugs in bitmap code and possible bitmap index corruption).\n> +\t */\n> +\tif (!pack_to_stdout)\n> +\t\tuse_bitmap_index_default = 0;\n> +\n> +\tif (use_bitmap_index < 0)\n> +\t\tuse_bitmap_index = use_bitmap_index_default;\n> +\n> +\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n> +\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n> \t\tuse_bitmap_index = 0;\n\nAnd that local/keep logic above would just become \"hard\" reasons\nincluded here.\n\n-Peff\n"},{"id":"292120","messageId":"20160725185313.GA13007@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160725184025.GA12297@sigill.intra.peff.net","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-25T18:53:13Z","receivedAt":"2016-07-25T18:53:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 25, 2016 at 02:40:25PM -0400, Jeff King wrote:\n\n> > @@ -1052,6 +1053,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> > {\n> > \tuint32_t index_pos;\n> > \n> > +\tif (local && has_loose_object_nonlocal(sha1))\n> > +\t\treturn 0;\n> > +\n> > \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> > \t\treturn 0;\n> \n> Hrm. Adding entries from the bitmap should ideally be very fast, but\n> here we're introducing extra lookups in the object database. I guess it\n> only kicks in when --local is given, though, which most bitmap-using\n> paths would not do.\n> \n> But is this check enough? The non-bitmap code path calls\n> want_object_in_pack, which checks not only loose objects, but also\n> non-local packs, and .keep.\n> \n> Those don't kick in for your use case. I wonder if we should simply have\n> something like:\n> \n> if (local || ignore_packed_keep)\n> \tuse_bitmap_index = 0;\n> \n> and just skip bitmaps for those cases. That's easy to reason about, and\n> I don't think anybody would care (your use case does not, and the repack\n> use case is already not going to use bitmaps).\n\nBTW, I thought we had more optimizations in this area, but I realized\nthat I had never sent them to the list. I just did, and you may want to\ntake a peek at:\n\n http://thread.gmane.org/gmane.comp.version-control.git/300218\n\nI doubt it will speed up your case much (unless you really do have tons\nof packs in your extraction). And I think it is still worth doing\ndisabling I showed above, even with the optimizations, just because it's\neasier to reason about.\n\nSo I _think_ those optimizations are orthogonal to what we're discussing\nhere, but I wanted to point you at them just in case.\n\n-Peff\n"},{"id":"292368","messageId":"20160727201506.GA15204@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160725185313.GA13007@sigill.intra.peff.net","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-27T20:15:06Z","receivedAt":"2016-07-27T20:15:37Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Mon, Jul 25, 2016 at 02:40:25PM -0400, Jeff King wrote:\n> On Wed, Jul 13, 2016 at 01:52:17PM +0300, Kirill Smelkov wrote:\n> \n> > > So I think if you were to repeatedly \"git repack -adb\" over time, you\n> > > would get worse and worse ordering as objects are added to the\n> > > repository.\n> > \n> > Jeff, first of all thanks for clarifying.\n> > \n> > So it is not-yet-packed-objects which make packing with bitmap less\n> > efficient. I was originally keeping in mind fresh repacked repository\n> > with just built bitmap index and for that case extracting pack with\n> > bitmap index seems to be just ok, but the more not-yet-packed objects we\n> > have the worse the result can be.\n> \n> Right. So I think your scheme is fine as long as you are doing your\n> regular \"pack all into one\" repacks with a real walk, and then\n> \"branching\" off of that with one-off bitmap-computed packs into files\n> (even if you later take a bunch of those files and pull them into a\n> single bitmapped, as long as that final \"all into one\" does the walk).\n> \n> Or I guess another way to think about it would be that if you're\n> computing bitmaps, you'd want to do the actual traversal.\n\nYes, exactly, and thanks for stating it clearly. We are doing repacks\nand recomputing bitmaps doing the real walk. As you say this should be\nfine.\n\n\n> > Yes, it also make sense. I saw write_reused_pack() in upstream git just\n> > copy raw bytes from original to destination pack. You mentioned you have\n> > something better for pack reuse - in your patch queue, in two words, is\n> > it now reusing pack based on object, not raw bytes, or is it something\n> > else?\n> > \n> > In other words in which way it works better? (I'm just curious here as\n> > it is interesting to know)\n> \n> The problem with the existing pack-reuse code is that it doesn't kick in\n> often enough. I think it looks to see that the client wants some\n> percentage of the pack (e.g., 90%), and then just sends the whole\n> beginning. This works especially badly if you have a bunch of related\n> repositories packed together (e.g., all of the forks of torvalds/linux\n> on GitHub), because you'll never hit 90% of that big pack; it has too\n> much unrelated cruft, even if most of the stuff you want _is_ at the\n> beginning. And \"percent of pack\" is not really a useful metric anyway.\n> \n> So the better scheme is more like:\n> \n> 1. Generate the bitmap of objects to send using reachability bitmaps.\n> \n> 2. Do a quick scan of their content in the packfile to see which can\n> be reused verbatim. If they're base objects, we can send them\n> as-is. If they're deltas, we can send them if their base is going\n> to be sent. This fills in another bitmap of \"reusable\" objects.\n> \n> After a long string of unusable objects, you can give up and set\n> the rest of the bitmap to zeroes.\n> \n> 3. Walk the \"reuse\" bitmap and send out the objects more-or-less\n> verbatim. You do have make adjustments to delta-base-offsets for\n> any \"holes\" (so if an object's entry says \"my base is 500 bytes\n> back\", but you omitted some objects in between, you have to adjust\n> that offset).\n> \n> The upside is that you can send out those objects without even making a\n> \"struct object_entry\" for them, which drastically reduces the memory\n> requirements for serving a clone. Any objects which didn't get marked\n> for reuse just get handled in the usual way (so stuff that was not close\n> by in the pack, or stuff that was pushed since your last big repack).\n\nThanks for clarifying. Yes, you are right, current upstream code checks\nto see whether >= 90% of pack is what destination wants and only reuse\nin such case. (I forgot about it, initially putting reuse at side in my\nhead as \"not applicable to git-backup\" because of that >= 90% reason).\n\nSo with the scheme you are drawing above it can be indeed more\nefficient, and applicable to both torvalds/linux+forks and git-backup\ncase (extracting packs from big pack of all repos).\n\nI'm looking forward to your patches on this topic. Please cc me on those\nif you find it convenient.\n\n> The downside is that because those objects aren't in our normal packing\n> list, they're not available as delta bases for the new objects we _do_\n> send. So it can make the resulting pack a little bit bigger.\n\nSo once again, the badness effect is the more, the more we have such\n\"new\" objects not in original main pack - i.e. as loose objects or\nobjects living in other smaller packs. The badness comes to zero in\nideal case of freshly repacked repo with only one big pack.\n\nAlso: after sending reused object, with more code effort, in principle\nwe can hook reused object for being considered as delta-bases for new\nobjects. I mean this should not be impossible in principle, or am I\nmissing something?\n\n\n> > Yes, for packing it is only hash which is used. And I assume name-hash\n> > for bitmap is not enabled by default for compatibility with JGit code.\n> >\n> > It would make sense to me to eventually enable name-hash bitmap\n> > extension by default, as packing result is much better with it. And\n> > those who care about compatibility with JGit can just turn it off in\n> > their git config.\n> \n> Correct, the defaults are for JGit compatibility. If you are not using\n> JGit, you should have it on all the time. We went with the conservative\n> default, but as more people using regular Git bitmaps, it would probably\n> be good to make them less arcane and confusing to use.\n\nI've just checked - JGit 3.7.1.201504261725-r (the version from Debian -\nquite old) does _not_ barf on seeing bitmaps with \"name hash\" section.\nI mean at least it does not error-exit on `jgit gc`, like\nt5310-pack-bitmaps.sh says it can:\n\n # jgit gc will barf if it does not like our bitmaps\n jgit gc\n\n\nI will be sending another mail with relevant JGit people cc'ed to turn\npack.writeBitmapHashCache=true by default.\n\n\n> > ---- 8< ----\n> > From: Kirill Smelkov \n> > Subject: [PATCH v3] pack-objects: Teach it to use reachability bitmap index when\n> > generating non-stdout pack too\n> > \n> > Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\n> > if a repository has bitmap index, pack-objects can nicely speedup\n> > \"Counting objects\" graph traversal phase. That however was done only for\n> > case when resultant pack is sent to stdout, not written into a file.\n> \n> I think we can give more motivation and context here with some of the\n> bits that have come out in our discussion. Like:\n> \n> The reason for this split is that pack-objects tries to determine how\n> \"careful\" it should be based on whether we are packing to disk or to\n> stdout. Packing to disk implies \"git repack\", and that we will likely\n> delete the old packs after finishing. We want to be more careful (so\n> as not to carry forward a corruption, and to generate a more optimal\n> pack), and we presumably run less frequently and can afford extra CPU.\n> Whereas packing to stdout implies serving a remote via \"git fetch\" or\n> \"git push\". This happens more frequently (e.g., a server handling many\n> fetching clients), and we assume the receiving end takes more\n> responsibility for verifying the data.\n> \n> But this isn't always the case. One might want to generate on-disk\n> packfiles for a specialized object transfer. Just using \"--stdout\" and\n> writing to a file is not optimal, as it will not generate the matching\n> pack index.\n> \n> So it would be useful to have some way of overriding this heuristic:\n> to tell pack-objects that even though it should generate on-disk\n> files, it is still OK to use the reachability bitmaps to do the\n> traversal.\n\nThanks, I'm adding this to the patch message in appropriate place.\n\n\n> > We can teach pack-objects to use bitmap index for initial object\n> > counting phase when generating resultant pack file too:\n> > \n> > - if we know bitmap index generation is not enabled for resultant pack:\n> > \n> > Current code has singleton bitmap_git so cannot work simultaneously\n> > with two bitmap indices.\n> \n> So one reason is that it is not currently possible with the\n> implementation. But I think it also gets to the above bit about\n> \"optimal\" packs. We do not want to generate bitmaps off of bitmaps,\n> because we lose information about the write order. That's probably worth\n> mentioning here.\n\nOk. I'm adding relevant note.\n\n\n> > - if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n> > \n> > Because on pack reuse raw entries are directly written out to destination\n> > pack by write_reused_pack() bypassing needed for pack index generation\n> > bookkeeping done by regular codepath in write_one() and friends.\n> > \n> > (at least that's my understanding after briefly looking at the code)\n> \n> Yes, that's right. We definitely want pack-reuse off for this case.\n\nOk, thanks for clarifying.\n\n> > NOTE\n> > \n> > Jeff King suggested that it might be not generally a good idea to\n> > use bitmap reachability index when repacking a repository. The reason\n> > here is for on-disk repack by default we want\n> > \n> > - to produce good pack (with bitmap index not-yet-packed objects are\n> > emitted to pack in suboptimal order).\n> > \n> > - to use more robust pack-generation codepath (avoiding possible\n> > bugs in bitmap code and possible bitmap index corruption).\n> \n> Ah, this kind of covers the bits I talked about above. I think it makes\n> more sense to introduce them as part of the motivation, though, rather\n> than as a note here.\n\nThanks, good idea (after we discussed the robustness issues and start to\ntake them into account). I'm moving this close to head of the\ndescription.\n\n\n> > Jeff also suggests that pack.useBitmaps was probably a mistake to\n> > introduce originally. This way we are not adding another config point,\n> > but instead just always default to-file pack-objects not to use bitmap\n> > index: Tools which need to generate on-disk packs with using bitmap, can\n> > pass --use-bitmap-index explicitly.\n> \n> This part is important, though. Basically the reason we respect the\n> command-line option is that we know that git-repack would never set it\n> explicitly, so it is the hint that pack-objects can use to know which\n> case we are serving: a careful repack of our data, or just extraction of\n> some objects.\n\nYes. To make this very clear I'm also adding explicit note git-repack\nnever passes --use-bitmap-index to pack-objects, so this way we can be\nsure regular on-disk repacking remains robust.\n\n\n> > @@ -1052,6 +1053,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> > {\n> > \tuint32_t index_pos;\n> > \n> > +\tif (local && has_loose_object_nonlocal(sha1))\n> > +\t\treturn 0;\n> > +\n> > \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> > \t\treturn 0;\n> \n> Hrm. Adding entries from the bitmap should ideally be very fast, but\n> here we're introducing extra lookups in the object database. I guess it\n> only kicks in when --local is given, though, which most bitmap-using\n> paths would not do.\n> \n> But is this check enough? The non-bitmap code path calls\n> want_object_in_pack, which checks not only loose objects, but also\n> non-local packs, and .keep.\n> \n> Those don't kick in for your use case. I wonder if we should simply have\n> something like:\n> \n> if (local || ignore_packed_keep)\n> \tuse_bitmap_index = 0;\n> \n> and just skip bitmaps for those cases. That's easy to reason about, and\n> I don't think anybody would care (your use case does not, and the repack\n> use case is already not going to use bitmaps).\n\nYou are right - this is not enough. Initially I did not delved into this\n--local case and only cared to make tests pass (which were failing without this\ncheck when initial patch was using --use-bitmap-index by default).\n\nI agree it is simpler to just not handle this case for now.\n\nActually after thinking about it a bit more, I can see that even current\ncode, allows `git pack-objects --stdout --local or --honor-pack-keep`\nand does not handle those options properly. Thus I suggest to apply the\nfollowing patch as the first one in this now series:\n\n---- 8< ----\nFrom: Kirill Smelkov \nSubject: [PATCH 1/2] pack-objects: Make sure use_bitmap_index is not active under\n --local or --honor-pack-keep\n\nSince 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep should be respected. In non-bitmapped codepath this\nis handled in want_object_in_pack(), but bitmapped codepath has simply\nno such checking at all.\n\nThe bitmapped codepath however was allowing to pass --local and\n--honor-pack-keep and bitmap indices were still used under such\nconditions - potentially giving wrong output (including objects from\nnon-local or .keep'ed pack).\n\nInstead of fixing bitmapped codepath to respect those options, since\ncurrently no one actually need or use them in combination with bitmaps,\nlet's just force use_bitmap_index=0 when any of --local or\n--honor-pack-keep are used and add appropriate comment about\nnot-checking for those in add_object_entry_from_bitmap()\n\nSuggested-by: Jeff King \n---\n builtin/pack-objects.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 15866d7..d7cf782 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1055,6 +1055,12 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\t/*\n+\t * for simplicity we always want object to be in pack, as\n+\t * use_bitmap_index codepath assumes neither --local nor --honor-pack-keep\n+\t * is active.\n+\t */\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\n@@ -2776,6 +2782,15 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n+\t/*\n+\t * \"lazy\" reasons not to use bitmaps; it is easier to reason about when\n+\t * neither --local nor --honor-pack-keep is in action, and so far no one\n+\t * needed nor implemented such support yet.\n+\t */\n+\tif (local || ignore_packed_keep)\n+\t\tuse_bitmap_index = 0;\n+\n+\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n-- \n2.9.0.431.g3cb5c84\n---- 8< ----\n\n\nOn Mon, Jul 25, 2016 at 02:53:13PM -0400, Jeff King wrote:\n> On Mon, Jul 25, 2016 at 02:40:25PM -0400, Jeff King wrote:\n> \n> > > @@ -1052,6 +1053,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> > > {\n> > > \tuint32_t index_pos;\n> > > \n> > > +\tif (local && has_loose_object_nonlocal(sha1))\n> > > +\t\treturn 0;\n> > > +\n> > > \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> > > \t\treturn 0;\n> > \n> > Hrm. Adding entries from the bitmap should ideally be very fast, but\n> > here we're introducing extra lookups in the object database. I guess it\n> > only kicks in when --local is given, though, which most bitmap-using\n> > paths would not do.\n> > \n> > But is this check enough? The non-bitmap code path calls\n> > want_object_in_pack, which checks not only loose objects, but also\n> > non-local packs, and .keep.\n> > \n> > Those don't kick in for your use case. I wonder if we should simply have\n> > something like:\n> > \n> > if (local || ignore_packed_keep)\n> > \tuse_bitmap_index = 0;\n> > \n> > and just skip bitmaps for those cases. That's easy to reason about, and\n> > I don't think anybody would care (your use case does not, and the repack\n> > use case is already not going to use bitmaps).\n> \n> BTW, I thought we had more optimizations in this area, but I realized\n> that I had never sent them to the list. I just did, and you may want to\n> take a peek at:\n> \n> http://thread.gmane.org/gmane.comp.version-control.git/300218\n>\n> I doubt it will speed up your case much (unless you really do have tons\n> of packs in your extraction).\n\n> And I think it is still worth doing disabling I showed above, even\n> with the optimizations, just because it's easier to reason about.\n> \n> So I _think_ those optimizations are orthogonal to what we're discussing\n> here, but I wanted to point you at them just in case.\n\nThanks for the head-ups and for sending it. Yes, for git-backup we\nusually do restore from freshly repacked repo, But the optimization is\nuseful in many other cases. After reading the patches I wonder why\ncurrent state was for so a long time. I frequently have close to 50\npacks in a repository, with only automatic gc triggering to do full\nrepack, and for that case looping always through whole 50 packs for\nevery object, even when we already found the pack an object lives in, is\njust a waste of time. And yes, on client side I almost never use\nalternate objects store and almost never do concurrent fetches etc (so if\nI understand correctly, no .keep files). Thanks for sending it.\n\n( Btw, if we are talking about optimizations, here is something related\n to pack extractions, I think it is worth mentioning just in case:\n\n https://lab.nexedi.com/kirr/git-backup/blob/ad6c6853/NOTES.restore\n\n It is a scheme how to compute \"non-overlapping\" set of packs when\n restoring repositories from big backup repo, so both disk size (same\n objects in many packs) and time (computing packs with many same objects\n many times) are not wasted. Then shared between repositories packs are\n just hardlinked to appropriate places.\n \n It is in line with e.g. \n \n https://git.kernel.org/cgit/git/git.git/commit/tree-diff.c?id=72441af7\n \n because it is algorithmical optimization, only for now I do not have\n working code implementing it yet. )\n\n\nanyway updated main patch goes below:\n\n(whole-patch interdiff)\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1ef85a6..f8b173d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1053,12 +1053,15 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n {\n uint32_t index_pos;\n \n- if (local && has_loose_object_nonlocal(sha1))\n- return 0;\n-\n if (have_duplicate_entry(sha1, 0, &index_pos))\n return 0;\n \n+ /*\n+ * for simplicity we always want object to be in pack, as\n+ * use_bitmap_index path assumes neither --local nor --honor-pack-keep\n+ * is active.\n+ */\n+\n create_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n display_progress(progress_state, nr_result);\n@@ -2796,6 +2799,15 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n if (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n use_bitmap_index = 0;\n \n+ /*\n+ * \"lazy\" reasons not to use bitmaps; it is easier to reason about when\n+ * neither --local nor --honor-pack-keep is in action, and so far no one\n+ * needed nor implemented such support yet.\n+ */\n+ if (local || ignore_packed_keep)\n+ use_bitmap_index = 0;\n+\n+\n if (pack_to_stdout || !rev_list_all)\n write_bitmap_index = 0;\n \n\n(log interdiff)\n@@ -5,29 +5,69 @@ if a repository has bitmap index, pack-objects can nicely speedup\n \"Counting objects\" graph traversal phase. That however was done only for\n case when resultant pack is sent to stdout, not written into a file.\n \n-We can teach pack-objects to use bitmap index for initial object\n+The reason here is for on-disk repack by default we want:\n+\n+- to produce good pack (with bitmap index not-yet-packed objects are\n+ emitted to pack in suboptimal order).\n+\n+- to use more robust pack-generation codepath (avoiding possible\n+ bugs in bitmap code and possible bitmap index corruption).\n+\n+Jeff Kind further explains:\n+\n+ The reason for this split is that pack-objects tries to determine how\n+ \"careful\" it should be based on whether we are packing to disk or to\n+ stdout. Packing to disk implies \"git repack\", and that we will likely\n+ delete the old packs after finishing. We want to be more careful (so\n+ as not to carry forward a corruption, and to generate a more optimal\n+ pack), and we presumably run less frequently and can afford extra CPU.\n+ Whereas packing to stdout implies serving a remote via \"git fetch\" or\n+ \"git push\". This happens more frequently (e.g., a server handling many\n+ fetching clients), and we assume the receiving end takes more\n+ responsibility for verifying the data.\n+\n+ But this isn't always the case. One might want to generate on-disk\n+ packfiles for a specialized object transfer. Just using \"--stdout\" and\n+ writing to a file is not optimal, as it will not generate the matching\n+ pack index.\n+\n+ So it would be useful to have some way of overriding this heuristic:\n+ to tell pack-objects that even though it should generate on-disk\n+ files, it is still OK to use the reachability bitmaps to do the\n+ traversal.\n+\n+So we can teach pack-objects to use bitmap index for initial object\n counting phase when generating resultant pack file too:\n \n+- if we care it is not activated under git-repack:\n+\n+ See above about repack robustness and not forward-carrying corruption.\n+\n - if we know bitmap index generation is not enabled for resultant pack:\n \n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n \n+ We also want to avoid (at least with current implementation)\n+ generating bitmaps off of bitmaps. The reason here is: when generating\n+ a pack, not-yet-packed objects will be emitted into pack in\n+ suboptimal order and added to tail of the bitmap as \"extended entries\".\n+ When the resultant pack + some new objects in associated repository\n+ are in turn used to generate another pack with bitmap, the situation\n+ repeats: new objects are again not emitted optimally and just added to\n+ bitmap tail - not in recency order.\n+\n+ So the pack badness can grow over time when at each step we have\n+ bitmapped pack + some other objects. That's why we want to avoid\n+ generating bitmaps off of bitmaps, not to let pack badness grow.\n+\n - if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n \n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n \n- (at least that's my understanding after briefly looking at the code)\n-\n-We also need to care and teach add_object_entry_from_bitmap() to respect\n---local via not adding nonlocal loose object to resultant pack (this\n-is bitmap-codepath counterpart of daae0625 (pack-objects: extend --local\n-to mean ignore non-local loose objects too) -- not to break 'loose\n-objects in alternate ODB are not repacked' in t7700-repack.sh .\n-\n-Otherwise all git tests pass, and for pack-objects -> file we get nice\n+This way all git tests pass, and for pack-objects -> file we get nice\n speedup:\n \n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n@@ -55,27 +95,62 @@ initial motivation for the patch.\n \n NOTE\n \n-Jeff King suggested that it might be not generally a good idea to\n-use bitmap reachability index when repacking a repository. The reason\n-here is for on-disk repack by default we want\n-\n-- to produce good pack (with bitmap index not-yet-packed objects are\n- emitted to pack in suboptimal order).\n-\n-- to use more robust pack-generation codepath (avoiding possible\n- bugs in bitmap code and possible bitmap index corruption).\n-\n Jeff also suggests that pack.useBitmaps was probably a mistake to\n introduce originally. This way we are not adding another config point,\n but instead just always default to-file pack-objects not to use bitmap\n index: Tools which need to generate on-disk packs with using bitmap, can\n-pass --use-bitmap-index explicitly.\n+pass --use-bitmap-index explicitly. And git-repack does never pass\n+--use-bitmap-index, so this way we can be sure regular on-disk repacking\n+remains robust.\n+\n+NOTE2\n+\n+`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\n+than `git pack-objects file.pack`. Extracting erp5.git pack from\n+lab.nexedi.com backup repository:\n+\n+---- 8< ----\n+$ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n+\n+real 0m22.309s\n+user 0m21.148s\n+sys 0m0.932s\n+\n+$ time git index-pack erp5pack-stdout.pack\n+\n+real 0m50.873s <-- more than 2 times slower than time to generate pack itself!\n+user 0m49.300s\n+sys 0m1.360s\n+---- 8< ----\n+\n+So the time for\n+\n+ `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n+\n+while\n+\n+ `pack-objects file.pack` which does both pack and index is 27s.\n+\n+And even\n+\n+ `pack-objects --no-use-bitmap-index file.pack` is 37s.\n+\n+Jeff explains:\n+\n+ The packfile does not carry the sha1 of the objects. A receiving\n+ index-pack has to compute them itself, including inflating and applying\n+ all of the deltas.\n+\n+that's why for `git-backup restore` we want to teach `git pack-objects\n+file.pack` to use bitmaps instead of using `git pack-objects --stdout\n+>file.pack` + `git index-pack file.pack`.\n \n More context:\n \n http://article.gmane.org/gmane.comp.version-control.git/299063\n http://article.gmane.org/gmane.comp.version-control.git/299107\n http://article.gmane.org/gmane.comp.version-control.git/299420\n+ http://article.gmane.org/gmane.comp.version-control.git/300217\n \n Cc: Vicent Marti \n Helped-by: Jeff King \n\n\n(patch itself)\n---- 8< ----\nFrom: Kirill Smelkov \nSubject: [PATCH 2/2] pack-objects: Teach it to use reachability bitmap index when\n generating non-stdout pack too\n\nStarting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nThe reason here is for on-disk repack by default we want:\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff Kind further explains:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\nSo we can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we care it is not activated under git-repack:\n\n See above about repack robustness and not forward-carrying corruption.\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n We also want to avoid (at least with current implementation)\n generating bitmaps off of bitmaps. The reason here is: when generating\n a pack, not-yet-packed objects will be emitted into pack in\n suboptimal order and added to tail of the bitmap as \"extended entries\".\n When the resultant pack + some new objects in associated repository\n are in turn used to generate another pack with bitmap, the situation\n repeats: new objects are again not emitted optimally and just added to\n bitmap tail - not in recency order.\n\n So the pack badness can grow over time when at each step we have\n bitmapped pack + some other objects. That's why we want to avoid\n generating bitmaps off of bitmaps, not to let pack badness grow.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\nThis way all git tests pass, and for pack-objects -> file we get nice\nspeedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly. And git-repack does never pass\n--use-bitmap-index, so this way we can be sure regular on-disk repacking\nremains robust.\n\nNOTE2\n\n`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n---- 8< ----\n$ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\nreal 0m22.309s\nuser 0m21.148s\nsys 0m0.932s\n\n$ time git index-pack erp5pack-stdout.pack\n\nreal 0m50.873s <-- more than 2 times slower than time to generate pack itself!\nuser 0m49.300s\nsys 0m1.360s\n---- 8< ----\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n\nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n\nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\nJeff explains:\n\n The packfile does not carry the sha1 of the objects. A receiving\n index-pack has to compute them itself, including inflating and applying\n all of the deltas.\n\nthat's why for `git-backup restore` we want to teach `git pack-objects\nfile.pack` to use bitmaps instead of using `git pack-objects --stdout\n>file.pack` + `git index-pack file.pack`.\n\nMore context:\n\n http://article.gmane.org/gmane.comp.version-control.git/299063\n http://article.gmane.org/gmane.comp.version-control.git/299107\n http://article.gmane.org/gmane.comp.version-control.git/299420\n http://article.gmane.org/gmane.comp.version-control.git/300217\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n Documentation/config.txt | 3 +++\n builtin/pack-objects.c | 25 +++++++++++++++++++++----\n t/t5310-pack-bitmaps.sh | 14 ++++++++++++++\n 3 files changed, 38 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex b0ed71f..39ab41d 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2244,6 +2244,9 @@ pack.useBitmaps::\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n++\n+*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n+\tpack bitmaps.\n \n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d7cf782..f8b173d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE;\n \n@@ -2231,7 +2232,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2494,7 +2495,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (prepare_bitmap_walk(revs) < 0)\n \t\treturn -1;\n \n-\tif (pack_options_allow_reuse() &&\n+\tif (pack_options_allow_reuse() && pack_to_stdout &&\n \t !reuse_partial_packfile_from_bitmap(\n \t\t\t&reuse_packfile,\n \t\t\t&reuse_packfile_objects,\n@@ -2779,7 +2780,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \t/*\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 0d03583..0802b7c 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -117,6 +117,20 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.verify &&\n+\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n+\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n+\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.0.431.g3cb5c84\n"},{"id":"292371","messageId":"xmqqlh0mvmpn.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160727201506.GA15204@teco.navytux.spb.ru","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-27T20:40:36Z","receivedAt":"2016-07-27T20:40:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> > From: Kirill Smelkov \n> Subject: [PATCH 1/2] pack-objects: Make sure use_bitmap_index is not active under\n> --local or --honor-pack-keep\n>\n> Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\n> are two codepaths in pack-objects: with & without using bitmap\n> reachability index.\n>\n> However add_object_entry_from_bitmap(), despite its non-bitmapped\n> counterpart add_object_entry(), in no way does check for whether --local\n> or --honor-pack-keep should be respected. In non-bitmapped codepath this\n> is handled in want_object_in_pack(), but bitmapped codepath has simply\n> no such checking at all.\n>\n> The bitmapped codepath however was allowing to pass --local and\n> --honor-pack-keep and bitmap indices were still used under such\n> conditions - potentially giving wrong output (including objects from\n> non-local or .keep'ed pack).\n>\n> Instead of fixing bitmapped codepath to respect those options, since\n> currently no one actually need or use them in combination with bitmaps,\n> let's just force use_bitmap_index=0 when any of --local or\n> --honor-pack-keep are used and add appropriate comment about\n> not-checking for those in add_object_entry_from_bitmap()\n>\n> Suggested-by: Jeff King \n> ---\n> builtin/pack-objects.c | 15 +++++++++++++++\n> 1 file changed, 15 insertions(+)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 15866d7..d7cf782 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1055,6 +1055,12 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> \t\treturn 0;\n> \n> +\t/*\n> +\t * for simplicity we always want object to be in pack, as\n> +\t * use_bitmap_index codepath assumes neither --local nor --honor-pack-keep\n> +\t * is active.\n> +\t */\n\nI am not sure this comment is useful to readers.\n\nUnless the readers are comparing add_object_entry() and this\nfunction and wondering why this side lacks a check here, iow, when\nthey are merely following from a caller of this function through\nthis function down to its callee to understand what goes on, this\ncomment would not help them and only confuse them.\n\nIf we were to say something to help those who are comparing these\ntwo functions, I think we should be more explicit, i.e.\n\n The caller disables use-bitmap-index when --local or\n --honor-pack-keep options are in effect because bitmap code is\n not prepared to handle them. Because the control does not reach\n here if these options are in effect, the check with\n want_object_in_pack() to skip objects is not done.\n\nor something like that.\n\nOr is the rest of the bitmap codepath prepared to handle these\noptions and it is just the matter of adding the missing check with\nwant_object_in_pack() here to make it work correctly?\n\n> \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n> \n> \tdisplay_progress(progress_state, nr_result);\n> @@ -2776,6 +2782,15 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n> \t\tuse_bitmap_index = 0;\n> \n> +\t/*\n> +\t * \"lazy\" reasons not to use bitmaps; it is easier to reason about when\n> +\t * neither --local nor --honor-pack-keep is in action, and so far no one\n> +\t * needed nor implemented such support yet.\n> +\t */\n\nJustifying comment like this is a good idea, but the comment above\ndoes not make it very clear that this is a correctness fix, i.e. if\nwe do not disable, the code will do a wrong thing.\n\nThe other logic to disable use of bitmap we can see in the\npre-context would also benefit from some description as to why;\n6b8fda2d (pack-objects: use bitmaps when packing objects,\n2013-12-21) didn't do a very good job in that---the reason is not\nclear in its log message, either.\n\n> +\tif (local || ignore_packed_keep)\n> +\t\tuse_bitmap_index = 0;\n> +\n> +\n\nI see one extra blank line here ;-)\n\n> \tif (pack_to_stdout || !rev_list_all)\n> \t\twrite_bitmap_index = 0;\n\nThanks.\n"},{"id":"292449","messageId":"20160728202237.GC4026@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqqlh0mvmpn.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-28T20:22:37Z","receivedAt":"2016-07-28T20:22:48Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Junio, first of all thanks for feedback,\n\nOn Wed, Jul 27, 2016 at 01:40:36PM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> > > From: Kirill Smelkov \n> > Subject: [PATCH 1/2] pack-objects: Make sure use_bitmap_index is not active under\n> > --local or --honor-pack-keep\n> >\n> > Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\n> > are two codepaths in pack-objects: with & without using bitmap\n> > reachability index.\n> >\n> > However add_object_entry_from_bitmap(), despite its non-bitmapped\n> > counterpart add_object_entry(), in no way does check for whether --local\n> > or --honor-pack-keep should be respected. In non-bitmapped codepath this\n> > is handled in want_object_in_pack(), but bitmapped codepath has simply\n> > no such checking at all.\n> >\n> > The bitmapped codepath however was allowing to pass --local and\n> > --honor-pack-keep and bitmap indices were still used under such\n> > conditions - potentially giving wrong output (including objects from\n> > non-local or .keep'ed pack).\n> >\n> > Instead of fixing bitmapped codepath to respect those options, since\n> > currently no one actually need or use them in combination with bitmaps,\n> > let's just force use_bitmap_index=0 when any of --local or\n> > --honor-pack-keep are used and add appropriate comment about\n> > not-checking for those in add_object_entry_from_bitmap()\n> >\n> > Suggested-by: Jeff King \n> > ---\n> > builtin/pack-objects.c | 15 +++++++++++++++\n> > 1 file changed, 15 insertions(+)\n> >\n> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> > index 15866d7..d7cf782 100644\n> > --- a/builtin/pack-objects.c\n> > +++ b/builtin/pack-objects.c\n> > @@ -1055,6 +1055,12 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> > \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> > \t\treturn 0;\n> > \n> > +\t/*\n> > +\t * for simplicity we always want object to be in pack, as\n> > +\t * use_bitmap_index codepath assumes neither --local nor --honor-pack-keep\n> > +\t * is active.\n> > +\t */\n> \n> I am not sure this comment is useful to readers.\n> \n> Unless the readers are comparing add_object_entry() and this\n> function and wondering why this side lacks a check here, iow, when\n> they are merely following from a caller of this function through\n> this function down to its callee to understand what goes on, this\n> comment would not help them and only confuse them.\n> \n> If we were to say something to help those who are comparing these\n> two functions, I think we should be more explicit, i.e.\n> \n> The caller disables use-bitmap-index when --local or\n> --honor-pack-keep options are in effect because bitmap code is\n> not prepared to handle them. Because the control does not reach\n> here if these options are in effect, the check with\n> want_object_in_pack() to skip objects is not done.\n> \n> or something like that.\n\nYou are probably right.\n\n\n> Or is the rest of the bitmap codepath prepared to handle these\n> options and it is just the matter of adding the missing check with\n> want_object_in_pack() here to make it work correctly?\n\nI'm waiting so long for main patch to be at least queued to pu, that I'm\nnow a bit frustrated and ready to do something not related to main goal :)\n(they say every joke contains part of a joke). Here is something from\nsleepy me:\n\n---- 8< ----\nFrom: Kirill Smelkov \nDate: Wed, 27 Jul 2016 22:18:04 +0300\nSubject: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to\n respect --local, --honor-pack-keep and --incremental\n\nSince 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep or --incremental should be respected. In\nnon-bitmapped codepath this is handled in want_object_in_pack(), but\nbitmapped codepath has simply no such checking at all.\n\nThe bitmapped codepath however was allowing to pass in all those options\nand with bitmap indices still being used under such conditions -\npotentially giving wrong output (e.g. including objects from non-local or\n.keep'ed pack).\n\nWe can easily fix this by noting the following: when an object comes to\nadd_object_entry_from_bitmap() it can come for two reasons:\n\n 1. entries coming from main pack covered by bitmap index, and\n 2. object coming from, possibly alternate, loose or other packs.\n\nFor \"2\" we always have pack not yet found by bitmap traversal code, and\nthus we can simply reuse non-bitmapped want_object_in_pack() to find in\nwhich pack an object lives and also for taking omitting decision.\n\nFor \"1\" we always have pack already found by bitmap traversal code and we\nonly need to check that pack for same omission criteria used in\nwant_object_in_pack() for found_pack.\n\nSuggested-by: Junio C Hamano \nDiscussed-with: Jeff King \n---\n builtin/pack-objects.c | 39 +++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 100 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 139 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a2f8cfd..34b3019 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -987,6 +987,42 @@ static int want_object_in_pack(const unsigned char *sha1,\n \treturn 1;\n }\n \n+/* Like want_object_in_pack() but for objects coming from-under bitmapped traversal */\n+static int want_object_in_pack_bitmap(const unsigned char *sha1,\n+\t\t\t\t struct packed_git **found_pack,\n+\t\t\t\t off_t *found_offset)\n+{\n+\tstruct packed_git *p = *found_pack;\n+\n+\t/*\n+\t * There are two types of requests coming here:\n+\t * 1. entries coming from main pack covered by bitmap index, and\n+\t * 2. object coming from, possibly alternate, loose or other packs.\n+\t *\n+\t * For \"1\" we always have *found_pack != NULL passed here from\n+\t * traverse_bitmap_commit_list(). (*found_pack is bitmap_git.pack\n+\t * actually).\n+\t *\n+\t * For \"2\" we always have *found_pack == NULL passed here from\n+\t * traverse_bitmap_commit_list() - since this is the way bitmap\n+\t * traversal passes here \"extended\" bitmap entries.\n+\t */\n+\n+\t/* objects not covered by bitmap */\n+\tif (!p)\n+\t\treturn want_object_in_pack(sha1, 0, found_pack, found_offset);\n+\n+\t/* objects covered by bitmap - we only have to check p wrt local and .keep */\n+\tif (incremental)\n+\t\treturn 0;\n+\tif (local && !p->pack_local)\n+\t\treturn 0;\n+\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n static void create_object_entry(const unsigned char *sha1,\n \t\t\t\tenum object_type type,\n \t\t\t\tuint32_t hash,\n@@ -1055,6 +1091,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\tif (!want_object_in_pack_bitmap(sha1, &pack, &offset))\n+\t\treturn 0;\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..a76f6ca 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -118,6 +118,88 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects respects --local (non-local loose)' '\n+\tmkdir -p alt_objects/pack &&\n+\techo $(pwd)/alt_objects > .git/objects/info/alternates &&\n+\techo content1 > file1 &&\n+\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n+\tgit add file1 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file1 &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >1.pack &&\n+\tgit index-pack 1.pack &&\n+\tgit verify-pack -v 1.pack >1.objects &&\n+\tif egrep \"^$objsha1\" 1.objects; then\n+\t\techo \"Non-local object present in pack generated with --local: $objsha1\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\techo content2 > file2 &&\n+\tobjsha2=$(git hash-object -w file2) &&\n+\tgit add file2 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file2 &&\n+\tpack2=$(echo $objsha2 | \\\n+\t\tgit pack-objects pack2) &&\n+\tmv pack2-$pack2.* .git/objects/pack/ &&\n+\ttouch .git/objects/pack/pack2-$pack2.keep &&\n+\trm $(objpath $objsha2) &&\n+\techo HEAD | \\\n+\tgit pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\tgit index-pack 2a.pack &&\n+\tgit verify-pack -v 2a.pack >2a.objects &&\n+\tif egrep \"^$objsha2\" 2a.objects; then\n+\t\techo \"Object from .keeped pack present in pack generated with --honor-pack-keep: $objsha2\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local pack)' '\n+\tmv .git/objects/pack/pack2-$pack2.* alt_objects/pack/ &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >2b.pack &&\n+\tgit index-pack 2b.pack &&\n+\tgit verify-pack -v 2b.pack >2b.objects &&\n+\tif egrep \"^$objsha2\" 2b.objects; then\n+\t\techo \"Non-local object present in pack generated with --local: $objsha2\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\tgit verify-pack -v .git/objects/pack/$packbitmap.pack >packbitmap.verify &&\n+\tgrep -o \"^$_x40\" packbitmap.verify |sort >packbitmap.objects &&\n+\ttouch .git/objects/pack/$packbitmap.keep &&\n+\techo HEAD | \\\n+\tgit pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\tgit index-pack 3a.pack &&\n+\tgit verify-pack -v 3a.pack >3a.objects &&\n+\tif grep -qFf packbitmap.objects 3a.objects; then\n+\t\techo \"Object from .keeped bitmapped pack present in pack generated with --honour-pack-keep\"\n+\t\treturn 1\n+\tfi &&\n+\trm .git/objects/pack/$packbitmap.keep\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\tmv .git/objects/pack/$packbitmap.* alt_objects/pack/ &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >3b.pack &&\n+\tgit index-pack 3b.pack &&\n+\tgit verify-pack -v 3b.pack >3b.objects &&\n+\tif grep -qFf packbitmap.objects 3b.objects; then\n+\t\techo \"Non-local object from bitmapped pack present in pack generated with --local\"\n+\t\treturn 1\n+\tfi &&\n+\tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n@@ -143,6 +225,24 @@ test_expect_success 'create objects for missing-HAVE tests' '\n \tEOF\n '\n \n+test_expect_success 'pack-objects respects --incremental' '\n+\tcat >revs2 <<-EOF &&\n+\tHEAD\n+\t$commit\n+\tEOF\n+\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\tgit index-pack 4.pack &&\n+\tgit verify-pack -v 4.pack >4.verify &&\n+\tgrep -o \"^$_x40\" 4.verify |sort >4.objects &&\n+\ttest_line_count = 4 4.objects &&\n+\tgit rev-list --objects $commit >revlist &&\n+\tgrep -o \"^$_x40\" revlist |sort >objects &&\n+\tif grep -qvFf objects 4.objects; then\n+\t\techo \"Expected objects not present in incremental pack\"\n+\t\treturn 1\n+\tfi\n+'\n+\n test_expect_success 'pack with missing blob' '\n \trm $(objpath $blob) &&\n \tgit pack-objects --stdout --revs /dev/null\n-- \n2.9.0.431.g3cb5c84\n---- 8< ----\n\n\nand main patch updated to avoid trivial conflicts\n\n---- 8< ----\nFrom: Kirill Smelkov \nDate: Thu, 7 Jul 2016 20:12:00 +0300\nSubject: [PATCH 2/2] pack-objects: Teach it to use reachability bitmap index\n when generating non-stdout pack too\n\nStarting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nThe reason here is for on-disk repack by default we want:\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff Kind further explains:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\nSo we can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we care it is not activated under git-repack:\n\n See above about repack robustness and not forward-carrying corruption.\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n We also want to avoid (at least with current implementation)\n generating bitmaps off of bitmaps. The reason here is: when generating\n a pack, not-yet-packed objects will be emitted into pack in\n suboptimal order and added to tail of the bitmap as \"extended entries\".\n When the resultant pack + some new objects in associated repository\n are in turn used to generate another pack with bitmap, the situation\n repeats: new objects are again not emitted optimally and just added to\n bitmap tail - not in recency order.\n\n So the pack badness can grow over time when at each step we have\n bitmapped pack + some other objects. That's why we want to avoid\n generating bitmaps off of bitmaps, not to let pack badness grow.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\nThis way all git tests pass, and for pack-objects -> file we get nice\nspeedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly. And git-repack does never pass\n--use-bitmap-index, so this way we can be sure regular on-disk repacking\nremains robust.\n\nNOTE2\n\n`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n---- 8< ----\n$ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\nreal 0m22.309s\nuser 0m21.148s\nsys 0m0.932s\n\n$ time git index-pack erp5pack-stdout.pack\n\nreal 0m50.873s <-- more than 2 times slower than time to generate pack itself!\nuser 0m49.300s\nsys 0m1.360s\n---- 8< ----\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n\nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n\nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\nJeff explains:\n\n The packfile does not carry the sha1 of the objects. A receiving\n index-pack has to compute them itself, including inflating and applying\n all of the deltas.\n\nthat's why for `git-backup restore` we want to teach `git pack-objects\nfile.pack` to use bitmaps instead of using `git pack-objects --stdout\n>file.pack` + `git index-pack file.pack`.\n\nMore context:\n\n http://article.gmane.org/gmane.comp.version-control.git/299063\n http://article.gmane.org/gmane.comp.version-control.git/299107\n http://article.gmane.org/gmane.comp.version-control.git/299420\n http://article.gmane.org/gmane.comp.version-control.git/300217\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n Documentation/config.txt | 3 +++\n builtin/pack-objects.c | 25 +++++++++++++++++++++----\n t/t5310-pack-bitmaps.sh | 14 ++++++++++++++\n 3 files changed, 38 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 8b1aee4..6a903c0 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2244,6 +2244,9 @@ pack.useBitmaps::\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n++\n+*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n+\tpack bitmaps.\n \n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 34b3019..2b2e74a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2264,7 +2265,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2527,7 +2528,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (prepare_bitmap_walk(revs) < 0)\n \t\treturn -1;\n \n-\tif (pack_options_allow_reuse() &&\n+\tif (pack_options_allow_reuse() && pack_to_stdout &&\n \t !reuse_partial_packfile_from_bitmap(\n \t\t\t&reuse_packfile,\n \t\t\t&reuse_packfile_objects,\n@@ -2812,7 +2813,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex a76f6ca..58c3b29 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -200,6 +200,20 @@ test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n \tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.verify &&\n+\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n+\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n+\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.0.431.g3cb5c84\n---- 8< ----\n\nThanks,\nKirill\n"},{"id":"292454","messageId":"xmqqh9b9qx5m.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160728202237.GC4026@teco.navytux.spb.ru","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-28T21:18:29Z","receivedAt":"2016-07-28T21:18:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> I'm waiting so long for main patch to be at least queued to pu, that I'm\n> now a bit frustrated and ready to do something not related to main goal :)\n\nPerhaps the first step would be to stop putting multiple patches in\na single e-mail buried after a few pages of discussion. I will not\neven find that there _are_ multiple patches in the message if I am\nnot involved directly in the discussion, and the discussion is still\nongoing, because it is likely that I'd skim just a few paragraphs at\nthe top before going on to other messages.\n\nI won't touch the message I am responding to, as your -- 8< -- cut\nmark does not even seem to be a reliable marker between patches\n(i.e. I see something like this that is clearly not a message\nboundary:\n\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n---- 8< ----\n$ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\nreal 0m22.309s\n...\n)\n\n"},{"id":"292495","messageId":"20160729074051.GA5987@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqqh9b9qx5m.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] pack-objects: Use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-29T07:40:52Z","receivedAt":"2016-07-29T07:41:05Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Thu, Jul 28, 2016 at 02:18:29PM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> > I'm waiting so long for main patch to be at least queued to pu, that I'm\n> > now a bit frustrated and ready to do something not related to main goal :)\n> \n> Perhaps the first step would be to stop putting multiple patches in\n> a single e-mail buried after a few pages of discussion. I will not\n> even find that there _are_ multiple patches in the message if I am\n> not involved directly in the discussion, and the discussion is still\n> ongoing, because it is likely that I'd skim just a few paragraphs at\n> the top before going on to other messages.\n> \n> I won't touch the message I am responding to, as your -- 8< -- cut\n> mark does not even seem to be a reliable marker between patches\n> (i.e. I see something like this that is clearly not a message\n> boundary:\n> \n> than `git pack-objects file.pack`. Extracting erp5.git pack from\n> lab.nexedi.com backup repository:\n> \n> ---- 8< ----\n> $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n> \n> real 0m22.309s\n> ...\n> )\n\nOk, makes sense and my fault. I'm resending each patch as separate\nmessage in reply to this mail.\n"},{"id":"292498","messageId":"20160729074656.23813-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160729074051.GA5987@teco.navytux.spb.ru","subject":"[PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-29T07:46:56Z","receivedAt":"2016-07-29T07:47:18Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep or --incremental should be respected. In\nnon-bitmapped codepath this is handled in want_object_in_pack(), but\nbitmapped codepath has simply no such checking at all.\n\nThe bitmapped codepath however was allowing to pass in all those options\nand with bitmap indices still being used under such conditions -\npotentially giving wrong output (e.g. including objects from non-local or\n.keep'ed pack).\n\nWe can easily fix this by noting the following: when an object comes to\nadd_object_entry_from_bitmap() it can come for two reasons:\n\n 1. entries coming from main pack covered by bitmap index, and\n 2. object coming from, possibly alternate, loose or other packs.\n\nFor \"2\" we always have pack not yet found by bitmap traversal code, and\nthus we can simply reuse non-bitmapped want_object_in_pack() to find in\nwhich pack an object lives and also for taking omitting decision.\n\nFor \"1\" we always have pack already found by bitmap traversal code and we\nonly need to check that pack for same criteria used in\nwant_object_in_pack() for found_pack.\n\nSuggested-by: Junio C Hamano \nDiscussed-with: Jeff King \n---\n builtin/pack-objects.c | 39 +++++++++++++++++++\n t/t5310-pack-bitmaps.sh | 100 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 139 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a2f8cfd..34b3019 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -987,6 +987,42 @@ static int want_object_in_pack(const unsigned char *sha1,\n \treturn 1;\n }\n \n+/* Like want_object_in_pack() but for objects coming from-under bitmapped traversal */\n+static int want_object_in_pack_bitmap(const unsigned char *sha1,\n+\t\t\t\t struct packed_git **found_pack,\n+\t\t\t\t off_t *found_offset)\n+{\n+\tstruct packed_git *p = *found_pack;\n+\n+\t/*\n+\t * There are two types of requests coming here:\n+\t * 1. entries coming from main pack covered by bitmap index, and\n+\t * 2. object coming from, possibly alternate, loose or other packs.\n+\t *\n+\t * For \"1\" we always have *found_pack != NULL passed here from\n+\t * traverse_bitmap_commit_list(). (*found_pack is bitmap_git.pack\n+\t * actually).\n+\t *\n+\t * For \"2\" we always have *found_pack == NULL passed here from\n+\t * traverse_bitmap_commit_list() - since this is the way bitmap\n+\t * traversal passes here \"extended\" bitmap entries.\n+\t */\n+\n+\t/* objects not covered by bitmap */\n+\tif (!p)\n+\t\treturn want_object_in_pack(sha1, 0, found_pack, found_offset);\n+\n+\t/* objects covered by bitmap - we only have to check p wrt local and .keep */\n+\tif (incremental)\n+\t\treturn 0;\n+\tif (local && !p->pack_local)\n+\t\treturn 0;\n+\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n static void create_object_entry(const unsigned char *sha1,\n \t\t\t\tenum object_type type,\n \t\t\t\tuint32_t hash,\n@@ -1055,6 +1091,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\tif (!want_object_in_pack_bitmap(sha1, &pack, &offset))\n+\t\treturn 0;\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..a76f6ca 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -118,6 +118,88 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects respects --local (non-local loose)' '\n+\tmkdir -p alt_objects/pack &&\n+\techo $(pwd)/alt_objects > .git/objects/info/alternates &&\n+\techo content1 > file1 &&\n+\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n+\tgit add file1 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file1 &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >1.pack &&\n+\tgit index-pack 1.pack &&\n+\tgit verify-pack -v 1.pack >1.objects &&\n+\tif egrep \"^$objsha1\" 1.objects; then\n+\t\techo \"Non-local object present in pack generated with --local: $objsha1\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\techo content2 > file2 &&\n+\tobjsha2=$(git hash-object -w file2) &&\n+\tgit add file2 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file2 &&\n+\tpack2=$(echo $objsha2 | \\\n+\t\tgit pack-objects pack2) &&\n+\tmv pack2-$pack2.* .git/objects/pack/ &&\n+\ttouch .git/objects/pack/pack2-$pack2.keep &&\n+\trm $(objpath $objsha2) &&\n+\techo HEAD | \\\n+\tgit pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\tgit index-pack 2a.pack &&\n+\tgit verify-pack -v 2a.pack >2a.objects &&\n+\tif egrep \"^$objsha2\" 2a.objects; then\n+\t\techo \"Object from .keeped pack present in pack generated with --honor-pack-keep: $objsha2\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local pack)' '\n+\tmv .git/objects/pack/pack2-$pack2.* alt_objects/pack/ &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >2b.pack &&\n+\tgit index-pack 2b.pack &&\n+\tgit verify-pack -v 2b.pack >2b.objects &&\n+\tif egrep \"^$objsha2\" 2b.objects; then\n+\t\techo \"Non-local object present in pack generated with --local: $objsha2\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\tgit verify-pack -v .git/objects/pack/$packbitmap.pack >packbitmap.verify &&\n+\tgrep -o \"^$_x40\" packbitmap.verify |sort >packbitmap.objects &&\n+\ttouch .git/objects/pack/$packbitmap.keep &&\n+\techo HEAD | \\\n+\tgit pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\tgit index-pack 3a.pack &&\n+\tgit verify-pack -v 3a.pack >3a.objects &&\n+\tif grep -qFf packbitmap.objects 3a.objects; then\n+\t\techo \"Object from .keeped bitmapped pack present in pack generated with --honour-pack-keep\"\n+\t\treturn 1\n+\tfi &&\n+\trm .git/objects/pack/$packbitmap.keep\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\tmv .git/objects/pack/$packbitmap.* alt_objects/pack/ &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >3b.pack &&\n+\tgit index-pack 3b.pack &&\n+\tgit verify-pack -v 3b.pack >3b.objects &&\n+\tif grep -qFf packbitmap.objects 3b.objects; then\n+\t\techo \"Non-local object from bitmapped pack present in pack generated with --local\"\n+\t\treturn 1\n+\tfi &&\n+\tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n@@ -143,6 +225,24 @@ test_expect_success 'create objects for missing-HAVE tests' '\n \tEOF\n '\n \n+test_expect_success 'pack-objects respects --incremental' '\n+\tcat >revs2 <<-EOF &&\n+\tHEAD\n+\t$commit\n+\tEOF\n+\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\tgit index-pack 4.pack &&\n+\tgit verify-pack -v 4.pack >4.verify &&\n+\tgrep -o \"^$_x40\" 4.verify |sort >4.objects &&\n+\ttest_line_count = 4 4.objects &&\n+\tgit rev-list --objects $commit >revlist &&\n+\tgrep -o \"^$_x40\" revlist |sort >objects &&\n+\tif grep -qvFf objects 4.objects; then\n+\t\techo \"Expected objects not present in incremental pack\"\n+\t\treturn 1\n+\tfi\n+'\n+\n test_expect_success 'pack with missing blob' '\n \trm $(objpath $blob) &&\n \tgit pack-objects --stdout --revs /dev/null\n-- \n2.9.0.431.g3cb5c84\n"},{"id":"292499","messageId":"20160729074746.31862-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160729074051.GA5987@teco.navytux.spb.ru","subject":"[PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-07-29T07:47:46Z","receivedAt":"2016-07-29T07:47:58Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nThe reason here is for on-disk repack by default we want:\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff Kind further explains:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\nSo we can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we care it is not activated under git-repack:\n\n See above about repack robustness and not forward-carrying corruption.\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n We also want to avoid (at least with current implementation)\n generating bitmaps off of bitmaps. The reason here is: when generating\n a pack, not-yet-packed objects will be emitted into pack in\n suboptimal order and added to tail of the bitmap as \"extended entries\".\n When the resultant pack + some new objects in associated repository\n are in turn used to generate another pack with bitmap, the situation\n repeats: new objects are again not emitted optimally and just added to\n bitmap tail - not in recency order.\n\n So the pack badness can grow over time when at each step we have\n bitmapped pack + some other objects. That's why we want to avoid\n generating bitmaps off of bitmaps, not to let pack badness grow.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\nThis way all git tests pass, and for pack-objects -> file we get nice\nspeedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly. And git-repack does never pass\n--use-bitmap-index, so this way we can be sure regular on-disk repacking\nremains robust.\n\nNOTE2\n\n`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\n real 0m22.309s\n user 0m21.148s\n sys 0m0.932s\n\n $ time git index-pack erp5pack-stdout.pack\n\n real 0m50.873s <-- more than 2 times slower than time to generate pack itself!\n user 0m49.300s\n sys 0m1.360s\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n\nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n\nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\nJeff explains:\n\n The packfile does not carry the sha1 of the objects. A receiving\n index-pack has to compute them itself, including inflating and applying\n all of the deltas.\n\nthat's why for `git-backup restore` we want to teach `git pack-objects\nfile.pack` to use bitmaps instead of using `git pack-objects --stdout\n>file.pack` + `git index-pack file.pack`.\n\nMore context:\n\n http://article.gmane.org/gmane.comp.version-control.git/299063\n http://article.gmane.org/gmane.comp.version-control.git/299107\n http://article.gmane.org/gmane.comp.version-control.git/299420\n http://article.gmane.org/gmane.comp.version-control.git/300217\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n Documentation/config.txt | 3 +++\n builtin/pack-objects.c | 25 +++++++++++++++++++++----\n t/t5310-pack-bitmaps.sh | 14 ++++++++++++++\n 3 files changed, 38 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 8b1aee4..6a903c0 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2244,6 +2244,9 @@ pack.useBitmaps::\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n++\n+*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n+\tpack bitmaps.\n \n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 34b3019..2b2e74a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2264,7 +2265,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2527,7 +2528,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (prepare_bitmap_walk(revs) < 0)\n \t\treturn -1;\n \n-\tif (pack_options_allow_reuse() &&\n+\tif (pack_options_allow_reuse() && pack_to_stdout &&\n \t !reuse_partial_packfile_from_bitmap(\n \t\t\t&reuse_packfile,\n \t\t\t&reuse_packfile_objects,\n@@ -2812,7 +2813,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex a76f6ca..58c3b29 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -200,6 +200,20 @@ test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n \tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.verify &&\n+\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n+\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n+\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.0.431.g3cb5c84\n"},{"id":"292772","messageId":"xmqqy44gjqv9.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160729074656.23813-1-kirr@nexedi.com","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-01T18:17:30Z","receivedAt":"2016-08-01T18:26:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\n> are two codepaths in pack-objects: with & without using bitmap\n> reachability index.\n>\n> However add_object_entry_from_bitmap(), despite its non-bitmapped\n> counterpart add_object_entry(), in no way does check for whether --local\n> or --honor-pack-keep or --incremental should be respected. In\n> non-bitmapped codepath this is handled in want_object_in_pack(), but\n> bitmapped codepath has simply no such checking at all.\n>\n> The bitmapped codepath however was allowing to pass in all those options\n> and with bitmap indices still being used under such conditions -\n> potentially giving wrong output (e.g. including objects from non-local or\n> .keep'ed pack).\n>\n> We can easily fix this by noting the following: when an object comes to\n> add_object_entry_from_bitmap() it can come for two reasons:\n>\n> 1. entries coming from main pack covered by bitmap index, and\n> 2. object coming from, possibly alternate, loose or other packs.\n>\n> For \"2\" we always have pack not yet found by bitmap traversal code, and\n> thus we can simply reuse non-bitmapped want_object_in_pack() to find in\n> which pack an object lives and also for taking omitting decision.\n>\n> For \"1\" we always have pack already found by bitmap traversal code and we\n> only need to check that pack for same criteria used in\n> want_object_in_pack() for found_pack.\n>\n> Suggested-by: Junio C Hamano \n> Discussed-with: Jeff King \n> ---\n\nI do not think I suggested much of this to deserve credit like this,\nthough, as I certainly haven't thought about the pros-and-cons\nbetween adding the same \"some object in pack may not want to be in\nthe output\" logic to the bitmap side, or punting the bitmap codepath\nwhen local/keep are involved.\n\n> +/* Like want_object_in_pack() but for objects coming from-under bitmapped traversal */\n> +static int want_object_in_pack_bitmap(const unsigned char *sha1,\n> +\t\t\t\t struct packed_git **found_pack,\n> +\t\t\t\t off_t *found_offset)\n> +{\n> +\tstruct packed_git *p = *found_pack;\n> +\n> +\t/*\n> +\t * There are two types of requests coming here:\n> +\t * 1. entries coming from main pack covered by bitmap index, and\n> +\t * 2. object coming from, possibly alternate, loose or other packs.\n> +\t *\n> +\t * For \"1\" we always have *found_pack != NULL passed here from\n> +\t * traverse_bitmap_commit_list(). (*found_pack is bitmap_git.pack\n> +\t * actually).\n> +\t *\n> +\t * For \"2\" we always have *found_pack == NULL passed here from\n> +\t * traverse_bitmap_commit_list() - since this is the way bitmap\n> +\t * traversal passes here \"extended\" bitmap entries.\n> +\t */\n> +\n> +\t/* objects not covered by bitmap */\n> +\tif (!p)\n> +\t\treturn want_object_in_pack(sha1, 0, found_pack, found_offset);\n> +\t/* objects covered by bitmap - we only have to check p wrt local and .keep */\n\nI am assuming that p != NULL only means \"this object exists in THIS\npack\", without saying anything about \"this object may also exist in\nother places\", but \"we only have to check\" implies that \"p != NULL\"\nmeans \"this object exists *ONLY* in this pack and nowhere else\".\n\nPuzzled.\n\n\n> +\tif (incremental)\n> +\t\treturn 0;\n> +\tif (local && !p->pack_local)\n> +\t\treturn 0;\n> +\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n> +\t\treturn 0;\n> +\n> +\treturn 1;\n> +}\n> +\n"},{"id":"293349","messageId":"20160808123734.GA3995@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqqy44gjqv9.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-08T12:37:35Z","receivedAt":"2016-08-08T12:37:47Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Mon, Aug 01, 2016 at 11:17:30AM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> > Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\n> > are two codepaths in pack-objects: with & without using bitmap\n> > reachability index.\n> >\n> > However add_object_entry_from_bitmap(), despite its non-bitmapped\n> > counterpart add_object_entry(), in no way does check for whether --local\n> > or --honor-pack-keep or --incremental should be respected. In\n> > non-bitmapped codepath this is handled in want_object_in_pack(), but\n> > bitmapped codepath has simply no such checking at all.\n> >\n> > The bitmapped codepath however was allowing to pass in all those options\n> > and with bitmap indices still being used under such conditions -\n> > potentially giving wrong output (e.g. including objects from non-local or\n> > .keep'ed pack).\n> >\n> > We can easily fix this by noting the following: when an object comes to\n> > add_object_entry_from_bitmap() it can come for two reasons:\n> >\n> > 1. entries coming from main pack covered by bitmap index, and\n> > 2. object coming from, possibly alternate, loose or other packs.\n> >\n> > For \"2\" we always have pack not yet found by bitmap traversal code, and\n> > thus we can simply reuse non-bitmapped want_object_in_pack() to find in\n> > which pack an object lives and also for taking omitting decision.\n> >\n> > For \"1\" we always have pack already found by bitmap traversal code and we\n> > only need to check that pack for same criteria used in\n> > want_object_in_pack() for found_pack.\n> >\n> > Suggested-by: Junio C Hamano \n> > Discussed-with: Jeff King \n> > ---\n> \n> I do not think I suggested much of this to deserve credit like this,\n> though, as I certainly haven't thought about the pros-and-cons\n> between adding the same \"some object in pack may not want to be in\n> the output\" logic to the bitmap side, or punting the bitmap codepath\n> when local/keep are involved.\n\nI understand. Still for me it was you who convinced me to add proper\nsupport for e.g. --local vs bitmap instead of special-casing it.\nI think we also can avoid punting the bitmap codepath - please see\nbelow.\n\n\n> > +/* Like want_object_in_pack() but for objects coming from-under bitmapped traversal */\n> > +static int want_object_in_pack_bitmap(const unsigned char *sha1,\n> > +\t\t\t\t struct packed_git **found_pack,\n> > +\t\t\t\t off_t *found_offset)\n> > +{\n> > +\tstruct packed_git *p = *found_pack;\n> > +\n> > +\t/*\n> > +\t * There are two types of requests coming here:\n> > +\t * 1. entries coming from main pack covered by bitmap index, and\n> > +\t * 2. object coming from, possibly alternate, loose or other packs.\n> > +\t *\n> > +\t * For \"1\" we always have *found_pack != NULL passed here from\n> > +\t * traverse_bitmap_commit_list(). (*found_pack is bitmap_git.pack\n> > +\t * actually).\n> > +\t *\n> > +\t * For \"2\" we always have *found_pack == NULL passed here from\n> > +\t * traverse_bitmap_commit_list() - since this is the way bitmap\n> > +\t * traversal passes here \"extended\" bitmap entries.\n> > +\t */\n> > +\n> > +\t/* objects not covered by bitmap */\n> > +\tif (!p)\n> > +\t\treturn want_object_in_pack(sha1, 0, found_pack, found_offset);\n> > +\t/* objects covered by bitmap - we only have to check p wrt local and .keep */\n> \n> I am assuming that p != NULL only means \"this object exists in THIS\n> pack\", without saying anything about \"this object may also exist in\n> other places\", but \"we only have to check\" implies that \"p != NULL\"\n> means \"this object exists *ONLY* in this pack and nowhere else\".\n> \n> Puzzled.\n\nYou are right. Being new to --local and .keep I've missed this. I've\nadded tests to cover cases like \"object lives in both bitmapped pack and\nnon-local loose or .keep'ed pack\" and made the adjustments. The checks\nare now live unified in want_object_in_pack() for both bitmapped and\nnon-bitmapped codepaths. Please apply the following corrected patch on\ntop of 56dfeb62 (jk/pack-objects-optim).\n\nThanks,\nKirill\n\n---- 8< ----\nFrom: Kirill Smelkov \nDate: Fri, 29 Jul 2016 10:46:56 +0300\nSubject: [PATCH v2] pack-objects: Teach --use-bitmap-index codepath to respect\n --local, --honor-pack-keep and --incremental\n\nSince 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep or --incremental should be respected. In\nnon-bitmapped codepath this is handled in want_object_in_pack(), but\nbitmapped codepath has simply no such checking at all.\n\nThe bitmapped codepath however was allowing to pass in all those options\nand with bitmap indices still being used under such conditions -\npotentially giving wrong output (e.g. including objects from non-local or\n.keep'ed pack).\n\nWe can easily fix this by noting the following: when an object comes to\nadd_object_entry_from_bitmap() it can come for two reasons:\n\n 1. entries coming from main pack covered by bitmap index, and\n 2. object coming from, possibly alternate, loose or other packs.\n\n\"2\" can be already handled by want_object_in_pack() and to cover\n\"1\" we can teach want_object_in_pack() to expect that *found_pack can be\nnon-NULL, meaning calling client already found object's pack entry.\n\nIn want_object_in_pack() we care to start the checks from already found\npack, if we have one, this way caring not to do more than 1 iteration\nin case neither --local nor --honour-pack-keep are active. In\nparticular, as p5310-pack-bitmaps.sh shows, we do not do harm to\nserved-with-bitmap clones performance-wise:\n\n Test 56dfeb62 this tree\n -----------------------------------------------------------------\n 5310.2: repack to disk 9.14(8.18+0.31) 8.89(7.92+0.28) -2.7%\n 5310.3: simulated clone 1.94(2.14+0.07) 1.91(2.08+0.08) -1.5%\n 5310.4: simulated fetch 0.75(1.01+0.02) 0.75(0.94+0.07) +0.0%\n 5310.6: partial bitmap 1.99(2.44+0.16) 1.95(2.40+0.14) -2.0%\n\nwith all differences strangely showing we are a bit faster now, but\nprobably all being within noise.\n\nSuggested-by: Junio C Hamano \nDiscussed-with: Jeff King \n---\n builtin/pack-objects.c | 36 ++++++++++++-----\n t/t5310-pack-bitmaps.sh | 103 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 130 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c4c2a3c..2c274d3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -948,9 +948,9 @@ static int have_duplicate_entry(const unsigned char *sha1,\n * Check whether we want the object in the pack (e.g., we do not want\n * objects found in non-local stores if the \"--local\" option was used).\n *\n- * As a side effect of this check, we will find the packed version of this\n- * object, if any. We therefore pass out the pack information to avoid having\n- * to look it up again later.\n+ * As a side effect of this check, if object's pack entry was not already found,\n+ * we will find the packed version of this object, if any. We therefore pass\n+ * out the pack information to avoid having to look it up again later.\n */\n static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t int exclude,\n@@ -958,15 +958,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t off_t *found_offset)\n {\n \tstruct packed_git *p;\n+\tstruct packed_git *pack1 = *found_pack;\n+\tint pack1_seen = !pack1;\n \n \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n \t\treturn 0;\n \n-\t*found_pack = NULL;\n-\t*found_offset = 0;\n+\t/*\n+\t * If we already know the pack object lives in, start checks from that\n+\t * pack - in the usual case when neither --local was given nor .keep files\n+\t * are present the loop will degenerate to have only 1 iteration.\n+\t */\n+\tfor (p = (pack1 ? pack1 : packed_git); p;\n+\t p = (pack1_seen ? p->next : packed_git), pack1_seen = 1) {\n+\t\toff_t offset;\n+\n+\t\tif (p == pack1) {\n+\t\t\tif (pack1_seen)\n+\t\t\t\tcontinue;\n+\t\t\toffset = *found_offset;\n+\t\t}\n+\t\telse {\n+\t\t\toffset = find_pack_entry_one(sha1, p);\n+\t\t}\n \n-\tfor (p = packed_git; p; p = p->next) {\n-\t\toff_t offset = find_pack_entry_one(sha1, p);\n \t\tif (offset) {\n \t\t\tif (!*found_pack) {\n \t\t\t\tif (!is_pack_valid(p))\n@@ -1039,8 +1054,8 @@ static const char no_closure_warning[] = N_(\n static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \t\t\t const char *name, int exclude)\n {\n-\tstruct packed_git *found_pack;\n-\toff_t found_offset;\n+\tstruct packed_git *found_pack = NULL;\n+\toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n@@ -1073,6 +1088,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n+\t\treturn 0;\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..1a61de4 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -16,6 +16,7 @@ test_expect_success 'setup repo with moderate-sized history' '\n \t\ttest_commit side-$i\n \tdone &&\n \tgit checkout master &&\n+\tbitmaptip=$(git show-ref -s master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n \tgit config repack.writebitmaps true &&\n@@ -118,6 +119,90 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects respects --local (non-local loose)' '\n+\tmkdir -p alt_objects/pack &&\n+\techo $(pwd)/alt_objects > .git/objects/info/alternates &&\n+\techo content1 > file1 &&\n+\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n+\tgit cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n+\tgit add file1 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file1 &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >1.pack &&\n+\tgit index-pack 1.pack &&\n+\tgit verify-pack -v 1.pack >1.objects &&\n+\techo -e \"$objsha1\\n$blob\" >nonlocal-loose &&\n+\tif grep -qFf nonlocal-loose 1.objects; then\n+\t\techo \"Non-local object present in pack generated with --local\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\techo content2 > file2 &&\n+\tobjsha2=$(git hash-object -w file2) &&\n+\tgit add file2 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file2 &&\n+\techo -e \"$objsha2\\n$bitmaptip\" >keepobjects &&\n+\tpack2=$(git pack-objects pack2 2a.pack &&\n+\tgit index-pack 2a.pack &&\n+\tgit verify-pack -v 2a.pack >2a.objects &&\n+\tif grep -qFf keepobjects 2a.objects; then\n+\t\techo \"Object from .keeped pack present in pack generated with --honor-pack-keep\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local pack)' '\n+\tmv .git/objects/pack/pack2-$pack2.* alt_objects/pack/ &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >2b.pack &&\n+\tgit index-pack 2b.pack &&\n+\tgit verify-pack -v 2b.pack >2b.objects &&\n+\tif grep -qFf keepobjects 2b.objects; then\n+\t\techo \"Non-local object present in pack generated with --local\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\tgit verify-pack -v .git/objects/pack/$packbitmap.pack >packbitmap.verify &&\n+\tgrep -o \"^$_x40\" packbitmap.verify |sort >packbitmap.objects &&\n+\ttouch .git/objects/pack/$packbitmap.keep &&\n+\techo HEAD | \\\n+\tgit pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\tgit index-pack 3a.pack &&\n+\tgit verify-pack -v 3a.pack >3a.objects &&\n+\tif grep -qFf packbitmap.objects 3a.objects; then\n+\t\techo \"Object from .keeped bitmapped pack present in pack generated with --honour-pack-keep\"\n+\t\treturn 1\n+\tfi &&\n+\trm .git/objects/pack/$packbitmap.keep\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\tmv .git/objects/pack/$packbitmap.* alt_objects/pack/ &&\n+\techo HEAD | \\\n+\tgit pack-objects --local --stdout --revs >3b.pack &&\n+\tgit index-pack 3b.pack &&\n+\tgit verify-pack -v 3b.pack >3b.objects &&\n+\tif grep -qFf packbitmap.objects 3b.objects; then\n+\t\techo \"Non-local object from bitmapped pack present in pack generated with --local\"\n+\t\treturn 1\n+\tfi &&\n+\tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n@@ -143,6 +228,24 @@ test_expect_success 'create objects for missing-HAVE tests' '\n \tEOF\n '\n \n+test_expect_success 'pack-objects respects --incremental' '\n+\tcat >revs2 <<-EOF &&\n+\tHEAD\n+\t$commit\n+\tEOF\n+\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\tgit index-pack 4.pack &&\n+\tgit verify-pack -v 4.pack >4.verify &&\n+\tgrep -o \"^$_x40\" 4.verify |sort >4.objects &&\n+\ttest_line_count = 4 4.objects &&\n+\tgit rev-list --objects $commit >revlist &&\n+\tgrep -o \"^$_x40\" revlist |sort >objects &&\n+\tif grep -qvFf objects 4.objects; then\n+\t\techo \"Expected objects not present in incremental pack\"\n+\t\treturn 1\n+\tfi\n+'\n+\n test_expect_success 'pack with missing blob' '\n \trm $(objpath $blob) &&\n \tgit pack-objects --stdout --revs /dev/null\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293350","messageId":"20160808135020.yggthtfekdiwq3lz@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160808123734.GA3995@teco.navytux.spb.ru","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-08T13:50:20Z","receivedAt":"2016-08-08T13:50:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 08, 2016 at 03:37:35PM +0300, Kirill Smelkov wrote:\n\n> @@ -958,15 +958,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n> \t\t\t off_t *found_offset)\n> {\n> \tstruct packed_git *p;\n> +\tstruct packed_git *pack1 = *found_pack;\n> +\tint pack1_seen = !pack1;\n> \n> \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n> \t\treturn 0;\n> \n> -\t*found_pack = NULL;\n> -\t*found_offset = 0;\n> +\t/*\n> +\t * If we already know the pack object lives in, start checks from that\n> +\t * pack - in the usual case when neither --local was given nor .keep files\n> +\t * are present the loop will degenerate to have only 1 iteration.\n> +\t */\n> +\tfor (p = (pack1 ? pack1 : packed_git); p;\n> +\t p = (pack1_seen ? p->next : packed_git), pack1_seen = 1) {\n> +\t\toff_t offset;\n\nHmm. So this is basically sticking the found-pack at the front of the\nloop.\n\nWe either need to look at zero packs here (we already know where the\nobject is, and we don't need to bother with --local or .keep lookups),\nor we need to look at all of them (to check for local/keep).\n\nI guess you structured it this way to try to reuse the \"can we break out\nearly\" logic from the middle of the loop. So we go through the loop one\ntime, and then break out. And then this:\n\n> +\t\tif (p == pack1) {\n> +\t\t\tif (pack1_seen)\n> +\t\t\t\tcontinue;\n> +\t\t\toffset = *found_offset;\n> +\t\t}\n> +\t\telse {\n> +\t\t\toffset = find_pack_entry_one(sha1, p);\n> +\t\t}\n\nis meant to make that one-time through the loop cheaper. So I don't\nthink it's wrong, but it's very confusing to me.\n\nWould it be simpler to stick that logic in a function like:\n\n static int want_found_object(int exclude, struct packed_git *pack)\n {\n\tif (exclude)\n\t\treturn 1;\n\tif (incremental)\n\t\treturn 0;\n\n\t/* if we can break early, then do so */\n\tif (!ignore_packed_keep &&\n\t (!local || !have_non_local_packs))\n\t\treturn 1;\n\n\tif (local && !p->pack_local)\n\t\treturn 0;\n\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n\t\treturn 0;\n\n\t/* indeterminate; keep looking for more packs */\n\treturn -1;\n }\n\n static int want_object_in_pack(...)\n {\n\t...\n\tif (!exclude && local && has_loose_object_nonlocal(sha1))\n\t\treturn 0;\n\n\tif (*found_pack) {\n\t\tint ret = want_found_object(exclude, *found_pack);\n\t\tif (ret != -1)\n\t\t\treturn ret;\n\t}\n\n\tfor (p = packed_git; p; p = p->next) {\n\t\toff_t offset;\n\n\t\tif (p == *found_pack)\n\t\t\toffset = *found_offset;\n\t\telse\n\t\t\toffset = find_pack_entry(sha1, p);\n\t\tif (offset) {\n\t\t\t... fill in *found_pack ...\n\t\t\tint ret = want_found_object(exclude, p);\n\t\t\tif (ret != -1)\n\t\t\t\treturn ret;\n\t\t}\n\t}\n\treturn 1;\n }\n\nThat's a little more verbose, but IMHO the flow is a lot easier to\nfollow (especially as the later re-rolls of that series actually muck\nwith the loop order more, but with this approach there's no conflict).\n\n> static int add_object_entry(const unsigned char *sha1, enum object_type type,\n> \t\t\t const char *name, int exclude)\n> {\n> -\tstruct packed_git *found_pack;\n> -\toff_t found_offset;\n> +\tstruct packed_git *found_pack = NULL;\n> +\toff_t found_offset = 0;\n> \tuint32_t index_pos;\n> \n> \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n> @@ -1073,6 +1088,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> \t\treturn 0;\n> \n> +\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n> +\t\treturn 0;\n> +\n\nThis part looks correct and easy to understand.\n\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index 3893afd..1a61de4 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -16,6 +16,7 @@ test_expect_success 'setup repo with moderate-sized history' '\n> \t\ttest_commit side-$i\n> \tdone &&\n> \tgit checkout master &&\n> +\tbitmaptip=$(git show-ref -s master) &&\n\nOur usual method for getting a sha1 is \"git rev-parse\". I don't think\nthere's anything wrong with your method, but it might be better to stick\nto the canonical one (I had to actually look up \"show-ref -s\").\n\n> @@ -118,6 +119,90 @@ test_expect_success 'incremental repack can disable bitmaps' '\n> \tgit repack -d --no-write-bitmap-index\n> '\n> \n> +test_expect_success 'pack-objects respects --local (non-local loose)' '\n> +\tmkdir -p alt_objects/pack &&\n> +\techo $(pwd)/alt_objects > .git/objects/info/alternates &&\n> +\techo content1 > file1 &&\n\nStyle: we don't put a space between \">\" and the filename.\n\n> +\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n> +\tgit cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n\nI'm not sure why we need two objects in the fake alt_objects repository.\nShouldn't one be enough to do the test?\n\n> +\tgit add file1 &&\n\nI think this will actually skip the writing of the loose object, because\nit's already available in the alternate object store. You probably want\nto do this before adding it there.\n\n> +\ttest_tick &&\n> +\tgit commit -m commit_file1 &&\n> +\techo HEAD | \\\n\nNo need for \"\\\" after a \"|\"; the shell knows it has to keep looking.\n\n> +\tgit pack-objects --local --stdout --revs >1.pack &&\n> +\tgit index-pack 1.pack &&\n\nI'd have expected you to use the non-stdout version here. Is this meant\nto be independent of your other patch (I think that's OK).\n\n> +\tgit verify-pack -v 1.pack >1.objects &&\n\nIt's cheaper to use \"git show-index <1.pack\", and the output is saner,\ntoo.\n\n> +\techo -e \"$objsha1\\n$blob\" >nonlocal-loose &&\n\n\"echo -e\" isn't portable. You can use \"printf\", or two echos like:\n\n {\n echo one &&\n echo two\n } >file\n\n(though I'm still not sure what we gain by checking both).\n\n> +\tif grep -qFf nonlocal-loose 1.objects; then\n> +\t\techo \"Non-local object present in pack generated with --local\"\n> +\t\treturn 1\n> +\tfi\n> +'\n\ngrep -f isn't portable. However, I think:\n\n echo $objsha1 >expect &&\n git show-index <1.pack | cut -d' ' -f2 >actual\n test_cmp expect actual\n\nwould work (if you do stick with two entries, you might need to sort\nyour \"expect\").\n\nI think similar comments apply to the other tests. I would have expected\n\"respects --local (non-local pack)\" to come next (i.e., to keep all of\nthe --local tests together). But you seem to interleave them with\n--honor-pack-keep.\n\n-Peff\n"},{"id":"293351","messageId":"20160808135145.7tsigbmbsoar372y@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160808135020.yggthtfekdiwq3lz@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-08T13:51:45Z","receivedAt":"2016-08-08T13:51:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 08, 2016 at 09:50:20AM -0400, Jeff King wrote:\n\n> > +\tgit pack-objects --local --stdout --revs >1.pack &&\n> > +\tgit index-pack 1.pack &&\n> \n> I'd have expected you to use the non-stdout version here. Is this meant\n> to be independent of your other patch (I think that's OK).\n\nOh, nevermind, I forgot this was meant to be a preparatory patch. So it\nmakes sense to use --stdout in the tests.\n\n-Peff\n"},{"id":"293352","messageId":"20160808135600.c6hdlqwwtqe7thd5@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160729074746.31862-1-kirr@nexedi.com","subject":"Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-08T13:56:00Z","receivedAt":"2016-08-08T13:56:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 29, 2016 at 10:47:46AM +0300, Kirill Smelkov wrote:\n\n> @@ -2527,7 +2528,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n> \tif (prepare_bitmap_walk(revs) < 0)\n> \t\treturn -1;\n> \n> -\tif (pack_options_allow_reuse() &&\n> +\tif (pack_options_allow_reuse() && pack_to_stdout &&\n> \t !reuse_partial_packfile_from_bitmap(\n\nShould pack_to_stdout just be part of pack_options_allow_reuse()?\n\n> @@ -2812,7 +2813,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n> \t\tunpack_unreachable_expiration = 0;\n> \n> -\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n> +\t/*\n> +\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n> +\t *\n> +\t * - to produce good pack (with bitmap index not-yet-packed objects are\n> +\t * packed in suboptimal order).\n> +\t *\n> +\t * - to use more robust pack-generation codepath (avoiding possible\n> +\t * bugs in bitmap code and possible bitmap index corruption).\n> +\t */\n> +\tif (!pack_to_stdout)\n> +\t\tuse_bitmap_index_default = 0;\n> +\n> +\tif (use_bitmap_index < 0)\n> +\t\tuse_bitmap_index = use_bitmap_index_default;\n> +\n> +\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n> +\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n> \t\tuse_bitmap_index = 0;\n\nThis all makes sense and looks good.\n\n> +test_expect_success 'pack-objects to file can use bitmap' '\n> +\t# make sure we still have 1 bitmap index from previous tests\n> +\tls .git/objects/pack/ | grep bitmap >output &&\n> +\ttest_line_count = 1 output &&\n> +\t# verify equivalent packs are generated with/without using bitmap index\n> +\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa +\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb +\tgit verify-pack -v packa-$packasha1.pack >packa.verify &&\n> +\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n> +\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n> +\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n> +\ttest_cmp packa.objects packb.objects\n> +'\n\nI don't think \"grep -o\" is portable. However, an easier way to do this\nis probably:\n\n # these are already in sorted order\n git show-index packa.objects &&\n git show-index packb.objects &&\n test_cmp packa.objects packb.objects\n\n-Peff\n"},{"id":"293357","messageId":"20160808154054.GB3995@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160808135600.c6hdlqwwtqe7thd5@sigill.intra.peff.net","subject":"Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-08T15:40:54Z","receivedAt":"2016-08-08T15:41:03Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Mon, Aug 08, 2016 at 09:56:00AM -0400, Jeff King wrote:\n> On Fri, Jul 29, 2016 at 10:47:46AM +0300, Kirill Smelkov wrote:\n> \n> > @@ -2527,7 +2528,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n> > \tif (prepare_bitmap_walk(revs) < 0)\n> > \t\treturn -1;\n> > \n> > -\tif (pack_options_allow_reuse() &&\n> > +\tif (pack_options_allow_reuse() && pack_to_stdout &&\n> > \t !reuse_partial_packfile_from_bitmap(\n> \n> Should pack_to_stdout just be part of pack_options_allow_reuse()?\n\nYes, makes sense; thanks for catching this.\n\n\n> > @@ -2812,7 +2813,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> > \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n> > \t\tunpack_unreachable_expiration = 0;\n> > \n> > -\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n> > +\t/*\n> > +\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n> > +\t *\n> > +\t * - to produce good pack (with bitmap index not-yet-packed objects are\n> > +\t * packed in suboptimal order).\n> > +\t *\n> > +\t * - to use more robust pack-generation codepath (avoiding possible\n> > +\t * bugs in bitmap code and possible bitmap index corruption).\n> > +\t */\n> > +\tif (!pack_to_stdout)\n> > +\t\tuse_bitmap_index_default = 0;\n> > +\n> > +\tif (use_bitmap_index < 0)\n> > +\t\tuse_bitmap_index = use_bitmap_index_default;\n> > +\n> > +\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n> > +\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n> > \t\tuse_bitmap_index = 0;\n> \n> This all makes sense and looks good.\n\nThanks.\n\n\n> > +test_expect_success 'pack-objects to file can use bitmap' '\n> > +\t# make sure we still have 1 bitmap index from previous tests\n> > +\tls .git/objects/pack/ | grep bitmap >output &&\n> > +\ttest_line_count = 1 output &&\n> > +\t# verify equivalent packs are generated with/without using bitmap index\n> > +\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa > +\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb > +\tgit verify-pack -v packa-$packasha1.pack >packa.verify &&\n> > +\tgit verify-pack -v packb-$packbsha1.pack >packb.verify &&\n> > +\tgrep -o \"^$_x40\" packa.verify |sort >packa.objects &&\n> > +\tgrep -o \"^$_x40\" packb.verify |sort >packb.objects &&\n> > +\ttest_cmp packa.objects packb.objects\n> > +'\n> \n> I don't think \"grep -o\" is portable. However, an easier way to do this\n> is probably:\n> \n> # these are already in sorted order\n> git show-index packa.objects &&\n> git show-index packb.objects &&\n> test_cmp packa.objects packb.objects\n\nThanks for the info. I did not knew about show-index when I was starting\nto work on this and later it just came out of sight. Please find\ncorrected patch below.\n\n---- 8< ----\nFrom: Kirill Smelkov \nDate: Fri, 29 Jul 2016 10:47:46 +0300\nSubject: [PATCH v5] pack-objects: Teach it to use reachability bitmap index when\n generating non-stdout pack too\n\nStarting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nThe reason here is for on-disk repack by default we want:\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff Kind further explains:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\nSo we can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we care it is not activated under git-repack:\n\n See above about repack robustness and not forward-carrying corruption.\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n We also want to avoid (at least with current implementation)\n generating bitmaps off of bitmaps. The reason here is: when generating\n a pack, not-yet-packed objects will be emitted into pack in\n suboptimal order and added to tail of the bitmap as \"extended entries\".\n When the resultant pack + some new objects in associated repository\n are in turn used to generate another pack with bitmap, the situation\n repeats: new objects are again not emitted optimally and just added to\n bitmap tail - not in recency order.\n\n So the pack badness can grow over time when at each step we have\n bitmapped pack + some other objects. That's why we want to avoid\n generating bitmaps off of bitmaps, not to let pack badness grow.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\nThis way for pack-objects -> file we get nice speedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly. And git-repack does never pass\n--use-bitmap-index, so this way we can be sure regular on-disk repacking\nremains robust.\n\nNOTE2\n\n`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\n real 0m22.309s\n user 0m21.148s\n sys 0m0.932s\n\n $ time git index-pack erp5pack-stdout.pack\n\n real 0m50.873s <-- more than 2 times slower than time to generate pack itself!\n user 0m49.300s\n sys 0m1.360s\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n\nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n\nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\nJeff explains:\n\n The packfile does not carry the sha1 of the objects. A receiving\n index-pack has to compute them itself, including inflating and applying\n all of the deltas.\n\nthat's why for `git-backup restore` we want to teach `git pack-objects\nfile.pack` to use bitmaps instead of using `git pack-objects --stdout\n>file.pack` + `git index-pack file.pack`.\n\nMore context:\n\n http://marc.info/?t=146792101400001&r=1&w=2\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n Documentation/config.txt | 3 +++\n builtin/pack-objects.c | 31 ++++++++++++++++++++++++-------\n t/t5310-pack-bitmaps.sh | 12 ++++++++++++\n 3 files changed, 39 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex bc1c433..4ba0c4a 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2244,6 +2244,9 @@ pack.useBitmaps::\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n++\n+*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n+\tpack bitmaps.\n \n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 92e2e5f..0a89e8d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2226,7 +2227,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2475,13 +2476,13 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n }\n \n /*\n- * This tracks any options which a reader of the pack might\n- * not understand, and which would therefore prevent blind reuse\n- * of what we have on disk.\n+ * This tracks any options which pack-reuse code expects to be on, or which a\n+ * reader of the pack might not understand, and which would therefore prevent\n+ * blind reuse of what we have on disk.\n */\n static int pack_options_allow_reuse(void)\n {\n-\treturn allow_ofs_delta;\n+\treturn pack_to_stdout && allow_ofs_delta;\n }\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n@@ -2774,7 +2775,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..ffecc6a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -118,6 +118,18 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\tgit show-index packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293358","messageId":"xmqqk2frgs4s.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160808135020.yggthtfekdiwq3lz@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-08T16:08:51Z","receivedAt":"2016-08-08T16:09:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Mon, Aug 08, 2016 at 03:37:35PM +0300, Kirill Smelkov wrote:\n> ...\n> static int want_object_in_pack(...)\n> {\n> \t...\n> \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n> \t\treturn 0;\n>\n> \tif (*found_pack) {\n> \t\tint ret = want_found_object(exclude, *found_pack);\n> \t\tif (ret != -1)\n> \t\t\treturn ret;\n> \t}\n>\n> \tfor (p = packed_git; p; p = p->next) {\n> \t\toff_t offset;\n>\n> \t\tif (p == *found_pack)\n> \t\t\toffset = *found_offset;\n> \t\telse\n> \t\t\toffset = find_pack_entry(sha1, p);\n> \t\tif (offset) {\n> \t\t\t... fill in *found_pack ...\n> \t\t\tint ret = want_found_object(exclude, p);\n> \t\t\tif (ret != -1)\n> \t\t\t\treturn ret;\n> \t\t}\n> \t}\n> \treturn 1;\n> }\n>\n> That's a little more verbose, but IMHO the flow is a lot easier to\n> follow (especially as the later re-rolls of that series actually muck\n> with the loop order more, but with this approach there's no conflict).\n\nI agree; Kirill's version was so confusing that I couldn't see what\nit was trying to do with \"pack1_seen\" flag that is reset every time\nloop repeats (at least, before got my coffee ;-). A helper function\nlike the above makes the logic a lot easier to grasp.\n\n>> static int add_object_entry(const unsigned char *sha1, enum object_type type,\n>> \t\t\t const char *name, int exclude)\n>> {\n>> -\tstruct packed_git *found_pack;\n>> -\toff_t found_offset;\n>> +\tstruct packed_git *found_pack = NULL;\n>> +\toff_t found_offset = 0;\n>> \tuint32_t index_pos;\n>> \n>> \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n>> @@ -1073,6 +1088,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n>> \tif (have_duplicate_entry(sha1, 0, &index_pos))\n>> \t\treturn 0;\n>> \n>> +\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n>> +\t\treturn 0;\n>> +\n>\n> This part looks correct and easy to understand.\n\nYes.\n"},{"id":"293359","messageId":"xmqqfuqfgrzq.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160808123734.GA3995@teco.navytux.spb.ru","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-08T16:11:53Z","receivedAt":"2016-08-08T16:12:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n>> > ...\n>> > Suggested-by: Junio C Hamano \n>> > Discussed-with: Jeff King \n>> > ---\n>> \n>> I do not think I suggested much of this to deserve credit like this,\n>> though, as I certainly haven't thought about the pros-and-cons\n>> between adding the same \"some object in pack may not want to be in\n>> the output\" logic to the bitmap side, or punting the bitmap codepath\n>> when local/keep are involved.\n>\n> I understand. Still for me it was you who convinced me to add proper\n> support for e.g. --local vs bitmap instead of special-casing it.\n\nOK, in such a case, it probably is more sensible to do it like:\n\n ...\n with all differences strangely showing we are a bit faster now, but\n probably all being within noise.\n\n Credit for inspiring this solution and discussing the design of\n the change goes to Junio and Jeff King.\n\n Signed-off-by: Kirill Smelkov \n ---\n builtin/pack-objects.c | 36 ++++++++++++-----\n t/t5310-pack-bitmaps.sh | 103 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 130 insertions(+), 9 deletions(-)\n\nDon't forget your own sign-off ;-)\n"},{"id":"293373","messageId":"xmqq60rbf80t.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160808154054.GB3995@teco.navytux.spb.ru","subject":"Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-08T18:08:34Z","receivedAt":"2016-08-08T18:08:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> Thanks for the info. I did not knew about show-index when I was starting\n> to work on this and later it just came out of sight. Please find\n> corrected patch below.\n>\n> ---- 8< ----\n> From: Kirill Smelkov \n> Date: Fri, 29 Jul 2016 10:47:46 +0300\n> Subject: [PATCH v5] pack-objects: Teach it to use reachability bitmap index when\n> generating non-stdout pack too\n\nPlease don't do this (not the patch text itself, but saying \"Please\nfind ...\" and attaching the patch AFTER 60+ lines of response).\nWhen going through old/read messages to see if there are patches\nthat fell through the cracks, if it is not immediately clear in the\ntop part of the message that it contains an updated patch, such a\npatch will certainly be missed.\n\nPlease say \"I'll follow up with a corrected patch\" instead of\n\"Please find ...\" and respond to that message with just the patch.\n"},{"id":"293374","messageId":"20160808181353.GC3995@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqq60rbf80t.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-08T18:13:53Z","receivedAt":"2016-08-08T18:14:01Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Mon, Aug 08, 2016 at 11:08:34AM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> > Thanks for the info. I did not knew about show-index when I was starting\n> > to work on this and later it just came out of sight. Please find\n> > corrected patch below.\n> >\n> > ---- 8< ----\n> > From: Kirill Smelkov \n> > Date: Fri, 29 Jul 2016 10:47:46 +0300\n> > Subject: [PATCH v5] pack-objects: Teach it to use reachability bitmap index when\n> > generating non-stdout pack too\n> \n> Please don't do this (not the patch text itself, but saying \"Please\n> find ...\" and attaching the patch AFTER 60+ lines of response).\n> When going through old/read messages to see if there are patches\n> that fell through the cracks, if it is not immediately clear in the\n> top part of the message that it contains an updated patch, such a\n> patch will certainly be missed.\n> \n> Please say \"I'll follow up with a corrected patch\" instead of\n> \"Please find ...\" and respond to that message with just the patch.\n\nOk, I see. Should I resend this v5 as separated one or only starting\nfrom next time?\n\nAnother question: I'm preparing another version of \"pack-objects: Teach\n--use-bitmap-index codepath to respect --local ...\" and was going to\nput\n\n ( updated patch is in the end of this mail )\n\nin the top of the message. Is it ok or better not to do so and just respin\nthe patch in its own separate mail?\n\nThanks beforehand for clarifying,\nKirill\n\nP.S. I put updated patches in the same mail not because I'm trying to\n make maintainer's life harder, but because this is the way I would\n expect and prefer them to be coming to me...\n"},{"id":"293376","messageId":"20160808181942.GD3995@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160808135020.yggthtfekdiwq3lz@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-08T18:19:42Z","receivedAt":"2016-08-08T18:19:51Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"\n( updated patch is in the end of this mail )\n\nJeff, first of all thanks for commenting,\n\nOn Mon, Aug 08, 2016 at 09:50:20AM -0400, Jeff King wrote:\n> On Mon, Aug 08, 2016 at 03:37:35PM +0300, Kirill Smelkov wrote:\n> \n> > @@ -958,15 +958,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n> > \t\t\t off_t *found_offset)\n> > {\n> > \tstruct packed_git *p;\n> > +\tstruct packed_git *pack1 = *found_pack;\n> > +\tint pack1_seen = !pack1;\n> > \n> > \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n> > \t\treturn 0;\n> > \n> > -\t*found_pack = NULL;\n> > -\t*found_offset = 0;\n> > +\t/*\n> > +\t * If we already know the pack object lives in, start checks from that\n> > +\t * pack - in the usual case when neither --local was given nor .keep files\n> > +\t * are present the loop will degenerate to have only 1 iteration.\n> > +\t */\n> > +\tfor (p = (pack1 ? pack1 : packed_git); p;\n> > +\t p = (pack1_seen ? p->next : packed_git), pack1_seen = 1) {\n> > +\t\toff_t offset;\n> \n> Hmm. So this is basically sticking the found-pack at the front of the\n> loop.\n> \n> We either need to look at zero packs here (we already know where the\n> object is, and we don't need to bother with --local or .keep lookups),\n> or we need to look at all of them (to check for local/keep).\n> \n> I guess you structured it this way to try to reuse the \"can we break out\n> early\" logic from the middle of the loop. So we go through the loop one\n> time, and then break out. And then this:\n> \n> > +\t\tif (p == pack1) {\n> > +\t\t\tif (pack1_seen)\n> > +\t\t\t\tcontinue;\n> > +\t\t\toffset = *found_offset;\n> > +\t\t}\n> > +\t\telse {\n> > +\t\t\toffset = find_pack_entry_one(sha1, p);\n> > +\t\t}\n> \n> is meant to make that one-time through the loop cheaper. So I don't\n> think it's wrong, but it's very confusing to me.\n> \n> Would it be simpler to stick that logic in a function like:\n> \n> static int want_found_object(int exclude, struct packed_git *pack)\n> {\n> \tif (exclude)\n> \t\treturn 1;\n> \tif (incremental)\n> \t\treturn 0;\n> \n> \t/* if we can break early, then do so */\n> \tif (!ignore_packed_keep &&\n> \t (!local || !have_non_local_packs))\n> \t\treturn 1;\n> \n> \tif (local && !p->pack_local)\n> \t\treturn 0;\n> \tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n> \t\treturn 0;\n> \n> \t/* indeterminate; keep looking for more packs */\n> \treturn -1;\n> }\n> \n> static int want_object_in_pack(...)\n> {\n> \t...\n> \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n> \t\treturn 0;\n> \n> \tif (*found_pack) {\n> \t\tint ret = want_found_object(exclude, *found_pack);\n> \t\tif (ret != -1)\n> \t\t\treturn ret;\n> \t}\n> \n> \tfor (p = packed_git; p; p = p->next) {\n> \t\toff_t offset;\n> \n> \t\tif (p == *found_pack)\n> \t\t\toffset = *found_offset;\n> \t\telse\n> \t\t\toffset = find_pack_entry(sha1, p);\n> \t\tif (offset) {\n> \t\t\t... fill in *found_pack ...\n> \t\t\tint ret = want_found_object(exclude, p);\n> \t\t\tif (ret != -1)\n> \t\t\t\treturn ret;\n> \t\t}\n> \t}\n> \treturn 1;\n> }\n> \n> That's a little more verbose, but IMHO the flow is a lot easier to\n> follow (especially as the later re-rolls of that series actually muck\n> with the loop order more, but with this approach there's no conflict).\n\nOn Mon, Aug 08, 2016 at 09:08:51AM -0700, Junio C Hamano wrote:\n> I agree; Kirill's version was so confusing that I couldn't see what\n> it was trying to do with \"pack1_seen\" flag that is reset every time\n> loop repeats (at least, before got my coffee ;-). A helper function\n> like the above makes the logic a lot easier to grasp.\n\nOk, at least I put today's record for the most confusing code. I agree\nwith your comments - it is better to simplify control-flow logic. Somehow\nmy head was refusing doing that and insisted on keeping the loop inside\nintact. Maybe I should have a bit of rest... Scratch all that in favour\nof want_found_object() and thanks for heads-up.\n\n\n\n> > static int add_object_entry(const unsigned char *sha1, enum object_type type,\n> > \t\t\t const char *name, int exclude)\n> > {\n> > -\tstruct packed_git *found_pack;\n> > -\toff_t found_offset;\n> > +\tstruct packed_git *found_pack = NULL;\n> > +\toff_t found_offset = 0;\n> > \tuint32_t index_pos;\n> > \n> > \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n> > @@ -1073,6 +1088,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> > \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> > \t\treturn 0;\n> > \n> > +\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n> > +\t\treturn 0;\n> > +\n> \n> This part looks correct and easy to understand.\n\nthanks.\n\n\n> > diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> > index 3893afd..1a61de4 100755\n> > --- a/t/t5310-pack-bitmaps.sh\n> > +++ b/t/t5310-pack-bitmaps.sh\n> > @@ -16,6 +16,7 @@ test_expect_success 'setup repo with moderate-sized history' '\n> > \t\ttest_commit side-$i\n> > \tdone &&\n> > \tgit checkout master &&\n> > +\tbitmaptip=$(git show-ref -s master) &&\n> \n> Our usual method for getting a sha1 is \"git rev-parse\". I don't think\n> there's anything wrong with your method, but it might be better to stick\n> to the canonical one (I had to actually look up \"show-ref -s\").\n\nok.\n\n\n> > @@ -118,6 +119,90 @@ test_expect_success 'incremental repack can disable bitmaps' '\n> > \tgit repack -d --no-write-bitmap-index\n> > '\n> > \n> > +test_expect_success 'pack-objects respects --local (non-local loose)' '\n> > +\tmkdir -p alt_objects/pack &&\n> > +\techo $(pwd)/alt_objects > .git/objects/info/alternates &&\n> > +\techo content1 > file1 &&\n> \n> Style: we don't put a space between \">\" and the filename.\n\nok, corrected.\n\n\n> > +\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n> > +\tgit cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n> \n> I'm not sure why we need two objects in the fake alt_objects repository.\n> Shouldn't one be enough to do the test?\n\nThose two objects are different: one is not present in main bitmapped\npack and another is present in main bitmapped pack. So the second one\ntests for case Junio caught - when bitmapped pack overlaps with\nnon-local loose object and with --local we want to avoid that object in\nresultant pack. I've adjusted the patch as\n\n+ # non-local loose object which is not present in bitmapped pack\n objsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n+ # non-local loose object which is also present in bitmapped pack\n git cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n\n> > +\tgit add file1 &&\n> \n> I think this will actually skip the writing of the loose object, because\n> it's already available in the alternate object store. You probably want\n> to do this before adding it there.\n\nIt does not want to add the object to local objects - it just wants to\nmake a commit with reference to that object, so that $objsha1 we added\nabove becomes referenced from HEAD and thus should be put in pack\nwithout --local (and with --local we test it is not put there).\n\n> > +\ttest_tick &&\n> > +\tgit commit -m commit_file1 &&\n> > +\techo HEAD | \\\n> \n> No need for \"\\\" after a \"|\"; the shell knows it has to keep looking.\n\nOk, thanks for the info. I've actually now folded those two lines into\none as it is not long.\n\n> \n> > +\tgit pack-objects --local --stdout --revs >1.pack &&\n> > +\tgit index-pack 1.pack &&\n> \n> I'd have expected you to use the non-stdout version here. Is this meant\n> to be independent of your other patch (I think that's OK).\n\nOn Mon, Aug 08, 2016 at 09:50:20AM -0400, Jeff King wrote:\n> Oh, nevermind, I forgot this was meant to be a preparatory patch. So it\n> makes sense to use --stdout in the tests.\n\nActually now these two patches:\n\n - to teach bitmapped pack-objects about --local & friends, and\n - to teach `pack-objects file` to use bitmaps\n\nare completely separated and orthogonal.\n\nI mean they work independently and can be reviewed / applied\nindependently, each solving its own task. Initially I was keeping them\ntogether because in the first version of `pack-objects file` the default was\nto always use bitmap index, and since repack was using it and there were\ntests for repack v.s. non-local objects those tests were failing.\n\nNow, since we figured we should have use_bitmap_index=0 by default when\npacking to file, the `bitmap + --local` part is not needed for the first\npatch. ( it is still good to have the `bitmap + --local` applied because\nit restores correctness and consistency and allows future paths for\nbrave soles to do repacking with bitmap index being on maybe )\n\nFor the current patch I think using --stdout in tests is ok as we know\n--stdout uses bitmap indices by default.\n\n> > +\tgit verify-pack -v 1.pack >1.objects &&\n> \n> It's cheaper to use \"git show-index <1.pack\", and the output is saner,\n> too.\n\nI've copied those verify-packs from t7700-repack.sh, but it ok to switch\nto show-index. Thanks for pointing this out.\n\n> > +\techo -e \"$objsha1\\n$blob\" >nonlocal-loose &&\n> \n> \"echo -e\" isn't portable. You can use \"printf\", or two echos like:\n> \n> {\n> echo one &&\n> echo two\n> } >file\n\nok, switching to printf. Thanks for portability hint.\n\n> (though I'm still not sure what we gain by checking both).\n\nPlease see above about those two objects serves for testing two\ndifferent scenarios: 1) non-local object which is not in bitmapped pack,\nand 2) non-local object which is also present in bitmapped pack.\n\n\n> > +\tif grep -qFf nonlocal-loose 1.objects; then\n> > +\t\techo \"Non-local object present in pack generated with --local\"\n> > +\t\treturn 1\n> > +\tfi\n> > +'\n> \n> grep -f isn't portable. However, I think:\n> \n> echo $objsha1 >expect &&\n> git show-index <1.pack | cut -d' ' -f2 >actual\n> test_cmp expect actual\n> \n> would work (if you do stick with two entries, you might need to sort\n> your \"expect\").\n\nThanks for pointing out grep -f is not portable (I did not knew nor\ncared about portability). However here and in similar places we are\nchecking that entries in nonlocal-loose are not present in 1.objects\nand that is not what test_cmp does as it would test nonlocal-loose and\n1.objects to be completely same or not.\n\nFor this reason I'm changing `grep -f ...` to `git grep --no-index -f ...`\nwhich we carry with us.\n\n> \n> I think similar comments apply to the other tests.\n\nI went through all tests in the patch and made similar adjustments\neverywhere.\n\n> I would have expected\n> \"respects --local (non-local pack)\" to come next (i.e., to keep all of\n> the --local tests together). But you seem to interleave them with\n> --honor-pack-keep.\n\nThere is a reason: \"respects --local (non-local pack)\" needs non-local\npack setup in alt_objects/ for its tests to run. And we setup such pack\nas byproduct of running \"pack-objects respects --honor-pack-keep (local\nnon-bitmapped pack)\". Similarly \"pack-objects respects --local\n(non-local bitmapped pack)\" for its testing needs (and moves to\nalt_objects/) main bitmapped pack, which was just analyzed in\n\"pack-objects respects --honor-pack-keep (local bitmapped pack)\"\n\nInitially I tried to cluster tests (i.e. all --local together and then\nall --honor-pack-keep together) but having tests interleaved turned out\nto be handy because one step checks something and prepares setup for its\nnext one.\n\nOn Mon, Aug 08, 2016 at 09:11:53AM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> >> > ...\n> >> > Suggested-by: Junio C Hamano \n> >> > Discussed-with: Jeff King \n> >> > ---\n> >> \n> >> I do not think I suggested much of this to deserve credit like this,\n> >> though, as I certainly haven't thought about the pros-and-cons\n> >> between adding the same \"some object in pack may not want to be in\n> >> the output\" logic to the bitmap side, or punting the bitmap codepath\n> >> when local/keep are involved.\n> >\n> > I understand. Still for me it was you who convinced me to add proper\n> > support for e.g. --local vs bitmap instead of special-casing it.\n> \n> OK, in such a case, it probably is more sensible to do it like:\n> \n> ...\n> with all differences strangely showing we are a bit faster now, but\n> probably all being within noise.\n> \n> Credit for inspiring this solution and discussing the design of\n> the change goes to Junio and Jeff King.\n\nOk, thanks for advice.\n\n> Don't forget your own sign-off ;-)\n\nOops, thanks for catching :) Obviously I forgot it and now corrected.\n\nThanks,\nKirill\n\n---- 8< ----\nFrom: Kirill Smelkov \nSubject: [PATCH v3] pack-objects: Teach --use-bitmap-index codepath to respect\n --local, --honor-pack-keep and --incremental\n\nSince 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep or --incremental should be respected. In\nnon-bitmapped codepath this is handled in want_object_in_pack(), but\nbitmapped codepath has simply no such checking at all.\n\nThe bitmapped codepath however was allowing to pass in all those options\nand with bitmap indices still being used under such conditions -\npotentially giving wrong output (e.g. including objects from non-local or\n.keep'ed pack).\n\nWe can easily fix this by noting the following: when an object comes to\nadd_object_entry_from_bitmap() it can come for two reasons:\n\n 1. entries coming from main pack covered by bitmap index, and\n 2. object coming from, possibly alternate, loose or other packs.\n\n\"2\" can be already handled by want_object_in_pack() and to cover\n\"1\" we can teach want_object_in_pack() to expect that *found_pack can be\nnon-NULL, meaning calling client already found object's pack entry.\n\nIn want_object_in_pack() we care to start the checks from already found\npack, if we have one, this way determining the answer right away\nin case neither --local nor --honour-pack-keep are active. In\nparticular, as p5310-pack-bitmaps.sh shows, we do not do harm to\nserved-with-bitmap clones performance-wise:\n\n Test 56dfeb62 this tree\n -----------------------------------------------------------------\n 5310.2: repack to disk 9.63(8.67+0.33) 9.47(8.55+0.28) -1.7%\n 5310.3: simulated clone 2.07(2.17+0.12) 2.03(2.14+0.12) -1.9%\n 5310.4: simulated fetch 0.78(1.03+0.02) 0.76(1.00+0.03) -2.6%\n 5310.6: partial bitmap 1.97(2.43+0.15) 1.92(2.36+0.14) -2.5%\n\nwith all differences strangely showing we are a bit faster now, but\nprobably all being within noise.\n\nAnd in the general case we care not to have duplicate\nfind_pack_entry_one(*found_pack) calls. Worst what can happen is we can\ncall want_found_object(*found_pack) -- newly introduced helper for\nchecking whether we want object -- twice, but since want_found_object()\nis very lightweight it does not make any difference.\n\nI appreciate help and discussing this change with Junio C Hamano and\nJeff King.\n\nSigned-off-by: Kirill Smelkov \n---\n builtin/pack-objects.c | 94 ++++++++++++++++++++++++++--------------\n t/t5310-pack-bitmaps.sh | 111 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 172 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c4c2a3c..e06c1bf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -944,13 +944,45 @@ static int have_duplicate_entry(const unsigned char *sha1,\n \treturn 1;\n }\n \n+static int want_found_object(int exclude, struct packed_git *p)\n+{\n+\tif (exclude)\n+\t\treturn 1;\n+\tif (incremental)\n+\t\treturn 0;\n+\n+\t/*\n+\t * When asked to do --local (do not include an\n+\t * object that appears in a pack we borrow\n+\t * from elsewhere) or --honor-pack-keep (do not\n+\t * include an object that appears in a pack marked\n+\t * with .keep), we need to make sure no copy of this\n+\t * object come from in _any_ pack that causes us to\n+\t * omit it, and need to complete this loop. When\n+\t * neither option is in effect, we know the object\n+\t * we just found is going to be packed, so break\n+\t * out of the search loop now.\n+\t */\n+\tif (!ignore_packed_keep &&\n+\t (!local || !have_non_local_packs))\n+\t\treturn 1;\n+\n+\tif (local && !p->pack_local)\n+\t\treturn 0;\n+\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\t\treturn 0;\n+\n+\t/* we don't know yet; keep looking for more packs */\n+\treturn -1;\n+}\n+\n /*\n * Check whether we want the object in the pack (e.g., we do not want\n * objects found in non-local stores if the \"--local\" option was used).\n *\n- * As a side effect of this check, we will find the packed version of this\n- * object, if any. We therefore pass out the pack information to avoid having\n- * to look it up again later.\n+ * As a side effect of this check, if object's pack entry was not already found,\n+ * we will find the packed version of this object, if any. We therefore pass\n+ * out the pack information to avoid having to look it up again later.\n */\n static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t int exclude,\n@@ -958,15 +990,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t off_t *found_offset)\n {\n \tstruct packed_git *p;\n+\tint want;\n \n \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n \t\treturn 0;\n \n-\t*found_pack = NULL;\n-\t*found_offset = 0;\n+\t/*\n+\t * If we already know the pack object lives in, start checks from that\n+\t * pack - in the usual case when neither --local was given nor .keep files\n+\t * are present we will determine the answer right now.\n+\t */\n+\tif (*found_pack) {\n+\t\twant = want_found_object(exclude, *found_pack);\n+\t\tif (want != -1)\n+\t\t\treturn want;\n+\t}\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\toff_t offset = find_pack_entry_one(sha1, p);\n+\t\toff_t offset;\n+\n+\t\tif (p == *found_pack)\n+\t\t\toffset = *found_offset;\n+\t\telse\n+\t\t\toffset = find_pack_entry_one(sha1, p);\n+\n \t\tif (offset) {\n \t\t\tif (!*found_pack) {\n \t\t\t\tif (!is_pack_valid(p))\n@@ -974,31 +1021,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t\t*found_offset = offset;\n \t\t\t\t*found_pack = p;\n \t\t\t}\n-\t\t\tif (exclude)\n-\t\t\t\treturn 1;\n-\t\t\tif (incremental)\n-\t\t\t\treturn 0;\n-\n-\t\t\t/*\n-\t\t\t * When asked to do --local (do not include an\n-\t\t\t * object that appears in a pack we borrow\n-\t\t\t * from elsewhere) or --honor-pack-keep (do not\n-\t\t\t * include an object that appears in a pack marked\n-\t\t\t * with .keep), we need to make sure no copy of this\n-\t\t\t * object come from in _any_ pack that causes us to\n-\t\t\t * omit it, and need to complete this loop. When\n-\t\t\t * neither option is in effect, we know the object\n-\t\t\t * we just found is going to be packed, so break\n-\t\t\t * out of the loop to return 1 now.\n-\t\t\t */\n-\t\t\tif (!ignore_packed_keep &&\n-\t\t\t (!local || !have_non_local_packs))\n-\t\t\t\tbreak;\n-\n-\t\t\tif (local && !p->pack_local)\n-\t\t\t\treturn 0;\n-\t\t\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n-\t\t\t\treturn 0;\n+\t\t\twant = want_found_object(exclude, p);\n+\t\t\tif (want != -1)\n+\t\t\t\treturn want;\n \t\t}\n \t}\n \n@@ -1039,8 +1064,8 @@ static const char no_closure_warning[] = N_(\n static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \t\t\t const char *name, int exclude)\n {\n-\tstruct packed_git *found_pack;\n-\toff_t found_offset;\n+\tstruct packed_git *found_pack = NULL;\n+\toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n@@ -1073,6 +1098,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n+\t\treturn 0;\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..e71caa4 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -7,6 +7,19 @@ objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n \n+# show objects present in pack ($1 should be associated *.idx)\n+packobjects () {\n+\tgit show-index <$1 | cut -d' ' -f2\n+}\n+\n+# hasany pattern-file content-file\n+# tests whether content-file has any entry from pattern-file with entries being\n+# whole lines.\n+hasany () {\n+\t# NOTE `grep -f` is not portable\n+\tgit grep --no-index -qFf $1 $2\n+}\n+\n test_expect_success 'setup repo with moderate-sized history' '\n \tfor i in $(test_seq 1 10); do\n \t\ttest_commit $i\n@@ -16,6 +29,7 @@ test_expect_success 'setup repo with moderate-sized history' '\n \t\ttest_commit side-$i\n \tdone &&\n \tgit checkout master &&\n+\tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n \tgit config repack.writebitmaps true &&\n@@ -118,6 +132,86 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects respects --local (non-local loose)' '\n+\tmkdir -p alt_objects/pack &&\n+\techo $(pwd)/alt_objects >.git/objects/info/alternates &&\n+\techo content1 >file1 &&\n+\t# non-local loose object which is not present in bitmapped pack\n+\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n+\t# non-local loose object which is also present in bitmapped pack\n+\tgit cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n+\tgit add file1 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file1 &&\n+\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\tgit index-pack 1.pack &&\n+\tpackobjects 1.idx >1.objects &&\n+\tprintf \"$objsha1\\n$blob\\n\" >nonlocal-loose &&\n+\tif hasany nonlocal-loose 1.objects; then\n+\t\techo \"Non-local object present in pack generated with --local\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\techo content2 >file2 &&\n+\tobjsha2=$(git hash-object -w file2) &&\n+\tgit add file2 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file2 &&\n+\tprintf \"$objsha2\\n$bitmaptip\\n\" >keepobjects &&\n+\tpack2=$(git pack-objects pack2 2a.pack &&\n+\tgit index-pack 2a.pack &&\n+\tpackobjects 2a.idx >2a.objects &&\n+\tif hasany keepobjects 2a.objects; then\n+\t\techo \"Object from .keeped pack present in pack generated with --honor-pack-keep\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local pack)' '\n+\tmv .git/objects/pack/pack2-$pack2.* alt_objects/pack/ &&\n+\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\tgit index-pack 2b.pack &&\n+\tpackobjects 2b.idx >2b.objects &&\n+\tif hasany keepobjects 2b.objects; then\n+\t\techo \"Non-local object present in pack generated with --local\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\tpackobjects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\ttouch .git/objects/pack/$packbitmap.keep &&\n+\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\tgit index-pack 3a.pack &&\n+\tpackobjects 3a.idx >3a.objects &&\n+\tif hasany packbitmap.objects 3a.objects; then\n+\t\techo \"Object from .keeped bitmapped pack present in pack generated with --honour-pack-keep\"\n+\t\treturn 1\n+\tfi &&\n+\trm .git/objects/pack/$packbitmap.keep\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\tmv .git/objects/pack/$packbitmap.* alt_objects/pack/ &&\n+\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\tgit index-pack 3b.pack &&\n+\tpackobjects 3b.idx >3b.objects &&\n+\tif hasany packbitmap.objects 3b.objects; then\n+\t\techo \"Non-local object from bitmapped pack present in pack generated with --local\"\n+\t\treturn 1\n+\tfi &&\n+\tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n@@ -143,6 +237,23 @@ test_expect_success 'create objects for missing-HAVE tests' '\n \tEOF\n '\n \n+test_expect_success 'pack-objects respects --incremental' '\n+\tcat >revs2 <<-EOF &&\n+\tHEAD\n+\t$commit\n+\tEOF\n+\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\tgit index-pack 4.pack &&\n+\tpackobjects 4.idx >4.objects &&\n+\ttest_line_count = 4 4.objects &&\n+\tgit rev-list --objects $commit >revlist &&\n+\tcut -d\" \" -f1 revlist |sort >objects &&\n+\tif !hasany objects 4.objects; then\n+\t\techo \"Expected objects not present in incremental pack\"\n+\t\treturn 1\n+\tfi\n+'\n+\n test_expect_success 'pack with missing blob' '\n \trm $(objpath $blob) &&\n \tgit pack-objects --stdout --revs /dev/null\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293378","messageId":"xmqq1t1zf74d.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160808181353.GC3995@teco.navytux.spb.ru","subject":"Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-08T18:28:02Z","receivedAt":"2016-08-08T18:28:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> Another question: I'm preparing another version of \"pack-objects: Teach\n> --use-bitmap-index codepath to respect --local ...\" and was going to\n> put\n>\n> ( updated patch is in the end of this mail )\n>\n> in the top of the message. Is it ok or better not to do so and just respin\n> the patch in its own separate mail?\n\nThat would force those who pick leftover bits to _open_ and read a\nfirst few lines.\n\nDefinitely it is better than burying a patch after 60+ lines, but a\nseparate patch with incremented \"[PATCH v6 1/2]\" on the subject line\nbeats it hands-down from discoverability's point of view.\n"},{"id":"293381","messageId":"20160808185541.6433-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160808154054.GB3995@teco.navytux.spb.ru","subject":"[PATCH v5] pack-objects: teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-08T18:55:41Z","receivedAt":"2016-08-08T18:55:58Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nThe reason here is for on-disk repack by default we want:\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff King further explains:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\nSo we can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we care it is not activated under git-repack:\n\n See above about repack robustness and not forward-carrying corruption.\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n We also want to avoid (at least with current implementation)\n generating bitmaps off of bitmaps. The reason here is: when generating\n a pack, not-yet-packed objects will be emitted into pack in\n suboptimal order and added to tail of the bitmap as \"extended entries\".\n When the resultant pack + some new objects in associated repository\n are in turn used to generate another pack with bitmap, the situation\n repeats: new objects are again not emitted optimally and just added to\n bitmap tail - not in recency order.\n\n So the pack badness can grow over time when at each step we have\n bitmapped pack + some other objects. That's why we want to avoid\n generating bitmaps off of bitmaps, not to let pack badness grow.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\nThis way for pack-objects -> file we get nice speedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly. And git-repack does never pass\n--use-bitmap-index, so this way we can be sure regular on-disk repacking\nremains robust.\n\nNOTE2\n\n`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\n real 0m22.309s\n user 0m21.148s\n sys 0m0.932s\n\n $ time git index-pack erp5pack-stdout.pack\n\n real 0m50.873s <-- more than 2 times slower than time to generate pack itself!\n user 0m49.300s\n sys 0m1.360s\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n\nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n\nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\nJeff explains:\n\n The packfile does not carry the sha1 of the objects. A receiving\n index-pack has to compute them itself, including inflating and applying\n all of the deltas.\n\nthat's why for `git-backup restore` we want to teach `git pack-objects\nfile.pack` to use bitmaps instead of using `git pack-objects --stdout\n>file.pack` + `git index-pack file.pack`.\n\nMore context:\n\n http://marc.info/?t=146792101400001&r=1&w=2\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n Documentation/config.txt | 3 +++\n builtin/pack-objects.c | 31 ++++++++++++++++++++++++-------\n t/t5310-pack-bitmaps.sh | 12 ++++++++++++\n 3 files changed, 39 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex bc1c433..4ba0c4a 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2244,6 +2244,9 @@ pack.useBitmaps::\n \tto stdout (e.g., during the server side of a fetch). Defaults to\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n++\n+*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n+\tpack bitmaps.\n \n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 92e2e5f..0a89e8d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2226,7 +2227,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2475,13 +2476,13 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n }\n \n /*\n- * This tracks any options which a reader of the pack might\n- * not understand, and which would therefore prevent blind reuse\n- * of what we have on disk.\n+ * This tracks any options which pack-reuse code expects to be on, or which a\n+ * reader of the pack might not understand, and which would therefore prevent\n+ * blind reuse of what we have on disk.\n */\n static int pack_options_allow_reuse(void)\n {\n-\treturn allow_ofs_delta;\n+\treturn pack_to_stdout && allow_ofs_delta;\n }\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n@@ -2774,7 +2775,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..ffecc6a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -118,6 +118,18 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\tgit show-index packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293382","messageId":"20160808185726.6766-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160808181942.GD3995@teco.navytux.spb.ru","subject":"[PATCH v3] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-08T18:57:26Z","receivedAt":"2016-08-08T18:57:41Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep or --incremental should be respected. In\nnon-bitmapped codepath this is handled in want_object_in_pack(), but\nbitmapped codepath has simply no such checking at all.\n\nThe bitmapped codepath however was allowing to pass in all those options\nand with bitmap indices still being used under such conditions -\npotentially giving wrong output (e.g. including objects from non-local or\n.keep'ed pack).\n\nWe can easily fix this by noting the following: when an object comes to\nadd_object_entry_from_bitmap() it can come for two reasons:\n\n 1. entries coming from main pack covered by bitmap index, and\n 2. object coming from, possibly alternate, loose or other packs.\n\n\"2\" can be already handled by want_object_in_pack() and to cover\n\"1\" we can teach want_object_in_pack() to expect that *found_pack can be\nnon-NULL, meaning calling client already found object's pack entry.\n\nIn want_object_in_pack() we care to start the checks from already found\npack, if we have one, this way determining the answer right away\nin case neither --local nor --honour-pack-keep are active. In\nparticular, as p5310-pack-bitmaps.sh shows, we do not do harm to\nserved-with-bitmap clones performance-wise:\n\n Test 56dfeb62 this tree\n -----------------------------------------------------------------\n 5310.2: repack to disk 9.63(8.67+0.33) 9.47(8.55+0.28) -1.7%\n 5310.3: simulated clone 2.07(2.17+0.12) 2.03(2.14+0.12) -1.9%\n 5310.4: simulated fetch 0.78(1.03+0.02) 0.76(1.00+0.03) -2.6%\n 5310.6: partial bitmap 1.97(2.43+0.15) 1.92(2.36+0.14) -2.5%\n\nwith all differences strangely showing we are a bit faster now, but\nprobably all being within noise.\n\nAnd in the general case we care not to have duplicate\nfind_pack_entry_one(*found_pack) calls. Worst what can happen is we can\ncall want_found_object(*found_pack) -- newly introduced helper for\nchecking whether we want object -- twice, but since want_found_object()\nis very lightweight it does not make any difference.\n\nI appreciate help and discussing this change with Junio C Hamano and\nJeff King.\n\nSigned-off-by: Kirill Smelkov \n---\n builtin/pack-objects.c | 94 ++++++++++++++++++++++++++--------------\n t/t5310-pack-bitmaps.sh | 111 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 172 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c4c2a3c..e06c1bf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -944,13 +944,45 @@ static int have_duplicate_entry(const unsigned char *sha1,\n \treturn 1;\n }\n \n+static int want_found_object(int exclude, struct packed_git *p)\n+{\n+\tif (exclude)\n+\t\treturn 1;\n+\tif (incremental)\n+\t\treturn 0;\n+\n+\t/*\n+\t * When asked to do --local (do not include an\n+\t * object that appears in a pack we borrow\n+\t * from elsewhere) or --honor-pack-keep (do not\n+\t * include an object that appears in a pack marked\n+\t * with .keep), we need to make sure no copy of this\n+\t * object come from in _any_ pack that causes us to\n+\t * omit it, and need to complete this loop. When\n+\t * neither option is in effect, we know the object\n+\t * we just found is going to be packed, so break\n+\t * out of the search loop now.\n+\t */\n+\tif (!ignore_packed_keep &&\n+\t (!local || !have_non_local_packs))\n+\t\treturn 1;\n+\n+\tif (local && !p->pack_local)\n+\t\treturn 0;\n+\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\t\treturn 0;\n+\n+\t/* we don't know yet; keep looking for more packs */\n+\treturn -1;\n+}\n+\n /*\n * Check whether we want the object in the pack (e.g., we do not want\n * objects found in non-local stores if the \"--local\" option was used).\n *\n- * As a side effect of this check, we will find the packed version of this\n- * object, if any. We therefore pass out the pack information to avoid having\n- * to look it up again later.\n+ * As a side effect of this check, if object's pack entry was not already found,\n+ * we will find the packed version of this object, if any. We therefore pass\n+ * out the pack information to avoid having to look it up again later.\n */\n static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t int exclude,\n@@ -958,15 +990,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t off_t *found_offset)\n {\n \tstruct packed_git *p;\n+\tint want;\n \n \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n \t\treturn 0;\n \n-\t*found_pack = NULL;\n-\t*found_offset = 0;\n+\t/*\n+\t * If we already know the pack object lives in, start checks from that\n+\t * pack - in the usual case when neither --local was given nor .keep files\n+\t * are present we will determine the answer right now.\n+\t */\n+\tif (*found_pack) {\n+\t\twant = want_found_object(exclude, *found_pack);\n+\t\tif (want != -1)\n+\t\t\treturn want;\n+\t}\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\toff_t offset = find_pack_entry_one(sha1, p);\n+\t\toff_t offset;\n+\n+\t\tif (p == *found_pack)\n+\t\t\toffset = *found_offset;\n+\t\telse\n+\t\t\toffset = find_pack_entry_one(sha1, p);\n+\n \t\tif (offset) {\n \t\t\tif (!*found_pack) {\n \t\t\t\tif (!is_pack_valid(p))\n@@ -974,31 +1021,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t\t*found_offset = offset;\n \t\t\t\t*found_pack = p;\n \t\t\t}\n-\t\t\tif (exclude)\n-\t\t\t\treturn 1;\n-\t\t\tif (incremental)\n-\t\t\t\treturn 0;\n-\n-\t\t\t/*\n-\t\t\t * When asked to do --local (do not include an\n-\t\t\t * object that appears in a pack we borrow\n-\t\t\t * from elsewhere) or --honor-pack-keep (do not\n-\t\t\t * include an object that appears in a pack marked\n-\t\t\t * with .keep), we need to make sure no copy of this\n-\t\t\t * object come from in _any_ pack that causes us to\n-\t\t\t * omit it, and need to complete this loop. When\n-\t\t\t * neither option is in effect, we know the object\n-\t\t\t * we just found is going to be packed, so break\n-\t\t\t * out of the loop to return 1 now.\n-\t\t\t */\n-\t\t\tif (!ignore_packed_keep &&\n-\t\t\t (!local || !have_non_local_packs))\n-\t\t\t\tbreak;\n-\n-\t\t\tif (local && !p->pack_local)\n-\t\t\t\treturn 0;\n-\t\t\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n-\t\t\t\treturn 0;\n+\t\t\twant = want_found_object(exclude, p);\n+\t\t\tif (want != -1)\n+\t\t\t\treturn want;\n \t\t}\n \t}\n \n@@ -1039,8 +1064,8 @@ static const char no_closure_warning[] = N_(\n static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \t\t\t const char *name, int exclude)\n {\n-\tstruct packed_git *found_pack;\n-\toff_t found_offset;\n+\tstruct packed_git *found_pack = NULL;\n+\toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n@@ -1073,6 +1098,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n+\t\treturn 0;\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..e71caa4 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -7,6 +7,19 @@ objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n \n+# show objects present in pack ($1 should be associated *.idx)\n+packobjects () {\n+\tgit show-index <$1 | cut -d' ' -f2\n+}\n+\n+# hasany pattern-file content-file\n+# tests whether content-file has any entry from pattern-file with entries being\n+# whole lines.\n+hasany () {\n+\t# NOTE `grep -f` is not portable\n+\tgit grep --no-index -qFf $1 $2\n+}\n+\n test_expect_success 'setup repo with moderate-sized history' '\n \tfor i in $(test_seq 1 10); do\n \t\ttest_commit $i\n@@ -16,6 +29,7 @@ test_expect_success 'setup repo with moderate-sized history' '\n \t\ttest_commit side-$i\n \tdone &&\n \tgit checkout master &&\n+\tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n \tgit config repack.writebitmaps true &&\n@@ -118,6 +132,86 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects respects --local (non-local loose)' '\n+\tmkdir -p alt_objects/pack &&\n+\techo $(pwd)/alt_objects >.git/objects/info/alternates &&\n+\techo content1 >file1 &&\n+\t# non-local loose object which is not present in bitmapped pack\n+\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n+\t# non-local loose object which is also present in bitmapped pack\n+\tgit cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n+\tgit add file1 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file1 &&\n+\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\tgit index-pack 1.pack &&\n+\tpackobjects 1.idx >1.objects &&\n+\tprintf \"$objsha1\\n$blob\\n\" >nonlocal-loose &&\n+\tif hasany nonlocal-loose 1.objects; then\n+\t\techo \"Non-local object present in pack generated with --local\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\techo content2 >file2 &&\n+\tobjsha2=$(git hash-object -w file2) &&\n+\tgit add file2 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file2 &&\n+\tprintf \"$objsha2\\n$bitmaptip\\n\" >keepobjects &&\n+\tpack2=$(git pack-objects pack2 2a.pack &&\n+\tgit index-pack 2a.pack &&\n+\tpackobjects 2a.idx >2a.objects &&\n+\tif hasany keepobjects 2a.objects; then\n+\t\techo \"Object from .keeped pack present in pack generated with --honor-pack-keep\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local pack)' '\n+\tmv .git/objects/pack/pack2-$pack2.* alt_objects/pack/ &&\n+\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\tgit index-pack 2b.pack &&\n+\tpackobjects 2b.idx >2b.objects &&\n+\tif hasany keepobjects 2b.objects; then\n+\t\techo \"Non-local object present in pack generated with --local\"\n+\t\treturn 1\n+\tfi\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\tpackobjects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\ttouch .git/objects/pack/$packbitmap.keep &&\n+\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\tgit index-pack 3a.pack &&\n+\tpackobjects 3a.idx >3a.objects &&\n+\tif hasany packbitmap.objects 3a.objects; then\n+\t\techo \"Object from .keeped bitmapped pack present in pack generated with --honour-pack-keep\"\n+\t\treturn 1\n+\tfi &&\n+\trm .git/objects/pack/$packbitmap.keep\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\tmv .git/objects/pack/$packbitmap.* alt_objects/pack/ &&\n+\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\tgit index-pack 3b.pack &&\n+\tpackobjects 3b.idx >3b.objects &&\n+\tif hasany packbitmap.objects 3b.objects; then\n+\t\techo \"Non-local object from bitmapped pack present in pack generated with --local\"\n+\t\treturn 1\n+\tfi &&\n+\tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n@@ -143,6 +237,23 @@ test_expect_success 'create objects for missing-HAVE tests' '\n \tEOF\n '\n \n+test_expect_success 'pack-objects respects --incremental' '\n+\tcat >revs2 <<-EOF &&\n+\tHEAD\n+\t$commit\n+\tEOF\n+\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\tgit index-pack 4.pack &&\n+\tpackobjects 4.idx >4.objects &&\n+\ttest_line_count = 4 4.objects &&\n+\tgit rev-list --objects $commit >revlist &&\n+\tcut -d\" \" -f1 revlist |sort >objects &&\n+\tif !hasany objects 4.objects; then\n+\t\techo \"Expected objects not present in incremental pack\"\n+\t\treturn 1\n+\tfi\n+'\n+\n test_expect_success 'pack with missing blob' '\n \trm $(objpath $blob) &&\n \tgit pack-objects --stdout --revs /dev/null\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293383","messageId":"20160808185846.GE3995@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqq1t1zf74d.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 2/2] pack-objects: Teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-08T18:58:47Z","receivedAt":"2016-08-08T18:58:53Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Mon, Aug 08, 2016 at 11:28:02AM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> > Another question: I'm preparing another version of \"pack-objects: Teach\n> > --use-bitmap-index codepath to respect --local ...\" and was going to\n> > put\n> >\n> > ( updated patch is in the end of this mail )\n> >\n> > in the top of the message. Is it ok or better not to do so and just respin\n> > the patch in its own separate mail?\n> \n> That would force those who pick leftover bits to _open_ and read a\n> first few lines.\n> \n> Definitely it is better than burying a patch after 60+ lines, but a\n> separate patch with incremented \"[PATCH v6 1/2]\" on the subject line\n> beats it hands-down from discoverability's point of view.\n\nThanks, I see. I've resent both patches as separate mails.\n"},{"id":"293384","messageId":"xmqqpopjdqsa.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160808135020.yggthtfekdiwq3lz@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-08T19:06:13Z","receivedAt":"2016-08-08T19:06:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> +\tif grep -qFf nonlocal-loose 1.objects; then\n>> +\t\techo \"Non-local object present in pack generated with --local\"\n>> +\t\treturn 1\n>> +\tfi\n>> +'\n>\n> grep -f isn't portable. However, I think:\n>\n> echo $objsha1 >expect &&\n> git show-index <1.pack | cut -d' ' -f2 >actual\n> test_cmp expect actual\n>\n> would work (if you do stick with two entries, you might need to sort\n> your \"expect\").\n\nHmph, are you sure? \"grep -f pattern_file\" is in POSIX.1.\n\n"},{"id":"293385","messageId":"xmqqoa53dpue.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160808181942.GD3995@teco.navytux.spb.ru","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-08T19:26:33Z","receivedAt":"2016-08-08T19:26:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> ---- 8< ----\n> From: Kirill Smelkov \n> Subject: [PATCH v3] pack-objects: Teach --use-bitmap-index codepath to respect\n> --local, --honor-pack-keep and --incremental\n\n(Not a question to Kirill)\n\nHmph. I suspect that handling of in-body header by mailinfo not\nprepared to see RFC2822 header folding. \"am -c\" gives a single line\nsubject with \" --local ...\" as its first line in the body.\n\nI'll leave it as a low-hanging fruit for somebody to fix ;-)\n\n\tSubject: pack-objects: respect --local, etc. when bitmap is in use\n\nmight be shorter and more to the point, anyway.\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index c4c2a3c..e06c1bf 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -944,13 +944,45 @@ static int have_duplicate_entry(const unsigned char *sha1,\n> \treturn 1;\n> }\n> \n> +static int want_found_object(int exclude, struct packed_git *p)\n> +{\n> +\tif (exclude)\n> +\t\treturn 1;\n> +\tif (incremental)\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * When asked to do --local (do not include an\n> +\t * object that appears in a pack we borrow\n> +\t * from elsewhere) or --honor-pack-keep (do not\n> +\t * include an object that appears in a pack marked\n> +\t * with .keep), we need to make sure no copy of this\n> +\t * object come from in _any_ pack that causes us to\n> +\t * omit it, and need to complete this loop. When\n> +\t * neither option is in effect, we know the object\n> +\t * we just found is going to be packed, so break\n> +\t * out of the search loop now.\n> +\t */\n\nThe blame is mine, but \"no copy of this object appears in _any_ pack\"\nwould be more correct and easier to read.\n\nThis code is no longer in a search loop; its caller is. Further\nrephrasing is needed. \"When asked to do ...these things..., finding\na pack that matches the criteria is sufficient for us to decide to\nomit it. However, even if this pack does not satisify the criteria,\nwe need to make sure no copy of this object appears in _any_ pack\nthat makes us to omit the object, so we need to check all the packs.\nSignal that by returning -1 to the caller.\" or something along that\nline.\n\n> /*\n> * Check whether we want the object in the pack (e.g., we do not want\n> * objects found in non-local stores if the \"--local\" option was used).\n> *\n> - * As a side effect of this check, we will find the packed version of this\n> - * object, if any. We therefore pass out the pack information to avoid having\n> - * to look it up again later.\n> + * As a side effect of this check, if object's pack entry was not already found,\n> + * we will find the packed version of this object, if any. We therefore pass\n> + * out the pack information to avoid having to look it up again later.\n\nThe reasoning leading to \"We therefore\" is understandable, but \"pass\nout the pack information\" is not quite. Is this meant to explain\nthe fact that *found_pack and *found_offset are in-out parameters?\n\nThe explanation to justify why *found_pack and *found_offset that\nused to be out parameters are made in-out parameters belongs to the\nlog message. We do not want this in-code comment to explain the\nupdated code relative to what the code used to do; that is not\nuseful to those who read the code for the first time in the context\nof the committed state.\n\n /* \n * Check whether we want to pack the object in the pack (e.g. ...).\n *\n * If the caller already knows an existing pack it wants to\n * take the object from, that is passed in *found_pack and\n * *found_offset; otherwise this function finds if there is\n * any pack that has the object and returns the pack and its\n * offset in these variables.\n */\n\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index 3893afd..e71caa4 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -7,6 +7,19 @@ objpath () {\n> \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n> }\n> \n> +# show objects present in pack ($1 should be associated *.idx)\n> +packobjects () {\n> +\tgit show-index <$1 | cut -d' ' -f2\n> +}\n\nThat is a misleading name for a helper function that produces a list\nof objects that were packed. \"list_packed_objects\", perhaps.\n\n> +# hasany pattern-file content-file\n> +# tests whether content-file has any entry from pattern-file with entries being\n> +# whole lines.\n> +hasany () {\n> +\t# NOTE `grep -f` is not portable\n> +\tgit grep --no-index -qFf $1 $2\n> +}\n\nI doubt \"grep -f pattern_file\" is not portable, but in any case, it\nis probably a good idea to have this helper function to make the\ncaller easier to read. Please name it \"has_any\", though, and quote\n\"$1\" and \"$2\" as they are meant to be able to take any filename.\n\n> +test_expect_success 'pack-objects respects --local (non-local loose)' '\n> +\tmkdir -p alt_objects/pack &&\n\nI'd really really prefer to see an empty repository created for\nthis. Even though the original intent was .git/objects/ alone,\ni.e. GIT_OBJECT_DIRECTORY can exist without associated refs, we\ndiscovered that it is in general not a good idea (think: \"gc\").\n\n> +\techo $(pwd)/alt_objects >.git/objects/info/alternates &&\n> +\techo content1 >file1 &&\n> +\t# non-local loose object which is not present in bitmapped pack\n> +\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n\nDon't say \"sha\" when you mean \"object name\". Otherwise you would\nend up introducing funky variable names like $objsha2 we see below\nthat is confusing (we don't use SHA-2).\n\n> +\t# non-local loose object which is also present in bitmapped pack\n> +\tgit cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n> +\tgit add file1 &&\n> +\ttest_tick &&\n> +\tgit commit -m commit_file1 &&\n> +\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n> +\tgit index-pack 1.pack &&\n> +\tpackobjects 1.idx >1.objects &&\n> +\tprintf \"$objsha1\\n$blob\\n\" >nonlocal-loose &&\n\nI think Peff meant to suggest this instead:\n\n\tprintf \"%s\\n\" \"$objsha1\" \"$blob\"\n\n> +\tif hasany nonlocal-loose 1.objects; then\n> +\t\techo \"Non-local object present in pack generated with --local\"\n> +\t\treturn 1\n> +\tfi\n\nJust saying\n\n\t! has_any nonlocal-loose 1.objects\n\nis sufficient. Same comment for all other uses of these verbose\noutput.\n\nBesides, we spell \"if/then/fi\" like this:\n\n\tif condition\n then\n \tbody\n\tfi\n\nwithout a semicolon.\n\n> +test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n> +...\n> +\ttouch .git/objects/pack/pack2-$pack2.keep &&\n\nPlease don't do \"touch\" _unless_ you care about the timestamp of the\nfile. Redirect an empty command into it, i.e.\n\n\t>.git/objects/pack/pack2-$pack2.keep\n\nor\n\n\techo \"reason to keep it\" >.git/objects/pack/pack2-$pack2.keep\n\ninstead.\n\n> +test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n> +\tls .git/objects/pack/ | grep bitmap >output &&\n> +\ttest_line_count = 1 output &&\n> +\tpackbitmap=$(basename $(cat output) .bitmap) &&\n> +\tpackobjects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n> +\ttouch .git/objects/pack/$packbitmap.keep &&\n> +\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n> +\tgit index-pack 3a.pack &&\n> +\tpackobjects 3a.idx >3a.objects &&\n> +\tif hasany packbitmap.objects 3a.objects; then\n> +\t\techo \"Object from .keeped bitmapped pack present in pack generated with --honour-pack-keep\"\n> +\t\treturn 1\n> +\tfi &&\n> +\trm .git/objects/pack/$packbitmap.keep\n\nArrange this removal to happen even when any earlier step fails, so\nthat later tests will not get affected by stray existence of this\nfile, by using test_when_finished. E.g.\n\n\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n\t>.git/objects/pack/$packbitmap.keep\" &&\n\n> +test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n> +\tmv .git/objects/pack/$packbitmap.* alt_objects/pack/ &&\n> +\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n> +\tgit index-pack 3b.pack &&\n> +\tpackobjects 3b.idx >3b.objects &&\n> +\tif hasany packbitmap.objects 3b.objects; then\n> +\t\techo \"Non-local object from bitmapped pack present in pack generated with --local\"\n> +\t\treturn 1\n> +\tfi &&\n> +\tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n\nDitto on potential use of test_when_finished.\n\n"},{"id":"293386","messageId":"20160808190950.x7ld2r4n7id57nof@sigill.intra.peff.net","threadId":"42795","inReplyTo":"xmqqpopjdqsa.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-08T19:09:51Z","receivedAt":"2016-08-08T19:55:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 08, 2016 at 12:06:13PM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> >> +\tif grep -qFf nonlocal-loose 1.objects; then\n> >> +\t\techo \"Non-local object present in pack generated with --local\"\n> >> +\t\treturn 1\n> >> +\tfi\n> >> +'\n> >\n> > grep -f isn't portable. However, I think:\n> >\n> > echo $objsha1 >expect &&\n> > git show-index <1.pack | cut -d' ' -f2 >actual\n> > test_cmp expect actual\n> >\n> > would work (if you do stick with two entries, you might need to sort\n> > your \"expect\").\n> \n> Hmph, are you sure? \"grep -f pattern_file\" is in POSIX.1.\n\nHmm, you're right. I specifically checked my local grep.1posix manpage,\nbut searching for \"-f\" didn't turn up anything, because it's formatted\nwith a Unicode minus sign (U+2212). Bleh.\n\n-Peff\n"},{"id":"293390","messageId":"xmqqbn13dltr.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160808185541.6433-1-kirr@nexedi.com","subject":"Re: [PATCH v5] pack-objects: teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-08T20:53:20Z","receivedAt":"2016-08-08T20:53:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index bc1c433..4ba0c4a 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -2244,6 +2244,9 @@ pack.useBitmaps::\n> \tto stdout (e.g., during the server side of a fetch). Defaults to\n> \ttrue. You should not generally need to turn this off unless\n> \tyou are debugging pack bitmaps.\n> ++\n> +*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n> +\tpack bitmaps.\n\nThis is a bit hard to read and understand.\n\nThe patched result starts with \"When true, git will use bitmap when\npacking to stdout\", i.e. when packing to file, git will not. So\nthis *NOTE* is repeating the same thing. The reader is made to\nwonder \"Why does it need to repeat the same thing? Does this mean\nwhen the variable is set, a pack sent to a disk uses the bitmap?\"\n\nI think what you actually do in the code is to make the variable\naffect _only_ the standard-output case, and users need a command\nline option if they want to use bitmap when writing to a file (the\ncode to do so looks correctly done).\n\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index 3893afd..ffecc6a 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -118,6 +118,18 @@ test_expect_success 'incremental repack can disable bitmaps' '\n> \tgit repack -d --no-write-bitmap-index\n> '\n> \n> +test_expect_success 'pack-objects to file can use bitmap' '\n> +\t# make sure we still have 1 bitmap index from previous tests\n> +\tls .git/objects/pack/ | grep bitmap >output &&\n> +\ttest_line_count = 1 output &&\n> +\t# verify equivalent packs are generated with/without using bitmap index\n> +\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa +\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb +\tgit show-index packa.objects &&\n> +\tgit show-index packb.objects &&\n> +\ttest_cmp packa.objects packb.objects\n> +'\n\nLooks good.\n"},{"id":"293469","messageId":"20160809112123.GA14273@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqqoa53dpue.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-09T11:21:27Z","receivedAt":"2016-08-09T11:21:39Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Junio, first of all thanks for feedback,\n\nOn Mon, Aug 08, 2016 at 12:26:33PM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n[...]\n> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> > index c4c2a3c..e06c1bf 100644\n> > --- a/builtin/pack-objects.c\n> > +++ b/builtin/pack-objects.c\n> > @@ -944,13 +944,45 @@ static int have_duplicate_entry(const unsigned char *sha1,\n> > \treturn 1;\n> > }\n> > \n> > +static int want_found_object(int exclude, struct packed_git *p)\n> > +{\n> > +\tif (exclude)\n> > +\t\treturn 1;\n> > +\tif (incremental)\n> > +\t\treturn 0;\n> > +\n> > +\t/*\n> > +\t * When asked to do --local (do not include an\n> > +\t * object that appears in a pack we borrow\n> > +\t * from elsewhere) or --honor-pack-keep (do not\n> > +\t * include an object that appears in a pack marked\n> > +\t * with .keep), we need to make sure no copy of this\n> > +\t * object come from in _any_ pack that causes us to\n> > +\t * omit it, and need to complete this loop. When\n> > +\t * neither option is in effect, we know the object\n> > +\t * we just found is going to be packed, so break\n> > +\t * out of the search loop now.\n> > +\t */\n> \n> The blame is mine, but \"no copy of this object appears in _any_ pack\"\n> would be more correct and easier to read.\n> \n> This code is no longer in a search loop; its caller is. Further\n> rephrasing is needed. \"When asked to do ...these things..., finding\n> a pack that matches the criteria is sufficient for us to decide to\n> omit it. However, even if this pack does not satisify the criteria,\n> we need to make sure no copy of this object appears in _any_ pack\n> that makes us to omit the object, so we need to check all the packs.\n> Signal that by returning -1 to the caller.\" or something along that\n> line.\n\nOk, I've rephrased it your way. Thanks for advising.\n\n> > /*\n> > * Check whether we want the object in the pack (e.g., we do not want\n> > * objects found in non-local stores if the \"--local\" option was used).\n> > *\n> > - * As a side effect of this check, we will find the packed version of this\n> > - * object, if any. We therefore pass out the pack information to avoid having\n> > - * to look it up again later.\n> > + * As a side effect of this check, if object's pack entry was not already found,\n> > + * we will find the packed version of this object, if any. We therefore pass\n> > + * out the pack information to avoid having to look it up again later.\n> \n> The reasoning leading to \"We therefore\" is understandable, but \"pass\n> out the pack information\" is not quite. Is this meant to explain\n> the fact that *found_pack and *found_offset are in-out parameters?\n> \n> The explanation to justify why *found_pack and *found_offset that\n> used to be out parameters are made in-out parameters belongs to the\n> log message. We do not want this in-code comment to explain the\n> updated code relative to what the code used to do; that is not\n> useful to those who read the code for the first time in the context\n> of the committed state.\n> \n> /* \n> * Check whether we want to pack the object in the pack (e.g. ...).\n> *\n> * If the caller already knows an existing pack it wants to\n> * take the object from, that is passed in *found_pack and\n> * *found_offset; otherwise this function finds if there is\n> * any pack that has the object and returns the pack and its\n> * offset in these variables.\n> */\n\nThe \"pass out the pack information ...\" is not my text - I only added\n\"if object's pack entry was not already found\" in the middle of the\nsentence and rewrapped this paragraph. The \"pass out the pack\ninformation ...\" comes from ce2bc424 (pack-objects: split\nadd_object_entry; 2013-12-21)\n\nI agree your text is more clear and it is better to adjust the comments.\n\n> > diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> > index 3893afd..e71caa4 100755\n> > --- a/t/t5310-pack-bitmaps.sh\n> > +++ b/t/t5310-pack-bitmaps.sh\n> > @@ -7,6 +7,19 @@ objpath () {\n> > \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n> > }\n> > \n> > +# show objects present in pack ($1 should be associated *.idx)\n> > +packobjects () {\n> > +\tgit show-index <$1 | cut -d' ' -f2\n> > +}\n> \n> That is a misleading name for a helper function that produces a list\n> of objects that were packed. \"list_packed_objects\", perhaps.\n\nI agree it is ambiguous wrt `git pack-objects` and sorry for choosing\nnot good name from the start. I'm changing it to pack_list_objects().\n( personally I would use pack_obj_list a-la git-rev-list, but let's try\n not to create another review step because of abbreviate vs not-abbreviate )\n\n> > +# hasany pattern-file content-file\n> > +# tests whether content-file has any entry from pattern-file with entries being\n> > +# whole lines.\n> > +hasany () {\n> > +\t# NOTE `grep -f` is not portable\n> > +\tgit grep --no-index -qFf $1 $2\n> > +}\n> \n> I doubt \"grep -f pattern_file\" is not portable, but in any case, it\n> is probably a good idea to have this helper function to make the\n> caller easier to read. Please name it \"has_any\", though, and quote\n> \"$1\" and \"$2\" as they are meant to be able to take any filename.\n\nOk, thanks for the info `grep -f` is portable\n\n\n> > +test_expect_success 'pack-objects respects --local (non-local loose)' '\n> > +\tmkdir -p alt_objects/pack &&\n> \n> I'd really really prefer to see an empty repository created for\n> this. Even though the original intent was .git/objects/ alone,\n> i.e. GIT_OBJECT_DIRECTORY can exist without associated refs, we\n> discovered that it is in general not a good idea (think: \"gc\").\n\nThe \"mkdir alt_objects/\" comes from t7700-repack.sh - e.g. from\n3c3df429 which I've ported from there. However as you say let's switch\nthis to having full another repo.\n\n\n> > +\techo $(pwd)/alt_objects >.git/objects/info/alternates &&\n> > +\techo content1 >file1 &&\n> > +\t# non-local loose object which is not present in bitmapped pack\n> > +\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n> \n> Don't say \"sha\" when you mean \"object name\". Otherwise you would\n> end up introducing funky variable names like $objsha2 we see below\n> that is confusing (we don't use SHA-2).\n\nOk makes sense, I've changed objsha1 to altblob and objsha2 to blob2.\nThanks for head-ups on this.\n\n\n> > +\t# non-local loose object which is also present in bitmapped pack\n> > +\tgit cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n> > +\tgit add file1 &&\n> > +\ttest_tick &&\n> > +\tgit commit -m commit_file1 &&\n> > +\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n> > +\tgit index-pack 1.pack &&\n> > +\tpackobjects 1.idx >1.objects &&\n> > +\tprintf \"$objsha1\\n$blob\\n\" >nonlocal-loose &&\n> \n> I think Peff meant to suggest this instead:\n> \n> \tprintf \"%s\\n\" \"$objsha1\" \"$blob\"\n\nOops, yes, my bad. Corrected.\n\n\n> > +\tif hasany nonlocal-loose 1.objects; then\n> > +\t\techo \"Non-local object present in pack generated with --local\"\n> > +\t\treturn 1\n> > +\tfi\n> \n> Just saying\n> \n> \t! has_any nonlocal-loose 1.objects\n> \n> is sufficient. Same comment for all other uses of these verbose\n> output.\n> \n> Besides, we spell \"if/then/fi\" like this:\n> \n> \tif condition\n> then\n> \tbody\n> \tfi\n> \n> without a semicolon.\n\nI initially copied this check-templates from t7700-repack.sh, e.g. from\n3289b9de (t7700: test that 'repack -a' packs alternate packed objects;\n2008-11-13) and other places.\n\nBut ok, let's switch the checks to oneliners like \"! has_any ...\"\n\n> > +test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n> > +...\n> > +\ttouch .git/objects/pack/pack2-$pack2.keep &&\n> \n> Please don't do \"touch\" _unless_ you care about the timestamp of the\n> file. Redirect an empty command into it, i.e.\n> \n> \t>.git/objects/pack/pack2-$pack2.keep\n> \n> or\n> \n> \techo \"reason to keep it\" >.git/objects/pack/pack2-$pack2.keep\n> \n> instead.\n\nOk, I've changed to >file as the reason here is obvious.\nWould you please explain why we should not use touch if we do not care\nabout timestamps? Simply style?\n\n> > +test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n> > +\tls .git/objects/pack/ | grep bitmap >output &&\n> > +\ttest_line_count = 1 output &&\n> > +\tpackbitmap=$(basename $(cat output) .bitmap) &&\n> > +\tpackobjects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n> > +\ttouch .git/objects/pack/$packbitmap.keep &&\n> > +\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n> > +\tgit index-pack 3a.pack &&\n> > +\tpackobjects 3a.idx >3a.objects &&\n> > +\tif hasany packbitmap.objects 3a.objects; then\n> > +\t\techo \"Object from .keeped bitmapped pack present in pack generated with --honour-pack-keep\"\n> > +\t\treturn 1\n> > +\tfi &&\n> > +\trm .git/objects/pack/$packbitmap.keep\n> \n> Arrange this removal to happen even when any earlier step fails, so\n> that later tests will not get affected by stray existence of this\n> file, by using test_when_finished. E.g.\n> \n> \tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n> \ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n> \t>.git/objects/pack/$packbitmap.keep\" &&\n\nOk, I did not knew about test_when_finished, and thanks for pointing\nthis out. Adjusted here and in similar place.\n\nWill send v4 patch as reply to this mail with below interdiff:\n\nThanks again,\nKirill\n\n---- 8< ---- (interdiff)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4c129bd..c92d7fc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -953,16 +953,14 @@ static int want_found_object(int exclude, struct packed_git *p)\n \t\treturn 0;\n \n \t/*\n-\t * When asked to do --local (do not include an\n-\t * object that appears in a pack we borrow\n-\t * from elsewhere) or --honor-pack-keep (do not\n-\t * include an object that appears in a pack marked\n-\t * with .keep), we need to make sure no copy of this\n-\t * object come from in _any_ pack that causes us to\n-\t * omit it, and need to complete this loop. When\n-\t * neither option is in effect, we know the object\n-\t * we just found is going to be packed, so break\n-\t * out of the search loop now.\n+\t * When asked to do --local (do not include an object that appears in a\n+\t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n+\t * an object that appears in a pack marked with .keep), finding a pack\n+\t * that matches the criteria is sufficient for us to decide to omit it.\n+\t * However, even if this pack does not satisfy the criteria, we need to\n+\t * make sure no copy of this object appears in _any_ pack that makes us\n+\t * to omit the object, so we need to check all the packs. Signal that by\n+\t * returning -1 to the caller.\n \t */\n \tif (!ignore_packed_keep &&\n \t (!local || !have_non_local_packs))\n@@ -981,9 +979,10 @@ static int want_found_object(int exclude, struct packed_git *p)\n * Check whether we want the object in the pack (e.g., we do not want\n * objects found in non-local stores if the \"--local\" option was used).\n *\n- * As a side effect of this check, if object's pack entry was not already found,\n- * we will find the packed version of this object, if any. We therefore pass\n- * out the pack information to avoid having to look it up again later.\n+ * If the caller already knows an existing pack it wants to take the object\n+ * from, that is passed in *found_pack and *found_offset; otherwise this\n+ * function finds if there is any pack that has the object and returns the pack\n+ * and its offset in these variables.\n */\n static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t int exclude,\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex cce95d8..44914ac 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -8,16 +8,15 @@ objpath () {\n }\n \n # show objects present in pack ($1 should be associated *.idx)\n-packobjects () {\n-\tgit show-index <$1 | cut -d' ' -f2\n+pack_list_objects () {\n+\tgit show-index <\"$1\" | cut -d' ' -f2\n }\n \n-# hasany pattern-file content-file\n+# has_any pattern-file content-file\n # tests whether content-file has any entry from pattern-file with entries being\n # whole lines.\n-hasany () {\n-\t# NOTE `grep -f` is not portable\n-\tgit grep --no-index -qFf $1 $2\n+has_any () {\n+\tgrep -qFf \"$1\" \"$2\"\n }\n \n test_expect_success 'setup repo with moderate-sized history' '\n@@ -133,83 +132,68 @@ test_expect_success 'incremental repack can disable bitmaps' '\n '\n \n test_expect_success 'pack-objects respects --local (non-local loose)' '\n-\tmkdir -p alt_objects/pack &&\n-\techo $(pwd)/alt_objects >.git/objects/info/alternates &&\n+\tgit init --bare alt.git &&\n+\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n \techo content1 >file1 &&\n \t# non-local loose object which is not present in bitmapped pack\n-\tobjsha1=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file1) &&\n+\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n \t# non-local loose object which is also present in bitmapped pack\n-\tgit cat-file blob $blob | GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w --stdin &&\n+\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n \tgit add file1 &&\n \ttest_tick &&\n \tgit commit -m commit_file1 &&\n \techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n \tgit index-pack 1.pack &&\n-\tpackobjects 1.idx >1.objects &&\n-\tprintf \"$objsha1\\n$blob\\n\" >nonlocal-loose &&\n-\tif hasany nonlocal-loose 1.objects; then\n-\t\techo \"Non-local object present in pack generated with --local\"\n-\t\treturn 1\n-\tfi\n+\tpack_list_objects 1.idx >1.objects &&\n+\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n+\t! has_any nonlocal-loose 1.objects\n '\n \n test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n \techo content2 >file2 &&\n-\tobjsha2=$(git hash-object -w file2) &&\n+\tblob2=$(git hash-object -w file2) &&\n \tgit add file2 &&\n \ttest_tick &&\n \tgit commit -m commit_file2 &&\n-\tprintf \"$objsha2\\n$bitmaptip\\n\" >keepobjects &&\n+\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n \tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n+\trm $(objpath $blob2) &&\n \techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n \tgit index-pack 2a.pack &&\n-\tpackobjects 2a.idx >2a.objects &&\n-\tif hasany keepobjects 2a.objects; then\n-\t\techo \"Object from .keeped pack present in pack generated with --honor-pack-keep\"\n-\t\treturn 1\n-\tfi\n+\tpack_list_objects 2a.idx >2a.objects &&\n+\t! has_any keepobjects 2a.objects\n '\n \n test_expect_success 'pack-objects respects --local (non-local pack)' '\n-\tmv .git/objects/pack/pack2-$pack2.* alt_objects/pack/ &&\n+\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n \techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n \tgit index-pack 2b.pack &&\n-\tpackobjects 2b.idx >2b.objects &&\n-\tif hasany keepobjects 2b.objects; then\n-\t\techo \"Non-local object present in pack generated with --local\"\n-\t\treturn 1\n-\tfi\n+\tpack_list_objects 2b.idx >2b.objects &&\n+\t! has_any keepobjects 2b.objects\n '\n \n test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n \tls .git/objects/pack/ | grep bitmap >output &&\n \ttest_line_count = 1 output &&\n \tpackbitmap=$(basename $(cat output) .bitmap) &&\n-\tpackobjects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n-\ttouch .git/objects/pack/$packbitmap.keep &&\n+\tpack_list_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n+\t>.git/objects/pack/$packbitmap.keep &&\n \techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n \tgit index-pack 3a.pack &&\n-\tpackobjects 3a.idx >3a.objects &&\n-\tif hasany packbitmap.objects 3a.objects; then\n-\t\techo \"Object from .keeped bitmapped pack present in pack generated with --honour-pack-keep\"\n-\t\treturn 1\n-\tfi &&\n-\trm .git/objects/pack/$packbitmap.keep\n+\tpack_list_objects 3a.idx >3a.objects &&\n+\t! has_any packbitmap.objects 3a.objects\n '\n \n test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n-\tmv .git/objects/pack/$packbitmap.* alt_objects/pack/ &&\n+\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n+\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n \techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n \tgit index-pack 3b.pack &&\n-\tpackobjects 3b.idx >3b.objects &&\n-\tif hasany packbitmap.objects 3b.objects; then\n-\t\techo \"Non-local object from bitmapped pack present in pack generated with --local\"\n-\t\treturn 1\n-\tfi &&\n-\tmv alt_objects/pack/$packbitmap.* .git/objects/pack/\n+\tpack_list_objects 3b.idx >3b.objects &&\n+\t! has_any packbitmap.objects 3b.objects\n '\n \n test_expect_success 'pack-objects to file can use bitmap' '\n@@ -256,14 +240,11 @@ test_expect_success 'pack-objects respects --incremental' '\n \tEOF\n \tgit pack-objects --incremental --stdout --revs 4.pack &&\n \tgit index-pack 4.pack &&\n-\tpackobjects 4.idx >4.objects &&\n+\tpack_list_objects 4.idx >4.objects &&\n \ttest_line_count = 4 4.objects &&\n \tgit rev-list --objects $commit >revlist &&\n \tcut -d\" \" -f1 revlist |sort >objects &&\n-\tif !hasany objects 4.objects; then\n-\t\techo \"Expected objects not present in incremental pack\"\n-\t\treturn 1\n-\tfi\n+\ttest_cmp 4.objects objects\n '\n \n test_expect_success 'pack with missing blob' '\n"},{"id":"293470","messageId":"20160809112143.GA21132@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqqbn13dltr.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5] pack-objects: teach it to use reachability bitmap index when generating non-stdout pack too","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-09T11:21:47Z","receivedAt":"2016-08-09T11:21:54Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Mon, Aug 08, 2016 at 01:53:20PM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> > diff --git a/Documentation/config.txt b/Documentation/config.txt\n> > index bc1c433..4ba0c4a 100644\n> > --- a/Documentation/config.txt\n> > +++ b/Documentation/config.txt\n> > @@ -2244,6 +2244,9 @@ pack.useBitmaps::\n> > \tto stdout (e.g., during the server side of a fetch). Defaults to\n> > \ttrue. You should not generally need to turn this off unless\n> > \tyou are debugging pack bitmaps.\n> > ++\n> > +*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n> > +\tpack bitmaps.\n> \n> This is a bit hard to read and understand.\n> \n> The patched result starts with \"When true, git will use bitmap when\n> packing to stdout\", i.e. when packing to file, git will not. So\n> this *NOTE* is repeating the same thing. The reader is made to\n> wonder \"Why does it need to repeat the same thing? Does this mean\n> when the variable is set, a pack sent to a disk uses the bitmap?\"\n> \n> I think what you actually do in the code is to make the variable\n> affect _only_ the standard-output case, and users need a command\n> line option if they want to use bitmap when writing to a file (the\n> code to do so looks correctly done).\n\nYes it is this way how it is programmed. But I've added the note because\nit is very implicit to me that \"When true, git will use bitmap when\npacking to stdout\" means 1) the default for packing-to-file is different\nand 2) there is no way to set the default for packing-to-file. That's\nwhy I added the explicit info.\n\nAnd especially since the config name \"pack.useBitmaps\" does not contain\n\"stdout\" at all it can be very confusing to people looking at this the\nfirst time (at least it was so this way for me). Also please recall you\nwondering why 6b8fda2d added bitmap support only for to-stdout case not\neven mentioning about why it is done only for that case and not for\nto-file case).\n\nI do not insist on the note however - I only thought it is better to\nhave it - so if you prefer we go without it - let us drop this note.\n\nWill send v6 as reply to this mail with below interdiff.\n\nThanks,\nKirill\n\n---- 8< ---- (interdiff)\n\n--- b/Documentation/config.txt\n+++ a/Documentation/config.txt\n@@ -2246,9 +2246,6 @@\n to stdout (e.g., during the server side of a fetch). Defaults to\n true. You should not generally need to turn this off unless\n you are debugging pack bitmaps.\n-+\n-*NOTE*: when packing to file (e.g., on repack) the default is always not to use\n- pack bitmaps.\n \n pack.writeBitmaps (deprecated)::\n This is a deprecated synonym for `repack.writeBitmaps`.\ndiff -u b/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n--- b/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -219,8 +219,8 @@\n # verify equivalent packs are generated with/without using bitmap index\n packasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n- git show-index packb.objects &&\n+ pack_list_objects packa.objects &&\n+ pack_list_objects packb.objects &&\n test_cmp packa.objects packb.objects\n '\n"},{"id":"293472","messageId":"20160809112547.9047-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160809112123.GA14273@teco.navytux.spb.ru","subject":"[PATCH 1/2 v4] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-09T11:25:47Z","receivedAt":"2016-08-09T11:26:10Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep or --incremental should be respected. In\nnon-bitmapped codepath this is handled in want_object_in_pack(), but\nbitmapped codepath has simply no such checking at all.\n\nThe bitmapped codepath however was allowing to pass in all those options\nand with bitmap indices still being used under such conditions -\npotentially giving wrong output (e.g. including objects from non-local or\n.keep'ed pack).\n\nWe can easily fix this by noting the following: when an object comes to\nadd_object_entry_from_bitmap() it can come for two reasons:\n\n 1. entries coming from main pack covered by bitmap index, and\n 2. object coming from, possibly alternate, loose or other packs.\n\n\"2\" can be already handled by want_object_in_pack() and to cover\n\"1\" we can teach want_object_in_pack() to expect that *found_pack can be\nnon-NULL, meaning calling client already found object's pack entry.\n\nIn want_object_in_pack() we care to start the checks from already found\npack, if we have one, this way determining the answer right away\nin case neither --local nor --honour-pack-keep are active. In\nparticular, as p5310-pack-bitmaps.sh shows, we do not do harm to\nserved-with-bitmap clones performance-wise:\n\n Test 56dfeb62 this tree\n -----------------------------------------------------------------\n 5310.2: repack to disk 9.63(8.67+0.33) 9.47(8.55+0.28) -1.7%\n 5310.3: simulated clone 2.07(2.17+0.12) 2.03(2.14+0.12) -1.9%\n 5310.4: simulated fetch 0.78(1.03+0.02) 0.76(1.00+0.03) -2.6%\n 5310.6: partial bitmap 1.97(2.43+0.15) 1.92(2.36+0.14) -2.5%\n\nwith all differences strangely showing we are a bit faster now, but\nprobably all being within noise.\n\nAnd in the general case we care not to have duplicate\nfind_pack_entry_one(*found_pack) calls. Worst what can happen is we can\ncall want_found_object(*found_pack) -- newly introduced helper for\nchecking whether we want object -- twice, but since want_found_object()\nis very lightweight it does not make any difference.\n\nI appreciate help and discussing this change with Junio C Hamano and\nJeff King.\n\nSigned-off-by: Kirill Smelkov \n---\n builtin/pack-objects.c | 93 +++++++++++++++++++++++++++++++------------------\n t/t5310-pack-bitmaps.sh | 92 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 152 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c4c2a3c..b1007f2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -944,13 +944,44 @@ static int have_duplicate_entry(const unsigned char *sha1,\n \treturn 1;\n }\n \n+static int want_found_object(int exclude, struct packed_git *p)\n+{\n+\tif (exclude)\n+\t\treturn 1;\n+\tif (incremental)\n+\t\treturn 0;\n+\n+\t/*\n+\t * When asked to do --local (do not include an object that appears in a\n+\t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n+\t * an object that appears in a pack marked with .keep), finding a pack\n+\t * that matches the criteria is sufficient for us to decide to omit it.\n+\t * However, even if this pack does not satisfy the criteria, we need to\n+\t * make sure no copy of this object appears in _any_ pack that makes us\n+\t * to omit the object, so we need to check all the packs. Signal that by\n+\t * returning -1 to the caller.\n+\t */\n+\tif (!ignore_packed_keep &&\n+\t (!local || !have_non_local_packs))\n+\t\treturn 1;\n+\n+\tif (local && !p->pack_local)\n+\t\treturn 0;\n+\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\t\treturn 0;\n+\n+\t/* we don't know yet; keep looking for more packs */\n+\treturn -1;\n+}\n+\n /*\n * Check whether we want the object in the pack (e.g., we do not want\n * objects found in non-local stores if the \"--local\" option was used).\n *\n- * As a side effect of this check, we will find the packed version of this\n- * object, if any. We therefore pass out the pack information to avoid having\n- * to look it up again later.\n+ * If the caller already knows an existing pack it wants to take the object\n+ * from, that is passed in *found_pack and *found_offset; otherwise this\n+ * function finds if there is any pack that has the object and returns the pack\n+ * and its offset in these variables.\n */\n static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t int exclude,\n@@ -958,15 +989,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t off_t *found_offset)\n {\n \tstruct packed_git *p;\n+\tint want;\n \n \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n \t\treturn 0;\n \n-\t*found_pack = NULL;\n-\t*found_offset = 0;\n+\t/*\n+\t * If we already know the pack object lives in, start checks from that\n+\t * pack - in the usual case when neither --local was given nor .keep files\n+\t * are present we will determine the answer right now.\n+\t */\n+\tif (*found_pack) {\n+\t\twant = want_found_object(exclude, *found_pack);\n+\t\tif (want != -1)\n+\t\t\treturn want;\n+\t}\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\toff_t offset = find_pack_entry_one(sha1, p);\n+\t\toff_t offset;\n+\n+\t\tif (p == *found_pack)\n+\t\t\toffset = *found_offset;\n+\t\telse\n+\t\t\toffset = find_pack_entry_one(sha1, p);\n+\n \t\tif (offset) {\n \t\t\tif (!*found_pack) {\n \t\t\t\tif (!is_pack_valid(p))\n@@ -974,31 +1020,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t\t*found_offset = offset;\n \t\t\t\t*found_pack = p;\n \t\t\t}\n-\t\t\tif (exclude)\n-\t\t\t\treturn 1;\n-\t\t\tif (incremental)\n-\t\t\t\treturn 0;\n-\n-\t\t\t/*\n-\t\t\t * When asked to do --local (do not include an\n-\t\t\t * object that appears in a pack we borrow\n-\t\t\t * from elsewhere) or --honor-pack-keep (do not\n-\t\t\t * include an object that appears in a pack marked\n-\t\t\t * with .keep), we need to make sure no copy of this\n-\t\t\t * object come from in _any_ pack that causes us to\n-\t\t\t * omit it, and need to complete this loop. When\n-\t\t\t * neither option is in effect, we know the object\n-\t\t\t * we just found is going to be packed, so break\n-\t\t\t * out of the loop to return 1 now.\n-\t\t\t */\n-\t\t\tif (!ignore_packed_keep &&\n-\t\t\t (!local || !have_non_local_packs))\n-\t\t\t\tbreak;\n-\n-\t\t\tif (local && !p->pack_local)\n-\t\t\t\treturn 0;\n-\t\t\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n-\t\t\t\treturn 0;\n+\t\t\twant = want_found_object(exclude, p);\n+\t\t\tif (want != -1)\n+\t\t\t\treturn want;\n \t\t}\n \t}\n \n@@ -1039,8 +1063,8 @@ static const char no_closure_warning[] = N_(\n static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \t\t\t const char *name, int exclude)\n {\n-\tstruct packed_git *found_pack;\n-\toff_t found_offset;\n+\tstruct packed_git *found_pack = NULL;\n+\toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n@@ -1073,6 +1097,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n+\t\treturn 0;\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..a50d867 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -7,6 +7,18 @@ objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n \n+# show objects present in pack ($1 should be associated *.idx)\n+pack_list_objects () {\n+\tgit show-index <\"$1\" | cut -d' ' -f2\n+}\n+\n+# has_any pattern-file content-file\n+# tests whether content-file has any entry from pattern-file with entries being\n+# whole lines.\n+has_any () {\n+\tgrep -qFf \"$1\" \"$2\"\n+}\n+\n test_expect_success 'setup repo with moderate-sized history' '\n \tfor i in $(test_seq 1 10); do\n \t\ttest_commit $i\n@@ -16,6 +28,7 @@ test_expect_success 'setup repo with moderate-sized history' '\n \t\ttest_commit side-$i\n \tdone &&\n \tgit checkout master &&\n+\tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n \tgit config repack.writebitmaps true &&\n@@ -118,6 +131,71 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects respects --local (non-local loose)' '\n+\tgit init --bare alt.git &&\n+\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n+\techo content1 >file1 &&\n+\t# non-local loose object which is not present in bitmapped pack\n+\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n+\t# non-local loose object which is also present in bitmapped pack\n+\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n+\tgit add file1 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file1 &&\n+\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\tgit index-pack 1.pack &&\n+\tpack_list_objects 1.idx >1.objects &&\n+\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n+\t! has_any nonlocal-loose 1.objects\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\techo content2 >file2 &&\n+\tblob2=$(git hash-object -w file2) &&\n+\tgit add file2 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file2 &&\n+\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n+\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n+\trm $(objpath $blob2) &&\n+\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\tgit index-pack 2a.pack &&\n+\tpack_list_objects 2a.idx >2a.objects &&\n+\t! has_any keepobjects 2a.objects\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local pack)' '\n+\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n+\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\tgit index-pack 2b.pack &&\n+\tpack_list_objects 2b.idx >2b.objects &&\n+\t! has_any keepobjects 2b.objects\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\tpack_list_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n+\t>.git/objects/pack/$packbitmap.keep &&\n+\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\tgit index-pack 3a.pack &&\n+\tpack_list_objects 3a.idx >3a.objects &&\n+\t! has_any packbitmap.objects 3a.objects\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n+\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n+\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\tgit index-pack 3b.pack &&\n+\tpack_list_objects 3b.idx >3b.objects &&\n+\t! has_any packbitmap.objects 3b.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n@@ -143,6 +221,20 @@ test_expect_success 'create objects for missing-HAVE tests' '\n \tEOF\n '\n \n+test_expect_success 'pack-objects respects --incremental' '\n+\tcat >revs2 <<-EOF &&\n+\tHEAD\n+\t$commit\n+\tEOF\n+\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\tgit index-pack 4.pack &&\n+\tpack_list_objects 4.idx >4.objects &&\n+\ttest_line_count = 4 4.objects &&\n+\tgit rev-list --objects $commit >revlist &&\n+\tcut -d\" \" -f1 revlist |sort >objects &&\n+\ttest_cmp 4.objects objects\n+'\n+\n test_expect_success 'pack with missing blob' '\n \trm $(objpath $blob) &&\n \tgit pack-objects --stdout --revs /dev/null\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293473","messageId":"20160809112641.9235-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160809112143.GA21132@teco.navytux.spb.ru","subject":"[PATCH 2/2 v6] pack-objects: use reachability bitmap index when generating non-stdout pack","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-09T11:26:41Z","receivedAt":"2016-08-09T11:27:09Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nThe reason here is for on-disk repack by default we want:\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff King further explains:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\nSo we can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we care it is not activated under git-repack:\n\n See above about repack robustness and not forward-carrying corruption.\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n We also want to avoid (at least with current implementation)\n generating bitmaps off of bitmaps. The reason here is: when generating\n a pack, not-yet-packed objects will be emitted into pack in\n suboptimal order and added to tail of the bitmap as \"extended entries\".\n When the resultant pack + some new objects in associated repository\n are in turn used to generate another pack with bitmap, the situation\n repeats: new objects are again not emitted optimally and just added to\n bitmap tail - not in recency order.\n\n So the pack badness can grow over time when at each step we have\n bitmapped pack + some other objects. That's why we want to avoid\n generating bitmaps off of bitmaps, not to let pack badness grow.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\nThis way for pack-objects -> file we get nice speedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly. And git-repack does never pass\n--use-bitmap-index, so this way we can be sure regular on-disk repacking\nremains robust.\n\nNOTE2\n\n`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\n real 0m22.309s\n user 0m21.148s\n sys 0m0.932s\n\n $ time git index-pack erp5pack-stdout.pack\n\n real 0m50.873s <-- more than 2 times slower than time to generate pack itself!\n user 0m49.300s\n sys 0m1.360s\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n\nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n\nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\nJeff explains:\n\n The packfile does not carry the sha1 of the objects. A receiving\n index-pack has to compute them itself, including inflating and applying\n all of the deltas.\n\nthat's why for `git-backup restore` we want to teach `git pack-objects\nfile.pack` to use bitmaps instead of using `git pack-objects --stdout\n>file.pack` + `git index-pack file.pack`.\n\nMore context:\n\n http://marc.info/?t=146792101400001&r=1&w=2\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n builtin/pack-objects.c | 31 ++++++++++++++++++++++++-------\n t/t5310-pack-bitmaps.sh | 12 ++++++++++++\n 2 files changed, 36 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b1007f2..c92d7fc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -67,7 +67,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2270,7 +2271,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2519,13 +2520,13 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n }\n \n /*\n- * This tracks any options which a reader of the pack might\n- * not understand, and which would therefore prevent blind reuse\n- * of what we have on disk.\n+ * This tracks any options which pack-reuse code expects to be on, or which a\n+ * reader of the pack might not understand, and which would therefore prevent\n+ * blind reuse of what we have on disk.\n */\n static int pack_options_allow_reuse(void)\n {\n-\treturn allow_ofs_delta;\n+\treturn pack_to_stdout && allow_ofs_delta;\n }\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n@@ -2818,7 +2819,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex a50d867..44914ac 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -196,6 +196,18 @@ test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n \t! has_any packbitmap.objects 3b.objects\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\tpack_list_objects packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293500","messageId":"xmqqtwetc2bh.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160809112123.GA14273@teco.navytux.spb.ru","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-09T16:52:18Z","receivedAt":"2016-08-09T16:52:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> Would you please explain why we should not use touch if we do not care\n> about timestamps? Simply style?\n\nTo help readers.\n\n\"touch A\" forcess the readers wonder \"does the timestamp of A\nmatter, and if so in what way?\" and \"does any later test care what\nis _in_ A, and if so in what way?\" Both of them is wasting their\ntime when there is no reason why \"touch\" should have been used. \n\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index cce95d8..44914ac 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -8,16 +8,15 @@ objpath () {\n> }\n> \n> # show objects present in pack ($1 should be associated *.idx)\n> -packobjects () {\n> -\tgit show-index <$1 | cut -d' ' -f2\n> +pack_list_objects () {\n> +\tgit show-index <\"$1\" | cut -d' ' -f2\n> }\n\npack-list-objects still sounds as if you are packing \"list objects\",\nthough. If you are listing packed objects (or objects in a pack),\nlist-packed-objects (or list-objects-in-pack) reads clearer and more\nto the point, at least to me.\n\n> -# hasany pattern-file content-file\n> +# has_any pattern-file content-file\n> # tests whether content-file has any entry from pattern-file with entries being\n> # whole lines.\n> -hasany () {\n> -\t# NOTE `grep -f` is not portable\n> -\tgit grep --no-index -qFf $1 $2\n> +has_any () {\n> +\tgrep -qFf \"$1\" \"$2\"\n\nOmitting \"-q\" would help those who have to debug breakage in this\ntest or the code that this test checks. What test_expect_success\noutputs is not shown by default, and running the test script with\n\"-v\" would show them as a debugging aid.\n\nThanks.\n"},{"id":"293533","messageId":"20160809192858.GA25822@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqqtwetc2bh.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-09T19:29:02Z","receivedAt":"2016-08-09T19:30:11Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Tue, Aug 09, 2016 at 09:52:18AM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> > Would you please explain why we should not use touch if we do not care\n> > about timestamps? Simply style?\n> \n> To help readers.\n> \n> \"touch A\" forcess the readers wonder \"does the timestamp of A\n> matter, and if so in what way?\" and \"does any later test care what\n> is _in_ A, and if so in what way?\" Both of them is wasting their\n> time when there is no reason why \"touch\" should have been used. \n\nI see, thanks for explaining. I used to read it a bit the other way;\nmaybe it is just an environment difference.\n\n\n> > diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> > index cce95d8..44914ac 100755\n> > --- a/t/t5310-pack-bitmaps.sh\n> > +++ b/t/t5310-pack-bitmaps.sh\n> > @@ -8,16 +8,15 @@ objpath () {\n> > }\n> > \n> > # show objects present in pack ($1 should be associated *.idx)\n> > -packobjects () {\n> > -\tgit show-index <$1 | cut -d' ' -f2\n> > +pack_list_objects () {\n> > +\tgit show-index <\"$1\" | cut -d' ' -f2\n> > }\n> \n> pack-list-objects still sounds as if you are packing \"list objects\",\n> though. If you are listing packed objects (or objects in a pack),\n> list-packed-objects (or list-objects-in-pack) reads clearer and more\n> to the point, at least to me.\n\nOk, let it be list_packed_objects().\n\n\n> > -# hasany pattern-file content-file\n> > +# has_any pattern-file content-file\n> > # tests whether content-file has any entry from pattern-file with entries being\n> > # whole lines.\n> > -hasany () {\n> > -\t# NOTE `grep -f` is not portable\n> > -\tgit grep --no-index -qFf $1 $2\n> > +has_any () {\n> > +\tgrep -qFf \"$1\" \"$2\"\n> \n> Omitting \"-q\" would help those who have to debug breakage in this\n> test or the code that this test checks. What test_expect_success\n> outputs is not shown by default, and running the test script with\n> \"-v\" would show them as a debugging aid.\n\nOk, makes sense. Both patches adjusted and will be reposted.\n\nThanks,\nKirill\n"},{"id":"293534","messageId":"20160809193143.32213-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160809192858.GA25822@teco.navytux.spb.ru","subject":"[PATCH 1/2 v5] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-09T19:31:43Z","receivedAt":"2016-08-09T19:33:26Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep or --incremental should be respected. In\nnon-bitmapped codepath this is handled in want_object_in_pack(), but\nbitmapped codepath has simply no such checking at all.\n\nThe bitmapped codepath however was allowing to pass in all those options\nand with bitmap indices still being used under such conditions -\npotentially giving wrong output (e.g. including objects from non-local or\n.keep'ed pack).\n\nWe can easily fix this by noting the following: when an object comes to\nadd_object_entry_from_bitmap() it can come for two reasons:\n\n 1. entries coming from main pack covered by bitmap index, and\n 2. object coming from, possibly alternate, loose or other packs.\n\n\"2\" can be already handled by want_object_in_pack() and to cover\n\"1\" we can teach want_object_in_pack() to expect that *found_pack can be\nnon-NULL, meaning calling client already found object's pack entry.\n\nIn want_object_in_pack() we care to start the checks from already found\npack, if we have one, this way determining the answer right away\nin case neither --local nor --honour-pack-keep are active. In\nparticular, as p5310-pack-bitmaps.sh shows, we do not do harm to\nserved-with-bitmap clones performance-wise:\n\n Test 56dfeb62 this tree\n -----------------------------------------------------------------\n 5310.2: repack to disk 9.63(8.67+0.33) 9.47(8.55+0.28) -1.7%\n 5310.3: simulated clone 2.07(2.17+0.12) 2.03(2.14+0.12) -1.9%\n 5310.4: simulated fetch 0.78(1.03+0.02) 0.76(1.00+0.03) -2.6%\n 5310.6: partial bitmap 1.97(2.43+0.15) 1.92(2.36+0.14) -2.5%\n\nwith all differences strangely showing we are a bit faster now, but\nprobably all being within noise.\n\nAnd in the general case we care not to have duplicate\nfind_pack_entry_one(*found_pack) calls. Worst what can happen is we can\ncall want_found_object(*found_pack) -- newly introduced helper for\nchecking whether we want object -- twice, but since want_found_object()\nis very lightweight it does not make any difference.\n\nI appreciate help and discussing this change with Junio C Hamano and\nJeff King.\n\nSigned-off-by: Kirill Smelkov \n---\n builtin/pack-objects.c | 93 +++++++++++++++++++++++++++++++------------------\n t/t5310-pack-bitmaps.sh | 92 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 152 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c4c2a3c..b1007f2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -944,13 +944,44 @@ static int have_duplicate_entry(const unsigned char *sha1,\n \treturn 1;\n }\n \n+static int want_found_object(int exclude, struct packed_git *p)\n+{\n+\tif (exclude)\n+\t\treturn 1;\n+\tif (incremental)\n+\t\treturn 0;\n+\n+\t/*\n+\t * When asked to do --local (do not include an object that appears in a\n+\t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n+\t * an object that appears in a pack marked with .keep), finding a pack\n+\t * that matches the criteria is sufficient for us to decide to omit it.\n+\t * However, even if this pack does not satisfy the criteria, we need to\n+\t * make sure no copy of this object appears in _any_ pack that makes us\n+\t * to omit the object, so we need to check all the packs. Signal that by\n+\t * returning -1 to the caller.\n+\t */\n+\tif (!ignore_packed_keep &&\n+\t (!local || !have_non_local_packs))\n+\t\treturn 1;\n+\n+\tif (local && !p->pack_local)\n+\t\treturn 0;\n+\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\t\treturn 0;\n+\n+\t/* we don't know yet; keep looking for more packs */\n+\treturn -1;\n+}\n+\n /*\n * Check whether we want the object in the pack (e.g., we do not want\n * objects found in non-local stores if the \"--local\" option was used).\n *\n- * As a side effect of this check, we will find the packed version of this\n- * object, if any. We therefore pass out the pack information to avoid having\n- * to look it up again later.\n+ * If the caller already knows an existing pack it wants to take the object\n+ * from, that is passed in *found_pack and *found_offset; otherwise this\n+ * function finds if there is any pack that has the object and returns the pack\n+ * and its offset in these variables.\n */\n static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t int exclude,\n@@ -958,15 +989,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t off_t *found_offset)\n {\n \tstruct packed_git *p;\n+\tint want;\n \n \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n \t\treturn 0;\n \n-\t*found_pack = NULL;\n-\t*found_offset = 0;\n+\t/*\n+\t * If we already know the pack object lives in, start checks from that\n+\t * pack - in the usual case when neither --local was given nor .keep files\n+\t * are present we will determine the answer right now.\n+\t */\n+\tif (*found_pack) {\n+\t\twant = want_found_object(exclude, *found_pack);\n+\t\tif (want != -1)\n+\t\t\treturn want;\n+\t}\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\toff_t offset = find_pack_entry_one(sha1, p);\n+\t\toff_t offset;\n+\n+\t\tif (p == *found_pack)\n+\t\t\toffset = *found_offset;\n+\t\telse\n+\t\t\toffset = find_pack_entry_one(sha1, p);\n+\n \t\tif (offset) {\n \t\t\tif (!*found_pack) {\n \t\t\t\tif (!is_pack_valid(p))\n@@ -974,31 +1020,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t\t*found_offset = offset;\n \t\t\t\t*found_pack = p;\n \t\t\t}\n-\t\t\tif (exclude)\n-\t\t\t\treturn 1;\n-\t\t\tif (incremental)\n-\t\t\t\treturn 0;\n-\n-\t\t\t/*\n-\t\t\t * When asked to do --local (do not include an\n-\t\t\t * object that appears in a pack we borrow\n-\t\t\t * from elsewhere) or --honor-pack-keep (do not\n-\t\t\t * include an object that appears in a pack marked\n-\t\t\t * with .keep), we need to make sure no copy of this\n-\t\t\t * object come from in _any_ pack that causes us to\n-\t\t\t * omit it, and need to complete this loop. When\n-\t\t\t * neither option is in effect, we know the object\n-\t\t\t * we just found is going to be packed, so break\n-\t\t\t * out of the loop to return 1 now.\n-\t\t\t */\n-\t\t\tif (!ignore_packed_keep &&\n-\t\t\t (!local || !have_non_local_packs))\n-\t\t\t\tbreak;\n-\n-\t\t\tif (local && !p->pack_local)\n-\t\t\t\treturn 0;\n-\t\t\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n-\t\t\t\treturn 0;\n+\t\t\twant = want_found_object(exclude, p);\n+\t\t\tif (want != -1)\n+\t\t\t\treturn want;\n \t\t}\n \t}\n \n@@ -1039,8 +1063,8 @@ static const char no_closure_warning[] = N_(\n static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \t\t\t const char *name, int exclude)\n {\n-\tstruct packed_git *found_pack;\n-\toff_t found_offset;\n+\tstruct packed_git *found_pack = NULL;\n+\toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n@@ -1073,6 +1097,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n+\t\treturn 0;\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..a278d30 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -7,6 +7,18 @@ objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n \n+# show objects present in pack ($1 should be associated *.idx)\n+list_packed_objects () {\n+\tgit show-index <\"$1\" | cut -d' ' -f2\n+}\n+\n+# has_any pattern-file content-file\n+# tests whether content-file has any entry from pattern-file with entries being\n+# whole lines.\n+has_any () {\n+\tgrep -Ff \"$1\" \"$2\"\n+}\n+\n test_expect_success 'setup repo with moderate-sized history' '\n \tfor i in $(test_seq 1 10); do\n \t\ttest_commit $i\n@@ -16,6 +28,7 @@ test_expect_success 'setup repo with moderate-sized history' '\n \t\ttest_commit side-$i\n \tdone &&\n \tgit checkout master &&\n+\tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n \tgit config repack.writebitmaps true &&\n@@ -118,6 +131,71 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects respects --local (non-local loose)' '\n+\tgit init --bare alt.git &&\n+\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n+\techo content1 >file1 &&\n+\t# non-local loose object which is not present in bitmapped pack\n+\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n+\t# non-local loose object which is also present in bitmapped pack\n+\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n+\tgit add file1 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file1 &&\n+\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\tgit index-pack 1.pack &&\n+\tlist_packed_objects 1.idx >1.objects &&\n+\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n+\t! has_any nonlocal-loose 1.objects\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\techo content2 >file2 &&\n+\tblob2=$(git hash-object -w file2) &&\n+\tgit add file2 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file2 &&\n+\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n+\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n+\trm $(objpath $blob2) &&\n+\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\tgit index-pack 2a.pack &&\n+\tlist_packed_objects 2a.idx >2a.objects &&\n+\t! has_any keepobjects 2a.objects\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local pack)' '\n+\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n+\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\tgit index-pack 2b.pack &&\n+\tlist_packed_objects 2b.idx >2b.objects &&\n+\t! has_any keepobjects 2b.objects\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n+\t>.git/objects/pack/$packbitmap.keep &&\n+\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\tgit index-pack 3a.pack &&\n+\tlist_packed_objects 3a.idx >3a.objects &&\n+\t! has_any packbitmap.objects 3a.objects\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n+\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n+\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\tgit index-pack 3b.pack &&\n+\tlist_packed_objects 3b.idx >3b.objects &&\n+\t! has_any packbitmap.objects 3b.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n@@ -143,6 +221,20 @@ test_expect_success 'create objects for missing-HAVE tests' '\n \tEOF\n '\n \n+test_expect_success 'pack-objects respects --incremental' '\n+\tcat >revs2 <<-EOF &&\n+\tHEAD\n+\t$commit\n+\tEOF\n+\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\tgit index-pack 4.pack &&\n+\tlist_packed_objects 4.idx >4.objects &&\n+\ttest_line_count = 4 4.objects &&\n+\tgit rev-list --objects $commit >revlist &&\n+\tcut -d\" \" -f1 revlist |sort >objects &&\n+\ttest_cmp 4.objects objects\n+'\n+\n test_expect_success 'pack with missing blob' '\n \trm $(objpath $blob) &&\n \tgit pack-objects --stdout --revs /dev/null\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293535","messageId":"20160809193217.32389-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160809192858.GA25822@teco.navytux.spb.ru","subject":"[PATCH 2/2 v7] pack-objects: use reachability bitmap index when generating non-stdout pack","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-08-09T19:32:17Z","receivedAt":"2016-08-09T19:34:18Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nThe reason here is for on-disk repack by default we want:\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff King further explains:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\nSo we can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we care it is not activated under git-repack:\n\n See above about repack robustness and not forward-carrying corruption.\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n Current code has singleton bitmap_git so cannot work simultaneously\n with two bitmap indices.\n\n We also want to avoid (at least with current implementation)\n generating bitmaps off of bitmaps. The reason here is: when generating\n a pack, not-yet-packed objects will be emitted into pack in\n suboptimal order and added to tail of the bitmap as \"extended entries\".\n When the resultant pack + some new objects in associated repository\n are in turn used to generate another pack with bitmap, the situation\n repeats: new objects are again not emitted optimally and just added to\n bitmap tail - not in recency order.\n\n So the pack badness can grow over time when at each step we have\n bitmapped pack + some other objects. That's why we want to avoid\n generating bitmaps off of bitmaps, not to let pack badness grow.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because on pack reuse raw entries are directly written out to destination\n pack by write_reused_pack() bypassing needed for pack index generation\n bookkeeping done by regular codepath in write_one() and friends.\n\nThis way for pack-objects -> file we get nice speedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly. And git-repack does never pass\n--use-bitmap-index, so this way we can be sure regular on-disk repacking\nremains robust.\n\nNOTE2\n\n`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\n real 0m22.309s\n user 0m21.148s\n sys 0m0.932s\n\n $ time git index-pack erp5pack-stdout.pack\n\n real 0m50.873s <-- more than 2 times slower than time to generate pack itself!\n user 0m49.300s\n sys 0m1.360s\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n\nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n\nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\nJeff explains:\n\n The packfile does not carry the sha1 of the objects. A receiving\n index-pack has to compute them itself, including inflating and applying\n all of the deltas.\n\nthat's why for `git-backup restore` we want to teach `git pack-objects\nfile.pack` to use bitmaps instead of using `git pack-objects --stdout\n>file.pack` + `git index-pack file.pack`.\n\nMore context:\n\n http://marc.info/?t=146792101400001&r=1&w=2\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \n---\n builtin/pack-objects.c | 31 ++++++++++++++++++++++++-------\n t/t5310-pack-bitmaps.sh | 12 ++++++++++++\n 2 files changed, 36 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b1007f2..c92d7fc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -67,7 +67,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2270,7 +2271,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2519,13 +2520,13 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n }\n \n /*\n- * This tracks any options which a reader of the pack might\n- * not understand, and which would therefore prevent blind reuse\n- * of what we have on disk.\n+ * This tracks any options which pack-reuse code expects to be on, or which a\n+ * reader of the pack might not understand, and which would therefore prevent\n+ * blind reuse of what we have on disk.\n */\n static int pack_options_allow_reuse(void)\n {\n-\treturn allow_ofs_delta;\n+\treturn pack_to_stdout && allow_ofs_delta;\n }\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n@@ -2818,7 +2819,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex a278d30..9602e9a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -196,6 +196,18 @@ test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n \t! has_any packbitmap.objects 3b.objects\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\tlist_packed_objects packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"293538","messageId":"xmqqmvkl90zu.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160809192858.GA25822@teco.navytux.spb.ru","subject":"Re: [PATCH 1/2] pack-objects: Teach --use-bitmap-index codepath to respect --local, --honor-pack-keep and --incremental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-09T19:49:09Z","receivedAt":"2016-08-09T19:55:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> On Tue, Aug 09, 2016 at 09:52:18AM -0700, Junio C Hamano wrote:\n>> \"touch A\" forcess the readers wonder \"does the timestamp of A\n>> matter, and if so in what way?\" and \"does any later test care what\n>> is _in_ A, and if so in what way?\" Both of them is wasting their\n>> time when there is no reason why \"touch\" should have been used. \n>\n> I see, thanks for explaining. I used to read it a bit the other way;\n\nSurely \">A\" may invite \"Hmm, is it important that A gets empty?\", so\nthe choice between the two is not so black-and-white. It just is\nthat \"touch\" has a more specific \"update the timestamp while keeping\nits contents intact\" meaning, compared to \">A\", which _could_ be\nread as \"make it empty and update its mtime\" but most people would\nnot (i.e. \"update its mtime\" is a side effect for any modification).\n\n> Ok, makes sense. Both patches adjusted and will be reposted.\n\nThanks.\n"},{"id":"299638","messageId":"20160818175222.bmm3ivjheokf2qzl@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160809193143.32213-1-kirr@nexedi.com","subject":"Re: [PATCH 1/2 v5] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-18T17:52:22Z","receivedAt":"2016-08-19T01:12:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 09, 2016 at 10:31:43PM +0300, Kirill Smelkov wrote:\n\n> Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\n> are two codepaths in pack-objects: with & without using bitmap\n> reachability index.\n\nSorry, I got distracted from reviewing these patches. I'll give them a\ndetailed look now and hopefully we can finalize the topic.\n\n> In want_object_in_pack() we care to start the checks from already found\n> pack, if we have one, this way determining the answer right away\n> in case neither --local nor --honour-pack-keep are active. In\n> particular, as p5310-pack-bitmaps.sh shows, we do not do harm to\n> served-with-bitmap clones performance-wise:\n> \n> Test 56dfeb62 this tree\n> -----------------------------------------------------------------\n> 5310.2: repack to disk 9.63(8.67+0.33) 9.47(8.55+0.28) -1.7%\n> 5310.3: simulated clone 2.07(2.17+0.12) 2.03(2.14+0.12) -1.9%\n> 5310.4: simulated fetch 0.78(1.03+0.02) 0.76(1.00+0.03) -2.6%\n> 5310.6: partial bitmap 1.97(2.43+0.15) 1.92(2.36+0.14) -2.5%\n> \n> with all differences strangely showing we are a bit faster now, but\n> probably all being within noise.\n\nGood to know there is no regression. It is curious that there is a\nslight _improvement_ across the board. Do we have an explanation for\nthat? It seems odd that noise would be so consistent.\n\n> And in the general case we care not to have duplicate\n> find_pack_entry_one(*found_pack) calls. Worst what can happen is we can\n> call want_found_object(*found_pack) -- newly introduced helper for\n> checking whether we want object -- twice, but since want_found_object()\n> is very lightweight it does not make any difference.\n\nI had trouble parsing this. I think maybe:\n\n In the general case we do not want to call find_pack_entry_one() more\n than once, because it is expensive. This patch splits the loop in\n want_object_in_pack() into two parts: finding the object and seeing if\n it impacts our choice to include it in the pack. We may call the\n inexpensive want_found_object() twice, but we will never call\n find_pack_entry_one() if we do not need to.\n\n> +static int want_found_object(int exclude, struct packed_git *p)\n> +{\n> +\tif (exclude)\n> +\t\treturn 1;\n> +\tif (incremental)\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * When asked to do --local (do not include an object that appears in a\n> +\t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n> +\t * an object that appears in a pack marked with .keep), finding a pack\n> +\t * that matches the criteria is sufficient for us to decide to omit it.\n> +\t * However, even if this pack does not satisfy the criteria, we need to\n> +\t * make sure no copy of this object appears in _any_ pack that makes us\n> +\t * to omit the object, so we need to check all the packs. Signal that by\n> +\t * returning -1 to the caller.\n> +\t */\n> +\tif (!ignore_packed_keep &&\n> +\t (!local || !have_non_local_packs))\n> +\t\treturn 1;\n\nHmm. The comment says \"-1\", but the return says \"1\". That is because the\ncomment is describing the return that happens at the end. :)\n\nI wonder if the last sentence should be:\n\n We can check here whether these options can possibly matter; if not,\n we can return early from the function here. Otherwise, we signal \"-1\"\n at the end to tell the caller that we do not know either way, and it\n needs to check more packs.\n\n> -\t*found_pack = NULL;\n> -\t*found_offset = 0;\n> +\t/*\n> +\t * If we already know the pack object lives in, start checks from that\n> +\t * pack - in the usual case when neither --local was given nor .keep files\n> +\t * are present we will determine the answer right now.\n> +\t */\n> +\tif (*found_pack) {\n> +\t\twant = want_found_object(exclude, *found_pack);\n> +\t\tif (want != -1)\n> +\t\t\treturn want;\n> +\t}\n\nLooks correct. Though it is not really \"start checks from...\" anymore,\nbut rather \"do a quick check to see if we can quit early, and otherwise\nstart the loop\". That might be nitpicking, though.\n\n> \tfor (p = packed_git; p; p = p->next) {\n> -\t\toff_t offset = find_pack_entry_one(sha1, p);\n> +\t\toff_t offset;\n> +\n> +\t\tif (p == *found_pack)\n> +\t\t\toffset = *found_offset;\n> +\t\telse\n> +\t\t\toffset = find_pack_entry_one(sha1, p);\n> +\n\nThis hunk will conflict with the MRU optimizations in 'next', but I\nthink the resolution should be pretty trivial.\n\n> static int add_object_entry(const unsigned char *sha1, enum object_type type,\n> \t\t\t const char *name, int exclude)\n> {\n> -\tstruct packed_git *found_pack;\n> -\toff_t found_offset;\n> +\tstruct packed_git *found_pack = NULL;\n> +\toff_t found_offset = 0;\n\nI think technically we don't need to initialize found_offset here (it is\nconsidered only if *found_pack is not NULL), but it doesn't hurt to make\nour starting assumptions clear.\n\n> @@ -1073,6 +1097,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> \t\treturn 0;\n> \n> +\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n> +\t\treturn 0;\n> +\n\nAnd this caller doesn't need to worry about initialization, because of\ncourse it knows it has a pack/offset already. Good.\n\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index 3893afd..a278d30 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n\nTests look OK. I saw a few style nitpicks, but I think they are not even\nagainst our style guide but more \"I would have written it like this\" and\nare not even worth quibbling over.\n\nSo I think the code here is fine, and I just had a few minor complaints\non comment and commit message clarity.\n\n-Peff\n"},{"id":"299639","messageId":"20160818180615.q25p57v35m2xxtww@sigill.intra.peff.net","threadId":"42795","inReplyTo":"20160809193217.32389-1-kirr@nexedi.com","subject":"Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when generating non-stdout pack","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-18T18:06:15Z","receivedAt":"2016-08-19T01:12:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 09, 2016 at 10:32:17PM +0300, Kirill Smelkov wrote:\n\n> Subject: Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when\n> generating non-stdout pack\n\nThis is v7, but as I understand your numbering, it goes with v5 of patch\n1/2 that I just reviewed (usually we just increment the version number\non the whole series and treat it as a unit, even if some patches didn't\nchange from version to version).\n\n> So we can teach pack-objects to use bitmap index for initial object\n> counting phase when generating resultant pack file too:\n> \n> - if we care it is not activated under git-repack:\n\nDo you mean \"if we take care that it is not...\" here?\n\n(I think you might just be getting tripped up in the English idioms;\n\"care\" means that we have a preference; \"to take care\" means that we are\nbeing careful).\n\n> - if we know bitmap index generation is not enabled for resultant pack:\n> \n> Current code has singleton bitmap_git so cannot work simultaneously\n> with two bitmap indices.\n\nMinor English fixes:\n\n The current code has a singleton bitmap_git, so it cannot work\n simultaneously with two bitmap indices.\n\n> - if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n> \n> Because on pack reuse raw entries are directly written out to destination\n> pack by write_reused_pack() bypassing needed for pack index generation\n> bookkeeping done by regular codepath in write_one() and friends.\n\nDitto on English:\n\n On pack reuse raw entries are directly written out to the destination\n pack by write_reused_pack(), bypassing the need for pack index\n generation bookkeeping done by the regular code path in write_one()\n and friends.\n\nI think this is missing the implication. Why wouldn't we want to reuse\nin this case? Certainly we don't when doing a \"careful\" on-disk repack.\nI suspect the answer is that we cannot write a \".idx\" off of the result\nof write_reused_pack(), and write-to-disk always includes the .idx.\n\n> More context:\n> \n> http://marc.info/?t=146792101400001&r=1&w=2\n\nCan we turn this into a link to public-inbox? We have just been bit by\nall of our old links to gmane dying, and they cannot easily be replaced\nbecause they use a gmane-specific article number. public-inbox URLs use\nmessage-ids, which should be usable for other archives if public-inbox\ngoes away.\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index b1007f2..c92d7fc 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n\nThe code here looks fine.\n\n> diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> index a278d30..9602e9a 100755\n> --- a/t/t5310-pack-bitmaps.sh\n> +++ b/t/t5310-pack-bitmaps.sh\n> @@ -196,6 +196,18 @@ test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n> \t! has_any packbitmap.objects 3b.objects\n> '\n> \n> +test_expect_success 'pack-objects to file can use bitmap' '\n> +\t# make sure we still have 1 bitmap index from previous tests\n> +\tls .git/objects/pack/ | grep bitmap >output &&\n> +\ttest_line_count = 1 output &&\n> +\t# verify equivalent packs are generated with/without using bitmap index\n> +\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa +\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb +\tlist_packed_objects packa.objects &&\n> +\tlist_packed_objects packb.objects &&\n> +\ttest_cmp packa.objects packb.objects\n> +'\n\nOf course we can't know if bitmaps were actually used, or if they were\nturned off under the hood. But at least this exercises the code a bit.\n\nYou could possibly add a perf test which shows off the improvement, but\nI don't think it's strictly necessary.\n\n-Peff\n"},{"id":"301598","messageId":"20160910145738.x6ihp2gqzpk7dbi3@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160818175222.bmm3ivjheokf2qzl@sigill.intra.peff.net","subject":"Re: [PATCH 1/2 v5] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-09-10T14:57:39Z","receivedAt":"2016-09-10T14:57:52Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Thu, Aug 18, 2016 at 01:52:22PM -0400, Jeff King wrote:\n> On Tue, Aug 09, 2016 at 10:31:43PM +0300, Kirill Smelkov wrote:\n> \n> > Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\n> > are two codepaths in pack-objects: with & without using bitmap\n> > reachability index.\n> \n> Sorry, I got distracted from reviewing these patches. I'll give them a\n> detailed look now and hopefully we can finalize the topic.\n\nJeff, thanks for feedback. On my side I'm sorry for the delay because I\nwas travelling and only recently got back to work.\n\n> > In want_object_in_pack() we care to start the checks from already found\n> > pack, if we have one, this way determining the answer right away\n> > in case neither --local nor --honour-pack-keep are active. In\n> > particular, as p5310-pack-bitmaps.sh shows, we do not do harm to\n> > served-with-bitmap clones performance-wise:\n> > \n> > Test 56dfeb62 this tree\n> > -----------------------------------------------------------------\n> > 5310.2: repack to disk 9.63(8.67+0.33) 9.47(8.55+0.28) -1.7%\n> > 5310.3: simulated clone 2.07(2.17+0.12) 2.03(2.14+0.12) -1.9%\n> > 5310.4: simulated fetch 0.78(1.03+0.02) 0.76(1.00+0.03) -2.6%\n> > 5310.6: partial bitmap 1.97(2.43+0.15) 1.92(2.36+0.14) -2.5%\n> > \n> > with all differences strangely showing we are a bit faster now, but\n> > probably all being within noise.\n> \n> Good to know there is no regression. It is curious that there is a\n> slight _improvement_ across the board. Do we have an explanation for\n> that? It seems odd that noise would be so consistent.\n\nYes, I too thought it and it turned out to be t/perf/run does not copy\nconfig.mak.autogen & friends to build/ and I'm using autoconf with\nCFLAGS=\"-march=native -O3 ...\"\n\nJunio, I could not resist to the following:\n\n---- 8< ----\nFrom: Kirill Smelkov \nSubject: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends\n to build area\n\nOtherwise for people who use autotools-based configure in main worktree,\nthe performance testing results will be inconsistent as work and build\ntrees could be using e.g. different optimization levels.\n\nSee e.g.\n\n\thttp://public-inbox.org/git/20160818175222.bmm3ivjheokf2qzl@sigill.intra.peff.net/\n\nfor example.\n\nNOTE config.status has to be copied because otherwise without it the build\nwould want to run reconfigure this way loosing just copied config.mak.autogen.\n\nSigned-off-by: Kirill Smelkov \n---\n t/perf/run | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/run b/t/perf/run\nindex cfd7012..aa383c2 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -30,7 +30,7 @@ unpack_git_rev () {\n }\n build_git_rev () {\n \trev=$1\n-\tcp ../../config.mak build/$rev/config.mak\n+\tcp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}\n \t(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||\n \tdie \"failed to build revision '$mydir'\"\n }\n-- \n2.9.2.701.gf965a18.dirty\n---- 8< ----\n\nWith corrected t/perf/run the timings are more realistic - e.g. 3\nconsecutive runs of `./run 56dfeb62 . ./p5310-pack-bitmaps.sh`:\n\nTest 56dfeb62 this tree\n-----------------------------------------------------------------\n5310.2: repack to disk 9.08(8.20+0.25) 9.09(8.14+0.32) +0.1%\n5310.3: simulated clone 1.92(2.12+0.08) 1.93(2.12+0.09) +0.5%\n5310.4: simulated fetch 0.82(1.07+0.04) 0.82(1.06+0.04) +0.0%\n5310.6: partial bitmap 1.96(2.42+0.13) 1.95(2.40+0.15) -0.5%\n\nTest 56dfeb62 this tree\n-----------------------------------------------------------------\n5310.2: repack to disk 9.11(8.16+0.32) 9.11(8.19+0.28) +0.0%\n5310.3: simulated clone 1.93(2.14+0.07) 1.92(2.11+0.10) -0.5%\n5310.4: simulated fetch 0.82(1.06+0.04) 0.82(1.04+0.05) +0.0%\n5310.6: partial bitmap 1.95(2.38+0.16) 1.94(2.39+0.14) -0.5%\n\nTest 56dfeb62 this tree\n-----------------------------------------------------------------\n5310.2: repack to disk 9.13(8.17+0.31) 9.07(8.13+0.28) -0.7%\n5310.3: simulated clone 1.92(2.13+0.07) 1.91(2.12+0.06) -0.5%\n5310.4: simulated fetch 0.82(1.08+0.03) 0.82(1.08+0.03) +0.0%\n5310.6: partial bitmap 1.96(2.43+0.14) 1.96(2.42+0.14) +0.0%\n\n\n\n> > And in the general case we care not to have duplicate\n> > find_pack_entry_one(*found_pack) calls. Worst what can happen is we can\n> > call want_found_object(*found_pack) -- newly introduced helper for\n> > checking whether we want object -- twice, but since want_found_object()\n> > is very lightweight it does not make any difference.\n> \n> I had trouble parsing this. I think maybe:\n> \n> In the general case we do not want to call find_pack_entry_one() more\n> than once, because it is expensive. This patch splits the loop in\n> want_object_in_pack() into two parts: finding the object and seeing if\n> it impacts our choice to include it in the pack. We may call the\n> inexpensive want_found_object() twice, but we will never call\n> find_pack_entry_one() if we do not need to.\n\nOk, thanks for the advice.\n\n> \n> > +static int want_found_object(int exclude, struct packed_git *p)\n> > +{\n> > +\tif (exclude)\n> > +\t\treturn 1;\n> > +\tif (incremental)\n> > +\t\treturn 0;\n> > +\n> > +\t/*\n> > +\t * When asked to do --local (do not include an object that appears in a\n> > +\t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n> > +\t * an object that appears in a pack marked with .keep), finding a pack\n> > +\t * that matches the criteria is sufficient for us to decide to omit it.\n> > +\t * However, even if this pack does not satisfy the criteria, we need to\n> > +\t * make sure no copy of this object appears in _any_ pack that makes us\n> > +\t * to omit the object, so we need to check all the packs. Signal that by\n> > +\t * returning -1 to the caller.\n> > +\t */\n> > +\tif (!ignore_packed_keep &&\n> > +\t (!local || !have_non_local_packs))\n> > +\t\treturn 1;\n> \n> Hmm. The comment says \"-1\", but the return says \"1\". That is because the\n> comment is describing the return that happens at the end. :)\n> \n> I wonder if the last sentence should be:\n> \n> We can check here whether these options can possibly matter; if not,\n> we can return early from the function here. Otherwise, we signal \"-1\"\n> at the end to tell the caller that we do not know either way, and it\n> needs to check more packs.\n\nThanks for the catch and hint. I've changed it to the following:\n\n\tWe can however first check whether these options can possible matter;\n\tif they do not matter we know we want the object in generated pack.\n\tOtherwise, we signal \"-1\" at the end to tell the caller that we do\n\tnot know either way, and it needs to check more packs.\n\nfull version:\n\n /*\n * When asked to do --local (do not include an object that appears in a\n * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n * an object that appears in a pack marked with .keep), finding a pack\n * that matches the criteria is sufficient for us to decide to omit it.\n * However, even if this pack does not satisfy the criteria, we need to\n * make sure no copy of this object appears in _any_ pack that makes us\n * to omit the object, so we need to check all the packs.\n *\n * We can however first check whether these options can possible matter;\n * if they do not matter we know we want the object in generated pack.\n * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n * not know either way, and it needs to check more packs.\n */\n\nHope it is ok.\n\n> > -\t*found_pack = NULL;\n> > -\t*found_offset = 0;\n> > +\t/*\n> > +\t * If we already know the pack object lives in, start checks from that\n> > +\t * pack - in the usual case when neither --local was given nor .keep files\n> > +\t * are present we will determine the answer right now.\n> > +\t */\n> > +\tif (*found_pack) {\n> > +\t\twant = want_found_object(exclude, *found_pack);\n> > +\t\tif (want != -1)\n> > +\t\t\treturn want;\n> > +\t}\n> \n> Looks correct. Though it is not really \"start checks from...\" anymore,\n> but rather \"do a quick check to see if we can quit early, and otherwise\n> start the loop\". That might be nitpicking, though.\n\nI see. Your version is ok, but to me 'start checks from ...' is a bit\nmore natural and explaining (yes, all subjective and depending on\ntaste), so if possible I'd prefer to leave it as is.\n\n> \n> > \tfor (p = packed_git; p; p = p->next) {\n> > -\t\toff_t offset = find_pack_entry_one(sha1, p);\n> > +\t\toff_t offset;\n> > +\n> > +\t\tif (p == *found_pack)\n> > +\t\t\toffset = *found_offset;\n> > +\t\telse\n> > +\t\t\toffset = find_pack_entry_one(sha1, p);\n> > +\n> \n> This hunk will conflict with the MRU optimizations in 'next', but I\n> think the resolution should be pretty trivial.\n\nYes.\n\n> > static int add_object_entry(const unsigned char *sha1, enum object_type type,\n> > \t\t\t const char *name, int exclude)\n> > {\n> > -\tstruct packed_git *found_pack;\n> > -\toff_t found_offset;\n> > +\tstruct packed_git *found_pack = NULL;\n> > +\toff_t found_offset = 0;\n> \n> I think technically we don't need to initialize found_offset here (it is\n> considered only if *found_pack is not NULL), but it doesn't hurt to make\n> our starting assumptions clear.\n\nYes, found_pack != NULL is indicator whether we have found_pack /\nfound_offset info, but it makes it much clear and defending from\nmistakes to set both found_{pack,offset} into known initial state.\n\n> > @@ -1073,6 +1097,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n> > \tif (have_duplicate_entry(sha1, 0, &index_pos))\n> > \t\treturn 0;\n> > \n> > +\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n> > +\t\treturn 0;\n> > +\n> \n> And this caller doesn't need to worry about initialization, because of\n> course it knows it has a pack/offset already. Good.\n\nYes, we have this info from bitmap walker calling us.\n\n\n> > diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> > index 3893afd..a278d30 100755\n> > --- a/t/t5310-pack-bitmaps.sh\n> > +++ b/t/t5310-pack-bitmaps.sh\n> \n> Tests look OK. I saw a few style nitpicks, but I think they are not even\n> against our style guide but more \"I would have written it like this\" and\n> are not even worth quibbling over.\n> \n> So I think the code here is fine, and I just had a few minor complaints\n> on comment and commit message clarity.\n\nThanks for feedback. Yes tastes can differ but your comments regarding\ncommit message and want_found_object() were objectively (imho) worth it\nand there I've made the adjustments.\n\nPlease expect updated patch to be send as reply to this mail.\n\nThanks again for feedback,\nKirill\n"},{"id":"301599","messageId":"20160910145925.xbbus7eck5ineika@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"20160818180615.q25p57v35m2xxtww@sigill.intra.peff.net","subject":"Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when generating non-stdout pack","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-09-10T14:59:25Z","receivedAt":"2016-09-10T14:59:33Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Thu, Aug 18, 2016 at 02:06:15PM -0400, Jeff King wrote:\n> On Tue, Aug 09, 2016 at 10:32:17PM +0300, Kirill Smelkov wrote:\n> \n> > Subject: Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when\n> > generating non-stdout pack\n> \n> This is v7, but as I understand your numbering, it goes with v5 of patch\n> 1/2 that I just reviewed (usually we just increment the version number\n> on the whole series and treat it as a unit, even if some patches didn't\n> change from version to version).\n\nThe reason those patches are having their own numbers is that they are\northogonal to each other and can be applied / rejected independently.\nSince I though Junio might want to pick them up as separate topics they\nwere versioned separately.\n\nBut ok, since now we have them considered both together, their next\nversions posted will be uniform v8.\n\n\n> > So we can teach pack-objects to use bitmap index for initial object\n> > counting phase when generating resultant pack file too:\n> > \n> > - if we care it is not activated under git-repack:\n> \n> Do you mean \"if we take care that it is not...\" here?\n> \n> (I think you might just be getting tripped up in the English idioms;\n> \"care\" means that we have a preference; \"to take care\" means that we are\n> being careful).\n\nOk, I've might have been tripped and thanks for the catch up. I've changed to\n\n\t\"if we take care to not let it be activated under git-repack\"\n\n> \n> > - if we know bitmap index generation is not enabled for resultant pack:\n> > \n> > Current code has singleton bitmap_git so cannot work simultaneously\n> > with two bitmap indices.\n> \n> Minor English fixes:\n> \n> The current code has a singleton bitmap_git, so it cannot work\n> simultaneously with two bitmap indices.\n\nok.\n\n> > - if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n> > \n> > Because on pack reuse raw entries are directly written out to destination\n> > pack by write_reused_pack() bypassing needed for pack index generation\n> > bookkeeping done by regular codepath in write_one() and friends.\n> \n> Ditto on English:\n> \n> On pack reuse raw entries are directly written out to the destination\n> pack by write_reused_pack(), bypassing the need for pack index\n> generation bookkeeping done by the regular code path in write_one()\n> and friends.\n> \n> I think this is missing the implication. Why wouldn't we want to reuse\n> in this case? Certainly we don't when doing a \"careful\" on-disk repack.\n> I suspect the answer is that we cannot write a \".idx\" off of the result\n> of write_reused_pack(), and write-to-disk always includes the .idx.\n\nYes, mentioning pack-to-file needs to generate .idx makes it more clear\nand thanks for pointing this out. I've changed this item to the\nfollowing (picking some of your English corrections):\n\n - if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because pack-to-file needs to generate index for destination pack, and\n currently on pack reuse raw entries are directly written out to the\n destination pack by write_reused_pack(), bypassing needed for pack index\n generation bookkeeping done by regular codepath in write_one() and\n friends.\n\n ( In the future we might teach pack-reuse code about cases when index\n also needs to be generated for resultant pack and remove\n pack-reuse-only-for-stdout limitation )\n\nHope it is ok.\n\n> > More context:\n> > \n> > http://marc.info/?t=146792101400001&r=1&w=2\n> \n> Can we turn this into a link to public-inbox? We have just been bit by\n> all of our old links to gmane dying, and they cannot easily be replaced\n> because they use a gmane-specific article number. public-inbox URLs use\n> message-ids, which should be usable for other archives if public-inbox\n> goes away.\n\nYes, makes sense to put msgid here. I've added\n\n\thttp://public-inbox.org/git/20160707190917.20011-1-kirr@nexedi.com/T/#t\n\n\n> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> > index b1007f2..c92d7fc 100644\n> > --- a/builtin/pack-objects.c\n> > +++ b/builtin/pack-objects.c\n> \n> The code here looks fine.\n\nThanks.\n\n> > diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\n> > index a278d30..9602e9a 100755\n> > --- a/t/t5310-pack-bitmaps.sh\n> > +++ b/t/t5310-pack-bitmaps.sh\n> > @@ -196,6 +196,18 @@ test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n> > \t! has_any packbitmap.objects 3b.objects\n> > '\n> > \n> > +test_expect_success 'pack-objects to file can use bitmap' '\n> > +\t# make sure we still have 1 bitmap index from previous tests\n> > +\tls .git/objects/pack/ | grep bitmap >output &&\n> > +\ttest_line_count = 1 output &&\n> > +\t# verify equivalent packs are generated with/without using bitmap index\n> > +\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa > +\tpackbsha1=$(git pack-objects --use-bitmap-index --all packb > +\tlist_packed_objects packa.objects &&\n> > +\tlist_packed_objects packb.objects &&\n> > +\ttest_cmp packa.objects packb.objects\n> > +'\n> \n> Of course we can't know if bitmaps were actually used, or if they were\n> turned off under the hood. But at least this exercises the code a bit.\n\nYes, I was thinking how to know the bitmap codepath was actually active,\nand without adding debugging points there is no way (at least I could\nnot find it).\n\n> You could possibly add a perf test which shows off the improvement, but\n> I don't think it's strictly necessary.\n\nGood idea. I've added this\n\n---- 8< ----\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex de2a224..bb91dbb 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -32,6 +32,14 @@ test_perf 'simulated fetch' '\n } | git pack-objects --revs --stdout >/dev/null\n '\n \n+test_perf 'pack to file' '\n+ git pack-objects --all pack1 /dev/null\n+'\n+\n+test_perf 'pack to file (bitmap)' '\n+ git pack-objects --use-bitmap-index --all pack1b /dev/null\n+'\n+\n test_expect_success 'create partial bitmap state' '\n # pick a commit to represent the repo tip in the past\n cutoff=$(git rev-list HEAD~100 -1) &&\n@@ -53,8 +61,12 @@ test_expect_success 'create partial bitmap state' '\n git update-ref HEAD $orig_tip\n '\n \n-test_perf 'partial bitmap' '\n+test_perf 'clone (partial bitmap)' '\n git pack-objects --stdout --all /dev/null\n '\n \n+test_perf 'pack to file (partial bitmap)' '\n+ git pack-objects --use-bitmap-index --all pack2b /dev/null\n+'\n+\n test_done\n---- 8< ----\n\n Test 56dfeb62 this tree\n --------------------------------------------------------------------------------\n 5310.2: repack to disk 8.98(8.05+0.29) 9.05(8.08+0.33) +0.8%\n 5310.3: simulated clone 2.02(2.27+0.09) 2.01(2.25+0.08) -0.5%\n 5310.4: simulated fetch 0.81(1.07+0.02) 0.81(1.05+0.04) +0.0%\n 5310.5: pack to file 7.58(7.04+0.28) 7.60(7.04+0.30) +0.3%\n 5310.6: pack to file (bitmap) 7.55(7.02+0.28) 3.25(2.82+0.18) -57.0%\n 5310.8: clone (partial bitmap) 1.83(2.26+0.12) 1.82(2.22+0.14) -0.5%\n 5310.9: pack to file (partial bitmap) 6.86(6.58+0.30) 2.87(2.74+0.20) -58.2%\n \n\nKirill\n"},{"id":"301600","messageId":"20160910150110.18844-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160910145738.x6ihp2gqzpk7dbi3@teco.navytux.spb.ru","subject":"[PATCH 1/2 v8] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-09-10T15:01:10Z","receivedAt":"2016-09-10T15:01:28Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Since 6b8fda2d (pack-objects: use bitmaps when packing objects) there\nare two codepaths in pack-objects: with & without using bitmap\nreachability index.\n\nHowever add_object_entry_from_bitmap(), despite its non-bitmapped\ncounterpart add_object_entry(), in no way does check for whether --local\nor --honor-pack-keep or --incremental should be respected. In\nnon-bitmapped codepath this is handled in want_object_in_pack(), but\nbitmapped codepath has simply no such checking at all.\n\nThe bitmapped codepath however was allowing to pass in all those options\nand with bitmap indices still being used under such conditions -\npotentially giving wrong output (e.g. including objects from non-local or\n.keep'ed pack).\n\nWe can easily fix this by noting the following: when an object comes to\nadd_object_entry_from_bitmap() it can come for two reasons:\n\n 1. entries coming from main pack covered by bitmap index, and\n 2. object coming from, possibly alternate, loose or other packs.\n\n\"2\" can be already handled by want_object_in_pack() and to cover\n\"1\" we can teach want_object_in_pack() to expect that *found_pack can be\nnon-NULL, meaning calling client already found object's pack entry.\n\nIn want_object_in_pack() we care to start the checks from already found\npack, if we have one, this way determining the answer right away\nin case neither --local nor --honour-pack-keep are active. In\nparticular, as p5310-pack-bitmaps.sh shows (3 consecutive runs), we do\nnot do harm to served-with-bitmap clones performance-wise:\n\n Test 56dfeb62 this tree\n -----------------------------------------------------------------\n 5310.2: repack to disk 9.08(8.20+0.25) 9.09(8.14+0.32) +0.1%\n 5310.3: simulated clone 1.92(2.12+0.08) 1.93(2.12+0.09) +0.5%\n 5310.4: simulated fetch 0.82(1.07+0.04) 0.82(1.06+0.04) +0.0%\n 5310.6: partial bitmap 1.96(2.42+0.13) 1.95(2.40+0.15) -0.5%\n\n Test 56dfeb62 this tree\n -----------------------------------------------------------------\n 5310.2: repack to disk 9.11(8.16+0.32) 9.11(8.19+0.28) +0.0%\n 5310.3: simulated clone 1.93(2.14+0.07) 1.92(2.11+0.10) -0.5%\n 5310.4: simulated fetch 0.82(1.06+0.04) 0.82(1.04+0.05) +0.0%\n 5310.6: partial bitmap 1.95(2.38+0.16) 1.94(2.39+0.14) -0.5%\n\n Test 56dfeb62 this tree\n -----------------------------------------------------------------\n 5310.2: repack to disk 9.13(8.17+0.31) 9.07(8.13+0.28) -0.7%\n 5310.3: simulated clone 1.92(2.13+0.07) 1.91(2.12+0.06) -0.5%\n 5310.4: simulated fetch 0.82(1.08+0.03) 0.82(1.08+0.03) +0.0%\n 5310.6: partial bitmap 1.96(2.43+0.14) 1.96(2.42+0.14) +0.0%\n\nwith delta timings showing they are all within noise from run to run.\n\nIn the general case we do not want to call find_pack_entry_one() more than\nonce, because it is expensive. This patch splits the loop in\nwant_object_in_pack() into two parts: finding the object and seeing if it\nimpacts our choice to include it in the pack. We may call the inexpensive\nwant_found_object() twice, but we will never call find_pack_entry_one() if we\ndo not need to.\n\nI appreciate help and discussing this change with Junio C Hamano and\nJeff King.\n\nSigned-off-by: Kirill Smelkov \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-objects.c | 97 ++++++++++++++++++++++++++++++++-----------------\n t/t5310-pack-bitmaps.sh | 92 ++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 156 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex c4c2a3c..19668d3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -944,13 +944,48 @@ static int have_duplicate_entry(const unsigned char *sha1,\n \treturn 1;\n }\n \n+static int want_found_object(int exclude, struct packed_git *p)\n+{\n+\tif (exclude)\n+\t\treturn 1;\n+\tif (incremental)\n+\t\treturn 0;\n+\n+\t/*\n+\t * When asked to do --local (do not include an object that appears in a\n+\t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n+\t * an object that appears in a pack marked with .keep), finding a pack\n+\t * that matches the criteria is sufficient for us to decide to omit it.\n+\t * However, even if this pack does not satisfy the criteria, we need to\n+\t * make sure no copy of this object appears in _any_ pack that makes us\n+\t * to omit the object, so we need to check all the packs.\n+\t *\n+\t * We can however first check whether these options can possible matter;\n+\t * if they do not matter we know we want the object in generated pack.\n+\t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n+\t * not know either way, and it needs to check more packs.\n+\t */\n+\tif (!ignore_packed_keep &&\n+\t (!local || !have_non_local_packs))\n+\t\treturn 1;\n+\n+\tif (local && !p->pack_local)\n+\t\treturn 0;\n+\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n+\t\treturn 0;\n+\n+\t/* we don't know yet; keep looking for more packs */\n+\treturn -1;\n+}\n+\n /*\n * Check whether we want the object in the pack (e.g., we do not want\n * objects found in non-local stores if the \"--local\" option was used).\n *\n- * As a side effect of this check, we will find the packed version of this\n- * object, if any. We therefore pass out the pack information to avoid having\n- * to look it up again later.\n+ * If the caller already knows an existing pack it wants to take the object\n+ * from, that is passed in *found_pack and *found_offset; otherwise this\n+ * function finds if there is any pack that has the object and returns the pack\n+ * and its offset in these variables.\n */\n static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t int exclude,\n@@ -958,15 +993,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t off_t *found_offset)\n {\n \tstruct packed_git *p;\n+\tint want;\n \n \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n \t\treturn 0;\n \n-\t*found_pack = NULL;\n-\t*found_offset = 0;\n+\t/*\n+\t * If we already know the pack object lives in, start checks from that\n+\t * pack - in the usual case when neither --local was given nor .keep files\n+\t * are present we will determine the answer right now.\n+\t */\n+\tif (*found_pack) {\n+\t\twant = want_found_object(exclude, *found_pack);\n+\t\tif (want != -1)\n+\t\t\treturn want;\n+\t}\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\toff_t offset = find_pack_entry_one(sha1, p);\n+\t\toff_t offset;\n+\n+\t\tif (p == *found_pack)\n+\t\t\toffset = *found_offset;\n+\t\telse\n+\t\t\toffset = find_pack_entry_one(sha1, p);\n+\n \t\tif (offset) {\n \t\t\tif (!*found_pack) {\n \t\t\t\tif (!is_pack_valid(p))\n@@ -974,31 +1024,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\t\t*found_offset = offset;\n \t\t\t\t*found_pack = p;\n \t\t\t}\n-\t\t\tif (exclude)\n-\t\t\t\treturn 1;\n-\t\t\tif (incremental)\n-\t\t\t\treturn 0;\n-\n-\t\t\t/*\n-\t\t\t * When asked to do --local (do not include an\n-\t\t\t * object that appears in a pack we borrow\n-\t\t\t * from elsewhere) or --honor-pack-keep (do not\n-\t\t\t * include an object that appears in a pack marked\n-\t\t\t * with .keep), we need to make sure no copy of this\n-\t\t\t * object come from in _any_ pack that causes us to\n-\t\t\t * omit it, and need to complete this loop. When\n-\t\t\t * neither option is in effect, we know the object\n-\t\t\t * we just found is going to be packed, so break\n-\t\t\t * out of the loop to return 1 now.\n-\t\t\t */\n-\t\t\tif (!ignore_packed_keep &&\n-\t\t\t (!local || !have_non_local_packs))\n-\t\t\t\tbreak;\n-\n-\t\t\tif (local && !p->pack_local)\n-\t\t\t\treturn 0;\n-\t\t\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n-\t\t\t\treturn 0;\n+\t\t\twant = want_found_object(exclude, p);\n+\t\t\tif (want != -1)\n+\t\t\t\treturn want;\n \t\t}\n \t}\n \n@@ -1039,8 +1067,8 @@ static const char no_closure_warning[] = N_(\n static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \t\t\t const char *name, int exclude)\n {\n-\tstruct packed_git *found_pack;\n-\toff_t found_offset;\n+\tstruct packed_git *found_pack = NULL;\n+\toff_t found_offset = 0;\n \tuint32_t index_pos;\n \n \tif (have_duplicate_entry(sha1, exclude, &index_pos))\n@@ -1073,6 +1101,9 @@ static int add_object_entry_from_bitmap(const unsigned char *sha1,\n \tif (have_duplicate_entry(sha1, 0, &index_pos))\n \t\treturn 0;\n \n+\tif (!want_object_in_pack(sha1, 0, &pack, &offset))\n+\t\treturn 0;\n+\n \tcreate_object_entry(sha1, type, name_hash, 0, 0, index_pos, pack, offset);\n \n \tdisplay_progress(progress_state, nr_result);\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 3893afd..a278d30 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -7,6 +7,18 @@ objpath () {\n \techo \".git/objects/$(echo \"$1\" | sed -e 's|\\(..\\)|\\1/|')\"\n }\n \n+# show objects present in pack ($1 should be associated *.idx)\n+list_packed_objects () {\n+\tgit show-index <\"$1\" | cut -d' ' -f2\n+}\n+\n+# has_any pattern-file content-file\n+# tests whether content-file has any entry from pattern-file with entries being\n+# whole lines.\n+has_any () {\n+\tgrep -Ff \"$1\" \"$2\"\n+}\n+\n test_expect_success 'setup repo with moderate-sized history' '\n \tfor i in $(test_seq 1 10); do\n \t\ttest_commit $i\n@@ -16,6 +28,7 @@ test_expect_success 'setup repo with moderate-sized history' '\n \t\ttest_commit side-$i\n \tdone &&\n \tgit checkout master &&\n+\tbitmaptip=$(git rev-parse master) &&\n \tblob=$(echo tagged-blob | git hash-object -w --stdin) &&\n \tgit tag tagged-blob $blob &&\n \tgit config repack.writebitmaps true &&\n@@ -118,6 +131,71 @@ test_expect_success 'incremental repack can disable bitmaps' '\n \tgit repack -d --no-write-bitmap-index\n '\n \n+test_expect_success 'pack-objects respects --local (non-local loose)' '\n+\tgit init --bare alt.git &&\n+\techo $(pwd)/alt.git/objects >.git/objects/info/alternates &&\n+\techo content1 >file1 &&\n+\t# non-local loose object which is not present in bitmapped pack\n+\taltblob=$(GIT_DIR=alt.git git hash-object -w file1) &&\n+\t# non-local loose object which is also present in bitmapped pack\n+\tgit cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin &&\n+\tgit add file1 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file1 &&\n+\techo HEAD | git pack-objects --local --stdout --revs >1.pack &&\n+\tgit index-pack 1.pack &&\n+\tlist_packed_objects 1.idx >1.objects &&\n+\tprintf \"%s\\n\" \"$altblob\" \"$blob\" >nonlocal-loose &&\n+\t! has_any nonlocal-loose 1.objects\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' '\n+\techo content2 >file2 &&\n+\tblob2=$(git hash-object -w file2) &&\n+\tgit add file2 &&\n+\ttest_tick &&\n+\tgit commit -m commit_file2 &&\n+\tprintf \"%s\\n\" \"$blob2\" \"$bitmaptip\" >keepobjects &&\n+\tpack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep &&\n+\trm $(objpath $blob2) &&\n+\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack &&\n+\tgit index-pack 2a.pack &&\n+\tlist_packed_objects 2a.idx >2a.objects &&\n+\t! has_any keepobjects 2a.objects\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local pack)' '\n+\tmv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ &&\n+\techo HEAD | git pack-objects --local --stdout --revs >2b.pack &&\n+\tgit index-pack 2b.pack &&\n+\tlist_packed_objects 2b.idx >2b.objects &&\n+\t! has_any keepobjects 2b.objects\n+'\n+\n+test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' '\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\tpackbitmap=$(basename $(cat output) .bitmap) &&\n+\tlist_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects &&\n+\ttest_when_finished \"rm -f .git/objects/pack/$packbitmap.keep\" &&\n+\t>.git/objects/pack/$packbitmap.keep &&\n+\techo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack &&\n+\tgit index-pack 3a.pack &&\n+\tlist_packed_objects 3a.idx >3a.objects &&\n+\t! has_any packbitmap.objects 3a.objects\n+'\n+\n+test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n+\tmv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ &&\n+\ttest_when_finished \"mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/\" &&\n+\techo HEAD | git pack-objects --local --stdout --revs >3b.pack &&\n+\tgit index-pack 3b.pack &&\n+\tlist_packed_objects 3b.idx >3b.objects &&\n+\t! has_any packbitmap.objects 3b.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n@@ -143,6 +221,20 @@ test_expect_success 'create objects for missing-HAVE tests' '\n \tEOF\n '\n \n+test_expect_success 'pack-objects respects --incremental' '\n+\tcat >revs2 <<-EOF &&\n+\tHEAD\n+\t$commit\n+\tEOF\n+\tgit pack-objects --incremental --stdout --revs 4.pack &&\n+\tgit index-pack 4.pack &&\n+\tlist_packed_objects 4.idx >4.objects &&\n+\ttest_line_count = 4 4.objects &&\n+\tgit rev-list --objects $commit >revlist &&\n+\tcut -d\" \" -f1 revlist |sort >objects &&\n+\ttest_cmp 4.objects objects\n+'\n+\n test_expect_success 'pack with missing blob' '\n \trm $(objpath $blob) &&\n \tgit pack-objects --stdout --revs /dev/null\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"301601","messageId":"20160910150144.19018-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160910145925.xbbus7eck5ineika@teco.navytux.spb.ru","subject":"[PATCH 2/2 v8] pack-objects: use reachability bitmap index when generating non-stdout pack","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-09-10T15:01:44Z","receivedAt":"2016-09-10T15:01:56Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Starting from 6b8fda2d (pack-objects: use bitmaps when packing objects)\nif a repository has bitmap index, pack-objects can nicely speedup\n\"Counting objects\" graph traversal phase. That however was done only for\ncase when resultant pack is sent to stdout, not written into a file.\n\nThe reason here is for on-disk repack by default we want:\n\n- to produce good pack (with bitmap index not-yet-packed objects are\n emitted to pack in suboptimal order).\n\n- to use more robust pack-generation codepath (avoiding possible\n bugs in bitmap code and possible bitmap index corruption).\n\nJeff King further explains:\n\n The reason for this split is that pack-objects tries to determine how\n \"careful\" it should be based on whether we are packing to disk or to\n stdout. Packing to disk implies \"git repack\", and that we will likely\n delete the old packs after finishing. We want to be more careful (so\n as not to carry forward a corruption, and to generate a more optimal\n pack), and we presumably run less frequently and can afford extra CPU.\n Whereas packing to stdout implies serving a remote via \"git fetch\" or\n \"git push\". This happens more frequently (e.g., a server handling many\n fetching clients), and we assume the receiving end takes more\n responsibility for verifying the data.\n\n But this isn't always the case. One might want to generate on-disk\n packfiles for a specialized object transfer. Just using \"--stdout\" and\n writing to a file is not optimal, as it will not generate the matching\n pack index.\n\n So it would be useful to have some way of overriding this heuristic:\n to tell pack-objects that even though it should generate on-disk\n files, it is still OK to use the reachability bitmaps to do the\n traversal.\n\nSo we can teach pack-objects to use bitmap index for initial object\ncounting phase when generating resultant pack file too:\n\n- if we take care to not let it be activated under git-repack:\n\n See above about repack robustness and not forward-carrying corruption.\n\n- if we know bitmap index generation is not enabled for resultant pack:\n\n The current code has singleton bitmap_git, so it cannot work\n simultaneously with two bitmap indices.\n\n We also want to avoid (at least with current implementation)\n generating bitmaps off of bitmaps. The reason here is: when generating\n a pack, not-yet-packed objects will be emitted into pack in\n suboptimal order and added to tail of the bitmap as \"extended entries\".\n When the resultant pack + some new objects in associated repository\n are in turn used to generate another pack with bitmap, the situation\n repeats: new objects are again not emitted optimally and just added to\n bitmap tail - not in recency order.\n\n So the pack badness can grow over time when at each step we have\n bitmapped pack + some other objects. That's why we want to avoid\n generating bitmaps off of bitmaps, not to let pack badness grow.\n\n- if we keep pack reuse enabled still only for \"send-to-stdout\" case:\n\n Because pack-to-file needs to generate index for destination pack, and\n currently on pack reuse raw entries are directly written out to the\n destination pack by write_reused_pack(), bypassing needed for pack index\n generation bookkeeping done by regular codepath in write_one() and\n friends.\n\n ( In the future we might teach pack-reuse code about cases when index\n also needs to be generated for resultant pack and remove\n pack-reuse-only-for-stdout limitation )\n\nThis way for pack-objects -> file we get nice speedup:\n\n erp5.git[1] (~230MB) extracted from ~ 5GB lab.nexedi.com backup\n repository managed by git-backup[2] via\n\n time echo 0186ac99 | git pack-objects --revs erp5pack\n\nbefore: 37.2s\nafter: 26.2s\n\nAnd for `git repack -adb` packed git.git\n\n time echo 5c589a73 | git pack-objects --revs gitpack\n\nbefore: 7.1s\nafter: 3.6s\n\ni.e. it can be 30% - 50% speedup for pack extraction.\n\ngit-backup extracts many packs on repositories restoration. That was my\ninitial motivation for the patch.\n\n[1] https://lab.nexedi.com/nexedi/erp5\n[2] https://lab.nexedi.com/kirr/git-backup\n\nNOTE\n\nJeff also suggests that pack.useBitmaps was probably a mistake to\nintroduce originally. This way we are not adding another config point,\nbut instead just always default to-file pack-objects not to use bitmap\nindex: Tools which need to generate on-disk packs with using bitmap, can\npass --use-bitmap-index explicitly. And git-repack does never pass\n--use-bitmap-index, so this way we can be sure regular on-disk repacking\nremains robust.\n\nNOTE2\n\n`git pack-objects --stdout >file.pack` + `git index-pack file.pack` is much slower\nthan `git pack-objects file.pack`. Extracting erp5.git pack from\nlab.nexedi.com backup repository:\n\n $ time echo 0186ac99 | git pack-objects --stdout --revs >erp5pack-stdout.pack\n\n real 0m22.309s\n user 0m21.148s\n sys 0m0.932s\n\n $ time git index-pack erp5pack-stdout.pack\n\n real 0m50.873s <-- more than 2 times slower than time to generate pack itself!\n user 0m49.300s\n sys 0m1.360s\n\nSo the time for\n\n `pack-object --stdout >file.pack` + `index-pack file.pack` is 72s,\n\nwhile\n\n `pack-objects file.pack` which does both pack and index is 27s.\n\nAnd even\n\n `pack-objects --no-use-bitmap-index file.pack` is 37s.\n\nJeff explains:\n\n The packfile does not carry the sha1 of the objects. A receiving\n index-pack has to compute them itself, including inflating and applying\n all of the deltas.\n\nthat's why for `git-backup restore` we want to teach `git pack-objects\nfile.pack` to use bitmaps instead of using `git pack-objects --stdout\n>file.pack` + `git index-pack file.pack`.\n\nNOTE3\n\nThe speedup is now tracked via t/perf/p5310-pack-bitmaps.sh\n\n Test 56dfeb62 this tree\n --------------------------------------------------------------------------------\n 5310.2: repack to disk 8.98(8.05+0.29) 9.05(8.08+0.33) +0.8%\n 5310.3: simulated clone 2.02(2.27+0.09) 2.01(2.25+0.08) -0.5%\n 5310.4: simulated fetch 0.81(1.07+0.02) 0.81(1.05+0.04) +0.0%\n 5310.5: pack to file 7.58(7.04+0.28) 7.60(7.04+0.30) +0.3%\n 5310.6: pack to file (bitmap) 7.55(7.02+0.28) 3.25(2.82+0.18) -57.0%\n 5310.8: clone (partial bitmap) 1.83(2.26+0.12) 1.82(2.22+0.14) -0.5%\n 5310.9: pack to file (partial bitmap) 6.86(6.58+0.30) 2.87(2.74+0.20) -58.2%\n\nMore context:\n\n http://marc.info/?t=146792101400001&r=1&w=2\n http://public-inbox.org/git/20160707190917.20011-1-kirr@nexedi.com/T/#t\n\nCc: Vicent Marti \nHelped-by: Jeff King \nSigned-off-by: Kirill Smelkov \nSigned-off-by: Junio C Hamano \n---\n builtin/pack-objects.c | 31 ++++++++++++++++++++++++-------\n t/perf/p5310-pack-bitmaps.sh | 14 +++++++++++++-\n t/t5310-pack-bitmaps.sh | 12 ++++++++++++\n 3 files changed, 49 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 19668d3..d48c290 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -67,7 +67,8 @@ static struct packed_git *reuse_packfile;\n static uint32_t reuse_packfile_objects;\n static off_t reuse_packfile_offset;\n \n-static int use_bitmap_index = 1;\n+static int use_bitmap_index_default = 1;\n+static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n@@ -2274,7 +2275,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\twrite_bitmap_options &= ~BITMAP_OPT_HASH_CACHE;\n \t}\n \tif (!strcmp(k, \"pack.usebitmaps\")) {\n-\t\tuse_bitmap_index = git_config_bool(k, v);\n+\t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2523,13 +2524,13 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n }\n \n /*\n- * This tracks any options which a reader of the pack might\n- * not understand, and which would therefore prevent blind reuse\n- * of what we have on disk.\n+ * This tracks any options which pack-reuse code expects to be on, or which a\n+ * reader of the pack might not understand, and which would therefore prevent\n+ * blind reuse of what we have on disk.\n */\n static int pack_options_allow_reuse(void)\n {\n-\treturn allow_ofs_delta;\n+\treturn pack_to_stdout && allow_ofs_delta;\n }\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n@@ -2822,7 +2823,23 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (!use_internal_rev_list || !pack_to_stdout || is_repository_shallow())\n+\t/*\n+\t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n+\t *\n+\t * - to produce good pack (with bitmap index not-yet-packed objects are\n+\t * packed in suboptimal order).\n+\t *\n+\t * - to use more robust pack-generation codepath (avoiding possible\n+\t * bugs in bitmap code and possible bitmap index corruption).\n+\t */\n+\tif (!pack_to_stdout)\n+\t\tuse_bitmap_index_default = 0;\n+\n+\tif (use_bitmap_index < 0)\n+\t\tuse_bitmap_index = use_bitmap_index_default;\n+\n+\t/* \"hard\" reasons not to use bitmaps; these just won't work at all */\n+\tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n \tif (pack_to_stdout || !rev_list_all)\ndiff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh\nindex de2a224..bb91dbb 100755\n--- a/t/perf/p5310-pack-bitmaps.sh\n+++ b/t/perf/p5310-pack-bitmaps.sh\n@@ -32,6 +32,14 @@ test_perf 'simulated fetch' '\n \t} | git pack-objects --revs --stdout >/dev/null\n '\n \n+test_perf 'pack to file' '\n+\tgit pack-objects --all pack1 /dev/null\n+'\n+\n+test_perf 'pack to file (bitmap)' '\n+\tgit pack-objects --use-bitmap-index --all pack1b /dev/null\n+'\n+\n test_expect_success 'create partial bitmap state' '\n \t# pick a commit to represent the repo tip in the past\n \tcutoff=$(git rev-list HEAD~100 -1) &&\n@@ -53,8 +61,12 @@ test_expect_success 'create partial bitmap state' '\n \tgit update-ref HEAD $orig_tip\n '\n \n-test_perf 'partial bitmap' '\n+test_perf 'clone (partial bitmap)' '\n \tgit pack-objects --stdout --all /dev/null\n '\n \n+test_perf 'pack to file (partial bitmap)' '\n+\tgit pack-objects --use-bitmap-index --all pack2b /dev/null\n+'\n+\n test_done\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex a278d30..9602e9a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -196,6 +196,18 @@ test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' '\n \t! has_any packbitmap.objects 3b.objects\n '\n \n+test_expect_success 'pack-objects to file can use bitmap' '\n+\t# make sure we still have 1 bitmap index from previous tests\n+\tls .git/objects/pack/ | grep bitmap >output &&\n+\ttest_line_count = 1 output &&\n+\t# verify equivalent packs are generated with/without using bitmap index\n+\tpackasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects &&\n+\tlist_packed_objects packb.objects &&\n+\ttest_cmp packa.objects packb.objects\n+'\n+\n test_expect_success 'full repack, reusing previous bitmaps' '\n \tgit repack -ad &&\n \tls .git/objects/pack/ | grep bitmap >output &&\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"301602","messageId":"20160910150512.19473-1-kirr@nexedi.com","threadId":"42795","inReplyTo":"20160910145738.x6ihp2gqzpk7dbi3@teco.navytux.spb.ru","subject":"[PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-09-10T15:05:12Z","receivedAt":"2016-09-10T15:05:20Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"Otherwise for people who use autotools-based configure in main worktree,\nthe performance testing results will be inconsistent as work and build\ntrees could be using e.g. different optimization levels.\n\nSee e.g.\n\n\thttp://public-inbox.org/git/20160818175222.bmm3ivjheokf2qzl@sigill.intra.peff.net/\n\nfor example.\n\nNOTE config.status has to be copied because otherwise without it the build\nwould want to run reconfigure this way loosing just copied config.mak.autogen.\n\nSigned-off-by: Kirill Smelkov \n---\n ( Resending as separate patch-mail, just in case )\n\n t/perf/run | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/perf/run b/t/perf/run\nindex cfd7012..aa383c2 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -30,7 +30,7 @@ unpack_git_rev () {\n }\n build_git_rev () {\n \trev=$1\n-\tcp ../../config.mak build/$rev/config.mak\n+\tcp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}\n \t(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||\n \tdie \"failed to build revision '$mydir'\"\n }\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"301733","messageId":"xmqqy42x2fc8.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160910145738.x6ihp2gqzpk7dbi3@teco.navytux.spb.ru","subject":"Re: [PATCH 1/2 v5] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-12T17:33:11Z","receivedAt":"2016-09-12T17:33:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> On Thu, Aug 18, 2016 at 01:52:22PM -0400, Jeff King wrote:\n> > \n> > Good to know there is no regression. It is curious that there is a\n> > slight _improvement_ across the board. Do we have an explanation for\n> > that? It seems odd that noise would be so consistent.\n>\n> Yes, I too thought it and it turned out to be t/perf/run does not copy\n> config.mak.autogen & friends to build/ and I'm using autoconf with\n> CFLAGS=\"-march=native -O3 ...\"\n>\n> Junio, I could not resist to the following:\n> ...\n> With corrected t/perf/run the timings are more realistic - e.g. 3\n> consecutive runs of `./run 56dfeb62 . ./p5310-pack-bitmaps.sh`:\n\nWow, that's what I call an exchange with quality during a review ;-)\n\nThanks for the curiosity and digging it to the root cause of the\nanomaly. Some GNUism/bashism in the way copying is spelled in the\npatch bothers me, but that is easily fixable.\n\nThanks.\n"},{"id":"301742","messageId":"xmqqh99l2aqt.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160910150512.19473-1-kirr@nexedi.com","subject":"Re: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-12T19:12:26Z","receivedAt":"2016-09-12T19:12:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> Otherwise for people who use autotools-based configure in main worktree,\n> the performance testing results will be inconsistent as work and build\n> trees could be using e.g. different optimization levels.\n>\n> See e.g.\n>\n> \thttp://public-inbox.org/git/20160818175222.bmm3ivjheokf2qzl@sigill.intra.peff.net/\n>\n> for example.\n>\n> NOTE config.status has to be copied because otherwise without it the build\n> would want to run reconfigure this way loosing just copied config.mak.autogen.\n>\n> Signed-off-by: Kirill Smelkov \n> ---\n> ( Resending as separate patch-mail, just in case )\n>\n> t/perf/run | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/t/perf/run b/t/perf/run\n> index cfd7012..aa383c2 100755\n> --- a/t/perf/run\n> +++ b/t/perf/run\n> @@ -30,7 +30,7 @@ unpack_git_rev () {\n> }\n> build_git_rev () {\n> \trev=$1\n> -\tcp ../../config.mak build/$rev/config.mak\n> +\tcp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}\n\nThat unfortunately is a GNUism -t with a bash-ism {a,b,c}; just keep\nit simple and stupid to make sure it is portable.\n\nThis is not even a part that we measure the runtime for anyway.\n\n> \t(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||\n> \tdie \"failed to build revision '$mydir'\"\n> }\n"},{"id":"301743","messageId":"xmqqd1k92aif.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"xmqqh99l2aqt.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-12T19:17:28Z","receivedAt":"2016-09-12T19:17:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> build_git_rev () {\n>> \trev=$1\n>> -\tcp ../../config.mak build/$rev/config.mak\n>> +\tcp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}\n>\n> That unfortunately is a GNUism -t with a bash-ism {a,b,c}; just keep\n> it simple and stupid to make sure it is portable.\n>\n> This is not even a part that we measure the runtime for anyway.\n\nIn other words, something along this line, perhaps.\n\n t/perf/run | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/run b/t/perf/run\nindex aa383c2..69a4714 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -30,7 +30,10 @@ unpack_git_rev () {\n }\n build_git_rev () {\n \trev=$1\n-\tcp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}\n+\tfor config in config.mak config.mak.autogen config.status\n+\tdo\n+\t\tcp \"../../$config\" \"build/$rev/\"\n+\tdone\n \t(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||\n \tdie \"failed to build revision '$mydir'\"\n }\n-- \n2.10.0-342-gc678130\n\n"},{"id":"301744","messageId":"xmqq8tuw3ow0.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160910145925.xbbus7eck5ineika@teco.navytux.spb.ru","subject":"Re: [PATCH 2/2 v7] pack-objects: use reachability bitmap index when generating non-stdout pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-12T19:21:35Z","receivedAt":"2016-09-12T19:21:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n>> This is v7, but as I understand your numbering, it goes with v5 of patch\n>> 1/2 that I just reviewed (usually we just increment the version number\n>> on the whole series and treat it as a unit, even if some patches didn't\n>> change from version to version).\n>\n> The reason those patches are having their own numbers is that they are\n> orthogonal to each other and can be applied / rejected independently.\n\nIn such a case, we wouldn't label them 1/2 and 2/2, which tells the\nreaders that these are two pieces that are to be applied together to\nform a single unit of change. That was what these numbered patches\nwith different version numbers confusing.\n\n> But ok, since now we have them considered both together, their next\n> versions posted will be uniform v8.\n\nOK. Thanks for clarifying.\n"},{"id":"301779","messageId":"xmqqpoo8yasu.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"xmqqd1k92aif.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-12T23:10:09Z","receivedAt":"2016-09-12T23:10:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> In other words, something along this line, perhaps.\n> ...\n\nNot quite. There is no guanratee that the user is using autoconf at\nall. It should be more like this, I think.\n\n t/perf/run | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/run b/t/perf/run\nindex aa383c2..7ec3734 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -30,7 +30,13 @@ unpack_git_rev () {\n }\n build_git_rev () {\n \trev=$1\n-\tcp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}\n+\tfor config in config.mak config.mak.autogen config.status\n+\tdo\n+\t\tif test -f \"../../$config\"\n+\t\tthen\n+\t\t\tcp \"../../$config\" \"build/$rev/\"\n+\t\tfi\n+\tdone\n \t(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||\n \tdie \"failed to build revision '$mydir'\"\n }\n\n\n"},{"id":"301835","messageId":"xmqqd1k8wc6h.fsf@gitster.mtv.corp.google.com","threadId":"42795","inReplyTo":"20160910150110.18844-1-kirr@nexedi.com","subject":"Re: [PATCH 1/2 v8] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-13T06:23:18Z","receivedAt":"2016-09-13T06:23:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kirill Smelkov writes:\n\n> +static int want_found_object(int exclude, struct packed_git *p)\n> +{\n> +\tif (exclude)\n> +\t\treturn 1;\n> +\tif (incremental)\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * When asked to do --local (do not include an object that appears in a\n> +\t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n> +\t * an object that appears in a pack marked with .keep), finding a pack\n> +\t * that matches the criteria is sufficient for us to decide to omit it.\n> +\t * However, even if this pack does not satisfy the criteria, we need to\n> +\t * make sure no copy of this object appears in _any_ pack that makes us\n> +\t * to omit the object, so we need to check all the packs.\n> +\t *\n> +\t * We can however first check whether these options can possible matter;\n> +\t * if they do not matter we know we want the object in generated pack.\n> +\t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n> +\t * not know either way, and it needs to check more packs.\n> +\t */\n> +\tif (!ignore_packed_keep &&\n> +\t (!local || !have_non_local_packs))\n> +\t\treturn 1;\n> +\n> +\tif (local && !p->pack_local)\n> +\t\treturn 0;\n> +\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n> +\t\treturn 0;\n> +\n> +\t/* we don't know yet; keep looking for more packs */\n> +\treturn -1;\n> +}\n\nMoving this logic out to this helper made the main logic in the\ncaller easier to grasp.\n\n> @@ -958,15 +993,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n> \t\t\t off_t *found_offset)\n> {\n> \tstruct packed_git *p;\n> +\tint want;\n> \n> \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n> \t\treturn 0;\n> \n> +\t/*\n> +\t * If we already know the pack object lives in, start checks from that\n> +\t * pack - in the usual case when neither --local was given nor .keep files\n> +\t * are present we will determine the answer right now.\n> +\t */\n> +\tif (*found_pack) {\n> +\t\twant = want_found_object(exclude, *found_pack);\n> +\t\tif (want != -1)\n> +\t\t\treturn want;\n> +\t}\n> \n> \tfor (p = packed_git; p; p = p->next) {\n> +\t\toff_t offset;\n> +\n> +\t\tif (p == *found_pack)\n> +\t\t\toffset = *found_offset;\n> +\t\telse\n> +\t\t\toffset = find_pack_entry_one(sha1, p);\n> +\n> \t\tif (offset) {\n> \t\t\tif (!*found_pack) {\n> \t\t\t\tif (!is_pack_valid(p))\n> @@ -974,31 +1024,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n> \t\t\t\t*found_offset = offset;\n> \t\t\t\t*found_pack = p;\n> \t\t\t}\n> +\t\t\twant = want_found_object(exclude, p);\n> +\t\t\tif (want != -1)\n> +\t\t\t\treturn want;\n> \t\t}\n> \t}\n\nAs Peff noted in his earlier review, however, MRU code needed to be\ngrafted in to the caller (an update to the MRU list was done in the\ncode that was moved to the want_found_object() helper). I think I\ndid it correctly, which ended up looking like this:\n\n want = want_found_object(exclude, p);\n if (!exclude && want > 0)\n mru_mark(packed_git_mru, entry);\n if (want != -1)\n return want;\n\nI somewhat feel that it is ugly that the helper knows about exclude\n(i.e. in the original code, we immediately returned 1 without\nfutzing with the MRU when we find an entry that is to be excluded,\nwhich now is done in the helper), and the caller also knows about\nexclude (i.e. the caller knows that the helper may return positive\nin two cases, it knows that MRU marking needs to happen only one of\nthe two cases, and it also knows that \"exclude\" is what\ndifferentiates between the two cases) at the same time.\n\nBut probably the reason why I feel it ugly is only because I knew\nhow the original looked like. I dunno.\n\n\n\n"},{"id":"301836","messageId":"20160913065800.cpye2rqgizszgsuc@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqqpoo8yasu.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends to build area","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-09-13T06:58:00Z","receivedAt":"2016-09-13T06:58:14Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Mon, Sep 12, 2016 at 04:10:09PM -0700, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > In other words, something along this line, perhaps.\n> > ...\n> \n> Not quite. There is no guanratee that the user is using autoconf at\n> all. It should be more like this, I think.\n> \n> t/perf/run | 8 +++++++-\n> 1 file changed, 7 insertions(+), 1 deletion(-)\n> \n> diff --git a/t/perf/run b/t/perf/run\n> index aa383c2..7ec3734 100755\n> --- a/t/perf/run\n> +++ b/t/perf/run\n> @@ -30,7 +30,13 @@ unpack_git_rev () {\n> }\n> build_git_rev () {\n> \trev=$1\n> -\tcp -t build/$rev ../../{config.mak,config.mak.autogen,config.status}\n> +\tfor config in config.mak config.mak.autogen config.status\n> +\tdo\n> +\t\tif test -f \"../../$config\"\n> +\t\tthen\n> +\t\t\tcp \"../../$config\" \"build/$rev/\"\n> +\t\tfi\n> +\tdone\n> \t(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||\n> \tdie \"failed to build revision '$mydir'\"\n> }\n\nJunio, thanks for encouraging feedback and for catching the *-isms. What\nyou propose is good (and we also automatically fix error when there was\nno config.mak - it was working but cp was giving an error to stderr but\nscript was continuing normally).\n\nI would amend your squash the following way:\n\n* `test -f` -> `test -e`, because -f tests whether a file exists _and_\n is regular file. Some people might have config.mak as a symlink for\n example. We don't want to miss them too.\n\nPlease find updated patch below:\n\n---- 8< ----\nFrom: Kirill Smelkov \nSubject: [PATCH] t/perf/run: Don't forget to copy config.mak.autogen & friends\n to build area\n\nOtherwise for people who use autotools-based configure in main worktree,\nthe performance testing results will be inconsistent as work and build\ntrees could be using e.g. different optimization levels.\n\nSee e.g.\n\n\thttp://public-inbox.org/git/20160818175222.bmm3ivjheokf2qzl@sigill.intra.peff.net/\n\nfor example.\n\nNOTE config.status has to be copied because otherwise without it the build\nwould want to run reconfigure this way loosing just copied config.mak.autogen.\n\nSigned-off-by: Kirill Smelkov \n---\n t/perf/run | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/run b/t/perf/run\nindex cfd7012..e8adeda 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -30,7 +30,13 @@ unpack_git_rev () {\n }\n build_git_rev () {\n \trev=$1\n-\tcp ../../config.mak build/$rev/config.mak\n+\tfor config in config.mak config.mak.autogen config.status\n+\tdo\n+\t\tif test -e \"../../$config\"\n+\t\tthen\n+\t\t\tcp \"../../$config\" \"build/$rev/\"\n+\t\tfi\n+\tdone\n \t(cd build/$rev && make $GIT_PERF_MAKE_OPTS) ||\n \tdie \"failed to build revision '$mydir'\"\n }\n-- \n2.9.2.701.gf965a18.dirty\n"},{"id":"301837","messageId":"20160913075004.xlteg44fgtemkcwh@teco.navytux.spb.ru","threadId":"42795","inReplyTo":"xmqqd1k8wc6h.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/2 v8] pack-objects: respect --local/--honor-pack-keep/--incremental when bitmap is in use","fromName":"Kirill Smelkov","fromEmail":"kirr@nexedi.com","sentAt":"2016-09-13T07:50:05Z","receivedAt":"2016-09-13T07:50:13Z","isPatch":true,"sender":{"key":"kirr@nexedi.com","avatar":"https://avatars.githubusercontent.com/u/1794248?v=4"},"body":"On Mon, Sep 12, 2016 at 11:23:18PM -0700, Junio C Hamano wrote:\n> Kirill Smelkov writes:\n> \n> > +static int want_found_object(int exclude, struct packed_git *p)\n> > +{\n> > +\tif (exclude)\n> > +\t\treturn 1;\n> > +\tif (incremental)\n> > +\t\treturn 0;\n> > +\n> > +\t/*\n> > +\t * When asked to do --local (do not include an object that appears in a\n> > +\t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n> > +\t * an object that appears in a pack marked with .keep), finding a pack\n> > +\t * that matches the criteria is sufficient for us to decide to omit it.\n> > +\t * However, even if this pack does not satisfy the criteria, we need to\n> > +\t * make sure no copy of this object appears in _any_ pack that makes us\n> > +\t * to omit the object, so we need to check all the packs.\n> > +\t *\n> > +\t * We can however first check whether these options can possible matter;\n> > +\t * if they do not matter we know we want the object in generated pack.\n> > +\t * Otherwise, we signal \"-1\" at the end to tell the caller that we do\n> > +\t * not know either way, and it needs to check more packs.\n> > +\t */\n> > +\tif (!ignore_packed_keep &&\n> > +\t (!local || !have_non_local_packs))\n> > +\t\treturn 1;\n> > +\n> > +\tif (local && !p->pack_local)\n> > +\t\treturn 0;\n> > +\tif (ignore_packed_keep && p->pack_local && p->pack_keep)\n> > +\t\treturn 0;\n> > +\n> > +\t/* we don't know yet; keep looking for more packs */\n> > +\treturn -1;\n> > +}\n> \n> Moving this logic out to this helper made the main logic in the\n> caller easier to grasp.\n> \n> > @@ -958,15 +993,30 @@ static int want_object_in_pack(const unsigned char *sha1,\n> > \t\t\t off_t *found_offset)\n> > {\n> > \tstruct packed_git *p;\n> > +\tint want;\n> > \n> > \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n> > \t\treturn 0;\n> > \n> > +\t/*\n> > +\t * If we already know the pack object lives in, start checks from that\n> > +\t * pack - in the usual case when neither --local was given nor .keep files\n> > +\t * are present we will determine the answer right now.\n> > +\t */\n> > +\tif (*found_pack) {\n> > +\t\twant = want_found_object(exclude, *found_pack);\n> > +\t\tif (want != -1)\n> > +\t\t\treturn want;\n> > +\t}\n> > \n> > \tfor (p = packed_git; p; p = p->next) {\n> > +\t\toff_t offset;\n> > +\n> > +\t\tif (p == *found_pack)\n> > +\t\t\toffset = *found_offset;\n> > +\t\telse\n> > +\t\t\toffset = find_pack_entry_one(sha1, p);\n> > +\n> > \t\tif (offset) {\n> > \t\t\tif (!*found_pack) {\n> > \t\t\t\tif (!is_pack_valid(p))\n> > @@ -974,31 +1024,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n> > \t\t\t\t*found_offset = offset;\n> > \t\t\t\t*found_pack = p;\n> > \t\t\t}\n> > +\t\t\twant = want_found_object(exclude, p);\n> > +\t\t\tif (want != -1)\n> > +\t\t\t\treturn want;\n> > \t\t}\n> > \t}\n> \n> As Peff noted in his earlier review, however, MRU code needed to be\n> grafted in to the caller (an update to the MRU list was done in the\n> code that was moved to the want_found_object() helper). I think I\n> did it correctly, which ended up looking like this:\n> \n> want = want_found_object(exclude, p);\n> if (!exclude && want > 0)\n> mru_mark(packed_git_mru, entry);\n> if (want != -1)\n> return want;\n> \n> I somewhat feel that it is ugly that the helper knows about exclude\n> (i.e. in the original code, we immediately returned 1 without\n> futzing with the MRU when we find an entry that is to be excluded,\n> which now is done in the helper), and the caller also knows about\n> exclude (i.e. the caller knows that the helper may return positive\n> in two cases, it knows that MRU marking needs to happen only one of\n> the two cases, and it also knows that \"exclude\" is what\n> differentiates between the two cases) at the same time.\n> \n> But probably the reason why I feel it ugly is only because I knew\n> how the original looked like. I dunno.\n\nJunio, the code above is correct semantic merge of pack-mru and my\ntopic, because in pack-mru if found and exclude=1, 1 was returned\nwithout marking found pack.\n\nBut I wonder: even if we exclude an object, we were still looking for it\nin packs, and when we found it, we found the corresponding pack too. So,\nthat pack _was_ most-recently-used, and it is correct to mark it as MRU.\n\nWe can do the simplification in the follow-up patch after the merge, so\nmerge does not change semantics and it is all bisectable, etc.\n\nJeff?\n"}]}