{"thread":{"id":"49479","subject":"We should add a \"git gc --auto\" after \"git clone\" due to commit graph","startedAt":"2018-10-03T13:24:02Z","lastAt":"2018-10-18T02:01:54Z","messageCount":78,"participants":["Ævar Arnfjörð Bjarmason","SZEDER Gábor","Derrick Stolee","Duy Nguyen","Stefan Beller","Jeff King","Junio C Hamano","Jonathan Tan"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"359518","messageId":"87tvm3go42.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":null,"subject":"We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-03T13:23:57Z","receivedAt":"2018-10-03T13:24:02Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Don't have time to patch this now, but thought I'd send a note / RFC\nabout this.\n\nNow that we have the commit graph it's nice to be able to set\ne.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n/etc/gitconfig to apply them to all repos.\n\nBut when I clone e.g. linux.git stuff like 'tag --contains' will be slow\nuntil whenever my first \"gc\" kicks in, which may be quite some time if\nI'm just using it passively.\n\nSo we should make \"git gc --auto\" be run on clone, and change the\nneed_to_gc() / cmd_gc() behavior so that we detect that the\ngc.writeCommitGraph=true setting is on, but we have no commit graph, and\nthen just generate that without doing a full repack.\n\nAs an aside such more granular \"gc\" would be nice for e.g. pack-refs\ntoo. It's possible for us to just have one pack, but to have 100k loose\nrefs.\n\nIt might also be good to have some gc.autoDetachOnClone option and have\nit false by default, so we don't have a race condition where \"clone\nlinux && git -C linux tag --contains\" is slow because the graph hasn't\nbeen generated yet, and generating the graph initially doesn't take that\nlong compared to the time to clone a large repo (and on a small one it\nwon't matter either way).\n\nI was going to say \"also for midx\", but of course after clone we have\njust one pack, so I can't imagine us needing this. But I can see us\nhaving other such optional side-indexes in the future generated by gc,\nand they'd also benefit from this.\n\n#leftoverbits\n"},{"id":"359519","messageId":"20181003133650.GN23446@localhost","threadId":"49479","inReplyTo":"87tvm3go42.fsf@evledraar.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-03T13:36:50Z","receivedAt":"2018-10-03T13:36:56Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Oct 03, 2018 at 03:23:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Don't have time to patch this now, but thought I'd send a note / RFC\n> about this.\n> \n> Now that we have the commit graph it's nice to be able to set\n> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n> /etc/gitconfig to apply them to all repos.\n> \n> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n> until whenever my first \"gc\" kicks in, which may be quite some time if\n> I'm just using it passively.\n> \n> So we should make \"git gc --auto\" be run on clone,\n\nThere is no garbage after 'git clone'...\n\n> and change the\n> need_to_gc() / cmd_gc() behavior so that we detect that the\n> gc.writeCommitGraph=true setting is on, but we have no commit graph, and\n> then just generate that without doing a full repack.\n\nOr just teach 'git clone' to run 'git commit-graph write ...'\n\n> As an aside such more granular \"gc\" would be nice for e.g. pack-refs\n> too. It's possible for us to just have one pack, but to have 100k loose\n> refs.\n> \n> It might also be good to have some gc.autoDetachOnClone option and have\n> it false by default, so we don't have a race condition where \"clone\n> linux && git -C linux tag --contains\" is slow because the graph hasn't\n> been generated yet, and generating the graph initially doesn't take that\n> long compared to the time to clone a large repo (and on a small one it\n> won't matter either way).\n> \n> I was going to say \"also for midx\", but of course after clone we have\n> just one pack, so I can't imagine us needing this. But I can see us\n> having other such optional side-indexes in the future generated by gc,\n> and they'd also benefit from this.\n> \n> #leftoverbits\n"},{"id":"359520","messageId":"836a8d28-0879-73a5-7e0b-75f438ff6785@gmail.com","threadId":"49479","inReplyTo":"20181003133650.GN23446@localhost","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-03T13:42:55Z","receivedAt":"2018-10-03T13:42:59Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/3/2018 9:36 AM, SZEDER Gábor wrote:\n> On Wed, Oct 03, 2018 at 03:23:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Don't have time to patch this now, but thought I'd send a note / RFC\n>> about this.\n>>\n>> Now that we have the commit graph it's nice to be able to set\n>> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n>> /etc/gitconfig to apply them to all repos.\n>>\n>> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n>> until whenever my first \"gc\" kicks in, which may be quite some time if\n>> I'm just using it passively.\n>>\n>> So we should make \"git gc --auto\" be run on clone,\n> There is no garbage after 'git clone'...\n\nAnd since there is no garbage, the gc will not write the commit-graph.\n\n>\n>> and change the\n>> need_to_gc() / cmd_gc() behavior so that we detect that the\n>> gc.writeCommitGraph=true setting is on, but we have no commit graph, and\n>> then just generate that without doing a full repack.\n> Or just teach 'git clone' to run 'git commit-graph write ...'\n\nI plan to add a 'fetch.writeCommitGraph' config setting. I was waiting \nuntil the file is incremental (on my to-do list soon), so the write is \nfast when only adding a few commits at a time. This would cover the \nclone case, too.\n\nThanks,\n-Stolee\n"},{"id":"359522","messageId":"87r2h7gmd7.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"20181003133650.GN23446@localhost","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-03T14:01:40Z","receivedAt":"2018-10-03T14:01:45Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 03 2018, SZEDER Gábor wrote:\n\n> On Wed, Oct 03, 2018 at 03:23:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Don't have time to patch this now, but thought I'd send a note / RFC\n>> about this.\n>>\n>> Now that we have the commit graph it's nice to be able to set\n>> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n>> /etc/gitconfig to apply them to all repos.\n>>\n>> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n>> until whenever my first \"gc\" kicks in, which may be quite some time if\n>> I'm just using it passively.\n>>\n>> So we should make \"git gc --auto\" be run on clone,\n>\n> There is no garbage after 'git clone'...\n\n\"git gc\" is really \"git gc-or-create-indexes\" these days.\n\n>> and change the\n>> need_to_gc() / cmd_gc() behavior so that we detect that the\n>> gc.writeCommitGraph=true setting is on, but we have no commit graph, and\n>> then just generate that without doing a full repack.\n>\n> Or just teach 'git clone' to run 'git commit-graph write ...'\n\nThen when adding something like the commit graph we'd need to patch both\ngit-clone and git-gc, it's much more straightforward to make\nneed_to_gc() more granular.\n\n>> As an aside such more granular \"gc\" would be nice for e.g. pack-refs\n>> too. It's possible for us to just have one pack, but to have 100k loose\n>> refs.\n>>\n>> It might also be good to have some gc.autoDetachOnClone option and have\n>> it false by default, so we don't have a race condition where \"clone\n>> linux && git -C linux tag --contains\" is slow because the graph hasn't\n>> been generated yet, and generating the graph initially doesn't take that\n>> long compared to the time to clone a large repo (and on a small one it\n>> won't matter either way).\n>>\n>> I was going to say \"also for midx\", but of course after clone we have\n>> just one pack, so I can't imagine us needing this. But I can see us\n>> having other such optional side-indexes in the future generated by gc,\n>> and they'd also benefit from this.\n>>\n>> #leftoverbits\n"},{"id":"359524","messageId":"20181003141732.GO23446@localhost","threadId":"49479","inReplyTo":"87r2h7gmd7.fsf@evledraar.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-03T14:17:32Z","receivedAt":"2018-10-03T14:17:39Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Oct 03, 2018 at 04:01:40PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Oct 03 2018, SZEDER Gábor wrote:\n> \n> > On Wed, Oct 03, 2018 at 03:23:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >> Don't have time to patch this now, but thought I'd send a note / RFC\n> >> about this.\n> >>\n> >> Now that we have the commit graph it's nice to be able to set\n> >> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n> >> /etc/gitconfig to apply them to all repos.\n> >>\n> >> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n> >> until whenever my first \"gc\" kicks in, which may be quite some time if\n> >> I'm just using it passively.\n> >>\n> >> So we should make \"git gc --auto\" be run on clone,\n> >\n> > There is no garbage after 'git clone'...\n> \n> \"git gc\" is really \"git gc-or-create-indexes\" these days.\n\nBecause it happens to be convenient to create those indexes at\ngc-time.  But that should not be an excuse to run gc when by\ndefinition no gc is needed.\n\n> >> and change the\n> >> need_to_gc() / cmd_gc() behavior so that we detect that the\n> >> gc.writeCommitGraph=true setting is on, but we have no commit graph, and\n> >> then just generate that without doing a full repack.\n> >\n> > Or just teach 'git clone' to run 'git commit-graph write ...'\n> \n> Then when adding something like the commit graph we'd need to patch both\n> git-clone and git-gc, it's much more straightforward to make\n> need_to_gc() more granular.\n> \n> >> As an aside such more granular \"gc\" would be nice for e.g. pack-refs\n> >> too. It's possible for us to just have one pack, but to have 100k loose\n> >> refs.\n> >>\n> >> It might also be good to have some gc.autoDetachOnClone option and have\n> >> it false by default, so we don't have a race condition where \"clone\n> >> linux && git -C linux tag --contains\" is slow because the graph hasn't\n> >> been generated yet, and generating the graph initially doesn't take that\n> >> long compared to the time to clone a large repo (and on a small one it\n> >> won't matter either way).\n> >>\n> >> I was going to say \"also for midx\", but of course after clone we have\n> >> just one pack, so I can't imagine us needing this. But I can see us\n> >> having other such optional side-indexes in the future generated by gc,\n> >> and they'd also benefit from this.\n> >>\n> >> #leftoverbits\n"},{"id":"359525","messageId":"87pnwrgll2.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"836a8d28-0879-73a5-7e0b-75f438ff6785@gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-03T14:18:33Z","receivedAt":"2018-10-03T14:18:39Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 03 2018, Derrick Stolee wrote:\n\n> On 10/3/2018 9:36 AM, SZEDER Gábor wrote:\n>> On Wed, Oct 03, 2018 at 03:23:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>> Don't have time to patch this now, but thought I'd send a note / RFC\n>>> about this.\n>>>\n>>> Now that we have the commit graph it's nice to be able to set\n>>> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n>>> /etc/gitconfig to apply them to all repos.\n>>>\n>>> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n>>> until whenever my first \"gc\" kicks in, which may be quite some time if\n>>> I'm just using it passively.\n>>>\n>>> So we should make \"git gc --auto\" be run on clone,\n>> There is no garbage after 'git clone'...\n>\n> And since there is no garbage, the gc will not write the commit-graph.\n\nI should probably have replied to this instead of SZEDER's in\nhttps://public-inbox.org/git/87r2h7gmd7.fsf@evledraar.gmail.com/ anyway\nmy 0.02 on that there.\n\n>>\n>>> and change the\n>>> need_to_gc() / cmd_gc() behavior so that we detect that the\n>>> gc.writeCommitGraph=true setting is on, but we have no commit graph, and\n>>> then just generate that without doing a full repack.\n>> Or just teach 'git clone' to run 'git commit-graph write ...'\n>\n> I plan to add a 'fetch.writeCommitGraph' config setting. I was waiting\n> until the file is incremental (on my to-do list soon), so the write is\n> fast when only adding a few commits at a time. This would cover the\n> clone case, too.\n\nIt's re-arranging deck chairs on the Titanic at this point, but this\napproach seems like the wrong way to go in this whole \"do we have crap\nto do?\" git-gc state-machine.\n\nIn my mind we should have only one entry point into that, and there\nshouldn't be magic like \"here's the gc-ish stuff we do on\nfetch\". Because if we care about a bunch of new commits being added on\n\"fetch\", that can also happen on \"commit\", \"am\", \"merge\", all of which\nrun \"gc --auto\" now.\n\nWhich is why I'm suggesting that we could add a sub-mode in need_to_gc()\nthat detects if a file we want to generate is entirely missing, which is\nextendable to future formats, and the only caveat at that point is if\nwe'd like that subset of \"gc\" to block and run in the foreground in the\n\"clone\" (or \"fetch\", ...) case.\n\nAnd then if we have a desire to incrementally add recently added commits\nto such formats, \"gc --auto\" could learn to consume reflogs or some\nother general inventory of \"stuff added since last gc\", and then we\nwouldn't have to instrument \"fetch\" specifically, the same would work\nfor \"commit\", \"am\", \"merge\" etc.\n"},{"id":"359526","messageId":"87o9cbglez.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"20181003141732.GO23446@localhost","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-03T14:22:12Z","receivedAt":"2018-10-03T14:22:18Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 03 2018, SZEDER Gábor wrote:\n\n> On Wed, Oct 03, 2018 at 04:01:40PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Wed, Oct 03 2018, SZEDER Gábor wrote:\n>>\n>> > On Wed, Oct 03, 2018 at 03:23:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> >> Don't have time to patch this now, but thought I'd send a note / RFC\n>> >> about this.\n>> >>\n>> >> Now that we have the commit graph it's nice to be able to set\n>> >> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n>> >> /etc/gitconfig to apply them to all repos.\n>> >>\n>> >> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n>> >> until whenever my first \"gc\" kicks in, which may be quite some time if\n>> >> I'm just using it passively.\n>> >>\n>> >> So we should make \"git gc --auto\" be run on clone,\n>> >\n>> > There is no garbage after 'git clone'...\n>>\n>> \"git gc\" is really \"git gc-or-create-indexes\" these days.\n>\n> Because it happens to be convenient to create those indexes at\n> gc-time.  But that should not be an excuse to run gc when by\n> definition no gc is needed.\n\nAh, I thought you just had an objection to the \"gc\" name being used for\nnon-gc stuff, but if you mean we shouldn't do a giant repack right after\nclone I agree. I meant that \"gc --auto\" would learn to do a subset of\nits work, instead of the current \"I have work to do, let's do all of\npack-refs/repack/commit-graph etc.\".\n\nSo we wouldn't be spending 5 minutes repacking linux.git right after\ncloning it, just ~10s generating the commit graph, and the same would\nhappen if you rm'd .git/objects/info/commit-graph and ran \"git commit\",\nwhich would kick of \"gc --auto\" in the background and do the same thing.\n"},{"id":"359527","messageId":"CACsJy8Dan1vRKYceBDVktC6FuzEiF4eHm+jE4gx09sTLL0ykyA@mail.gmail.com","threadId":"49479","inReplyTo":"87r2h7gmd7.fsf@evledraar.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-10-03T14:32:48Z","receivedAt":"2018-10-03T14:33:18Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Oct 3, 2018 at 4:01 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> >> and change the\n> >> need_to_gc() / cmd_gc() behavior so that we detect that the\n> >> gc.writeCommitGraph=true setting is on, but we have no commit graph, and\n> >> then just generate that without doing a full repack.\n> >\n> > Or just teach 'git clone' to run 'git commit-graph write ...'\n>\n> Then when adding something like the commit graph we'd need to patch both\n> git-clone and git-gc, it's much more straightforward to make\n> need_to_gc() more granular.\n\nIt is straightforward and misleading. If we organize the code well,\npatching both would not take much more effort and it reduces wtf?\nmoments reading the code.\n-- \nDuy\n"},{"id":"359528","messageId":"20181003145308.GP23446@localhost","threadId":"49479","inReplyTo":"87o9cbglez.fsf@evledraar.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-03T14:53:08Z","receivedAt":"2018-10-03T14:53:16Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Oct 03, 2018 at 04:22:12PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Oct 03 2018, SZEDER Gábor wrote:\n> \n> > On Wed, Oct 03, 2018 at 04:01:40PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >>\n> >> On Wed, Oct 03 2018, SZEDER Gábor wrote:\n> >>\n> >> > On Wed, Oct 03, 2018 at 03:23:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >> >> Don't have time to patch this now, but thought I'd send a note / RFC\n> >> >> about this.\n> >> >>\n> >> >> Now that we have the commit graph it's nice to be able to set\n> >> >> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n> >> >> /etc/gitconfig to apply them to all repos.\n> >> >>\n> >> >> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n> >> >> until whenever my first \"gc\" kicks in, which may be quite some time if\n> >> >> I'm just using it passively.\n> >> >>\n> >> >> So we should make \"git gc --auto\" be run on clone,\n> >> >\n> >> > There is no garbage after 'git clone'...\n> >>\n> >> \"git gc\" is really \"git gc-or-create-indexes\" these days.\n> >\n> > Because it happens to be convenient to create those indexes at\n> > gc-time.  But that should not be an excuse to run gc when by\n> > definition no gc is needed.\n> \n> Ah, I thought you just had an objection to the \"gc\" name being used for\n> non-gc stuff,\n\nBut you thought right, I do have an objection against that.  'git gc'\nshould, well, collect garbage.  Any non-gc stuff is already violating\nseparation of concerns.\n\n>  but if you mean we shouldn't do a giant repack right after\n> clone I agree.\n\nAnd, I also mean that since 'git clone' knows that there can't\npossibly be any garbage in the first place, then it shouldn't call 'gc\n--auto' at all.  However, since it also knows that there is a lot of\nnew stuff, then it should create a commit-graph if enabled.\n\n> I meant that \"gc --auto\" would learn to do a subset of\n> its work, instead of the current \"I have work to do, let's do all of\n> pack-refs/repack/commit-graph etc.\".\n> \n> So we wouldn't be spending 5 minutes repacking linux.git right after\n> cloning it, just ~10s generating the commit graph, and the same would\n> happen if you rm'd .git/objects/info/commit-graph and ran \"git commit\",\n> which would kick of \"gc --auto\" in the background and do the same thing.\n"},{"id":"359532","messageId":"87murvgir6.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"20181003145308.GP23446@localhost","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-03T15:19:41Z","receivedAt":"2018-10-03T15:19:47Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 03 2018, SZEDER Gábor wrote:\n\n> On Wed, Oct 03, 2018 at 04:22:12PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Wed, Oct 03 2018, SZEDER Gábor wrote:\n>>\n>> > On Wed, Oct 03, 2018 at 04:01:40PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> >>\n>> >> On Wed, Oct 03 2018, SZEDER Gábor wrote:\n>> >>\n>> >> > On Wed, Oct 03, 2018 at 03:23:57PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> >> >> Don't have time to patch this now, but thought I'd send a note / RFC\n>> >> >> about this.\n>> >> >>\n>> >> >> Now that we have the commit graph it's nice to be able to set\n>> >> >> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n>> >> >> /etc/gitconfig to apply them to all repos.\n>> >> >>\n>> >> >> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n>> >> >> until whenever my first \"gc\" kicks in, which may be quite some time if\n>> >> >> I'm just using it passively.\n>> >> >>\n>> >> >> So we should make \"git gc --auto\" be run on clone,\n>> >> >\n>> >> > There is no garbage after 'git clone'...\n>> >>\n>> >> \"git gc\" is really \"git gc-or-create-indexes\" these days.\n>> >\n>> > Because it happens to be convenient to create those indexes at\n>> > gc-time.  But that should not be an excuse to run gc when by\n>> > definition no gc is needed.\n>>\n>> Ah, I thought you just had an objection to the \"gc\" name being used for\n>> non-gc stuff,\n>\n> But you thought right, I do have an objection against that.  'git gc'\n> should, well, collect garbage.  Any non-gc stuff is already violating\n> separation of concerns.\n\nEver since git-gc was added back in 30f610b7b0 (\"Create 'git gc' to\nperform common maintenance operations.\", 2006-12-27) it has been\ndescribed as:\n\n    git-gc - Cleanup unnecessary files and optimize the local repository\n\nCreating these indexes like the commit-graph falls under \"optimize the\nlocal repository\", and 3rd party tools (e.g. the repo tool doing this\ncame up on list recently) have been calling \"gc --auto\" with this\nassumption.\n\n>>  but if you mean we shouldn't do a giant repack right after\n>> clone I agree.\n>\n> And, I also mean that since 'git clone' knows that there can't\n> possibly be any garbage in the first place, then it shouldn't call 'gc\n> --auto' at all.  However, since it also knows that there is a lot of\n> new stuff, then it should create a commit-graph if enabled.\n\nIs this something you think just because the tool isn't called\ngit-gc-and-optimzie, or do you think this regardless of what it's\ncalled?\n\nI don't see how splitting up the entry points for \"detect if we need to\ncleanup or optimize the repo\" leaves us with a better codebase for the\nreasons noted in\nhttps://public-inbox.org/git/87pnwrgll2.fsf@evledraar.gmail.com/\n"},{"id":"359550","messageId":"CACsJy8Ap0s-8NSM3MW9TtRifUUod0bDhGnzaeJuTSzXjmJA=pQ@mail.gmail.com","threadId":"49479","inReplyTo":"87tvm3go42.fsf@evledraar.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-10-03T16:45:46Z","receivedAt":"2018-10-03T16:46:15Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Oct 3, 2018 at 3:23 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n>\n> Don't have time to patch this now, but thought I'd send a note / RFC\n> about this.\n>\n> Now that we have the commit graph it's nice to be able to set\n> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n> /etc/gitconfig to apply them to all repos.\n>\n> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n> until whenever my first \"gc\" kicks in, which may be quite some time if\n> I'm just using it passively.\n\nSince you have core.hooksPath, you can already force gc (even in\ndetached mode) in post-checkout hook. I'm adding a new\n\"post-init-repo\" hook to allow more customization right after clone or\ninit-db which may also be useful for other things than gc.\n-- \nDuy\n"},{"id":"359551","messageId":"20181003165926.GR23446@localhost","threadId":"49479","inReplyTo":"87murvgir6.fsf@evledraar.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-03T16:59:26Z","receivedAt":"2018-10-03T16:59:32Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Oct 03, 2018 at 05:19:41PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >> >> >> So we should make \"git gc --auto\" be run on clone,\n> >> >> >\n> >> >> > There is no garbage after 'git clone'...\n> >> >>\n> >> >> \"git gc\" is really \"git gc-or-create-indexes\" these days.\n> >> >\n> >> > Because it happens to be convenient to create those indexes at\n> >> > gc-time.  But that should not be an excuse to run gc when by\n> >> > definition no gc is needed.\n> >>\n> >> Ah, I thought you just had an objection to the \"gc\" name being used for\n> >> non-gc stuff,\n> >\n> > But you thought right, I do have an objection against that.  'git gc'\n> > should, well, collect garbage.  Any non-gc stuff is already violating\n> > separation of concerns.\n> \n> Ever since git-gc was added back in 30f610b7b0 (\"Create 'git gc' to\n> perform common maintenance operations.\", 2006-12-27) it has been\n> described as:\n> \n>     git-gc - Cleanup unnecessary files and optimize the local repository\n> \n> Creating these indexes like the commit-graph falls under \"optimize the\n> local repository\",\n\nBut it doesn't fall under \"cleanup unnecessary files\", which the\ncommit-graph file is, since, strictly speaking, it's purely\noptimization.\n\nThat description came about, because cleaning up unnecessary files,\nnotably combining lots of loose refs into a single packed-refs file\nand combining lots of loose objects and pack files into a single pack\nfile, could not only make the repository smaller (barring too many\nexploding unreachable objects), but, as it turned out, could also make\nGit operations in that repository faster.\n\nTo me, the main goal of the command is cleanup.  Optimization, however\nbeneficial, is its side effect, and I assume the \"optimize\" part was\nadded to the description mainly to inform and \"encourage\" users.\nAfter all, the command is called 'git gc', not 'git optimize-repo'.\n\n> and 3rd party tools (e.g. the repo tool doing this\n> came up on list recently) have been calling \"gc --auto\" with this\n> assumption.\n> \n> >>  but if you mean we shouldn't do a giant repack right after\n> >> clone I agree.\n> >\n> > And, I also mean that since 'git clone' knows that there can't\n> > possibly be any garbage in the first place, then it shouldn't call 'gc\n> > --auto' at all.  However, since it also knows that there is a lot of\n> > new stuff, then it should create a commit-graph if enabled.\n> \n> Is this something you think just because the tool isn't called\n> git-gc-and-optimzie, or do you think this regardless of what it's\n> called?\n\nWell, that still has 'gc' in it...\n\n> I don't see how splitting up the entry points for \"detect if we need to\n> cleanup or optimize the repo\" leaves us with a better codebase for the\n> reasons noted in\n> https://public-inbox.org/git/87pnwrgll2.fsf@evledraar.gmail.com/\n\nSuch a separation would be valuable for those having gc.auto = 0 in\ntheir config.  Or, in general, to have a clearly marked entry point to\nupdate all the enabled \"purely-optimization\" files without 'gc'\nexploding a bunch of \"just-became-unreachable\" objects from deleted\nreflog entries and packfiles, or without performing a comparatively\nexpensive repacking.  Note the \"clearly marked\"; I don't think\nteaching 'gc [--auto]' various tricks to only create/update these\nfiles without doing what it is fundamentally supposed to do qualifies\nfor that.\n\n\n"},{"id":"359558","messageId":"CAGZ79kbYX79Pk=xR3hY6NHaRQd7KMWwvacNVyW8=QpLorzXihQ@mail.gmail.com","threadId":"49479","inReplyTo":"87o9cbglez.fsf@evledraar.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-10-03T17:47:28Z","receivedAt":"2018-10-03T17:47:43Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> So we wouldn't be spending 5 minutes repacking linux.git right after\n> cloning it, just ~10s generating the commit graph, and the same would\n> happen if you rm'd .git/objects/info/commit-graph and ran \"git commit\",\n> which would kick of \"gc --auto\" in the background and do the same thing.\n\nOr generating local bitmaps or pack idx files as well?\n"},{"id":"359562","messageId":"87lg7ehnps.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"CAGZ79kbYX79Pk=xR3hY6NHaRQd7KMWwvacNVyW8=QpLorzXihQ@mail.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-03T18:47:11Z","receivedAt":"2018-10-03T18:47:18Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 03 2018, Stefan Beller wrote:\n\n>> So we wouldn't be spending 5 minutes repacking linux.git right after\n>> cloning it, just ~10s generating the commit graph, and the same would\n>> happen if you rm'd .git/objects/info/commit-graph and ran \"git commit\",\n>> which would kick of \"gc --auto\" in the background and do the same thing.\n>\n> Or generating local bitmaps or pack idx files as well?\n\nI'm less familiar with this area, but when I clone I get a pack *.idx\nfile, why does it need to be regenerated?\n\nBut yeah, in principle this would be a sensible addition, but I'm not\naware of cases where clients get significant benefits from bitmaps (see\nhttps://githubengineering.com/counting-objects/), and I don't turn it on\nfor clients, but maybe I've missed something.\n"},{"id":"359564","messageId":"20181003185156.GA20709@sigill.intra.peff.net","threadId":"49479","inReplyTo":"87lg7ehnps.fsf@evledraar.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-03T18:51:57Z","receivedAt":"2018-10-03T18:52:00Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 03, 2018 at 08:47:11PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Oct 03 2018, Stefan Beller wrote:\n> \n> >> So we wouldn't be spending 5 minutes repacking linux.git right after\n> >> cloning it, just ~10s generating the commit graph, and the same would\n> >> happen if you rm'd .git/objects/info/commit-graph and ran \"git commit\",\n> >> which would kick of \"gc --auto\" in the background and do the same thing.\n> >\n> > Or generating local bitmaps or pack idx files as well?\n> \n> I'm less familiar with this area, but when I clone I get a pack *.idx\n> file, why does it need to be regenerated?\n> \n> But yeah, in principle this would be a sensible addition, but I'm not\n> aware of cases where clients get significant benefits from bitmaps (see\n> https://githubengineering.com/counting-objects/), and I don't turn it on\n> for clients, but maybe I've missed something.\n\nThey don't help yet, and there's no good reason to enable bitmaps for\nclients. I have a few patches that use bitmaps for things like\nahead/behind and --contains checks, but the utility of those may be\nlessened quite a bit by Stolee's commit-graph work.  And if it isn't,\nI'm mildly in favor of replacing the existing .bitmap format with\nsomething better integrated with commit-graphs (which would give us an\nopportunity to clean up some of the rough edges).\n\n-Peff\n"},{"id":"359566","messageId":"a300acae-c7f2-eace-5196-381a99d62c13@gmail.com","threadId":"49479","inReplyTo":"20181003185156.GA20709@sigill.intra.peff.net","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-03T18:59:34Z","receivedAt":"2018-10-03T18:59:39Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/3/2018 2:51 PM, Jeff King wrote:\n> On Wed, Oct 03, 2018 at 08:47:11PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> On Wed, Oct 03 2018, Stefan Beller wrote:\n>>\n>>>> So we wouldn't be spending 5 minutes repacking linux.git right after\n>>>> cloning it, just ~10s generating the commit graph, and the same would\n>>>> happen if you rm'd .git/objects/info/commit-graph and ran \"git commit\",\n>>>> which would kick of \"gc --auto\" in the background and do the same thing.\n>>> Or generating local bitmaps or pack idx files as well?\n>> I'm less familiar with this area, but when I clone I get a pack *.idx\n>> file, why does it need to be regenerated?\n>>\n>> But yeah, in principle this would be a sensible addition, but I'm not\n>> aware of cases where clients get significant benefits from bitmaps (see\n>> https://githubengineering.com/counting-objects/), and I don't turn it on\n>> for clients, but maybe I've missed something.\n> They don't help yet, and there's no good reason to enable bitmaps for\n> clients. I have a few patches that use bitmaps for things like\n> ahead/behind and --contains checks, but the utility of those may be\n> lessened quite a bit by Stolee's commit-graph work.  And if it isn't,\n> I'm mildly in favor of replacing the existing .bitmap format with\n> something better integrated with commit-graphs (which would give us an\n> opportunity to clean up some of the rough edges).\n\nIf the commit-graph doesn't improve enough on those applications, then \nwe could consider adding a commit-to-commit reachability bitmap inside \nthe commit-graph. ;)\n\n-Stolee\n"},{"id":"359568","messageId":"CAGZ79kbyCZcMuj=_Od+C+tHg=PzzDd8QV+9RqS7iFcZS+Ht+Bg@mail.gmail.com","threadId":"49479","inReplyTo":"20181003145308.GP23446@localhost","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-10-03T19:08:15Z","receivedAt":"2018-10-03T19:08:30Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">\n> But you thought right, I do have an objection against that.  'git gc'\n> should, well, collect garbage.  Any non-gc stuff is already violating\n> separation of concerns.\n\nI share these concerns in a slightly more abstract way, as\nI would bucket the actions into two separate bins:\n\nOne bin that throws away information.\nthis would include removing expired reflog entries (which\nI do not think are garbage, or collection thereof), but their\nusefulness is questionable.\n\nThe other bin would be actions that optimize but\ndo not throw away any information, repacking (without\ndropping files) would be part of it, or the new\n\"write additional files\".\n\nMaybe we can move all actions of the second bin into a new\n\"git optimize\" command, and git gc would do first the \"throw away\nthings\" and then the optimize action, whereas clone would only\ngo for the second optimizing part?\n"},{"id":"359569","messageId":"20181003191805.GB16666@sigill.intra.peff.net","threadId":"49479","inReplyTo":"a300acae-c7f2-eace-5196-381a99d62c13@gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-03T19:18:05Z","receivedAt":"2018-10-03T19:18:09Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 03, 2018 at 02:59:34PM -0400, Derrick Stolee wrote:\n\n> > They don't help yet, and there's no good reason to enable bitmaps for\n> > clients. I have a few patches that use bitmaps for things like\n> > ahead/behind and --contains checks, but the utility of those may be\n> > lessened quite a bit by Stolee's commit-graph work.  And if it isn't,\n> > I'm mildly in favor of replacing the existing .bitmap format with\n> > something better integrated with commit-graphs (which would give us an\n> > opportunity to clean up some of the rough edges).\n> \n> If the commit-graph doesn't improve enough on those applications, then we\n> could consider adding a commit-to-commit reachability bitmap inside the\n> commit-graph. ;)\n\nThat unfortunately wouldn't be enough for us to ditch the existing\n.bitmap files, since we need full object reachability for some cases\n(including packing). And commit-to-commit reachability is a trivial\nsubset of that. I'm not sure if it would be better to just leave\n.bitmaps in place as a server-side thing, and grow a new thing for\ncommit-to-commit reachability (since it would presumably be easier).\n\nI'm still excited about the prospect of a bloom filter for paths which\neach commit touches. I think that's the next big frontier in getting\nthings like \"git log -- path\" to a reasonable run-time.\n\n-Peff\n"},{"id":"359570","messageId":"20181003192127.GC16666@sigill.intra.peff.net","threadId":"49479","inReplyTo":"CAGZ79kbyCZcMuj=_Od+C+tHg=PzzDd8QV+9RqS7iFcZS+Ht+Bg@mail.gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-03T19:21:27Z","receivedAt":"2018-10-03T19:21:30Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 03, 2018 at 12:08:15PM -0700, Stefan Beller wrote:\n\n> I share these concerns in a slightly more abstract way, as\n> I would bucket the actions into two separate bins:\n> \n> One bin that throws away information.\n> this would include removing expired reflog entries (which\n> I do not think are garbage, or collection thereof), but their\n> usefulness is questionable.\n> \n> The other bin would be actions that optimize but\n> do not throw away any information, repacking (without\n> dropping files) would be part of it, or the new\n> \"write additional files\".\n> \n> Maybe we can move all actions of the second bin into a new\n> \"git optimize\" command, and git gc would do first the \"throw away\n> things\" and then the optimize action, whereas clone would only\n> go for the second optimizing part?\n\nOne problem with that world-view is that some of the operations do\n_both_, for efficiency. E.g., repacking will drop unreachable objects in\ntoo-old packs. We could actually be more aggressive in combining things\nhere. For instance, a full object graph walk in linux.git takes 30-60\nseconds, depending on your CPU. But we do it at least twice during a gc:\nonce to repack, and then again to determine reachability for pruning.\n\nIf you generate bitmaps during the repack step, you can use them during\nthe prune step. But by itself, the cost of generating the bitmaps\ngenerally outweighs the extra walk. So it's not worth generating them\n_just_ for this (but is an obvious optimization for a server which would\nbe generating them anyway).\n\n-Peff\n"},{"id":"359588","messageId":"87k1myhipe.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"20181003192127.GC16666@sigill.intra.peff.net","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-03T20:35:25Z","receivedAt":"2018-10-03T20:35:31Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 03 2018, Jeff King wrote:\n\n> On Wed, Oct 03, 2018 at 12:08:15PM -0700, Stefan Beller wrote:\n>\n>> I share these concerns in a slightly more abstract way, as\n>> I would bucket the actions into two separate bins:\n>>\n>> One bin that throws away information.\n>> this would include removing expired reflog entries (which\n>> I do not think are garbage, or collection thereof), but their\n>> usefulness is questionable.\n>>\n>> The other bin would be actions that optimize but\n>> do not throw away any information, repacking (without\n>> dropping files) would be part of it, or the new\n>> \"write additional files\".\n>>\n>> Maybe we can move all actions of the second bin into a new\n>> \"git optimize\" command, and git gc would do first the \"throw away\n>> things\" and then the optimize action, whereas clone would only\n>> go for the second optimizing part?\n>\n> One problem with that world-view is that some of the operations do\n> _both_, for efficiency. E.g., repacking will drop unreachable objects in\n> too-old packs. We could actually be more aggressive in combining things\n> here. For instance, a full object graph walk in linux.git takes 30-60\n> seconds, depending on your CPU. But we do it at least twice during a gc:\n> once to repack, and then again to determine reachability for pruning.\n>\n> If you generate bitmaps during the repack step, you can use them during\n> the prune step. But by itself, the cost of generating the bitmaps\n> generally outweighs the extra walk. So it's not worth generating them\n> _just_ for this (but is an obvious optimization for a server which would\n> be generating them anyway).\n\nI don't mean to fan the flames of this obviously controversial \"git gc\ndoes optimization\" topic (which I didn't suspect there would be a debate\nabout...), but a related thing I was wondering about the other day is\nwhether we could have a gc.fsck option, and in the background do fsck\nwhile we were at it, and report this back via some facility like\ngc.log[1].\n\nThat would also fall into this category of more work we could do while\nwe're doing a full walk anyway, but as with what you're suggesting would\nrequire some refactoring.\n\n1. Well, one that doesn't suck, see\n   https://public-inbox.org/git/87inc89j38.fsf@evledraar.gmail.com/ /\n   https://public-inbox.org/git/87d0vmck55.fsf@evledraar.gmail.com/ etc.\n"},{"id":"359636","messageId":"87in2hgzin.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"87tvm3go42.fsf@evledraar.gmail.com","subject":"[RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-04T21:42:08Z","receivedAt":"2018-10-04T21:42:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 03 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> Don't have time to patch this now, but thought I'd send a note / RFC\n> about this.\n>\n> Now that we have the commit graph it's nice to be able to set\n> e.g. core.commitGraph=true & gc.writeCommitGraph=true in ~/.gitconfig or\n> /etc/gitconfig to apply them to all repos.\n>\n> But when I clone e.g. linux.git stuff like 'tag --contains' will be slow\n> until whenever my first \"gc\" kicks in, which may be quite some time if\n> I'm just using it passively.\n>\n> So we should make \"git gc --auto\" be run on clone, and change the\n> need_to_gc() / cmd_gc() behavior so that we detect that the\n> gc.writeCommitGraph=true setting is on, but we have no commit graph, and\n> then just generate that without doing a full repack.\n>\n> As an aside such more granular \"gc\" would be nice for e.g. pack-refs\n> too. It's possible for us to just have one pack, but to have 100k loose\n> refs.\n>\n> It might also be good to have some gc.autoDetachOnClone option and have\n> it false by default, so we don't have a race condition where \"clone\n> linux && git -C linux tag --contains\" is slow because the graph hasn't\n> been generated yet, and generating the graph initially doesn't take that\n> long compared to the time to clone a large repo (and on a small one it\n> won't matter either way).\n>\n> I was going to say \"also for midx\", but of course after clone we have\n> just one pack, so I can't imagine us needing this. But I can see us\n> having other such optional side-indexes in the future generated by gc,\n> and they'd also benefit from this.\n\nI don't have time to polish this up for submission now, but here's a WIP\npatch that implements this, highlights:\n\n * There's a gc.clone.autoDetach=false default setting which overrides\n   gc.autoDetach if 'git gc --auto' is run via git-clone (we just pass a\n   --cloning option to indicate this).\n\n * A clone of say git.git with gc.writeCommitGraph=true looks like:\n\n   [...]\n   Receiving objects: 100% (255262/255262), 100.49 MiB | 17.78 MiB/s, done.\n   Resolving deltas: 100% (188947/188947), done.\n   Computing commit graph generation numbers: 100% (55210/55210), done.\n\n * The 'git gc --auto' command also knows to (only) run the commit-graph\n   (and space is left for future optimization steps) if general GC isn't\n   needed, but we need \"optimization\":\n\n   $ rm .git/objects/info/commit-graph; ~/g/git/git --exec-path=$PWD -c gc.writeCommitGraph=true -c gc.autoDetach=false gc --auto;\n   Annotating commits in commit graph: 341229, done.\n   Computing commit graph generation numbers: 100% (165969/165969), done.\n   $\n\n * The patch to gc.c looks less scary with -w, most of it is indenting\n   the existing pack-refs etc. with a \"!auto_gc || should_gc\" condition.\n\n * I added a commit_graph_exists() exists function and only care if I\n   get ENOENT for the purposes of this gc mode. This would need to be\n   tweaked for the incremental mode Derrick talks about, but if we just\n   set \"should_optimize\" that'll also work as far as gc --auto is\n   concerned (e.g. on fetch, am etc.)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 1546833213..5759fbb067 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1621,7 +1621,19 @@ gc.autoPackLimit::\n\n gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n-\tif the system supports it. Default is true.\n+\tif the system supports it. Default is true. Overridden by\n+\t`gc.clone.autoDetach` when running linkgit:git-clone[1].\n+\n+gc.clone.autoDetach::\n+\tMake `git gc --auto` return immediately and run in background\n+\tif the system supports it when run via\n+\tlinkgit:git-clone[1]. Default is false.\n++\n+The reason this defaults to false is because the only time we'll have\n+work to do after a 'git clone' is if something like\n+`gc.writeCommitGraph` is true, in that case we'd like to compute the\n+optimized file before returning, so that say commands that benefit\n+from commit graph aren't slow until it's generated in the background.\n\n gc.bigPackThreshold::\n \tIf non-zero, all packs larger than this limit are kept when\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 15b142d646..824c130ba5 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -897,6 +897,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstruct remote *remote;\n \tint err = 0, complete_refs_before_fetch = 1;\n \tint submodule_progress;\n+\tconst char *argv_gc_auto[]       = {\"gc\", \"--auto\", \"--cloning\", NULL};\n+\tconst char *argv_gc_auto_quiet[] = {\"gc\", \"--auto\", \"--cloning\", \"--quiet\", NULL};\n\n \tstruct refspec rs = REFSPEC_INIT_FETCH;\n \tstruct argv_array ref_prefixes = ARGV_ARRAY_INIT;\n@@ -1245,5 +1247,11 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n\n \trefspec_clear(&rs);\n \targv_array_clear(&ref_prefixes);\n+\n+\tif (0 <= option_verbosity)\n+\t\trun_command_v_opt_cd_env(argv_gc_auto, RUN_GIT_CMD, git_dir, NULL);\n+\telse\n+\t\trun_command_v_opt_cd_env(argv_gc_auto_quiet, RUN_GIT_CMD, git_dir, NULL);\n+\n \treturn err;\n }\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 6591ddbe83..27be03890a 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -43,6 +43,7 @@ static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n static int gc_write_commit_graph;\n static int detach_auto = 1;\n+static int detach_clone_auto = 0;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n static const char *prune_expire = \"2.weeks.ago\";\n@@ -133,6 +134,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n \tgit_config_get_bool(\"gc.writecommitgraph\", &gc_write_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n+\tgit_config_get_bool(\"gc.clone.autodetach\", &detach_clone_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n@@ -157,9 +159,6 @@ static int too_many_loose_objects(void)\n \tint num_loose = 0;\n \tint needed = 0;\n\n-\tif (gc_auto_threshold <= 0)\n-\t\treturn 0;\n-\n \tdir = opendir(git_path(\"objects/17\"));\n \tif (!dir)\n \t\treturn 0;\n@@ -369,10 +368,21 @@ static int need_to_gc(void)\n \t\treturn 0;\n\n \tif (run_hook_le(NULL, \"pre-auto-gc\", NULL))\n-\t\treturn 0;\n+\t\treturn -1;\n \treturn 1;\n }\n\n+static int need_to_optimize(void) {\n+\tif (gc_write_commit_graph) {\n+\t\tchar *obj_dir = get_object_directory();\n+\t\tchar *graph_name = get_commit_graph_filename(obj_dir);\n+\n+\t\tif (commit_graph_exists(graph_name) == 0) /* ENOENT */\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n /* return NULL on success, else hostname running the gc */\n static const char *lock_repo_for_gc(int force, pid_t* ret_pid)\n {\n@@ -491,6 +501,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n {\n \tint aggressive = 0;\n \tint auto_gc = 0;\n+\tint cloning = 0;\n \tint quiet = 0;\n \tint force = 0;\n \tconst char *name;\n@@ -498,6 +509,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tint daemonized = 0;\n \tint keep_base_pack = -1;\n \ttimestamp_t dummy;\n+\tint should_gc;\n+\tint should_optimize;\n\n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n@@ -507,6 +520,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL_F(0, \"auto\", &auto_gc, N_(\"enable auto-gc mode\"),\n \t\t\t   PARSE_OPT_NOCOMPLETE),\n+\t\tOPT_BOOL_F(0, \"cloning\", &cloning, N_(\"enable cloning mode\"),\n+\t\t\t   PARSE_OPT_NOCOMPLETE),\n \t\tOPT_BOOL_F(0, \"force\", &force,\n \t\t\t   N_(\"force running gc even if there may be another gc running\"),\n \t\t\t   PARSE_OPT_NOCOMPLETE),\n@@ -555,22 +570,27 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tif (!need_to_gc())\n+\t\tshould_gc = need_to_gc();\n+\t\tif (should_gc == -1)\n+\t\t\treturn 0;\n+\t\tshould_optimize = need_to_optimize();\n+\t\tif (!should_gc && !should_optimize)\n \t\t\treturn 0;\n-\t\tif (!quiet) {\n+\t\tif (!quiet && should_gc) {\n \t\t\tif (detach_auto)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n \t\t\telse\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository for optimum performance.\\n\"));\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n-\t\tif (detach_auto) {\n+\t\tif (detach_auto &&\n+\t\t    (!cloning || (cloning && detach_clone_auto))) {\n \t\t\tif (report_last_gc_error())\n \t\t\t\treturn -1;\n\n \t\t\tif (lock_repo_for_gc(force, &pid))\n \t\t\t\treturn 0;\n-\t\t\tif (gc_before_repack())\n+\t\t\tif (should_gc && gc_before_repack())\n \t\t\t\treturn -1;\n \t\t\tdelete_tempfile(&pidfile);\n\n@@ -611,45 +631,48 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tatexit(process_log_file_at_exit);\n \t}\n\n-\tif (gc_before_repack())\n-\t\treturn -1;\n-\n-\tif (!repository_format_precious_objects) {\n-\t\tclose_all_packs(the_repository->objects);\n-\t\tif (run_command_v_opt(repack.argv, RUN_GIT_CMD))\n-\t\t\treturn error(FAILED_RUN, repack.argv[0]);\n-\n-\t\tif (prune_expire) {\n-\t\t\targv_array_push(&prune, prune_expire);\n-\t\t\tif (quiet)\n-\t\t\t\targv_array_push(&prune, \"--no-progress\");\n-\t\t\tif (repository_format_partial_clone)\n-\t\t\t\targv_array_push(&prune,\n-\t\t\t\t\t\t\"--exclude-promisor-objects\");\n-\t\t\tif (run_command_v_opt(prune.argv, RUN_GIT_CMD))\n-\t\t\t\treturn error(FAILED_RUN, prune.argv[0]);\n+\tif (!auto_gc || should_gc) {\n+\t\tif (gc_before_repack())\n+\t\t\treturn -1;\n+\n+\t\tif (!repository_format_precious_objects) {\n+\t\t\tclose_all_packs(the_repository->objects);\n+\t\t\tif (run_command_v_opt(repack.argv, RUN_GIT_CMD))\n+\t\t\t\treturn error(FAILED_RUN, repack.argv[0]);\n+\n+\t\t\tif (prune_expire) {\n+\t\t\t\targv_array_push(&prune, prune_expire);\n+\t\t\t\tif (quiet)\n+\t\t\t\t\targv_array_push(&prune, \"--no-progress\");\n+\t\t\t\tif (repository_format_partial_clone)\n+\t\t\t\t\targv_array_push(&prune,\n+\t\t\t\t\t\t\t\"--exclude-promisor-objects\");\n+\t\t\t\tif (run_command_v_opt(prune.argv, RUN_GIT_CMD))\n+\t\t\t\t\treturn error(FAILED_RUN, prune.argv[0]);\n+\t\t\t}\n \t\t}\n-\t}\n\n-\tif (prune_worktrees_expire) {\n-\t\targv_array_push(&prune_worktrees, prune_worktrees_expire);\n-\t\tif (run_command_v_opt(prune_worktrees.argv, RUN_GIT_CMD))\n-\t\t\treturn error(FAILED_RUN, prune_worktrees.argv[0]);\n-\t}\n\n-\tif (run_command_v_opt(rerere.argv, RUN_GIT_CMD))\n-\t\treturn error(FAILED_RUN, rerere.argv[0]);\n+\t\tif (prune_worktrees_expire) {\n+\t\t\targv_array_push(&prune_worktrees, prune_worktrees_expire);\n+\t\t\tif (run_command_v_opt(prune_worktrees.argv, RUN_GIT_CMD))\n+\t\t\t\treturn error(FAILED_RUN, prune_worktrees.argv[0]);\n+\t\t}\n\n-\treport_garbage = report_pack_garbage;\n-\treprepare_packed_git(the_repository);\n-\tif (pack_garbage.nr > 0)\n-\t\tclean_pack_garbage();\n+\t\tif (run_command_v_opt(rerere.argv, RUN_GIT_CMD))\n+\t\t\treturn error(FAILED_RUN, rerere.argv[0]);\n+\n+\t\treport_garbage = report_pack_garbage;\n+\t\treprepare_packed_git(the_repository);\n+\t\tif (pack_garbage.nr > 0)\n+\t\t\tclean_pack_garbage();\n+\t}\n\n \tif (gc_write_commit_graph)\n \t\twrite_commit_graph_reachable(get_object_directory(), 0,\n \t\t\t\t\t     !quiet && !daemonized);\n\n-\tif (auto_gc && too_many_loose_objects())\n+\tif (auto_gc && should_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5908bd4e34..a4a7c94cec 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -57,6 +57,18 @@ static struct commit_graph *alloc_commit_graph(void)\n \treturn g;\n }\n\n+int commit_graph_exists(const char *graph_file)\n+{\n+\tstruct stat st;\n+\tif (stat(graph_file, &st)) {\n+\t\tif (errno == ENOENT)\n+\t\t\treturn 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t}\n+\treturn 1;\n+}\n+\n struct commit_graph *load_commit_graph_one(const char *graph_file)\n {\n \tvoid *graph_map;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 5678a8f4ca..a251f1bc32 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -11,6 +11,7 @@\n struct commit;\n\n char *get_commit_graph_filename(const char *obj_dir);\n+int commit_graph_exists(const char *graph_file);\n\n /*\n  * Given a commit struct, try to fill the commit struct info, including:\n"},{"id":"359651","messageId":"xmqqpnwo3ow9.fsf@gitster-ct.c.googlers.com","threadId":"49479","inReplyTo":"20181003165926.GR23446@localhost","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-05T06:09:58Z","receivedAt":"2018-10-05T06:10:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor <szeder.dev@gmail.com> writes:\n\n>>     git-gc - Cleanup unnecessary files and optimize the local repository\n>> \n>> Creating these indexes like the commit-graph falls under \"optimize the\n>> local repository\",\n>\n> But it doesn't fall under \"cleanup unnecessary files\", which the\n> commit-graph file is, since, strictly speaking, it's purely\n> optimization.\n\nI won't be actively engaged in this discussion soon, but I must say\nthat \"git gc\" doing \"garbage collection\" is merely an implementation\ndetail of optimizing the repository for further use.  And from that\npoint of view, what needs to be updated is the synopsis of the\ngit-gc doc.  It states \"X and Y\" above, but it actually is \"Y by\ndoing X and other things\".\n\nI understand your \"by definition there is no garbage immediately\nafter clone\" position, and also I would understand if you find it\n(perhaps philosophically) disturbing that \"git clone\" may give users\na suboptimal repository that immediately needs optimizing [*1*].\n\nBut that bridge was crossed long time ago ever since pack transfer\nwas invented.  The data source sends only the pack data stream, and\nthe receiving end is responsible for spending cycles to build .idx\nfile.  Theoretically, .pack should be all that is needed---you\nshould be able to locate any necessary object by parsing the .pack\nfile every time you open it, and .idx is mere optimization.  You can\nthink of the .midx and graph files the same way.\n\nI'd consider it a growing pain that these two recent inventions were\nand are still built as a totally optional and separate features,\nrequiring completely separate full enumeration of objects in the\nrepository that needs to happen anyway when we build .idx out of the\nreceived .pack.\n\nI would not be surprised by a future in which the initial index-pack\nthat is responsible for receiving the incoming pack stream and\nstoring that in .pack file(s) while creating corresponding .idx\nfile(s) becomes also responsible for building .midx and graph files\nin the same pass, or at least smaller number of passes.  Once we\ngain experience and confidence with these new auxiliary files, that\nought to happen naturally.  And at that point, we won't be having\nthis discussion---we'd all happily run index-pack to receive the\npack data, because that is pretty much the fundamental requirement\nto make use of the data.\n\n[Footnote]\n\n*1* Even without considering these recent invention of auxiliary\n    files, cloning from a sloppily packed server whose primary focus\n    is to avoid spending cycles by not computing better deltas will\n    give the cloner a suboptimal repository.  If we truly want to\n    have an optimized repository ready to be used after cloning, we\n    should run an equivalent of \"repack -a -d -f\" immediately after\n    \"git clone\".\n\n"},{"id":"359666","messageId":"dcb8f115-ce3c-64fa-50cc-dd03569c0164@gmail.com","threadId":"49479","inReplyTo":"87in2hgzin.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-05T12:05:55Z","receivedAt":"2018-10-05T12:06:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/4/2018 5:42 PM, Ævar Arnfjörð Bjarmason wrote:\n> I don't have time to polish this up for submission now, but here's a WIP\n> patch that implements this, highlights:\n>\n>   * There's a gc.clone.autoDetach=false default setting which overrides\n>     gc.autoDetach if 'git gc --auto' is run via git-clone (we just pass a\n>     --cloning option to indicate this).\n\nI'll repeat that it could make sense to do the same thing on clone _and_ \nfetch. Perhaps a \"--post-fetch\" flag would be good here to communicate \nthat we just downloaded a pack from a remote.\n\n>   * A clone of say git.git with gc.writeCommitGraph=true looks like:\n>\n>     [...]\n>     Receiving objects: 100% (255262/255262), 100.49 MiB | 17.78 MiB/s, done.\n>     Resolving deltas: 100% (188947/188947), done.\n>     Computing commit graph generation numbers: 100% (55210/55210), done.\n\nThis looks like good UX. Thanks for the progress here!\n\n>   * The 'git gc --auto' command also knows to (only) run the commit-graph\n>     (and space is left for future optimization steps) if general GC isn't\n>     needed, but we need \"optimization\":\n>\n>     $ rm .git/objects/info/commit-graph; ~/g/git/git --exec-path=$PWD -c gc.writeCommitGraph=true -c gc.autoDetach=false gc --auto;\n>     Annotating commits in commit graph: 341229, done.\n>     Computing commit graph generation numbers: 100% (165969/165969), done.\n>     $\n\nWill this also trigger a full commit-graph rewrite on every 'git commit' \ncommand? Or is there some way we can compute the staleness of the \ncommit-graph in order to only update if we get too far ahead? \nPreviously, this was solved by relying on the auto-GC threshold.\n\n>   * The patch to gc.c looks less scary with -w, most of it is indenting\n>     the existing pack-refs etc. with a \"!auto_gc || should_gc\" condition.\n>\n>   * I added a commit_graph_exists() exists function and only care if I\n>     get ENOENT for the purposes of this gc mode. This would need to be\n>     tweaked for the incremental mode Derrick talks about, but if we just\n>     set \"should_optimize\" that'll also work as far as gc --auto is\n>     concerned (e.g. on fetch, am etc.)\n\nThe incremental mode would operate the same as split-index, which means \nwe will still look for .git/objects/info/commit-graph. That file may \npoint us to more files.\n\n> +int commit_graph_exists(const char *graph_file)\n> +{\n> +\tstruct stat st;\n> +\tif (stat(graph_file, &st)) {\n> +\t\tif (errno == ENOENT)\n> +\t\t\treturn 0;\n> +\t\telse\n> +\t\t\treturn -1;\n> +\t}\n> +\treturn 1;\n> +}\n> +\n\nThis method serves a very similar purpose to \ngeneration_numbers_enabled(), except your method only cares about the \nfile existing. It ignores information like `core.commitGraph`, which \nshould keep us from doing anything with the commit-graph file if false.\n\nNothing about your method is specific to the commit-graph file, since \nyou provide a filename as a parameter. It could easily be \"int \nfile_exists(const char *filename)\".\n\nThanks,\n\n-Stolee\n\n\n"},{"id":"359673","messageId":"87ftxkh7bf.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"dcb8f115-ce3c-64fa-50cc-dd03569c0164@gmail.com","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-05T13:05:56Z","receivedAt":"2018-10-05T13:06:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 05 2018, Derrick Stolee wrote:\n\n> On 10/4/2018 5:42 PM, Ævar Arnfjörð Bjarmason wrote:\n>> I don't have time to polish this up for submission now, but here's a WIP\n>> patch that implements this, highlights:\n>>\n>>   * There's a gc.clone.autoDetach=false default setting which overrides\n>>     gc.autoDetach if 'git gc --auto' is run via git-clone (we just pass a\n>>     --cloning option to indicate this).\n>\n> I'll repeat that it could make sense to do the same thing on clone\n> _and_ fetch. Perhaps a \"--post-fetch\" flag would be good here to\n> communicate that we just downloaded a pack from a remote.\n\nI don't think that makes sense, but let's talk about why, because maybe\nI've missed something, you're certainly more familiar with the\ncommit-graph than I am.\n\nThe reason to do it on clone as a special-case or when the file is\nmissing, is because we know the file is desired (via the GC config), and\npresumably is expected to help performance, and we have 0% of it. So by\ngoing from 0% to 100% on clone we'll get fast --contains and other\ngoodies the graph helps with.\n\nBut when we're doing a fetch, or really anything else that runs \"git gc\n--auto\" we can safely assume that we have a recent enough graph, because\nit will have been run whenever auto-gc kicked in.\n\nI.e.:\n\n    # Slow, if we assume background forked commit-graph generation\n    # (which I'm avoiding)\n    git clone x && cd x && git tag --contains\n    # Fast enough, since we have an existing commit-graph\n    cd x && git fetch && git tag --contains\n\nI *do* think it might make sense to in general split off parts of \"gc\n--auto\" that we'd like to be more aggressive about, simply because the\nratio of how long it takes to do, and how much it helps with performance\nmakes more sense than a full repack, which is what the current heuristic\nis based on.\n\nAnd maybe when we run in that mode we should run in the foreground, but\nI don't see why git-fetch should be a special case there, and in this\nregard, the gc.clone.autoDetach=false setting I've made doesn't make\nmuch sence. I.e. maybe we should also skip forking to the background in\nsuch a mode when we trigger such a \"mini gc\" via git-commit or whatever.\n\n>>   * A clone of say git.git with gc.writeCommitGraph=true looks like:\n>>\n>>     [...]\n>>     Receiving objects: 100% (255262/255262), 100.49 MiB | 17.78 MiB/s, done.\n>>     Resolving deltas: 100% (188947/188947), done.\n>>     Computing commit graph generation numbers: 100% (55210/55210), done.\n>\n> This looks like good UX. Thanks for the progress here!\n>\n>>   * The 'git gc --auto' command also knows to (only) run the commit-graph\n>>     (and space is left for future optimization steps) if general GC isn't\n>>     needed, but we need \"optimization\":\n>>\n>>     $ rm .git/objects/info/commit-graph; ~/g/git/git --exec-path=$PWD -c gc.writeCommitGraph=true -c gc.autoDetach=false gc --auto;\n>>     Annotating commits in commit graph: 341229, done.\n>>     Computing commit graph generation numbers: 100% (165969/165969), done.\n>>     $\n>\n> Will this also trigger a full commit-graph rewrite on every 'git\n> commit' command?\n\nNope, because \"git commit\" can safely be assumed to have some\ncommit-graph anyway, and I'm just special casing the case where it\ndoesn't exist.\n\nBut if it doesn't exist and you do a \"git commit\" then \"gc --auto\" will\nbe run, and we'll fork to the background and generate it...\n\n>  Or is there some way we can compute the staleness of\n> the commit-graph in order to only update if we get too far ahead?\n> Previously, this was solved by relying on the auto-GC threshold.\n\nSo re the \"I don't think that makes sense...\" at the start of my E-Mail,\nisn't it fine to rely on the default thresholds here, or should we be\nmore aggressive?\n\n>>   * The patch to gc.c looks less scary with -w, most of it is indenting\n>>     the existing pack-refs etc. with a \"!auto_gc || should_gc\" condition.\n>>\n>>   * I added a commit_graph_exists() exists function and only care if I\n>>     get ENOENT for the purposes of this gc mode. This would need to be\n>>     tweaked for the incremental mode Derrick talks about, but if we just\n>>     set \"should_optimize\" that'll also work as far as gc --auto is\n>>     concerned (e.g. on fetch, am etc.)\n>\n> The incremental mode would operate the same as split-index, which\n> means we will still look for .git/objects/info/commit-graph. That file\n> may point us to more files.\n\nAh!\n\n>> +int commit_graph_exists(const char *graph_file)\n>> +{\n>> +\tstruct stat st;\n>> +\tif (stat(graph_file, &st)) {\n>> +\t\tif (errno == ENOENT)\n>> +\t\t\treturn 0;\n>> +\t\telse\n>> +\t\t\treturn -1;\n>> +\t}\n>> +\treturn 1;\n>> +}\n>> +\n>\n> This method serves a very similar purpose to\n> generation_numbers_enabled(), except your method only cares about the\n> file existing. It ignores information like `core.commitGraph`, which\n> should keep us from doing anything with the commit-graph file if\n> false.\n>\n> Nothing about your method is specific to the commit-graph file, since\n> you provide a filename as a parameter. It could easily be \"int\n> file_exists(const char *filename)\".\n\nI was being paranoid about not doing this if it didn't exist but it was\nsomething else than ENOENT (e.g. permission error?), but in retrospect\nthat's silly. I'll drop this helper and just use file_exists().\n"},{"id":"359686","messageId":"05fefab0-4012-4ef0-5df1-2430eea2bf4d@gmail.com","threadId":"49479","inReplyTo":"87ftxkh7bf.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-05T13:45:47Z","receivedAt":"2018-10-05T13:45:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/5/2018 9:05 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Fri, Oct 05 2018, Derrick Stolee wrote:\n>\n>> On 10/4/2018 5:42 PM, Ævar Arnfjörð Bjarmason wrote:\n>>> I don't have time to polish this up for submission now, but here's a WIP\n>>> patch that implements this, highlights:\n>>>\n>>>    * There's a gc.clone.autoDetach=false default setting which overrides\n>>>      gc.autoDetach if 'git gc --auto' is run via git-clone (we just pass a\n>>>      --cloning option to indicate this).\n>> I'll repeat that it could make sense to do the same thing on clone\n>> _and_ fetch. Perhaps a \"--post-fetch\" flag would be good here to\n>> communicate that we just downloaded a pack from a remote.\n> I don't think that makes sense, but let's talk about why, because maybe\n> I've missed something, you're certainly more familiar with the\n> commit-graph than I am.\n>\n> The reason to do it on clone as a special-case or when the file is\n> missing, is because we know the file is desired (via the GC config), and\n> presumably is expected to help performance, and we have 0% of it. So by\n> going from 0% to 100% on clone we'll get fast --contains and other\n> goodies the graph helps with.\n>\n> But when we're doing a fetch, or really anything else that runs \"git gc\n> --auto\" we can safely assume that we have a recent enough graph, because\n> it will have been run whenever auto-gc kicked in.\n>\n> I.e.:\n>\n>      # Slow, if we assume background forked commit-graph generation\n>      # (which I'm avoiding)\n>      git clone x && cd x && git tag --contains\n>      # Fast enough, since we have an existing commit-graph\n>      cd x && git fetch && git tag --contains\n>\n> I *do* think it might make sense to in general split off parts of \"gc\n> --auto\" that we'd like to be more aggressive about, simply because the\n> ratio of how long it takes to do, and how much it helps with performance\n> makes more sense than a full repack, which is what the current heuristic\n> is based on.\n>\n> And maybe when we run in that mode we should run in the foreground, but\n> I don't see why git-fetch should be a special case there, and in this\n> regard, the gc.clone.autoDetach=false setting I've made doesn't make\n> much sence. I.e. maybe we should also skip forking to the background in\n> such a mode when we trigger such a \"mini gc\" via git-commit or whatever.\n\nMy misunderstanding was that your proposed change to gc computes the \ncommit-graph in either of these two cases:\n\n(1) The auto-GC threshold is met.\n\n(2) There is no commit-graph file.\n\nAnd what I hope to have instead of (2) is (3):\n\n(3) The commit-graph file is \"sufficiently behind\" the tip refs.\n\nThis condition is intentionally vague at the moment. It could be that we \nhint that (3) holds by saying \"--post-fetch\" (i.e. \"We just downloaded a \npack, and it probably contains a lot of new commits\") or we could create \nsome more complicated condition based on counting reachable commits with \ninfinite generation number (the number of commits not in the \ncommit-graph file).\n\nI like that you are moving forward to make the commit-graph be written \nmore frequently, but I'm trying to push us in a direction of writing it \neven more often than your proposed strategy. We should avoid creating \ntoo many orthogonal conditions that trigger the commit-graph write, \nwhich is why I'm pushing on your design here.\n\nAnyone else have thoughts on this direction?\n\nThanks,\n\n-Stolee\n\n"},{"id":"359688","messageId":"87efd4h4ma.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"05fefab0-4012-4ef0-5df1-2430eea2bf4d@gmail.com","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-05T14:04:13Z","receivedAt":"2018-10-05T14:04:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 05 2018, Derrick Stolee wrote:\n\n> On 10/5/2018 9:05 AM, Ævar Arnfjörð Bjarmason wrote:\n>> On Fri, Oct 05 2018, Derrick Stolee wrote:\n>>\n>>> On 10/4/2018 5:42 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>> I don't have time to polish this up for submission now, but here's a WIP\n>>>> patch that implements this, highlights:\n>>>>\n>>>>    * There's a gc.clone.autoDetach=false default setting which overrides\n>>>>      gc.autoDetach if 'git gc --auto' is run via git-clone (we just pass a\n>>>>      --cloning option to indicate this).\n>>> I'll repeat that it could make sense to do the same thing on clone\n>>> _and_ fetch. Perhaps a \"--post-fetch\" flag would be good here to\n>>> communicate that we just downloaded a pack from a remote.\n>> I don't think that makes sense, but let's talk about why, because maybe\n>> I've missed something, you're certainly more familiar with the\n>> commit-graph than I am.\n>>\n>> The reason to do it on clone as a special-case or when the file is\n>> missing, is because we know the file is desired (via the GC config), and\n>> presumably is expected to help performance, and we have 0% of it. So by\n>> going from 0% to 100% on clone we'll get fast --contains and other\n>> goodies the graph helps with.\n>>\n>> But when we're doing a fetch, or really anything else that runs \"git gc\n>> --auto\" we can safely assume that we have a recent enough graph, because\n>> it will have been run whenever auto-gc kicked in.\n>>\n>> I.e.:\n>>\n>>      # Slow, if we assume background forked commit-graph generation\n>>      # (which I'm avoiding)\n>>      git clone x && cd x && git tag --contains\n>>      # Fast enough, since we have an existing commit-graph\n>>      cd x && git fetch && git tag --contains\n>>\n>> I *do* think it might make sense to in general split off parts of \"gc\n>> --auto\" that we'd like to be more aggressive about, simply because the\n>> ratio of how long it takes to do, and how much it helps with performance\n>> makes more sense than a full repack, which is what the current heuristic\n>> is based on.\n>>\n>> And maybe when we run in that mode we should run in the foreground, but\n>> I don't see why git-fetch should be a special case there, and in this\n>> regard, the gc.clone.autoDetach=false setting I've made doesn't make\n>> much sence. I.e. maybe we should also skip forking to the background in\n>> such a mode when we trigger such a \"mini gc\" via git-commit or whatever.\n>\n> My misunderstanding was that your proposed change to gc computes the\n> commit-graph in either of these two cases:\n>\n> (1) The auto-GC threshold is met.\n>\n> (2) There is no commit-graph file.\n>\n> And what I hope to have instead of (2) is (3):\n>\n> (3) The commit-graph file is \"sufficiently behind\" the tip refs.\n>\n> This condition is intentionally vague at the moment. It could be that\n> we hint that (3) holds by saying \"--post-fetch\" (i.e. \"We just\n> downloaded a pack, and it probably contains a lot of new commits\") or\n> we could create some more complicated condition based on counting\n> reachable commits with infinite generation number (the number of\n> commits not in the commit-graph file).\n>\n> I like that you are moving forward to make the commit-graph be written\n> more frequently, but I'm trying to push us in a direction of writing\n> it even more often than your proposed strategy. We should avoid\n> creating too many orthogonal conditions that trigger the commit-graph\n> write, which is why I'm pushing on your design here.\n>\n> Anyone else have thoughts on this direction?\n\nAh. I see. I think #3 makes perfect sense, but probably makes sense to\ndo as a follow-up, or maybe you'd like to stick a patch on top of the\nseries I have when I send it. I don't know how to write the \"I'm not\nquite happy about the commit graph\" code :)\n\nWhat I will do is refactor gc.c a bit and leave it in a state where it's\ngoing to be really easy to change the existing \"we have no commit graph,\nand thus should do the optimization step\" to have some more complex\ncondition instead of \"we have no commit graph\", i.e. your \"we just\ngrabbed a lot of data\".\n\nAlso, I'll drop the gc.clone.autoDetach=false setting and name it\nsomething more general. maybe gc.AutoDetachOnBigOptimization=false?\nAnyway something more generic so that \"clone\" will always pass in some\noption saying \"expect a large % commit graph update\" (100% in its case),\nand then in \"fetch\" we could have some detection of how big what we just\ngot from the server is, and do the same.\n\nThis seems to be to be the most general thing that would make sense, and\ncould also be extended e.g. to \"git commit\" and other users of gc\n--auto. If I started with a README file in an empty repo, and then made\na commit where I added 1 million files all in one commit, in which case\nwe'd (depending on that setting) also block in the foreground and\ngenerate the commit-graph.\n"},{"id":"359721","messageId":"20181005192154.GD17482@sigill.intra.peff.net","threadId":"49479","inReplyTo":"05fefab0-4012-4ef0-5df1-2430eea2bf4d@gmail.com","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-05T19:21:54Z","receivedAt":"2018-10-05T19:21:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 05, 2018 at 09:45:47AM -0400, Derrick Stolee wrote:\n\n> My misunderstanding was that your proposed change to gc computes the\n> commit-graph in either of these two cases:\n> \n> (1) The auto-GC threshold is met.\n> \n> (2) There is no commit-graph file.\n> \n> And what I hope to have instead of (2) is (3):\n> \n> (3) The commit-graph file is \"sufficiently behind\" the tip refs.\n> \n> This condition is intentionally vague at the moment. It could be that we\n> hint that (3) holds by saying \"--post-fetch\" (i.e. \"We just downloaded a\n> pack, and it probably contains a lot of new commits\") or we could create\n> some more complicated condition based on counting reachable commits with\n> infinite generation number (the number of commits not in the commit-graph\n> file).\n> \n> I like that you are moving forward to make the commit-graph be written more\n> frequently, but I'm trying to push us in a direction of writing it even more\n> often than your proposed strategy. We should avoid creating too many\n> orthogonal conditions that trigger the commit-graph write, which is why I'm\n> pushing on your design here.\n> \n> Anyone else have thoughts on this direction?\n\nYes, I think measuring \"sufficiently behind\" is the right thing.\nEverything else is a proxy or heuristic, and will run into corner cases.\nE.g., I have some small number of objects and then do a huge fetch, and\nnow my commit-graph only covers 5% of what's available.\n\nWe know how many objects are in the graph already. And it's not too\nexpensive to get the number of objects in the repository. We can do the\nsame sampling for loose objects that \"gc --auto\" does, and counting\npacked objects just involves opening up the .idx files (that can be slow\nif you have a ton of packs, but you'd want to either repack or use a\n.midx in that case anyway, either of which would help here).\n\nSo can we really just take (total_objects - commit_graph_objects) and\ncompare it to some threshold?\n\n-Peff\n"},{"id":"359725","messageId":"934fa00e-f6df-c333-4968-3e9acffab22d@gmail.com","threadId":"49479","inReplyTo":"20181005192154.GD17482@sigill.intra.peff.net","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-05T19:41:40Z","receivedAt":"2018-10-05T19:41:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/5/2018 3:21 PM, Jeff King wrote:\n> On Fri, Oct 05, 2018 at 09:45:47AM -0400, Derrick Stolee wrote:\n>\n>> My misunderstanding was that your proposed change to gc computes the\n>> commit-graph in either of these two cases:\n>>\n>> (1) The auto-GC threshold is met.\n>>\n>> (2) There is no commit-graph file.\n>>\n>> And what I hope to have instead of (2) is (3):\n>>\n>> (3) The commit-graph file is \"sufficiently behind\" the tip refs.\n>>\n>> This condition is intentionally vague at the moment. It could be that we\n>> hint that (3) holds by saying \"--post-fetch\" (i.e. \"We just downloaded a\n>> pack, and it probably contains a lot of new commits\") or we could create\n>> some more complicated condition based on counting reachable commits with\n>> infinite generation number (the number of commits not in the commit-graph\n>> file).\n>>\n>> I like that you are moving forward to make the commit-graph be written more\n>> frequently, but I'm trying to push us in a direction of writing it even more\n>> often than your proposed strategy. We should avoid creating too many\n>> orthogonal conditions that trigger the commit-graph write, which is why I'm\n>> pushing on your design here.\n>>\n>> Anyone else have thoughts on this direction?\n> Yes, I think measuring \"sufficiently behind\" is the right thing.\n> Everything else is a proxy or heuristic, and will run into corner cases.\n> E.g., I have some small number of objects and then do a huge fetch, and\n> now my commit-graph only covers 5% of what's available.\n>\n> We know how many objects are in the graph already. And it's not too\n> expensive to get the number of objects in the repository. We can do the\n> same sampling for loose objects that \"gc --auto\" does, and counting\n> packed objects just involves opening up the .idx files (that can be slow\n> if you have a ton of packs, but you'd want to either repack or use a\n> .midx in that case anyway, either of which would help here).\n>\n> So can we really just take (total_objects - commit_graph_objects) and\n> compare it to some threshold?\n\nThe commit-graph only stores the number of _commits_, not total objects.\n\nAzure Repos' commit-graph does store the total number of objects, and \nthat is how we trigger updating the graph, so it is not unreasonable to \nuse that as a heuristic.\n\nThanks,\n\n-Stolee\n\n"},{"id":"359729","messageId":"20181005194703.GA19735@sigill.intra.peff.net","threadId":"49479","inReplyTo":"934fa00e-f6df-c333-4968-3e9acffab22d@gmail.com","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-05T19:47:03Z","receivedAt":"2018-10-05T19:47:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 05, 2018 at 03:41:40PM -0400, Derrick Stolee wrote:\n\n> > So can we really just take (total_objects - commit_graph_objects) and\n> > compare it to some threshold?\n> \n> The commit-graph only stores the number of _commits_, not total objects.\n\nOh, right, of course. That does throw a monkey wrench in that line of\nthought. ;)\n\nThere's unfortunately not a fast way of doing that. One option would be\nto keep a counter of \"ungraphed commit objects\", and have callers update\nit. Anybody admitting a pack via index-pack or unpack-objects can easily\nget this information. Commands like fast-import can do likewise, and\n\"git commit\" obviously increments it by one.\n\nI'm not excited about adding a new global on-disk data structure (and\nthe accompanying lock).\n\n-Peff\n"},{"id":"359731","messageId":"14529aaa-fee0-e839-4629-bf87cdd13e31@gmail.com","threadId":"49479","inReplyTo":"20181005194703.GA19735@sigill.intra.peff.net","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-05T20:00:12Z","receivedAt":"2018-10-05T20:00:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/5/2018 3:47 PM, Jeff King wrote:\n> On Fri, Oct 05, 2018 at 03:41:40PM -0400, Derrick Stolee wrote:\n>\n>>> So can we really just take (total_objects - commit_graph_objects) and\n>>> compare it to some threshold?\n>> The commit-graph only stores the number of _commits_, not total objects.\n> Oh, right, of course. That does throw a monkey wrench in that line of\n> thought. ;)\n>\n> There's unfortunately not a fast way of doing that. One option would be\n> to keep a counter of \"ungraphed commit objects\", and have callers update\n> it. Anybody admitting a pack via index-pack or unpack-objects can easily\n> get this information. Commands like fast-import can do likewise, and\n> \"git commit\" obviously increments it by one.\n>\n> I'm not excited about adding a new global on-disk data structure (and\n> the accompanying lock).\n\nIf we want, then we can add an optional chunk to the commit-graph file \nthat stores the object count.\n\n"},{"id":"359732","messageId":"875zyggo2s.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"20181005194703.GA19735@sigill.intra.peff.net","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-05T20:01:31Z","receivedAt":"2018-10-05T20:01:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 05 2018, Jeff King wrote:\n\n> On Fri, Oct 05, 2018 at 03:41:40PM -0400, Derrick Stolee wrote:\n>\n>> > So can we really just take (total_objects - commit_graph_objects) and\n>> > compare it to some threshold?\n>>\n>> The commit-graph only stores the number of _commits_, not total objects.\n>\n> Oh, right, of course. That does throw a monkey wrench in that line of\n> thought. ;)\n>\n> There's unfortunately not a fast way of doing that. One option would be\n> to keep a counter of \"ungraphed commit objects\", and have callers update\n> it. Anybody admitting a pack via index-pack or unpack-objects can easily\n> get this information. Commands like fast-import can do likewise, and\n> \"git commit\" obviously increments it by one.\n>\n> I'm not excited about adding a new global on-disk data structure (and\n> the accompanying lock).\n\nYou don't really need a new global datastructure to solve this\nproblem. It would be sufficient to have git-gc itself write out a 4-line\ntext file after it runs saying how many tags, commits, trees and blobs\nit found on its last run.\n\nYou can then fuzzily compare object counts v.s. commit counts for the\npurposes of deciding whether something like the commit-graph needs to be\nupdated, while assuming that whatever new data you have has similar\nenough ratios of those as your existing data.\n\nThat's an assumption that'll hold well enough for big repos where this\nmatters the most, and who tend to grow in fairly uniform ways as far as\ntheir object type ratios go.\n\nDatabases like MySQL, PostgreSQL etc. pull similar tricks with their\nfuzzy table statistics.\n"},{"id":"359733","messageId":"20181005200208.GA21032@sigill.intra.peff.net","threadId":"49479","inReplyTo":"14529aaa-fee0-e839-4629-bf87cdd13e31@gmail.com","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-05T20:02:08Z","receivedAt":"2018-10-05T20:02:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 05, 2018 at 04:00:12PM -0400, Derrick Stolee wrote:\n\n> On 10/5/2018 3:47 PM, Jeff King wrote:\n> > On Fri, Oct 05, 2018 at 03:41:40PM -0400, Derrick Stolee wrote:\n> > \n> > > > So can we really just take (total_objects - commit_graph_objects) and\n> > > > compare it to some threshold?\n> > > The commit-graph only stores the number of _commits_, not total objects.\n> > Oh, right, of course. That does throw a monkey wrench in that line of\n> > thought. ;)\n> > \n> > There's unfortunately not a fast way of doing that. One option would be\n> > to keep a counter of \"ungraphed commit objects\", and have callers update\n> > it. Anybody admitting a pack via index-pack or unpack-objects can easily\n> > get this information. Commands like fast-import can do likewise, and\n> > \"git commit\" obviously increments it by one.\n> > \n> > I'm not excited about adding a new global on-disk data structure (and\n> > the accompanying lock).\n> \n> If we want, then we can add an optional chunk to the commit-graph file that\n> stores the object count.\n\nYeah, that's probably a saner route, since we have to do the write then\nanyway.\n\n-Peff\n"},{"id":"359734","messageId":"20181005200948.GB21032@sigill.intra.peff.net","threadId":"49479","inReplyTo":"875zyggo2s.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-05T20:09:48Z","receivedAt":"2018-10-05T20:09:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 05, 2018 at 10:01:31PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > There's unfortunately not a fast way of doing that. One option would be\n> > to keep a counter of \"ungraphed commit objects\", and have callers update\n> > it. Anybody admitting a pack via index-pack or unpack-objects can easily\n> > get this information. Commands like fast-import can do likewise, and\n> > \"git commit\" obviously increments it by one.\n> >\n> > I'm not excited about adding a new global on-disk data structure (and\n> > the accompanying lock).\n> \n> You don't really need a new global datastructure to solve this\n> problem. It would be sufficient to have git-gc itself write out a 4-line\n> text file after it runs saying how many tags, commits, trees and blobs\n> it found on its last run.\n>\n> You can then fuzzily compare object counts v.s. commit counts for the\n> purposes of deciding whether something like the commit-graph needs to be\n> updated, while assuming that whatever new data you have has similar\n> enough ratios of those as your existing data.\n\nI think this is basically the same thing as Stolee's suggestion to keep\nthe total object count in the commit-graph file. The only difference is\nhere is that we know the actual ratio of commit to blobs for this\nparticular repository. But I don't think we need to know that. As you\nsaid, this is fuzzy anyway, so a single number for \"update the graph\nwhen there are N new objects\" is likely enough.\n\nIf you had a repository with an unusually large tree, you'd end up\nrebuilding the graph more often. But I think it would probably be OK, as\nwe're primarily trying not to waste time doing a graph rebuild when\nwe've only done a small amount of other work. But if we just shoved a\nton of objects through index-pack then we did a lot of work, whether\nthose were commit objects or not.\n\n-Peff\n"},{"id":"359836","messageId":"20181008164141.GZ23446@szeder.dev","threadId":"49479","inReplyTo":"20181003191805.GB16666@sigill.intra.peff.net","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-08T16:41:41Z","receivedAt":"2018-10-08T16:41:48Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Oct 03, 2018 at 03:18:05PM -0400, Jeff King wrote:\n> I'm still excited about the prospect of a bloom filter for paths which\n> each commit touches. I think that's the next big frontier in getting\n> things like \"git log -- path\" to a reasonable run-time.\n\nThere is certainly potential there.  With a (very) rough PoC\nexperiment, a 8MB bloom filter, and a carefully choosen path I can\nachieve a nice, almost 25x speedup:\n\n  $ time git rev-list --count HEAD -- t/valgrind/valgrind.sh\n  6\n\n  real    0m1.563s\n  user    0m1.519s\n  sys     0m0.045s\n\n  $ time GIT_USE_POC_BLOOM_FILTER=y ~/src/git/git rev-list --count HEAD -- t/valgrind/valgrind.sh\n  6\n\n  real    0m0.063s\n  user    0m0.043s\n  sys     0m0.020s\n\n  bloom filter total queries: 16269 definitely not: 16195 maybe: 74 false positives: 64 fp ratio: 0.003934\n\nBut I'm afraid it will take a while until I get around to turn it into\nsomething presentable...\n\n"},{"id":"359837","messageId":"a66afe22-0523-c785-91e6-bf545683c67d@gmail.com","threadId":"49479","inReplyTo":"20181008164141.GZ23446@szeder.dev","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-08T16:57:34Z","receivedAt":"2018-10-08T16:57:37Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/8/2018 12:41 PM, SZEDER Gábor wrote:\n> On Wed, Oct 03, 2018 at 03:18:05PM -0400, Jeff King wrote:\n>> I'm still excited about the prospect of a bloom filter for paths which\n>> each commit touches. I think that's the next big frontier in getting\n>> things like \"git log -- path\" to a reasonable run-time.\n> There is certainly potential there.  With a (very) rough PoC\n> experiment, a 8MB bloom filter, and a carefully choosen path I can\n> achieve a nice, almost 25x speedup:\n>\n>    $ time git rev-list --count HEAD -- t/valgrind/valgrind.sh\n>    6\n>\n>    real    0m1.563s\n>    user    0m1.519s\n>    sys     0m0.045s\n>\n>    $ time GIT_USE_POC_BLOOM_FILTER=y ~/src/git/git rev-list --count HEAD -- t/valgrind/valgrind.sh\n>    6\n>\n>    real    0m0.063s\n>    user    0m0.043s\n>    sys     0m0.020s\n>\n>    bloom filter total queries: 16269 definitely not: 16195 maybe: 74 false positives: 64 fp ratio: 0.003934\nNice! These numbers make sense to me, in terms of how many TREESAME \nqueries we actually need to perform for such a query.\n> But I'm afraid it will take a while until I get around to turn it into\n> something presentable...\nDo you have the code pushed somewhere public where one could take a \nlook? I could provide some early feedback.\n\nThanks,\n-Stolee\n"},{"id":"359846","messageId":"20181008181015.GA23446@szeder.dev","threadId":"49479","inReplyTo":"a66afe22-0523-c785-91e6-bf545683c67d@gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-08T18:10:15Z","receivedAt":"2018-10-08T18:10:23Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 08, 2018 at 12:57:34PM -0400, Derrick Stolee wrote:\n> On 10/8/2018 12:41 PM, SZEDER Gábor wrote:\n> >On Wed, Oct 03, 2018 at 03:18:05PM -0400, Jeff King wrote:\n> >>I'm still excited about the prospect of a bloom filter for paths which\n> >>each commit touches. I think that's the next big frontier in getting\n> >>things like \"git log -- path\" to a reasonable run-time.\n> >There is certainly potential there.  With a (very) rough PoC\n> >experiment, a 8MB bloom filter, and a carefully choosen path I can\n> >achieve a nice, almost 25x speedup:\n> >\n> >   $ time git rev-list --count HEAD -- t/valgrind/valgrind.sh\n> >   6\n> >\n> >   real    0m1.563s\n> >   user    0m1.519s\n> >   sys     0m0.045s\n> >\n> >   $ time GIT_USE_POC_BLOOM_FILTER=y ~/src/git/git rev-list --count HEAD -- t/valgrind/valgrind.sh\n> >   6\n> >\n> >   real    0m0.063s\n> >   user    0m0.043s\n> >   sys     0m0.020s\n> >\n> >   bloom filter total queries: 16269 definitely not: 16195 maybe: 74 false positives: 64 fp ratio: 0.003934\n\n> Nice! These numbers make sense to me, in terms of how many TREESAME queries\n> we actually need to perform for such a query.\n\nYeah...  because you didn't notice that I deliberately cheated :)\n\nAs it turned out, it's not just about the number of diff queries that\nwe can spare, but, for the speedup _ratio_, it's more about how\nexpensive those diff queries are.\n\ngit.git has a rather flat hierarchy, and 't/' is the 372th entry in\nthe current root tree object, while 'valgrind/' is the 923th entry,\nand the diff machinery spends considerable time wading through the\nprevious entries.  Notice the \"carefully chosen path\" remark in my\nprevious email; I think this particular path has the highest number of\npreceeding tree entries, and, in addition, 't/' changes rather\nfrequently, so the diff machinery often has to scan two relatively big\ntree objects.  Had I chosen 'Documentation/RelNotes/1.5.0.1.txt'\ninstead, i.e. another path two directories deep, but whose leading\npath components are both near the beginning of the tree objects, the\nspeedup would be much less impressive: 0.282s vs. 0.049s, i.e. \"only\"\n~5.7x instead of ~24.8x.\n\n> >But I'm afraid it will take a while until I get around to turn it into\n> >something presentable...\n> Do you have the code pushed somewhere public where one could take a look? I\n> Do you have the code pushed somewhere public where one could take a \n> look? I could provide some early feedback.\n\nNah, definitely not...  I know full well how embarassingly broken this\nimplementation is, I don't need others to tell me that ;)\n\n"},{"id":"359848","messageId":"9ad5f166-f7c5-de79-0f86-f1f952cd33d2@gmail.com","threadId":"49479","inReplyTo":"20181008181015.GA23446@szeder.dev","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-08T18:29:47Z","receivedAt":"2018-10-08T18:29:50Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/8/2018 2:10 PM, SZEDER Gábor wrote:\n> On Mon, Oct 08, 2018 at 12:57:34PM -0400, Derrick Stolee wrote:\n>> Nice! These numbers make sense to me, in terms of how many TREESAME queries\n>> we actually need to perform for such a query.\n> Yeah...  because you didn't notice that I deliberately cheated :)\n>\n> As it turned out, it's not just about the number of diff queries that\n> we can spare, but, for the speedup _ratio_, it's more about how\n> expensive those diff queries are.\n>\n> git.git has a rather flat hierarchy, and 't/' is the 372th entry in\n> the current root tree object, while 'valgrind/' is the 923th entry,\n> and the diff machinery spends considerable time wading through the\n> previous entries.  Notice the \"carefully chosen path\" remark in my\n> previous email; I think this particular path has the highest number of\n> preceeding tree entries, and, in addition, 't/' changes rather\n> frequently, so the diff machinery often has to scan two relatively big\n> tree objects.  Had I chosen 'Documentation/RelNotes/1.5.0.1.txt'\n> instead, i.e. another path two directories deep, but whose leading\n> path components are both near the beginning of the tree objects, the\n> speedup would be much less impressive: 0.282s vs. 0.049s, i.e. \"only\"\n> ~5.7x instead of ~24.8x.\n\nThis is expected. The performance ratio is better when the path is any \nof the following:\n\n1. A very deep path (need to walk multiple trees to answer TREESAME)\n\n2. An entry is late in a very wide tree (need to spend extra time \nparsing tree object)\n\n3. The path doesn't change very often (need to inspect many TREESAME \npairs before finding enough interesting commits)\n\n4. Some sub-path changes often (so the TREESAME comparison needs to \nparse beyond that sub-path often)\n\nOur standard examples (Git and Linux repos) don't have many paths that \nhave these properties. But: they do exist. In other projects, this is \nactually typical. Think about Java projects that frequently have ~5 \nlevels of folders before actually touching a code file.\n\nWhen I was implementing the Bloom filter feature for Azure Repos, I ran \nperformance tests on the Linux repo using a random sampling of paths. \nThe typical speedup was 5x while some outliers were in the 25x range.\n\n>\n>>> But I'm afraid it will take a while until I get around to turn it into\n>>> something presentable...\n>> Do you have the code pushed somewhere public where one could take a look? I\n>> Do you have the code pushed somewhere public where one could take a\n>> look? I could provide some early feedback.\n> Nah, definitely not...  I know full well how embarassingly broken this\n> implementation is, I don't need others to tell me that ;)\nThere are two questions that I was hoping to answer by looking at your code:\n\n1. How do you store your Bloom filter? Is it connected to the \ncommit-graph and split on a commit-by-commit basis (storing \"$path\" as a \nkey), or is it one huge Bloom filter (storing \"$commitid:$path\" as key)?\n\n2. Where does your Bloom filter check plug into the TREESAME logic? I \nhaven't investigated this part, but hopefully it isn't too complicated.\n\nThanks,\n-Stolee\n"},{"id":"359881","messageId":"xmqqd0skt53o.fsf@gitster-ct.c.googlers.com","threadId":"49479","inReplyTo":"20181008164141.GZ23446@szeder.dev","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-08T23:02:03Z","receivedAt":"2018-10-08T23:02:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor <szeder.dev@gmail.com> writes:\n\n> There is certainly potential there.  With a (very) rough PoC\n> experiment, a 8MB bloom filter, and a carefully choosen path I can\n> achieve a nice, almost 25x speedup:\n>\n>   $ time git rev-list --count HEAD -- t/valgrind/valgrind.sh\n>   6\n>\n>   real    0m1.563s\n>   user    0m1.519s\n>   sys     0m0.045s\n>\n>   $ time GIT_USE_POC_BLOOM_FILTER=y ~/src/git/git rev-list --count HEAD -- t/valgrind/valgrind.sh\n>   6\n>\n>   real    0m0.063s\n>   user    0m0.043s\n>   sys     0m0.020s\n\nEven though I somehow sense a sign of exaggeration in [v] in the\npathname, it still is quite respectable.\n\n>   bloom filter total queries: 16269 definitely not: 16195 maybe: 74 false positives: 64 fp ratio: 0.003934\n>\n> But I'm afraid it will take a while until I get around to turn it into\n> something presentable...\n\nThat's OK.  This is an encouraging result.\n\nJust from curiousity, how are you keying the filter?  tree object\nname of the top-level and full path concatenated or something like\nthat?\n"},{"id":"359889","messageId":"20181009030803.GA6250@sigill.intra.peff.net","threadId":"49479","inReplyTo":"9ad5f166-f7c5-de79-0f86-f1f952cd33d2@gmail.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-09T03:08:03Z","receivedAt":"2018-10-09T03:08:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 08, 2018 at 02:29:47PM -0400, Derrick Stolee wrote:\n\n> > > > But I'm afraid it will take a while until I get around to turn it into\n> > > > something presentable...\n> > > Do you have the code pushed somewhere public where one could take a look? I\n> > > Do you have the code pushed somewhere public where one could take a\n> > > look? I could provide some early feedback.\n> > Nah, definitely not...  I know full well how embarassingly broken this\n> > implementation is, I don't need others to tell me that ;)\n> There are two questions that I was hoping to answer by looking at your code:\n> \n> 1. How do you store your Bloom filter? Is it connected to the commit-graph\n> and split on a commit-by-commit basis (storing \"$path\" as a key), or is it\n> one huge Bloom filter (storing \"$commitid:$path\" as key)?\n\nI guess you've probably thought all of this through for your\nimplementation, but let me pontificate.\n\nI'd have done it as one fixed-size filter per commit. Then you should be\nable to hash the path keys once, and apply the result as a bitwise query\nto each individual commit (I'm assuming that it's constant-time to\naccess the filter for each, as an index into an mmap'd array, with the\noffset coming from a commit-graph entry we'd be able to look up anyway).\n\nI think it would also be easier to deal with maintenance, since each\nfilter is independent (IIRC, you cannot delete from a bloom filter\nwithout re-adding all of the other keys).\n\nThe obvious downside is that it's O(commits) storage instead of O(1).\n\nNow let me ponder a bit further afield. A bloom filter lets us answer\nthe question \"did this commit (probably) touch these paths?\". But it\ndoes not let us answer \"which paths did this commit touch?\".\n\nThat second one is less useful than you might think, because we almost\nalways care about not just the names of the paths, but their actual\nobject ids. Think about a --raw diff, or even traversing for\nreachability (where if we knew the tree-diff cheaply, we could avoid\nasking \"have we seen this yet?\" on most of the tree entries). The names\nalone can make that a bit faster, but in the worst case you still have\nto walk the whole tree to find their entries.\n\nBut there's also a related question: how do we match pathspec patterns?\nFor a changed path like \"foo/bar/baz\", I imagine a bloom filter would\nmark all of \"foo\", \"foo/bar\", and \"foo/bar/baz\". But what about \"*.c\"? I\ndon't think a bloom filter can answer that.\n\nAt least not by itself. If we imagine that the commit-graph also had an\nalphabetized list of every path in every tree, then it's easy: apply the\nglob to that list once to get a set of concrete paths, and then query\nthe bloom filters for those. And that list actually isn't too big. The\ncomplete set of paths in linux.git is only about 300k gzipped (I think\nthat's the most relevant measure, since it's an obvious win to avoid\nrepeating shared prefixes of long paths).\n\nImagine we have that list. Is a bloom filter still the best data\nstructure for each commit? At the point that we have the complete\nuniverse of paths, we could give each commit a bitmap of changed paths.\nThat lets us ask \"did this commit touch these paths\" (collect the bits\nfrom the list of paths, then check for 1's), as well as \"what paths did\nwe touch\" (collect the 1 bits, and then index the path list).  Those\nbitmaps should compress very well via EWAH or similar (most of them\nwould be huge stretches of 0's punctuated by short runs of 1's).\n\nSo that seems promising to me (or at least not an obvious dead-end). I\ndo think maintenance gets to be a headache, though. Adding new paths\npotentially means reordering the bitmaps, which means O(commits) work to\n\"incrementally\" update the structure. (Unless you always add the new\npaths at the end, but then you lose fast lookups in the list; that might\nbe an acceptable tradeoff).\n\nAnd finally, there's one more radical option: could we actually store a\nreal per-commit tree-diff cache? I.e., imagine that each commit had the\nequivalent of a --raw diff easily accessible, including object ids. That\nwould allow:\n\n  - fast pathspec matches, including globs\n\n  - fast --raw output (and faster -p output, since we can skip the tree\n    entirely)\n\n  - fast reachability traversals (we only need to bother to look at the\n    objects for changed entries)\n\nwhere \"fast\" is basically O(size of commit's changes), rather than\nO(size of whole tree). This was one of the big ideas of packv4 that\nnever materialized. You can _almost_ do it with packv2, since after all,\nwe end up storing many trees as deltas. But those deltas are byte-wise\nso it's hard for a reader to convert them directly into a pure-tree\ndiff (they also don't mention the \"deleted\" data, so it's really only\nhalf a diff).\n\nSo let's imagine we'd store such a cache external to the regular object\ndata (i.e., as a commit-graph entry). The \"log --raw\" diff of linux.git\nhas 1.7M entries. The paths should easily compress to a single 32-bit\ninteger (e.g., as an index into a big path list). The oids are 20 bytes.\nAdd a few bytes for modes. That's about 80MB. Big, but not impossibly\nso. Maybe pushing it for true gigantic repos, though.\n\nThose numbers are ignoring merges, too. The meaning of \"did this commit\ntouch that path\" is a lot trickier for a merge commit, and I think may\ndepend on context. I'm not sure how even a bloom filter solution would\nhandle that (I was assuming we'd mostly punt and let merges fall back to\nopening up the trees).\n\nPhew. That was a lot. I don't want to derail any useful work either of\nyou is doing. These are just things I've been thinking over (or even in\nsome cases experimenting with), and I think it's worth laying all the\noptions on the table. I won't be surprised if you'd considered and\nrejected any of these alternate approaches, but I'd be curious to hear\nthe counter-arguments. :)\n\n-Peff\n"},{"id":"359921","messageId":"f877020c-3098-e4c4-ad64-cca57f764b91@gmail.com","threadId":"49479","inReplyTo":"20181009030803.GA6250@sigill.intra.peff.net","subject":"Bloom Filters (was Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-09T13:48:20Z","receivedAt":"2018-10-09T13:48:26Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"(Changing title to reflect the new topic.)\n\nOn 10/8/2018 11:08 PM, Jeff King wrote:\n> On Mon, Oct 08, 2018 at 02:29:47PM -0400, Derrick Stolee wrote:\n>\n>> There are two questions that I was hoping to answer by looking at \n>> your code:\n>> 1. How do you store your Bloom filter? Is it connected to the commit-graph\n>> and split on a commit-by-commit basis (storing \"$path\" as a key), or is it\n>> one huge Bloom filter (storing \"$commitid:$path\" as key)?\n> I guess you've probably thought all of this through for your\n> implementation, but let me pontificate.\n>\n> I'd have done it as one fixed-size filter per commit. Then you should be\n> able to hash the path keys once, and apply the result as a bitwise query\n> to each individual commit (I'm assuming that it's constant-time to\n> access the filter for each, as an index into an mmap'd array, with the\n> offset coming from a commit-graph entry we'd be able to look up anyway).\n\nYou're right that we want to hash the path a constant number of times. \nAdd in that we want to re-use information already serialized when \nupdating the file, and we see that having a commit-by-commit Bloom \nfilter is a good idea. Using (commit, path) pairs requires lots of \nre-hashing, repeated work when extending the filter, and poor locality \nwhen evaluating membership of a single key.\n\nThe nice thing is that you can generate k 32-bit hash values based on \ntwo 32-bit hash values that are \"independent enough\" (see [1]). We used \nMurmur3 with two different seed values to generate hashes a & b, then \nused the arithmetic progression a, a + b, a + 2b, ..., a + (k-1)b as our \nk hash values. These can be computed up front and then dropped into any \nsize filter using a simple modulo operation. This allows flexible sizes \nin our filters.\n\nI don't think fixed size filters are a good idea, and instead would opt \nfor flex-sized filters with a maximum size. The typical parameters use 7 \nhash functions and a filter size of (at least) 10 bits per entry. For \nmost commits (say 60-70%), 256 bits (32 bytes) is enough. Using a \nmaximum of 512 bytes covers 99% of commits. We will want these bounds to \nbe configurable via config. If we had a fixed size, then we either make \nit too small (and don't have sufficient coverage of commits) or too \nlarge (and waste a lot of space on the commits that change very little).\n\nWe can store these flex-sized filters in the commit-graph using two \ncolumns of data (two new optional chunks):\n\n* Bloom filter data: stores the binary data of each commit's Bloom \nfilter, concatenated together in the same order as the commits appear in \nthe commit-graph.\n\n* Bloom filter positions: The ith position of this column stores the \nstart of the (i+1)th Bloom filter (The 0th filter starts at byte 0). A \nBloom filter of size 0 is intended to mean \"we didn't store this filter \nbecause it would be too large\". We can compute the lengths of the filter \nby inspecting adjacent values.\n\nIn order to be flexible, we will want to encode some basic information \ninto the Bloom filter data chunk, such as a tuple of (hash version, num \nhash bits, num bits per entry). This allows us to change the parameters \nin config but still be able to read a serialized filter. Here I assume \nthat all filters share the same parameters. The \"hash version\" here is \ndifferent than the_hash_algo, because we don't care about cryptographic \nsecurity, only a uniform distrubution (hence, Murmur3 is a good, fast \noption).\n\n[1] \nhttps://web.archive.org/web/20090131053735/http://www.eecs.harvard.edu/~kirsch/pubs/bbbf/esa06.pdf\n\n>\n> I think it would also be easier to deal with maintenance, since each\n> filter is independent (IIRC, you cannot delete from a bloom filter\n> without re-adding all of the other keys).\n>\n> The obvious downside is that it's O(commits) storage instead of O(1).\nIt would always be O(changes), as the Bloom filter needs to grow in size \nas the number of entries grows.\n> Now let me ponder a bit further afield. A bloom filter lets us answer\n> the question \"did this commit (probably) touch these paths?\". But it\n> does not let us answer \"which paths did this commit touch?\".\n>\n> That second one is less useful than you might think, because we almost\n> always care about not just the names of the paths, but their actual\n> object ids. Think about a --raw diff, or even traversing for\n> reachability (where if we knew the tree-diff cheaply, we could avoid\n> asking \"have we seen this yet?\" on most of the tree entries). The names\n> alone can make that a bit faster, but in the worst case you still have\n> to walk the whole tree to find their entries.\n>\n> But there's also a related question: how do we match pathspec patterns?\n> For a changed path like \"foo/bar/baz\", I imagine a bloom filter would\n> mark all of \"foo\", \"foo/bar\", and \"foo/bar/baz\". But what about \"*.c\"? I\n> don't think a bloom filter can answer that.\n\nThe filter needs to store every path that would be considered \"not \nTREESAME\". It can't store wildcards, so you would need to evaluate the \nwildcard and test all of those paths individually (not a good idea).\n\n> At least not by itself. If we imagine that the commit-graph also had an\n> alphabetized list of every path in every tree, then it's easy: apply the\n> glob to that list once to get a set of concrete paths, and then query\n> the bloom filters for those. And that list actually isn't too big. The\n> complete set of paths in linux.git is only about 300k gzipped (I think\n> that's the most relevant measure, since it's an obvious win to avoid\n> repeating shared prefixes of long paths).\nAs you mention below, we would actually want a list of \"every path that \nhas ever appeared in the repo\".\n> Imagine we have that list. Is a bloom filter still the best data\n> structure for each commit? At the point that we have the complete\n> universe of paths, we could give each commit a bitmap of changed paths.\n> That lets us ask \"did this commit touch these paths\" (collect the bits\n> from the list of paths, then check for 1's), as well as \"what paths did\n> we touch\" (collect the 1 bits, and then index the path list).  Those\n> bitmaps should compress very well via EWAH or similar (most of them\n> would be huge stretches of 0's punctuated by short runs of 1's).\n\nI'm not convinced we would frequently have runs of 1's, and the bitmap \nwould not compress much better than simply listing the positions. For \nexample, a path \"foo/bar\" that resolves to a tree would only start a run \nif the next changes are the initial section of entries in that tree \n(sorted lexicographically) such as \"foo/bar/a, foo/bar/b\". If we deepen \ninto a tree, then we will break the run of 1's unless we changed every \npath deeper than that tree.\n\n> So that seems promising to me (or at least not an obvious dead-end). I\n> do think maintenance gets to be a headache, though. Adding new paths\n> potentially means reordering the bitmaps, which means O(commits) work to\n> \"incrementally\" update the structure. (Unless you always add the new\n> paths at the end, but then you lose fast lookups in the list; that might\n> be an acceptable tradeoff).\n>\n> And finally, there's one more radical option: could we actually store a\n> real per-commit tree-diff cache? I.e., imagine that each commit had the\n> equivalent of a --raw diff easily accessible, including object ids. That\n> would allow:\n>\n>    - fast pathspec matches, including globs\n>\n>    - fast --raw output (and faster -p output, since we can skip the tree\n>      entirely)\n>\n>    - fast reachability traversals (we only need to bother to look at the\n>      objects for changed entries)\n>\n> where \"fast\" is basically O(size of commit's changes), rather than\n> O(size of whole tree). This was one of the big ideas of packv4 that\n> never materialized. You can _almost_ do it with packv2, since after all,\n> we end up storing many trees as deltas. But those deltas are byte-wise\n> so it's hard for a reader to convert them directly into a pure-tree\n> diff (they also don't mention the \"deleted\" data, so it's really only\n> half a diff).\n>\n> So let's imagine we'd store such a cache external to the regular object\n> data (i.e., as a commit-graph entry). The \"log --raw\" diff of linux.git\n> has 1.7M entries. The paths should easily compress to a single 32-bit\n> integer (e.g., as an index into a big path list). The oids are 20 bytes.\n> Add a few bytes for modes. That's about 80MB. Big, but not impossibly\n> so. Maybe pushing it for true gigantic repos, though.\n\nAbove, I mentioned my gut reaction that storing a \"changed path bitmap\" \nper commit would not compress well. That puts that implementation very \nclose to the one you suggest here (except we also store the OID changes).\n\nI just want to compare your 80MB here to ~4MB it would take to store \nthose changed paths in Bloom filters (10 bits per entry -> ~2MB, but \nadding some slop for the commit-by-commit storage).\n\n> Those numbers are ignoring merges, too. The meaning of \"did this commit\n> touch that path\" is a lot trickier for a merge commit, and I think may\n> depend on context. I'm not sure how even a bloom filter solution would\n> handle that (I was assuming we'd mostly punt and let merges fall back to\n> opening up the trees).\n\nMy solution here is to always store the list of paths changed against \nthe first parent. If we evaluate TREESAME against our first parent while \ncomputing simplified file history, then we continue along first-parent \nhistory. It is possible to store filters for every parent, but I don't \nrecommend it. The merge commit will typically have many more change \npaths against the second parent, since the second parent is usually \nbringing in a small change done by few to catch up to the work done in \nparallel by many. Those diffs will frequently run over our limit.\n\n> Phew. That was a lot. I don't want to derail any useful work either of\n> you is doing. These are just things I've been thinking over (or even in\n> some cases experimenting with), and I think it's worth laying all the\n> options on the table. I won't be surprised if you'd considered and\n> rejected any of these alternate approaches, but I'd be curious to hear\n> the counter-arguments. :)\n\nThis is a good discussion to have, since the commit-graph feature is \ngetting to a stable place. We still have ongoing algorithm work with \ngeneration numbers, but this Bloom filter discussion (and \nimplementation) can happen in parallel.\n\nThanks,\n\n-Stolee\n\n"},{"id":"359937","messageId":"87ftxfez7s.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"f877020c-3098-e4c4-ad64-cca57f764b91@gmail.com","subject":"Re: Bloom Filters (was Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-09T18:45:11Z","receivedAt":"2018-10-09T18:45:17Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Oct 09 2018, Derrick Stolee wrote:\n\n> The filter needs to store every path that would be considered \"not\n> TREESAME\". It can't store wildcards, so you would need to evaluate the\n> wildcard and test all of those paths individually (not a good idea).\n\nIf full paths are stored, yes, But have you considered instead of\nstoring paths, storing all trigrams that can be extracted from changed\npaths at that commit?\n\nI.e. instead of a change to \"t/t0000-basic.sh\" storing\n\"t/t0000-basic.sh\" we'd store [\"t/t\", \"/t0\", \"t00\", \"000\", \"00-\" ...]\netc.\n\nThat sort of approach would mean that e.g. \"t*000*\", \"*asi*.sh\"\netc. could all be indexed, and as long as we could find three\nconsecutive bytes of fixed string we'd have a chance to short-circuit,\nbut would need to degrade to a full tree unpack for e.g. \"t*\". We could\nalso special-case certain sub-three-char indexes, or to\n\"bi-grams\". E.g. to be able to index '*.c' or 't*' (first char at the\nbeginning of a string only).\n\nIt would mean having to check more things in the bloom filter for each\ncommit, but that's going to be hot in cache at that point so it'll\nprobably beat unpacking trees by far, and we could short-circuit exit at\nthe first one that returned false.\n"},{"id":"359938","messageId":"20181009184647.GA7014@sigill.intra.peff.net","threadId":"49479","inReplyTo":"f877020c-3098-e4c4-ad64-cca57f764b91@gmail.com","subject":"Re: Bloom Filters (was Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-09T18:46:47Z","receivedAt":"2018-10-09T18:46:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 09, 2018 at 09:48:20AM -0400, Derrick Stolee wrote:\n\n> [I snipped all of the parts about bloom filters that seemed entirely\n>  reasonable to me ;) ]\n\n> > Imagine we have that list. Is a bloom filter still the best data\n> > structure for each commit? At the point that we have the complete\n> > universe of paths, we could give each commit a bitmap of changed paths.\n> > That lets us ask \"did this commit touch these paths\" (collect the bits\n> > from the list of paths, then check for 1's), as well as \"what paths did\n> > we touch\" (collect the 1 bits, and then index the path list).  Those\n> > bitmaps should compress very well via EWAH or similar (most of them\n> > would be huge stretches of 0's punctuated by short runs of 1's).\n> \n> I'm not convinced we would frequently have runs of 1's, and the bitmap would\n> not compress much better than simply listing the positions. For example, a\n> path \"foo/bar\" that resolves to a tree would only start a run if the next\n> changes are the initial section of entries in that tree (sorted\n> lexicographically) such as \"foo/bar/a, foo/bar/b\". If we deepen into a tree,\n> then we will break the run of 1's unless we changed every path deeper than\n> that tree.\n\nYeah, I doubt we'd really have runs of 1's (by short, I just mean 1 or\n2). I agree that listing the positions could work, though I sort of\nassumed that was more or less what a decent compressed bitmap would\nturn into. E.g., if bit N is set, we should be able to say \"N-1\nzeroes, 1 one\" in about the same size as we could say \"position N\".\n\nEWAH seems pretty awful in that regard. Or at least its serialized\nformat is (or maybe it's our implementation that is bad).\n\nThe patch below generates a bitmap for each commit in a repository (it\ndoesn't output the total list of paths; I've left that as an exercise\nfor the reader). On linux.git, the result is 57MB. But when I look at\nthe individual bitmap sizes (via GIT_TRACE), they're quite silly.\nStoring a single set bit takes 28 bytes in serialized form!\n\nThere are only around 120k unique paths (including prefix trees).\nNaively using run-length encoding and varints, our worst case should be\nsomething like 18-20 bits to say \"120k zeroes, then a 1, then all\nzeroes\".  And the average case should be better (you don't even need to\nsay \"120k\", but some smaller number).\n\nI wonder if Roaring does better here.\n\nGzipping the resulting bitmaps drops the total size to about 7.5MB.\nThat's not a particularly important number, but I think it shows that\nthe built-in ewah compression is far from ideal.\n\nJust listing the positions with a series of varints would generally be\nfine, since we expect sparse bitmaps. I just hoped that a good\nRLE scheme would degrade to roughly that for the sparse case, but also\nperform well for more dense cases.\n\nSo at any rate, I do think it would not be out of the question to store\nbitmaps like this. I'm much more worried about the maintenance cost of\nadding new entries incrementally. I think it's only feasible if we give\nup sorting, and then I wonder what other problems that might cause.\n\n-Peff\n\n-- >8 --\ndiff --git a/Makefile b/Makefile\nindex 13e1c52478..f6e823f2d6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -751,6 +751,7 @@ TEST_PROGRAMS_NEED_X += test-parse-options\n TEST_PROGRAMS_NEED_X += test-pkt-line\n TEST_PROGRAMS_NEED_X += test-svn-fe\n TEST_PROGRAMS_NEED_X += test-tool\n+TEST_PROGRAMS_NEED_X += test-tree-bitmap\n \n TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\n \ndiff --git a/t/helper/test-tree-bitmap.c b/t/helper/test-tree-bitmap.c\nnew file mode 100644\nindex 0000000000..bc5cf0e514\n--- /dev/null\n+++ b/t/helper/test-tree-bitmap.c\n@@ -0,0 +1,167 @@\n+#include \"cache.h\"\n+#include \"revision.h\"\n+#include \"diffcore.h\"\n+#include \"argv-array.h\"\n+#include \"ewah/ewok.h\"\n+\n+/* map of pathnames to bit positions */\n+struct pathmap_entry {\n+\tstruct hashmap_entry ent;\n+\tunsigned pos;\n+\tchar path[FLEX_ARRAY];\n+};\n+\n+static int pathmap_entry_hashcmp(const void *unused_cmp_data,\n+\t\t\t\t const void *entry,\n+\t\t\t\t const void *entry_or_key,\n+\t\t\t\t const void *keydata)\n+{\n+\tconst struct pathmap_entry *a = entry;\n+\tconst struct pathmap_entry *b = entry_or_key;\n+\tconst char *key = keydata;\n+\n+\treturn strcmp(a->path, key ? key : b->path);\n+}\n+\n+static int pathmap_entry_strcmp(const void *va, const void *vb)\n+{\n+\tstruct pathmap_entry *a = *(struct pathmap_entry **)va;\n+\tstruct pathmap_entry *b = *(struct pathmap_entry **)vb;\n+\treturn strcmp(a->path, b->path);\n+}\n+\n+struct walk_paths_data {\n+\tstruct hashmap *paths;\n+\tstruct commit *commit;\n+};\n+\n+static void walk_paths(diff_format_fn_t fn, struct hashmap *paths)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct rev_info revs;\n+\tstruct walk_paths_data data;\n+\tstruct commit *commit;\n+\n+\targv_array_pushl(&argv, \"rev-list\",\n+\t\t\t \"--all\", \"-t\", \"--no-renames\",\n+\t\t\t NULL);\n+\tinit_revisions(&revs, NULL);\n+\tsetup_revisions(argv.argc, argv.argv, &revs, NULL);\n+\trevs.diffopt.output_format = DIFF_FORMAT_CALLBACK;\n+\trevs.diffopt.format_callback = fn;\n+\trevs.diffopt.format_callback_data = &data;\n+\n+\tdata.paths = paths;\n+\n+\tprepare_revision_walk(&revs);\n+\twhile ((commit = get_revision(&revs))) {\n+\t\tdata.commit = commit;\n+\t\tdiff_tree_combined_merge(commit, 0, &revs);\n+\t}\n+\n+\treset_revision_walk();\n+\targv_array_clear(&argv);\n+}\n+\n+static void collect_commit_paths(struct diff_queue_struct *q,\n+\t\t\t\t struct diff_options *opts,\n+\t\t\t\t void *vdata)\n+{\n+\tstruct walk_paths_data *data = vdata;\n+\tint i;\n+\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tconst char *path = p->one->path;\n+\t\tstruct pathmap_entry *entry;\n+\t\tstruct hashmap_entry lookup;\n+\n+\t\thashmap_entry_init(&lookup, strhash(path));\n+\t\tentry = hashmap_get(data->paths, &lookup, path);\n+\t\tif (entry)\n+\t\t\tcontinue; /* already present */\n+\n+\t\tFLEX_ALLOC_STR(entry, path, path);\n+\t\tentry->ent = lookup;\n+\t\thashmap_put(data->paths, entry);\n+\t}\n+}\n+\n+/* assign a bit position to all possible paths */\n+static void collect_paths(struct hashmap *paths)\n+{\n+\tstruct pathmap_entry **sorted;\n+\tsize_t i, n;\n+\tstruct hashmap_iter iter;\n+\tstruct pathmap_entry *entry;\n+\n+\t/* grab all unique paths */\n+\thashmap_init(paths, pathmap_entry_hashcmp, NULL, 0);\n+\twalk_paths(collect_commit_paths, paths);\n+\n+\t/* and assign them bits in sorted order */\n+\tn = hashmap_get_size(paths);\n+\tALLOC_ARRAY(sorted, n);\n+\ti = 0;\n+\tfor (entry = hashmap_iter_first(paths, &iter);\n+\t     entry;\n+\t     entry = hashmap_iter_next(&iter)) {\n+\t\tassert(i < n);\n+\t\tsorted[i++] = entry;\n+\t}\n+\tQSORT(sorted, i, pathmap_entry_strcmp);\n+\tfor (i = 0; i < n; i++)\n+\t\tsorted[i]->pos = i;\n+\tfree(sorted);\n+}\n+\n+/* generate the bitmap for a single commit */\n+static void generate_bitmap(struct diff_queue_struct *q,\n+\t\t\t    struct diff_options *opts,\n+\t\t\t    void *vdata)\n+{\n+\tstruct walk_paths_data *data = vdata;\n+\tstruct bitmap *bitmap = bitmap_new();\n+\tstruct ewah_bitmap *ewah;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tsize_t i;\n+\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tconst char *path = p->one->path;\n+\t\tstruct pathmap_entry *entry;\n+\t\tstruct hashmap_entry lookup;\n+\n+\t\thashmap_entry_init(&lookup, strhash(path));\n+\t\tentry = hashmap_get(data->paths, &lookup, path);\n+\t\tif (!entry)\n+\t\t\tBUG(\"mysterious path appeared: %s\", path);\n+\n+\t\tbitmap_set(bitmap, entry->pos);\n+\t}\n+\n+\tewah = bitmap_to_ewah(bitmap);\n+\tewah_serialize_strbuf(ewah, &out);\n+\tfwrite(out.buf, 1, out.len, stdout);\n+\n+\ttrace_printf(\"bitmap %s %u %u\",\n+\t\t     oid_to_hex(&data->commit->object.oid),\n+\t\t     (unsigned)q->nr,\n+\t\t     (unsigned)out.len);\n+\n+\tstrbuf_release(&out);\n+\tewah_free(ewah);\n+\tbitmap_free(bitmap);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tstruct hashmap paths;\n+\n+\tsetup_git_directory();\n+\tcollect_paths(&paths);\n+\n+\twalk_paths(generate_bitmap, &paths);\n+\n+\treturn 0;\n+}\n"},{"id":"359939","messageId":"ec3ca377-0434-322e-4ab9-49e27f96f4af@gmail.com","threadId":"49479","inReplyTo":"20181009184647.GA7014@sigill.intra.peff.net","subject":"Re: Bloom Filters (was Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-09T19:03:08Z","receivedAt":"2018-10-09T19:03:15Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/9/2018 2:46 PM, Jeff King wrote:\n> On Tue, Oct 09, 2018 at 09:48:20AM -0400, Derrick Stolee wrote:\n>\n>> [I snipped all of the parts about bloom filters that seemed entirely\n>>   reasonable to me ;) ]\n>>> Imagine we have that list. Is a bloom filter still the best data\n>>> structure for each commit? At the point that we have the complete\n>>> universe of paths, we could give each commit a bitmap of changed paths.\n>>> That lets us ask \"did this commit touch these paths\" (collect the bits\n>>> from the list of paths, then check for 1's), as well as \"what paths did\n>>> we touch\" (collect the 1 bits, and then index the path list).  Those\n>>> bitmaps should compress very well via EWAH or similar (most of them\n>>> would be huge stretches of 0's punctuated by short runs of 1's).\n>> I'm not convinced we would frequently have runs of 1's, and the bitmap would\n>> not compress much better than simply listing the positions. For example, a\n>> path \"foo/bar\" that resolves to a tree would only start a run if the next\n>> changes are the initial section of entries in that tree (sorted\n>> lexicographically) such as \"foo/bar/a, foo/bar/b\". If we deepen into a tree,\n>> then we will break the run of 1's unless we changed every path deeper than\n>> that tree.\n> Yeah, I doubt we'd really have runs of 1's (by short, I just mean 1 or\n> 2). I agree that listing the positions could work, though I sort of\n> assumed that was more or less what a decent compressed bitmap would\n> turn into. E.g., if bit N is set, we should be able to say \"N-1\n> zeroes, 1 one\" in about the same size as we could say \"position N\".\n>\n> EWAH seems pretty awful in that regard. Or at least its serialized\n> format is (or maybe it's our implementation that is bad).\n>\n> The patch below generates a bitmap for each commit in a repository (it\n> doesn't output the total list of paths; I've left that as an exercise\n> for the reader). On linux.git, the result is 57MB. But when I look at\n> the individual bitmap sizes (via GIT_TRACE), they're quite silly.\n> Storing a single set bit takes 28 bytes in serialized form!\n>\n> There are only around 120k unique paths (including prefix trees).\n> Naively using run-length encoding and varints, our worst case should be\n> something like 18-20 bits to say \"120k zeroes, then a 1, then all\n> zeroes\".  And the average case should be better (you don't even need to\n> say \"120k\", but some smaller number).\n>\n> I wonder if Roaring does better here.\n\nIn these sparse cases, usually Roaring will organize the data as \"array \nchunks\" which are simply lists of the values. The thing that makes this \nstill compressible is that we store two bytes per entry, as the entries \nare grouped by a common most-significant two bytes. SInce you say ~120k \nunique paths, the Roaring bitmap would have two or three chunks per \nbitmap (and those chunks could be empty). The overhead to store the \nchunk positions, types, and lengths does come at a cost, but it's more \nlike 32 bytes _per commit_.\n\n> Gzipping the resulting bitmaps drops the total size to about 7.5MB.\n> That's not a particularly important number, but I think it shows that\n> the built-in ewah compression is far from ideal.\n>\n> Just listing the positions with a series of varints would generally be\n> fine, since we expect sparse bitmaps. I just hoped that a good\n> RLE scheme would degrade to roughly that for the sparse case, but also\n> perform well for more dense cases.\n>\n> So at any rate, I do think it would not be out of the question to store\n> bitmaps like this. I'm much more worried about the maintenance cost of\n> adding new entries incrementally. I think it's only feasible if we give\n> up sorting, and then I wonder what other problems that might cause.\nThe patch below gives me a starting point to try the Bloom filter \napproach and see what the numbers are like. You did all the \"git\" stuff \nlike computing the changed paths, so thanks!\n>\n> -- >8 --\n> diff --git a/Makefile b/Makefile\n> index 13e1c52478..f6e823f2d6 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -751,6 +751,7 @@ TEST_PROGRAMS_NEED_X += test-parse-options\n>   TEST_PROGRAMS_NEED_X += test-pkt-line\n>   TEST_PROGRAMS_NEED_X += test-svn-fe\n>   TEST_PROGRAMS_NEED_X += test-tool\n> +TEST_PROGRAMS_NEED_X += test-tree-bitmap\n>   \n>   TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\n>   \n> diff --git a/t/helper/test-tree-bitmap.c b/t/helper/test-tree-bitmap.c\n> new file mode 100644\n> index 0000000000..bc5cf0e514\n> --- /dev/null\n> +++ b/t/helper/test-tree-bitmap.c\n> @@ -0,0 +1,167 @@\n> +#include \"cache.h\"\n> +#include \"revision.h\"\n> +#include \"diffcore.h\"\n> +#include \"argv-array.h\"\n> +#include \"ewah/ewok.h\"\n> +\n> +/* map of pathnames to bit positions */\n> +struct pathmap_entry {\n> +\tstruct hashmap_entry ent;\n> +\tunsigned pos;\n> +\tchar path[FLEX_ARRAY];\n> +};\n> +\n> +static int pathmap_entry_hashcmp(const void *unused_cmp_data,\n> +\t\t\t\t const void *entry,\n> +\t\t\t\t const void *entry_or_key,\n> +\t\t\t\t const void *keydata)\n> +{\n> +\tconst struct pathmap_entry *a = entry;\n> +\tconst struct pathmap_entry *b = entry_or_key;\n> +\tconst char *key = keydata;\n> +\n> +\treturn strcmp(a->path, key ? key : b->path);\n> +}\n> +\n> +static int pathmap_entry_strcmp(const void *va, const void *vb)\n> +{\n> +\tstruct pathmap_entry *a = *(struct pathmap_entry **)va;\n> +\tstruct pathmap_entry *b = *(struct pathmap_entry **)vb;\n> +\treturn strcmp(a->path, b->path);\n> +}\n> +\n> +struct walk_paths_data {\n> +\tstruct hashmap *paths;\n> +\tstruct commit *commit;\n> +};\n> +\n> +static void walk_paths(diff_format_fn_t fn, struct hashmap *paths)\n> +{\n> +\tstruct argv_array argv = ARGV_ARRAY_INIT;\n> +\tstruct rev_info revs;\n> +\tstruct walk_paths_data data;\n> +\tstruct commit *commit;\n> +\n> +\targv_array_pushl(&argv, \"rev-list\",\n> +\t\t\t \"--all\", \"-t\", \"--no-renames\",\n> +\t\t\t NULL);\n> +\tinit_revisions(&revs, NULL);\n> +\tsetup_revisions(argv.argc, argv.argv, &revs, NULL);\n> +\trevs.diffopt.output_format = DIFF_FORMAT_CALLBACK;\n> +\trevs.diffopt.format_callback = fn;\n> +\trevs.diffopt.format_callback_data = &data;\n> +\n> +\tdata.paths = paths;\n> +\n> +\tprepare_revision_walk(&revs);\n> +\twhile ((commit = get_revision(&revs))) {\n> +\t\tdata.commit = commit;\n> +\t\tdiff_tree_combined_merge(commit, 0, &revs);\n> +\t}\n> +\n> +\treset_revision_walk();\n> +\targv_array_clear(&argv);\n> +}\n> +\n> +static void collect_commit_paths(struct diff_queue_struct *q,\n> +\t\t\t\t struct diff_options *opts,\n> +\t\t\t\t void *vdata)\n> +{\n> +\tstruct walk_paths_data *data = vdata;\n> +\tint i;\n> +\n> +\tfor (i = 0; i < q->nr; i++) {\n> +\t\tstruct diff_filepair *p = q->queue[i];\n> +\t\tconst char *path = p->one->path;\n> +\t\tstruct pathmap_entry *entry;\n> +\t\tstruct hashmap_entry lookup;\n> +\n> +\t\thashmap_entry_init(&lookup, strhash(path));\n> +\t\tentry = hashmap_get(data->paths, &lookup, path);\n> +\t\tif (entry)\n> +\t\t\tcontinue; /* already present */\n> +\n> +\t\tFLEX_ALLOC_STR(entry, path, path);\n> +\t\tentry->ent = lookup;\n> +\t\thashmap_put(data->paths, entry);\n> +\t}\n> +}\n> +\n> +/* assign a bit position to all possible paths */\n> +static void collect_paths(struct hashmap *paths)\n> +{\n> +\tstruct pathmap_entry **sorted;\n> +\tsize_t i, n;\n> +\tstruct hashmap_iter iter;\n> +\tstruct pathmap_entry *entry;\n> +\n> +\t/* grab all unique paths */\n> +\thashmap_init(paths, pathmap_entry_hashcmp, NULL, 0);\n> +\twalk_paths(collect_commit_paths, paths);\n> +\n> +\t/* and assign them bits in sorted order */\n> +\tn = hashmap_get_size(paths);\n> +\tALLOC_ARRAY(sorted, n);\n> +\ti = 0;\n> +\tfor (entry = hashmap_iter_first(paths, &iter);\n> +\t     entry;\n> +\t     entry = hashmap_iter_next(&iter)) {\n> +\t\tassert(i < n);\n> +\t\tsorted[i++] = entry;\n> +\t}\n> +\tQSORT(sorted, i, pathmap_entry_strcmp);\n> +\tfor (i = 0; i < n; i++)\n> +\t\tsorted[i]->pos = i;\n> +\tfree(sorted);\n> +}\n> +\n> +/* generate the bitmap for a single commit */\n> +static void generate_bitmap(struct diff_queue_struct *q,\n> +\t\t\t    struct diff_options *opts,\n> +\t\t\t    void *vdata)\n> +{\n> +\tstruct walk_paths_data *data = vdata;\n> +\tstruct bitmap *bitmap = bitmap_new();\n> +\tstruct ewah_bitmap *ewah;\n> +\tstruct strbuf out = STRBUF_INIT;\n> +\tsize_t i;\n> +\n> +\tfor (i = 0; i < q->nr; i++) {\n> +\t\tstruct diff_filepair *p = q->queue[i];\n> +\t\tconst char *path = p->one->path;\n> +\t\tstruct pathmap_entry *entry;\n> +\t\tstruct hashmap_entry lookup;\n> +\n> +\t\thashmap_entry_init(&lookup, strhash(path));\n> +\t\tentry = hashmap_get(data->paths, &lookup, path);\n> +\t\tif (!entry)\n> +\t\t\tBUG(\"mysterious path appeared: %s\", path);\n> +\n> +\t\tbitmap_set(bitmap, entry->pos);\n> +\t}\n> +\n> +\tewah = bitmap_to_ewah(bitmap);\n> +\tewah_serialize_strbuf(ewah, &out);\n> +\tfwrite(out.buf, 1, out.len, stdout);\n> +\n> +\ttrace_printf(\"bitmap %s %u %u\",\n> +\t\t     oid_to_hex(&data->commit->object.oid),\n> +\t\t     (unsigned)q->nr,\n> +\t\t     (unsigned)out.len);\n> +\n> +\tstrbuf_release(&out);\n> +\tewah_free(ewah);\n> +\tbitmap_free(bitmap);\n> +}\n> +\n> +int cmd_main(int argc, const char **argv)\n> +{\n> +\tstruct hashmap paths;\n> +\n> +\tsetup_git_directory();\n> +\tcollect_paths(&paths);\n> +\n> +\twalk_paths(generate_bitmap, &paths);\n> +\n> +\treturn 0;\n> +}\n"},{"id":"359940","messageId":"20181009193445.21908-1-szeder.dev@gmail.com","threadId":"49479","inReplyTo":"a66afe22-0523-c785-91e6-bf545683c67d@gmail.com","subject":"[PATCH 0/4] Bloom filter experiment","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-09T19:34:41Z","receivedAt":"2018-10-09T19:35:13Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"To keep the ball rolling, here is my proof of concept in a somewhat\ncleaned-up form, with still plenty of rough edges.\n\nYou can play around with it like this:\n\n  $ GIT_USE_POC_BLOOM_FILTER=$((8*1024*1024*8)) git commit-graph write\n  Computing commit graph generation numbers: 100% (52801/52801), done.\n  Computing bloom filter: 100% (52801/52801), done.\n  # Yeah, I even added progress indicator! :)\n  $ GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y git rev-list --count --full-history HEAD -- t/valgrind/valgrind.sh\n  886\n  20:40:24.783699 revision.c:486          bloom filter total queries: 66095 definitely not: 64953 maybe: 1142 false positives: 256 fp ratio: 0.003873\n\nThe value of $GIT_USE_POC_BLOOM_FILTER only really matters when writing\nthe Bloom filter, and it specifies the number of bits in the filter's\nbitmap, IOW the above command creates a 8MB Bloom filter.  To make use\nof the filter the variable can be anything non-empty.\n\nWriting the Bloom filter is very slow as it is (yeah, that's why\nbothered with the progress indicator ;).  I wrote about it in patch 2's\ncommit message: the cause for about half of the slowness is rather\nobvious, but I don't (yet) know what's responsible for the other half.\n\n\nNot a single test...  but I've run loops over all files in git.git\ncomparing 'git rev-list HEAD -- $file's output with and without the\nBloom filter, and, surprisingly, they match.  My quick'n'dirty\nexperiments usually don't fare this well...\n\n\nIt's also available at:\n\n  https://github.com/szeder/git bloom-filter-experiment\n\n\nSZEDER Gábor (4):\n  Add a (very) barebones Bloom filter implementation\n  commit-graph: write a Bloom filter containing changed paths for each\n    commit\n  revision.c: use the Bloom filter to speed up path-limited revision\n    walks\n  revision.c: add GIT_TRACE_BLOOM_FILTER for a bit of statistics\n\n Makefile       |   1 +\n bloom-filter.c | 103 +++++++++++++++++++++++++++++++++++++++\n bloom-filter.h |  39 +++++++++++++++\n commit-graph.c | 116 ++++++++++++++++++++++++++++++++++++++++++++\n pathspec.h     |   1 +\n revision.c     | 129 +++++++++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 389 insertions(+)\n create mode 100644 bloom-filter.c\n create mode 100644 bloom-filter.h\n\n-- \n2.19.1.409.g0a0ee5eb6b\n\n"},{"id":"359941","messageId":"20181009193445.21908-2-szeder.dev@gmail.com","threadId":"49479","inReplyTo":"20181009193445.21908-1-szeder.dev@gmail.com","subject":"[PATCH 1/4] Add a (very) barebones Bloom filter implementation","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-09T19:34:42Z","receivedAt":"2018-10-09T19:35:17Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"---\n Makefile       |   1 +\n bloom-filter.c | 103 +++++++++++++++++++++++++++++++++++++++++++++++++\n bloom-filter.h |  39 +++++++++++++++++++\n 3 files changed, 143 insertions(+)\n create mode 100644 bloom-filter.c\n create mode 100644 bloom-filter.h\n\ndiff --git a/Makefile b/Makefile\nindex 13e1c52478..850eafb3ee 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -827,6 +827,7 @@ LIB_OBJS += base85.o\n LIB_OBJS += bisect.o\n LIB_OBJS += blame.o\n LIB_OBJS += blob.o\n+LIB_OBJS += bloom-filter.o\n LIB_OBJS += branch.o\n LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle.o\ndiff --git a/bloom-filter.c b/bloom-filter.c\nnew file mode 100644\nindex 0000000000..7dce0e35fa\n--- /dev/null\n+++ b/bloom-filter.c\n@@ -0,0 +1,103 @@\n+#include \"cache.h\"\n+#include \"bloom-filter.h\"\n+\n+void bloom_filter_init(struct bloom_filter *bf, uint32_t bit_size)\n+{\n+\tif (bit_size % CHAR_BIT)\n+\t\tBUG(\"invalid size for bloom filter\");\n+\n+\tbf->nr_entries = 0;\n+\tbf->bit_size = bit_size;\n+\tbf->bits = xmalloc(bit_size / CHAR_BIT);\n+}\n+\n+void bloom_filter_free(struct bloom_filter *bf)\n+{\n+\tbf->nr_entries = 0;\n+\tbf->bit_size = 0;\n+\tFREE_AND_NULL(bf->bits);\n+}\n+\n+\n+void bloom_filter_set_bits(struct bloom_filter *bf, const uint32_t *offsets,\n+\t\t\t   int nr_offsets, int nr_entries)\n+{\n+\tint i;\n+\tfor (i = 0; i < nr_offsets; i++) {\n+\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size) / CHAR_BIT;\n+\t\tunsigned char mask = 1 << offsets[i] % CHAR_BIT;\n+\t\tbf->bits[byte_offset] |= mask;\n+\t}\n+\tbf->nr_entries += nr_entries;\n+}\n+\n+int bloom_filter_check_bits(struct bloom_filter *bf, const uint32_t *offsets,\n+\t\t\t    int nr)\n+{\n+\tint i;\n+\tfor (i = 0; i < nr; i++) {\n+\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size) / CHAR_BIT;\n+\t\tunsigned char mask = 1 << offsets[i] % CHAR_BIT;\n+\t\tif (!(bf->bits[byte_offset] & mask))\n+\t\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n+\n+void bloom_filter_add_hash(struct bloom_filter *bf, const unsigned char *hash)\n+{\n+\tuint32_t offsets[GIT_MAX_RAWSZ / sizeof(uint32_t)];\n+\thashcpy((unsigned char*)offsets, hash);\n+\tbloom_filter_set_bits(bf, offsets,\n+\t\t\t     the_hash_algo->rawsz / sizeof(*offsets), 1);\n+}\n+\n+int bloom_filter_check_hash(struct bloom_filter *bf, const unsigned char *hash)\n+{\n+\tuint32_t offsets[GIT_MAX_RAWSZ / sizeof(uint32_t)];\n+\thashcpy((unsigned char*)offsets, hash);\n+\treturn bloom_filter_check_bits(bf, offsets,\n+\t\t\tthe_hash_algo->rawsz / sizeof(*offsets));\n+}\n+\n+void hashxor(const unsigned char *hash1, const unsigned char *hash2,\n+\t     unsigned char *out)\n+{\n+\tint i;\n+\tfor (i = 0; i < the_hash_algo->rawsz; i++)\n+\t\tout[i] = hash1[i] ^ hash2[i];\n+}\n+\n+/* hardcoded for now... */\n+static GIT_PATH_FUNC(git_path_bloom, \"objects/info/bloom\")\n+\n+int bloom_filter_load(struct bloom_filter *bf)\n+{\n+\tint fd = open(git_path_bloom(), O_RDONLY);\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\n+\tread_in_full(fd, &bf->nr_entries, sizeof(bf->nr_entries));\n+\tread_in_full(fd, &bf->bit_size, sizeof(bf->bit_size));\n+\tif (bf->bit_size % CHAR_BIT)\n+\t\tBUG(\"invalid size for bloom filter\");\n+\tbf->bits = xmalloc(bf->bit_size / CHAR_BIT);\n+\tread_in_full(fd, bf->bits, bf->bit_size / CHAR_BIT);\n+\n+\tclose(fd);\n+\n+\treturn 0;\n+}\n+\n+void bloom_filter_write(struct bloom_filter *bf)\n+{\n+\tint fd = xopen(git_path_bloom(), O_WRONLY | O_CREAT | O_TRUNC, 0666);\n+\n+\twrite_in_full(fd, &bf->nr_entries, sizeof(bf->nr_entries));\n+\twrite_in_full(fd, &bf->bit_size, sizeof(bf->bit_size));\n+\twrite_in_full(fd, bf->bits, bf->bit_size / CHAR_BIT);\n+\n+\tclose(fd);\n+}\ndiff --git a/bloom-filter.h b/bloom-filter.h\nnew file mode 100644\nindex 0000000000..94d0af1708\n--- /dev/null\n+++ b/bloom-filter.h\n@@ -0,0 +1,39 @@\n+#ifndef BLOOM_FILTER_H\n+#define BLOOM_FILTER_H\n+\n+#include \"git-compat-util.h\"\n+\n+struct bloom_filter {\n+\tuint32_t nr_entries;\n+\tuint32_t bit_size;\n+\tunsigned char *bits;\n+};\n+\n+\n+void bloom_filter_init(struct bloom_filter *bf, uint32_t bit_size);\n+void bloom_filter_free(struct bloom_filter *bf);\n+\n+void bloom_filter_set_bits(struct bloom_filter *bf, const uint32_t *offsets,\n+\t\t\t   int nr_offsets, int nr_enries);\n+int bloom_filter_check_bits(struct bloom_filter *bf, const uint32_t *offsets,\n+\t\t\t    int nr);\n+\n+/*\n+ * Turns the given (SHA1) hash into 5 unsigned ints, and sets the bits at\n+ * those positions (modulo the bitmap's size) in the Bloom filter.\n+ */\n+void bloom_filter_add_hash(struct bloom_filter *bf, const unsigned char *hash);\n+/*\n+ * Turns the given (SHA1) hash into 5 unsigned ints, and checks the bits at\n+ * those positions (modulo the bitmap's size) in the Bloom filter.\n+ * Returns 1 if all those bits are set, 0 otherwise.\n+ */\n+int bloom_filter_check_hash(struct bloom_filter *bf, const unsigned char *hash);\n+\n+void hashxor(const unsigned char *hash1, const unsigned char *hash2,\n+\t     unsigned char *out);\n+\n+int bloom_filter_load(struct bloom_filter *bf);\n+void bloom_filter_write(struct bloom_filter *bf);\n+\n+#endif\n-- \n2.19.1.409.g0a0ee5eb6b\n\n"},{"id":"359942","messageId":"20181009193445.21908-3-szeder.dev@gmail.com","threadId":"49479","inReplyTo":"20181009193445.21908-1-szeder.dev@gmail.com","subject":"[PATCH 2/4] commit-graph: write a Bloom filter containing changed paths for each commit","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-09T19:34:43Z","receivedAt":"2018-10-09T19:35:19Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"You can create a Bloom filter containing changed paths for each commit\nin the history by running:\n\n  $ GIT_USE_POC_BLOOM_FILTER=$((8*1024*1024*8)) git commit-graph write\n\nwhere the value of $GIT_USE_POC_BLOOM_FILTER must specify the number\nof bits used in the Bloom filter's bitmap.\n\nWriting the Bloom filter is tied into the 'git commit-graph' command,\nmainly because that's where it might end up anyway, if it turns out to\nbe useful, but for now it's written to a different file\n('object/info/bloom').  No incremental updates yet, the Bloom filter\nis regenerated from scratch each time.\n\nThere is one single, big Bloom filter for the whole history (mainly\nbecause that was the simplest way to get this PoC experiment up and\nrunning).  The Bloom filter stores tuples of (path, parent-oid,\ncommit-oid) using the hash function:\n\n  XOR(SHA1(path), XOR(parent-oid, commit-oid))\n\nThe resulting 20 bytes are turned into 5 unsigned 32 bit ints, which\nthen specify the positions of the bits to set or check in the Bloom\nfilter's bitmap (modulo the bitmap's size).\n\nThe parent oid is taken into account, because during revision walking\nthe diff is checked in rev_compare_tree(), which compares one commit\nto _one_ of its parents, and in case of merge commits there are\nmultiple rev_compare_tree() calls with the same commit but with\ndifferent parent parameters.\n\nCombining hashes with XOR is, in general, frowned upon, because of its\nintrinsic properties:\n\n  XOR(A, A) = 0\n  XOR(A, B) = XOR(B, A)\n\nIn this case it should be fine, because all of XOR's operands are\ncryptographic hashes, so we can safely assume that they'll never be\nthe same.\n\nAdd each leading directory of the changed file, i.e. for\n'dir/subdir/file' add 'dir' and 'dir/subdir' as well, so the Bloom\nfilter could be used to speed up commands like 'git log dir/subdir',\ntoo.\n\nCreating the Bloom filter is sloooow.  Running it on git.git takes\nabout 23s on my hardware, while\n\n  git log --format='%H%n%P' --name-only --all >/dev/null\n\ngathers all the information necessary for that in about 5.3s.\n\nAbout 30% of the runtime is wasted by naively hashing and rehashing\nthe same paths over and over again.  A hash function faster than SHA1\ncould help with that; I just haven't yet bothered with spicing up\nmemhash() and friends to produce 5 ints, and neither wanted to\nintroduce another hash function with wideer output just yet.  Or\nperhaps our hashmap mapping paths of files to their SHAs and the SHAs\nof their leading directories...\n\nThat's not the only factor though.  After ripping out all the loops\nfrom add_changes_to_bloom_filter() there are no repeated SHA1(path)\ncalculations and no writes to the Bloom filter at all, i.e. all what\nremains is revision walking and diffing, yet it still takes about 16s,\ni.e. aroung 3 times more than the above mentioned 'git log' command.\nI guess some other fields in 'struct rev_info' or 'struct\ndiff_options' need to be set, but both of those are huge, and I\nhaven't yet spotted which ones.\n---\n commit-graph.c | 116 +++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 116 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a1454c52a6..f415d3b41f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -14,6 +14,9 @@\n #include \"object-store.h\"\n #include \"alloc.h\"\n #include \"progress.h\"\n+#include \"bloom-filter.h\"\n+#include \"diff.h\"\n+#include \"diffcore.h\"\n \n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n@@ -709,6 +712,117 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n+static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n+\t\t\t\t\tstruct commit *parent,\n+\t\t\t\t\tstruct commit *commit,\n+\t\t\t\t\tstruct diff_options *diffopt)\n+{\n+\tunsigned char p_c_hash[GIT_MAX_RAWSZ];\n+\tint i;\n+\n+\thashxor(parent->object.oid.hash, commit->object.oid.hash, p_c_hash);\n+\n+\tdiff_tree_oid(&parent->object.oid, &commit->object.oid, \"\", diffopt);\n+\tdiffcore_std(diffopt);\n+\n+\tfor (i = 0; i < diff_queued_diff.nr; i++) {\n+\t\tconst char *path = diff_queued_diff.queue[i]->two->path;\n+\t\tconst char *p = path;\n+\n+\t\t/*\n+\t\t * Add each leading directory of the changed file, i.e. for\n+\t\t * 'dir/subdir/file' add 'dir' and 'dir/subdir' as well, so\n+\t\t * the Bloom filter could be used to speed up commands like\n+\t\t * 'git log dir/subdir', too.\n+\t\t *\n+\t\t * Note that directories are added without the trailing '/'.\n+\t\t */\n+\t\tdo {\n+\t\t\tgit_hash_ctx ctx;\n+\t\t\tunsigned char name_hash[GIT_MAX_RAWSZ];\n+\t\t\tunsigned char hash[GIT_MAX_RAWSZ];\n+\n+\t\t\tp = strchrnul(p + 1, '/');\n+\n+\t\t\t/*\n+\t\t\t * Beware all the wasted CPU cycles!\n+\t\t\t *\n+\t\t\t * Most paths change (a lot) more than once in the\n+\t\t\t * history of a repository, so this hashes the same\n+\t\t\t * paths over and over again, accounting for almost\n+\t\t\t * 40% of the runtime.\n+\t\t\t */\n+\t\t\tthe_hash_algo->init_fn(&ctx);\n+\t\t\tthe_hash_algo->update_fn(&ctx, path, p - path);\n+\t\t\tthe_hash_algo->final_fn(name_hash, &ctx);\n+\n+\t\t\thashxor(name_hash, p_c_hash, hash);\n+\t\t\tbloom_filter_add_hash(bf, hash);\n+\t\t} while (*p);\n+\n+\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n+\t}\n+\n+\tfree(diff_queued_diff.queue);\n+\tDIFF_QUEUE_CLEAR(&diff_queued_diff);\n+}\n+\n+static void fill_bloom_filter(struct bloom_filter *bf,\n+\t\t\t\t    struct progress *progress)\n+{\n+\tstruct rev_info revs;\n+\tconst char *revs_argv[] = {NULL, \"--all\", NULL};\n+\tstruct commit *commit;\n+\tint i = 0;\n+\n+\t/* We (re-)create the bloom filter from scratch every time for now. */\n+\tinit_revisions(&revs, NULL);\n+\trevs.diffopt.flags.recursive = 1;\n+\tsetup_revisions(2, revs_argv, &revs, NULL);\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(\"revision walk setup failed while preparing bloom filter\");\n+\n+\twhile ((commit = get_revision(&revs))) {\n+\t\tstruct commit_list *parent;\n+\n+\t\tfor (parent = commit->parents; parent; parent = parent->next)\n+\t\t\tadd_changes_to_bloom_filter(bf, parent->item, commit,\n+\t\t\t\t\t\t    &revs.diffopt);\n+\n+\t\tdisplay_progress(progress, ++i);\n+\t}\n+}\n+\n+static void write_bloom_filter(int report_progress, int commit_nr)\n+{\n+\tstruct bloom_filter bf;\n+\tstruct progress *progress = NULL;\n+\tconst char *v = getenv(\"GIT_USE_POC_BLOOM_FILTER\");\n+\tunsigned int bitsize;\n+\tchar *end;\n+\n+\tif (!v)\n+\t\treturn;\n+\n+\tbitsize = strtol(v, &end, 10);\n+\tif (*end)\n+\t\tdie(\"GIT_USE_POC_BLOOM_FILTER must specify the number of bits in the bloom filter (multiple of 8, n < 2^32)\");\n+\n+\tbloom_filter_init(&bf, bitsize);\n+\n+\tif (report_progress)\n+\t\tprogress = start_progress(_(\"Computing bloom filter\"),\n+\t\t\t\t\t  commit_nr);\n+\n+\tfill_bloom_filter(&bf, progress);\n+\n+\tbloom_filter_write(&bf);\n+\tbloom_filter_free(&bf);\n+\n+\tstop_progress(&progress);\n+}\n+\n void write_commit_graph_reachable(const char *obj_dir, int append,\n \t\t\t\t  int report_progress)\n {\n@@ -916,6 +1030,8 @@ void write_commit_graph(const char *obj_dir,\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n+\twrite_bloom_filter(report_progress, commits.nr);\n+\n \tfree(graph_name);\n \tfree(commits.list);\n \tfree(oids.list);\n-- \n2.19.1.409.g0a0ee5eb6b\n\n"},{"id":"359943","messageId":"20181009193445.21908-4-szeder.dev@gmail.com","threadId":"49479","inReplyTo":"20181009193445.21908-1-szeder.dev@gmail.com","subject":"[PATCH 3/4] revision.c: use the Bloom filter to speed up path-limited revision walks","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-09T19:34:44Z","receivedAt":"2018-10-09T19:35:21Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"When $GIT_USE_POC_BLOOM_FILTER is set to a non-empty value, the\nrevision walk will use the Bloom filter to speed up a path-limited\nrevision walk.\n\nLoad the Bloom filter in prepare_revision_walk(); probably not the\nbest place for it, but it should suffice for experiementing with 'git\nrev-list'.\n\nChecking the Bloom filter is plugged into rev_compare_tree(), the\nfunction that compares the given paths in a commit to one of its\nparents.  If checking the Bloom filter returns that the interesting\npaths did not change, then it won't bother with the running the\nexpensive diff.\n\nAdd a new field to 'struct pathspec' to hold the SHA of the path, so\nthat hash is computed only once and then reused when checking each\ncommit.\n---\n pathspec.h |  1 +\n revision.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 98 insertions(+)\n\ndiff --git a/pathspec.h b/pathspec.h\nindex a6525a6551..565a26d91e 100644\n--- a/pathspec.h\n+++ b/pathspec.h\n@@ -47,6 +47,7 @@ struct pathspec {\n \t\t\t} match_mode;\n \t\t} *attr_match;\n \t\tstruct attr_check *attr_check;\n+\t\tunsigned char name_hash[GIT_MAX_RAWSZ];\n \t} *items;\n };\n \ndiff --git a/revision.c b/revision.c\nindex c5d0cb6599..3565785ca6 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -27,6 +27,7 @@\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n #include \"prio-queue.h\"\n+#include \"bloom-filter.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -463,6 +464,62 @@ static void file_change(struct diff_options *options,\n \toptions->flags.has_changes = 1;\n }\n \n+/* Another static... */\n+static struct bloom_filter bf;\n+\n+static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n+\t\t\t\t\t\t struct commit *parent,\n+\t\t\t\t\t\t struct commit *commit)\n+{\n+\tunsigned char p_c_hash[GIT_MAX_RAWSZ];\n+\tint i;\n+\n+\tif (!bf.bits)\n+\t\treturn -1;\n+\t/*\n+\t * If a commit is not in the 'commit-graph' file, then it's not in\n+\t * the Bloom filter either, so any query into that would report\n+\t * back a false negative, which is unacceptable.\n+\t *\n+\t * The writer of the Bloom filter must ensure that all commits that\n+\t * go into the 'commit-graph' go into the Bloom filter as well.\n+\t *\n+\t * If we won't tie the Bloom filter to the commit-graph tightly,\n+\t * then we'll have to come up with another means to prevent such\n+\t * false negatives.\n+\t */\n+\tif (!the_repository->objects->commit_graph)\n+\t\treturn -1;\n+\tif (commit->generation == GENERATION_NUMBER_INFINITY)\n+\t\treturn -1;\n+\n+\thashxor(parent->object.oid.hash, commit->object.oid.hash, p_c_hash);\n+\n+\tfor (i = 0; i < revs->pruning.pathspec.nr; i++) {\n+\t\tstruct pathspec_item *pi = &revs->pruning.pathspec.items[i];\n+\t\tunsigned char hash[GIT_MAX_RAWSZ];\n+\n+\t\thashxor(pi->name_hash, p_c_hash, hash);\n+\t\tif (bloom_filter_check_hash(&bf, hash)) {\n+\t\t\t/*\n+\t\t\t * At least one of the interesting pathspecs differs,\n+\t\t\t * so we can return early and let the diff machinery\n+\t\t\t * make sure that they indeed differ.\n+\t\t\t *\n+\t\t\t * Note: the diff machinery will look at all the given\n+\t\t\t * paths; a possible future optimization might bring\n+\t\t\t * the Bloom filter and the diff machinery closer to\n+\t\t\t * each other, so the diff won't waste time looking\n+\t\t\t * at those paths that the Bloom filter have found\n+\t\t\t * unchanged.\n+\t\t\t */\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int rev_compare_tree(struct rev_info *revs,\n \t\t\t    struct commit *parent, struct commit *commit)\n {\n@@ -492,6 +549,9 @@ static int rev_compare_tree(struct rev_info *revs,\n \t\t\treturn REV_TREE_SAME;\n \t}\n \n+\tif (!check_maybe_different_in_bloom_filter(revs, parent, commit))\n+\t\treturn REV_TREE_SAME;\n+\n \ttree_difference = REV_TREE_SAME;\n \trevs->pruning.flags.has_changes = 0;\n \tif (diff_tree_oid(&t1->object.oid, &t2->object.oid, \"\",\n@@ -3106,6 +3166,40 @@ static void expand_topo_walk(struct rev_info *revs, struct commit *commit)\n \t}\n }\n \n+void prepare_to_use_bloom_filter(struct rev_info *revs)\n+{\n+\tconst char *env = getenv(\"GIT_USE_POC_BLOOM_FILTER\");\n+\tint i;\n+\n+\tif (!env || !*env)\n+\t\treturn;\n+\n+\tfor (i = 0; i < revs->pruning.pathspec.nr; i++) {\n+\t\tstruct pathspec_item *pi = &revs->pruning.pathspec.items[i];\n+\t\tconst char *path = pi->match;\n+\t\tgit_hash_ctx ctx;\n+\t\tsize_t len = strlen(path);\n+\n+\t\t/*\n+\t\t * TODO: What about wildcards?  We'd probably just want to\n+\t\t * ignore the Bloom filter then.\n+\t\t */\n+\t\tthe_hash_algo->init_fn(&ctx);\n+\t\tthe_hash_algo->update_fn(&ctx, path,\n+\t\t\t\t\t path[len - 1] == '/' ? len - 1 : len);\n+\t\tthe_hash_algo->final_fn(pi->name_hash, &ctx);\n+\t}\n+\n+\tif (bf.bits)\n+\t\t/* Already loaded. */\n+\t\treturn;\n+\n+\tif (bloom_filter_load(&bf) < 0) {\n+\t\twarning(\"you wanted to use the Bloom filter, but it couldn't be loaded\");\n+\t\treturn;\n+\t}\n+}\n+\n int prepare_revision_walk(struct rev_info *revs)\n {\n \tint i;\n@@ -3155,6 +3249,9 @@ int prepare_revision_walk(struct rev_info *revs)\n \t\tsimplify_merges(revs);\n \tif (revs->children.name)\n \t\tset_children(revs);\n+\n+\tprepare_to_use_bloom_filter(revs);\n+\n \treturn 0;\n }\n \n-- \n2.19.1.409.g0a0ee5eb6b\n\n"},{"id":"359944","messageId":"20181009193445.21908-5-szeder.dev@gmail.com","threadId":"49479","inReplyTo":"20181009193445.21908-1-szeder.dev@gmail.com","subject":"[PATCH 4/4] revision.c: add GIT_TRACE_BLOOM_FILTER for a bit of statistics","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-09T19:34:45Z","receivedAt":"2018-10-09T19:35:22Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"It will output something like this:\n\n  $ GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y git rev-list --count --full-history HEAD -- t/valgrind/valgrind.sh\n  886\n  17:24:42.915053 revision.c:484          bloom filter total queries: 66095 definitely not: 64953 maybe: 1142 false positives: 256 fp ratio: 0.003873\n---\n revision.c | 34 +++++++++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/revision.c b/revision.c\nindex 3565785ca6..2f3f73b4dd 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -467,6 +467,25 @@ static void file_change(struct diff_options *options,\n /* Another static... */\n static struct bloom_filter bf;\n \n+static struct trace_key trace_bloom_filter = TRACE_KEY_INIT(BLOOM_FILTER);\n+static int trace_bloom_filter_atexit_registered;\n+static unsigned int bloom_filter_count_maybe;\n+static unsigned int bloom_filter_count_definitely_not;\n+static unsigned int bloom_filter_count_false_positive;\n+\n+static void print_bloom_filter_stats_atexit(void)\n+{\n+\tunsigned int total = bloom_filter_count_maybe +\n+\t\t\t     bloom_filter_count_definitely_not;\n+\ttrace_printf_key(&trace_bloom_filter,\n+\t\t\t \"bloom filter total queries: %d definitely not: %d maybe: %d false positives: %d fp ratio: %f\\n\",\n+\t\t\t total,\n+\t\t\t bloom_filter_count_definitely_not,\n+\t\t\t bloom_filter_count_maybe,\n+\t\t\t bloom_filter_count_false_positive,\n+\t\t\t (1.0 * bloom_filter_count_false_positive) / total);\n+}\n+\n static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\t\t\t\t\t struct commit *parent,\n \t\t\t\t\t\t struct commit *commit)\n@@ -513,10 +532,12 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\t\t * at those paths that the Bloom filter have found\n \t\t\t * unchanged.\n \t\t\t */\n+\t\t\tbloom_filter_count_maybe++;\n \t\t\treturn 1;\n \t\t}\n \t}\n \n+\tbloom_filter_count_definitely_not++;\n \treturn 0;\n }\n \n@@ -525,6 +546,7 @@ static int rev_compare_tree(struct rev_info *revs,\n {\n \tstruct tree *t1 = get_commit_tree(parent);\n \tstruct tree *t2 = get_commit_tree(commit);\n+\tint bloom_ret;\n \n \tif (!t1)\n \t\treturn REV_TREE_NEW;\n@@ -549,7 +571,8 @@ static int rev_compare_tree(struct rev_info *revs,\n \t\t\treturn REV_TREE_SAME;\n \t}\n \n-\tif (!check_maybe_different_in_bloom_filter(revs, parent, commit))\n+\tbloom_ret = check_maybe_different_in_bloom_filter(revs, parent, commit);\n+\tif (bloom_ret == 0)\n \t\treturn REV_TREE_SAME;\n \n \ttree_difference = REV_TREE_SAME;\n@@ -557,6 +580,8 @@ static int rev_compare_tree(struct rev_info *revs,\n \tif (diff_tree_oid(&t1->object.oid, &t2->object.oid, \"\",\n \t\t\t   &revs->pruning) < 0)\n \t\treturn REV_TREE_DIFFERENT;\n+\tif (bloom_ret == 1 && tree_difference == REV_TREE_SAME)\n+\t\tbloom_filter_count_false_positive++;\n \treturn tree_difference;\n }\n \n@@ -3198,6 +3223,13 @@ void prepare_to_use_bloom_filter(struct rev_info *revs)\n \t\twarning(\"you wanted to use the Bloom filter, but it couldn't be loaded\");\n \t\treturn;\n \t}\n+\n+\tif (trace_want(&trace_bloom_filter)) {\n+\t\tif (!trace_bloom_filter_atexit_registered) {\n+\t\t\tatexit(print_bloom_filter_stats_atexit);\n+\t\t\ttrace_bloom_filter_atexit_registered = 1;\n+\t\t}\n+\t}\n }\n \n int prepare_revision_walk(struct rev_info *revs)\n-- \n2.19.1.409.g0a0ee5eb6b\n\n"},{"id":"359945","messageId":"8dd5ab67-8f9b-fddb-b4ae-b4187eeffaac@gmail.com","threadId":"49479","inReplyTo":"20181009193445.21908-1-szeder.dev@gmail.com","subject":"Re: [PATCH 0/4] Bloom filter experiment","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-09T19:47:15Z","receivedAt":"2018-10-09T19:47:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/9/2018 3:34 PM, SZEDER Gábor wrote:\n> To keep the ball rolling, here is my proof of concept in a somewhat\n> cleaned-up form, with still plenty of rough edges.\n>\n> You can play around with it like this:\n>\n>    $ GIT_USE_POC_BLOOM_FILTER=$((8*1024*1024*8)) git commit-graph write\n>    Computing commit graph generation numbers: 100% (52801/52801), done.\n>    Computing bloom filter: 100% (52801/52801), done.\n>    # Yeah, I even added progress indicator! :)\n>    $ GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y git rev-list --count --full-history HEAD -- t/valgrind/valgrind.sh\n>    886\n>    20:40:24.783699 revision.c:486          bloom filter total queries: 66095 definitely not: 64953 maybe: 1142 false positives: 256 fp ratio: 0.003873\n>\n> The value of $GIT_USE_POC_BLOOM_FILTER only really matters when writing\n> the Bloom filter, and it specifies the number of bits in the filter's\n> bitmap, IOW the above command creates a 8MB Bloom filter.  To make use\n> of the filter the variable can be anything non-empty.\n>\n> Writing the Bloom filter is very slow as it is (yeah, that's why\n> bothered with the progress indicator ;).  I wrote about it in patch 2's\n> commit message: the cause for about half of the slowness is rather\n> obvious, but I don't (yet) know what's responsible for the other half.\n>\n>\n> Not a single test...  but I've run loops over all files in git.git\n> comparing 'git rev-list HEAD -- $file's output with and without the\n> Bloom filter, and, surprisingly, they match.  My quick'n'dirty\n> experiments usually don't fare this well...\n>\n>\n> It's also available at:\n>\n>    https://github.com/szeder/git bloom-filter-experiment\n\nThanks! I will take a close look at this tomorrow and start playing with it.\n\n-Stolee\n\n"},{"id":"359949","messageId":"20181009210620.GA9563@sigill.intra.peff.net","threadId":"49479","inReplyTo":"20181009193445.21908-3-szeder.dev@gmail.com","subject":"Re: [PATCH 2/4] commit-graph: write a Bloom filter containing changed paths for each commit","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-09T21:06:20Z","receivedAt":"2018-10-09T21:06:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 09, 2018 at 09:34:43PM +0200, SZEDER Gábor wrote:\n\n> Creating the Bloom filter is sloooow.  Running it on git.git takes\n> about 23s on my hardware, while\n> \n>   git log --format='%H%n%P' --name-only --all >/dev/null\n> \n> gathers all the information necessary for that in about 5.3s.\n\nThat command won't open the trees for merges at all. But your\nimplementation here looks like it does a diff against each parent of a\nmerge. Adding \"-m\" would be a more accurate comparison, I think.\n\nThough I find that puzzling, because \"-m --name-only\" seems to take\nabout 20x longer, not 3x. So perhaps I'm missing something.\n\n-Peff\n"},{"id":"359951","messageId":"20181009211449.GB9563@sigill.intra.peff.net","threadId":"49479","inReplyTo":"ec3ca377-0434-322e-4ab9-49e27f96f4af@gmail.com","subject":"Re: Bloom Filters (was Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-09T21:14:50Z","receivedAt":"2018-10-09T21:14:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 09, 2018 at 03:03:08PM -0400, Derrick Stolee wrote:\n\n> > I wonder if Roaring does better here.\n> \n> In these sparse cases, usually Roaring will organize the data as \"array\n> chunks\" which are simply lists of the values. The thing that makes this\n> still compressible is that we store two bytes per entry, as the entries are\n> grouped by a common most-significant two bytes. SInce you say ~120k unique\n> paths, the Roaring bitmap would have two or three chunks per bitmap (and\n> those chunks could be empty). The overhead to store the chunk positions,\n> types, and lengths does come at a cost, but it's more like 32 bytes _per\n> commit_.\n\nHmph. It really sounds like we could do better with a custom RLE\nsolution. But that makes me feel like I'm missing something, because\nsurely I can't invent something better than the state of the art in a\nsimple thought experiment, right?\n\nI know what I'm proposing would be quite bad for random access, but my\nimpression is that EWAH is the same. For the scale of bitmaps we're\ntalking about, I think linear/streaming access through the bitmap would\nbe OK.\n\n> > So at any rate, I do think it would not be out of the question to store\n> > bitmaps like this. I'm much more worried about the maintenance cost of\n> > adding new entries incrementally. I think it's only feasible if we give\n> > up sorting, and then I wonder what other problems that might cause.\n> The patch below gives me a starting point to try the Bloom filter approach\n> and see what the numbers are like. You did all the \"git\" stuff like\n> computing the changed paths, so thanks!\n\nGreat, I hope it can be useful. I almost wrote it as perl consuming the\noutput of \"log --format=%h --name-only\", but realized I didn't have a\nperl ewah implementation handy.\n\nYou'll probably want to tweak this part:\n\n> > +\tprepare_revision_walk(&revs);\n> > +\twhile ((commit = get_revision(&revs))) {\n> > +\t\tdata.commit = commit;\n> > +\t\tdiff_tree_combined_merge(commit, 0, &revs);\n> > +\t}\n\n...to handle merges in a particular way. This will actually ignore\nmerges totally. You could add \"-m\" to the revision arguments to get a\nper-parent diff, but of course you'd see those in your callback\nindividually. If you want to do _just_ the first parent diff, I think\nyou'll have to pick it apart manually, like:\n\n  while ((commit = get_revision(&revs))) {\n\tstruct object_id *parent_oid;\n\n\t/* ignore non-first parents, but handle root commits like --root */\n\tif (commit->parents)\n\t\tparent = &commit->parents->item->object.oid;\n\telse\n\t\tparent = the_hash_algo->empty_tree;\n\n\tdiff_tree_oid(parent, &commit->oid, ...);\n  }\n\n-Peff\n"},{"id":"359952","messageId":"20181009213007.GB23446@szeder.dev","threadId":"49479","inReplyTo":"20181009030803.GA6250@sigill.intra.peff.net","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-09T21:30:07Z","receivedAt":"2018-10-09T21:30:14Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 08, 2018 at 11:08:03PM -0400, Jeff King wrote:\n> I'd have done it as one fixed-size filter per commit. Then you should be\n> able to hash the path keys once, and apply the result as a bitwise query\n> to each individual commit (I'm assuming that it's constant-time to\n> access the filter for each, as an index into an mmap'd array, with the\n> offset coming from a commit-graph entry we'd be able to look up anyway).\n\nI used one big Bloom filter for the whole history, because that was\nthe simplest way to get going, and because I was primarily interested\nin the potential benefits instead of the cost of generating and\nmaintaining it.\n\nUsing a 8MB filter for git.git results in a false positive rate\nbetween 0.21% - 0.53%.  Splitting that up for ~53k commits we get ~160\nbytes for each.  On first sight that seems like rather small, but\nrunning a bit statistics shows that 99% of our commits don't change\nmore than 10 files.\n\nOne advantage of the \"one Bloom filter for each commit\" is that if a\ncommit doesn't have a corresponding Bloom filter, then, well, we can't\nquery the non-existing filter.  OTOH, with one big Bloom filter we\nhave to be careful to only ever query it with commits whose changes\nhave already been added, otherwise we can get false negatives.\n\n> I think it would also be easier to deal with maintenance, since each\n> filter is independent (IIRC, you cannot delete from a bloom filter\n> without re-adding all of the other keys).\n\nAccumulating entries related to unreachable commits will eventually\nincrease the false positive rate, but otherwise it won't cause false\nnegatives, and won't increase the size of the Bloom filter or the time\nnecessary to query it.  So not deleting those entries right away is\nnot an issue, and I think it could be postponed until bigger gc runs.\n\n[...]\n\n> But there's also a related question: how do we match pathspec patterns?\n> For a changed path like \"foo/bar/baz\", I imagine a bloom filter would\n> mark all of \"foo\", \"foo/bar\", and \"foo/bar/baz\".\n\nIndeed, that's what I did.\n\n> But what about \"*.c\"? I\n> don't think a bloom filter can answer that.\n\nSurely not, but it could easily return \"maybe\", and thus simply fall\nback to look at the diff.\n\nHowever, I've looked through the output of\n\n  grep '^git log[^|]*[\\[*?]' ~/.bash_history\n\nand haven't found a single case where I used Git's globbing.  When I\ndid use globbing, then I always used the shell's.  Yeah, just one data\npoint, and others surely use it differently, etc...  but I think we\nshould consider whether it's common enough to worry about and to\nincrease complexity because of it.\n\n[...]\n\n> So let's imagine we'd store such a cache external to the regular object\n> data (i.e., as a commit-graph entry). The \"log --raw\" diff of linux.git\n> has 1.7M entries. The paths should easily compress to a single 32-bit\n> integer (e.g., as an index into a big path list). The oids are 20 bytes.\n> Add a few bytes for modes. That's about 80MB. Big, but not impossibly\n> so. Maybe pushing it for true gigantic repos, though.\n\nIn my experiments with the Linux repo a 256MB Bloom filter has ~0.3%\nfalse positive rate, while a 128MB filter had 3-4%.  Even bigger,\nthough compared to the size of the checkout of the full kernel tree,\narguably not that much.\n\n> Those numbers are ignoring merges, too. The meaning of \"did this commit\n> touch that path\" is a lot trickier for a merge commit, and I think may\n> depend on context. I'm not sure how even a bloom filter solution would\n> handle that (I was assuming we'd mostly punt and let merges fall back to\n> opening up the trees).\n\nDuring revision walking rev_compare_tree() checks whether the given\npaths changed between a commit and _one_ of its parents, and in case\nof merge commits it's invoked multiple times with the same commit but\nwith different parent parameters.  By storing (changed-path,\nparent-oid, commit-oid) tuples in the Bloom filter it can deal with\nmerges, too.\n\n"},{"id":"359953","messageId":"20181009213724.GC23446@szeder.dev","threadId":"49479","inReplyTo":"20181009210620.GA9563@sigill.intra.peff.net","subject":"Re: [PATCH 2/4] commit-graph: write a Bloom filter containing changed paths for each commit","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-09T21:37:24Z","receivedAt":"2018-10-09T21:37:31Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Oct 09, 2018 at 05:06:20PM -0400, Jeff King wrote:\n> On Tue, Oct 09, 2018 at 09:34:43PM +0200, SZEDER Gábor wrote:\n> \n> > Creating the Bloom filter is sloooow.  Running it on git.git takes\n> > about 23s on my hardware, while\n> > \n> >   git log --format='%H%n%P' --name-only --all >/dev/null\n> > \n> > gathers all the information necessary for that in about 5.3s.\n> \n> That command won't open the trees for merges at all. But your\n> implementation here looks like it does a diff against each parent of a\n> merge.\n\nYeah, it does so, because that is what try_to_simplify_commit() /\nrev_compare_tree() will do while traversing the history.\n\n> Adding \"-m\" would be a more accurate comparison, I think.\n> \n> Though I find that puzzling, because \"-m --name-only\" seems to take\n> about 20x longer, not 3x. So perhaps I'm missing something.\n\nUgh, indeed.\n\n"},{"id":"359960","messageId":"20181009231250.GA19342@sigill.intra.peff.net","threadId":"49479","inReplyTo":"20181009211449.GB9563@sigill.intra.peff.net","subject":"Re: Bloom Filters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-09T23:12:50Z","receivedAt":"2018-10-09T23:12:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 09, 2018 at 05:14:50PM -0400, Jeff King wrote:\n\n> Hmph. It really sounds like we could do better with a custom RLE\n> solution. But that makes me feel like I'm missing something, because\n> surely I can't invent something better than the state of the art in a\n> simple thought experiment, right?\n> \n> I know what I'm proposing would be quite bad for random access, but my\n> impression is that EWAH is the same. For the scale of bitmaps we're\n> talking about, I think linear/streaming access through the bitmap would\n> be OK.\n\nThinking on it more, what I was missing is that for truly dense random\nbitmaps, this will perform much worse. Because it will use a byte to say\n\"there's one 1\", rather than a bit.\n\nBut I think it does OK in practice for the very sparse bitmaps we tend\nto see in this application.  I was able to generate a complete output\nthat can reproduce \"log --name-status -t\" for linux.git in 32MB. But:\n\n  - 15MB of that is commit sha1s, which will be stored elsewhere in a\n    \"real\" system\n\n  - 5MB of that is path list (which should shrink by a factor of 10 with\n    prefix compression, and is really a function of a tree size less\n    than history depth)\n\nSo the per-commit cost is not too bad. That's still not counting merges,\nthough, which would add another 10-15% (or maybe more; their bitmaps are\nless sparse).\n\nI don't know if this is a fruitful path at all or not. I was mostly just\nsatisfying my own curiosity on the bitmap encoding question. But I'll\npost the patches, just to show my work. The first one is the same\ninitial proof of concept I showed earlier.\n\n  [1/3]: initial tree-bitmap proof of concept\n  [2/3]: test-tree-bitmap: add \"dump\" mode\n  [3/3]: test-tree-bitmap: replace ewah with custom rle encoding\n\n Makefile                    |   1 +\n t/helper/test-tree-bitmap.c | 344 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 345 insertions(+)\n create mode 100644 t/helper/test-tree-bitmap.c\n\n-Peff\n"},{"id":"359961","messageId":"20181009231331.GA23730@sigill.intra.peff.net","threadId":"49479","inReplyTo":"20181009231250.GA19342@sigill.intra.peff.net","subject":"[PoC -- do not apply 1/3] initial tree-bitmap proof of concept","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-09T23:13:32Z","receivedAt":"2018-10-09T23:13:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Signed-off-by: Jeff King <peff@peff.net>\n---\n Makefile                    |   1 +\n t/helper/test-tree-bitmap.c | 167 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 168 insertions(+)\n create mode 100644 t/helper/test-tree-bitmap.c\n\ndiff --git a/Makefile b/Makefile\nindex 13e1c52478..f6e823f2d6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -751,6 +751,7 @@ TEST_PROGRAMS_NEED_X += test-parse-options\n TEST_PROGRAMS_NEED_X += test-pkt-line\n TEST_PROGRAMS_NEED_X += test-svn-fe\n TEST_PROGRAMS_NEED_X += test-tool\n+TEST_PROGRAMS_NEED_X += test-tree-bitmap\n \n TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\n \ndiff --git a/t/helper/test-tree-bitmap.c b/t/helper/test-tree-bitmap.c\nnew file mode 100644\nindex 0000000000..bc5cf0e514\n--- /dev/null\n+++ b/t/helper/test-tree-bitmap.c\n@@ -0,0 +1,167 @@\n+#include \"cache.h\"\n+#include \"revision.h\"\n+#include \"diffcore.h\"\n+#include \"argv-array.h\"\n+#include \"ewah/ewok.h\"\n+\n+/* map of pathnames to bit positions */\n+struct pathmap_entry {\n+\tstruct hashmap_entry ent;\n+\tunsigned pos;\n+\tchar path[FLEX_ARRAY];\n+};\n+\n+static int pathmap_entry_hashcmp(const void *unused_cmp_data,\n+\t\t\t\t const void *entry,\n+\t\t\t\t const void *entry_or_key,\n+\t\t\t\t const void *keydata)\n+{\n+\tconst struct pathmap_entry *a = entry;\n+\tconst struct pathmap_entry *b = entry_or_key;\n+\tconst char *key = keydata;\n+\n+\treturn strcmp(a->path, key ? key : b->path);\n+}\n+\n+static int pathmap_entry_strcmp(const void *va, const void *vb)\n+{\n+\tstruct pathmap_entry *a = *(struct pathmap_entry **)va;\n+\tstruct pathmap_entry *b = *(struct pathmap_entry **)vb;\n+\treturn strcmp(a->path, b->path);\n+}\n+\n+struct walk_paths_data {\n+\tstruct hashmap *paths;\n+\tstruct commit *commit;\n+};\n+\n+static void walk_paths(diff_format_fn_t fn, struct hashmap *paths)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct rev_info revs;\n+\tstruct walk_paths_data data;\n+\tstruct commit *commit;\n+\n+\targv_array_pushl(&argv, \"rev-list\",\n+\t\t\t \"--all\", \"-t\", \"--no-renames\",\n+\t\t\t NULL);\n+\tinit_revisions(&revs, NULL);\n+\tsetup_revisions(argv.argc, argv.argv, &revs, NULL);\n+\trevs.diffopt.output_format = DIFF_FORMAT_CALLBACK;\n+\trevs.diffopt.format_callback = fn;\n+\trevs.diffopt.format_callback_data = &data;\n+\n+\tdata.paths = paths;\n+\n+\tprepare_revision_walk(&revs);\n+\twhile ((commit = get_revision(&revs))) {\n+\t\tdata.commit = commit;\n+\t\tdiff_tree_combined_merge(commit, 0, &revs);\n+\t}\n+\n+\treset_revision_walk();\n+\targv_array_clear(&argv);\n+}\n+\n+static void collect_commit_paths(struct diff_queue_struct *q,\n+\t\t\t\t struct diff_options *opts,\n+\t\t\t\t void *vdata)\n+{\n+\tstruct walk_paths_data *data = vdata;\n+\tint i;\n+\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tconst char *path = p->one->path;\n+\t\tstruct pathmap_entry *entry;\n+\t\tstruct hashmap_entry lookup;\n+\n+\t\thashmap_entry_init(&lookup, strhash(path));\n+\t\tentry = hashmap_get(data->paths, &lookup, path);\n+\t\tif (entry)\n+\t\t\tcontinue; /* already present */\n+\n+\t\tFLEX_ALLOC_STR(entry, path, path);\n+\t\tentry->ent = lookup;\n+\t\thashmap_put(data->paths, entry);\n+\t}\n+}\n+\n+/* assign a bit position to all possible paths */\n+static void collect_paths(struct hashmap *paths)\n+{\n+\tstruct pathmap_entry **sorted;\n+\tsize_t i, n;\n+\tstruct hashmap_iter iter;\n+\tstruct pathmap_entry *entry;\n+\n+\t/* grab all unique paths */\n+\thashmap_init(paths, pathmap_entry_hashcmp, NULL, 0);\n+\twalk_paths(collect_commit_paths, paths);\n+\n+\t/* and assign them bits in sorted order */\n+\tn = hashmap_get_size(paths);\n+\tALLOC_ARRAY(sorted, n);\n+\ti = 0;\n+\tfor (entry = hashmap_iter_first(paths, &iter);\n+\t     entry;\n+\t     entry = hashmap_iter_next(&iter)) {\n+\t\tassert(i < n);\n+\t\tsorted[i++] = entry;\n+\t}\n+\tQSORT(sorted, i, pathmap_entry_strcmp);\n+\tfor (i = 0; i < n; i++)\n+\t\tsorted[i]->pos = i;\n+\tfree(sorted);\n+}\n+\n+/* generate the bitmap for a single commit */\n+static void generate_bitmap(struct diff_queue_struct *q,\n+\t\t\t    struct diff_options *opts,\n+\t\t\t    void *vdata)\n+{\n+\tstruct walk_paths_data *data = vdata;\n+\tstruct bitmap *bitmap = bitmap_new();\n+\tstruct ewah_bitmap *ewah;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tsize_t i;\n+\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tconst char *path = p->one->path;\n+\t\tstruct pathmap_entry *entry;\n+\t\tstruct hashmap_entry lookup;\n+\n+\t\thashmap_entry_init(&lookup, strhash(path));\n+\t\tentry = hashmap_get(data->paths, &lookup, path);\n+\t\tif (!entry)\n+\t\t\tBUG(\"mysterious path appeared: %s\", path);\n+\n+\t\tbitmap_set(bitmap, entry->pos);\n+\t}\n+\n+\tewah = bitmap_to_ewah(bitmap);\n+\tewah_serialize_strbuf(ewah, &out);\n+\tfwrite(out.buf, 1, out.len, stdout);\n+\n+\ttrace_printf(\"bitmap %s %u %u\",\n+\t\t     oid_to_hex(&data->commit->object.oid),\n+\t\t     (unsigned)q->nr,\n+\t\t     (unsigned)out.len);\n+\n+\tstrbuf_release(&out);\n+\tewah_free(ewah);\n+\tbitmap_free(bitmap);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tstruct hashmap paths;\n+\n+\tsetup_git_directory();\n+\tcollect_paths(&paths);\n+\n+\twalk_paths(generate_bitmap, &paths);\n+\n+\treturn 0;\n+}\n-- \n2.19.1.550.g7610f1eecb\n\n"},{"id":"359962","messageId":"20181009231405.GB23730@sigill.intra.peff.net","threadId":"49479","inReplyTo":"20181009231250.GA19342@sigill.intra.peff.net","subject":"[PoC -- do not apply 2/3] test-tree-bitmap: add \"dump\" mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-09T23:14:05Z","receivedAt":"2018-10-09T23:14:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This teaches \"gen\" mode (formerly the only mode) to include\nthe list of paths, and to prefix each bitmap with its\nmatching oid.\n\nThe \"dump\" mode can then read that back in and generate the\nlist of changed paths. This should be almost identical to:\n\n  git rev-list --all |\n  git diff-tree --stdin --name-only -t\n\nThe one difference is the sort order: git's diff output is\nin tree-sort order, so a subtree \"foo\" sorts like \"foo/\",\nwhich is after \"foo.bar\". Whereas the bitmap path list has a\ntrue byte sort, which puts \"foo.bar\" after \"foo\".\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n t/helper/test-tree-bitmap.c | 104 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 102 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-tree-bitmap.c b/t/helper/test-tree-bitmap.c\nindex bc5cf0e514..6f8833344a 100644\n--- a/t/helper/test-tree-bitmap.c\n+++ b/t/helper/test-tree-bitmap.c\n@@ -112,6 +112,14 @@ static void collect_paths(struct hashmap *paths)\n \tQSORT(sorted, i, pathmap_entry_strcmp);\n \tfor (i = 0; i < n; i++)\n \t\tsorted[i]->pos = i;\n+\n+\t/* dump it while we have the sorted order in memory */\n+\tfor (i = 0; i < n; i++) {\n+\t\tprintf(\"%s\", sorted[i]->path);\n+\t\tputchar('\\0');\n+\t}\n+\tputchar('\\0');\n+\n \tfree(sorted);\n }\n \n@@ -142,6 +150,8 @@ static void generate_bitmap(struct diff_queue_struct *q,\n \n \tewah = bitmap_to_ewah(bitmap);\n \tewah_serialize_strbuf(ewah, &out);\n+\n+\tfwrite(data->commit->object.oid.hash, 1, GIT_SHA1_RAWSZ, stdout);\n \tfwrite(out.buf, 1, out.len, stdout);\n \n \ttrace_printf(\"bitmap %s %u %u\",\n@@ -154,14 +164,104 @@ static void generate_bitmap(struct diff_queue_struct *q,\n \tbitmap_free(bitmap);\n }\n \n-int cmd_main(int argc, const char **argv)\n+static void do_gen(void)\n {\n \tstruct hashmap paths;\n-\n \tsetup_git_directory();\n \tcollect_paths(&paths);\n \n \twalk_paths(generate_bitmap, &paths);\n+}\n+\n+static void show_path(size_t pos, void *data)\n+{\n+\tconst char **paths = data;\n+\n+\t/* assert(pos < nr_paths), but we didn't pass the latter in */\n+\tprintf(\"%s\\n\", paths[pos]);\n+}\n+\n+static void do_dump(void)\n+{\n+\tstruct strbuf in = STRBUF_INIT;\n+\tconst char *cur;\n+\tsize_t remain;\n+\n+\tconst char **paths = NULL;\n+\tsize_t alloc_paths = 0, nr_paths = 0;\n+\n+\t/* slurp stdin; in the real world we'd mmap all this */\n+\tstrbuf_read(&in, 0, 0);\n+\tcur = in.buf;\n+\tremain = in.len;\n+\n+\t/* read path for each bit; in the real world this would be separate */\n+\twhile (remain) {\n+\t\tconst char *end = memchr(cur, '\\0', remain);\n+\t\tif (!end) {\n+\t\t\terror(\"truncated input while reading path\");\n+\t\t\tgoto out;\n+\t\t}\n+\t\tif (end == cur) {\n+\t\t\t/* empty field signals end of paths */\n+\t\t\tcur++;\n+\t\t\tremain--;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tALLOC_GROW(paths, nr_paths + 1, alloc_paths);\n+\t\tpaths[nr_paths++] = cur;\n+\n+\t\tremain -= end - cur + 1;\n+\t\tcur = end + 1;\n+\t}\n+\n+\t/* read the bitmap for each commit */\n+\twhile (remain) {\n+\t\tstruct object_id oid;\n+\t\tstruct ewah_bitmap *ewah;\n+\t\tssize_t len;\n+\n+\t\tif (remain < GIT_SHA1_RAWSZ) {\n+\t\t\terror(\"truncated input reading oid\");\n+\t\t\tgoto out;\n+\t\t}\n+\t\thashcpy(oid.hash, (const unsigned char *)cur);\n+\t\tcur += GIT_SHA1_RAWSZ;\n+\t\tremain -= GIT_SHA1_RAWSZ;\n+\n+\t\tewah = ewah_new();\n+\t\tlen = ewah_read_mmap(ewah, cur, remain);\n+\t\tif (len < 0) {\n+\t\t\tewah_free(ewah);\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n+\t\tewah_each_bit(ewah, show_path, paths);\n+\n+\t\tewah_free(ewah);\n+\t\tcur += len;\n+\t\tremain -= len;\n+\t}\n+\n+out:\n+\tfree(paths);\n+\tstrbuf_release(&in);\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tconst char *usage_msg = \"test-tree-bitmap <gen|dump>\";\n+\n+\tif (!argv[1])\n+\t\tusage(usage_msg);\n+\telse if (!strcmp(argv[1], \"gen\"))\n+\t\tdo_gen();\n+\telse if (!strcmp(argv[1], \"dump\"))\n+\t\tdo_dump();\n+\telse\n+\t\tusage(usage_msg);\n \n \treturn 0;\n }\n-- \n2.19.1.550.g7610f1eecb\n\n"},{"id":"359963","messageId":"20181009231441.GC23730@sigill.intra.peff.net","threadId":"49479","inReplyTo":"20181009231250.GA19342@sigill.intra.peff.net","subject":"[PoC -- do not apply 3/3] test-tree-bitmap: replace ewah with custom rle encoding","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-09T23:14:41Z","receivedAt":"2018-10-09T23:14:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"The rules are basically:\n\n - each bitmap is a series of counts of runs of 0/1\n\n - each count is one of our standard varints\n\n - each bitmap must have at least one initial count of\n   zeroes (which may itself be a zero-length count, if the\n   first bit is set)\n\n - a zero-length count anywhere else marks the end of\n   the bitmap\n\nFor a sparse bitmap, these will tend to be quite short,\nbecause long runs are encoded as fairly small counts. The\nworst case is an alternate 0/1/0/1 bitmap, where we will\nspend a full byte to specify each bit (thus bloating it by a\nfactor of 8 over an uncompressed bitmap).\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n t/helper/test-tree-bitmap.c | 105 +++++++++++++++++++++++++++++++-----\n 1 file changed, 91 insertions(+), 14 deletions(-)\n\ndiff --git a/t/helper/test-tree-bitmap.c b/t/helper/test-tree-bitmap.c\nindex 6f8833344a..36f19ed464 100644\n--- a/t/helper/test-tree-bitmap.c\n+++ b/t/helper/test-tree-bitmap.c\n@@ -3,6 +3,7 @@\n #include \"diffcore.h\"\n #include \"argv-array.h\"\n #include \"ewah/ewok.h\"\n+#include \"varint.h\"\n \n /* map of pathnames to bit positions */\n struct pathmap_entry {\n@@ -123,6 +124,49 @@ static void collect_paths(struct hashmap *paths)\n \tfree(sorted);\n }\n \n+static void strbuf_add_varint(struct strbuf *out, uintmax_t val)\n+{\n+\tsize_t len;\n+\tstrbuf_grow(out, 16); /* enough for any varint */\n+\tlen = encode_varint(val, (unsigned char *)out->buf + out->len);\n+\tstrbuf_setlen(out, out->len + len);\n+}\n+\n+static void bitmap_to_rle(struct strbuf *out, struct bitmap *bitmap)\n+{\n+\tint curval = 0; /* count zeroes, then ones, then zeroes, etc */\n+\tsize_t run = 0;\n+\tsize_t word;\n+\tsize_t orig_len = out->len;\n+\n+\tfor (word = 0; word < bitmap->word_alloc; word++) {\n+\t\tint bit;\n+\n+\t\tfor (bit = 0; bit < BITS_IN_EWORD; bit++) {\n+\t\t\tint val = !!(bitmap->words[word] & (((eword_t)1) << bit));\n+\t\t\tif (val == curval)\n+\t\t\t\trun++;\n+\t\t\telse {\n+\t\t\t\tstrbuf_add_varint(out, run);\n+\t\t\t\tcurval = 1 - curval; /* flip 0/1 */\n+\t\t\t\trun = 1;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * complete the run, but do not bother with trailing zeroes, unless we\n+\t * failed to write even an initial run of 0's.\n+\t */\n+\tif (curval && run)\n+\t\tstrbuf_add_varint(out, run);\n+\telse if (orig_len == out->len)\n+\t\tstrbuf_add_varint(out, 0);\n+\n+\t/* signal end-of-input with an empty run */\n+\tstrbuf_add_varint(out, 0);\n+}\n+\n /* generate the bitmap for a single commit */\n static void generate_bitmap(struct diff_queue_struct *q,\n \t\t\t    struct diff_options *opts,\n@@ -130,7 +174,6 @@ static void generate_bitmap(struct diff_queue_struct *q,\n {\n \tstruct walk_paths_data *data = vdata;\n \tstruct bitmap *bitmap = bitmap_new();\n-\tstruct ewah_bitmap *ewah;\n \tstruct strbuf out = STRBUF_INIT;\n \tsize_t i;\n \n@@ -148,8 +191,7 @@ static void generate_bitmap(struct diff_queue_struct *q,\n \t\tbitmap_set(bitmap, entry->pos);\n \t}\n \n-\tewah = bitmap_to_ewah(bitmap);\n-\tewah_serialize_strbuf(ewah, &out);\n+\tbitmap_to_rle(&out, bitmap);\n \n \tfwrite(data->commit->object.oid.hash, 1, GIT_SHA1_RAWSZ, stdout);\n \tfwrite(out.buf, 1, out.len, stdout);\n@@ -160,7 +202,6 @@ static void generate_bitmap(struct diff_queue_struct *q,\n \t\t     (unsigned)out.len);\n \n \tstrbuf_release(&out);\n-\tewah_free(ewah);\n \tbitmap_free(bitmap);\n }\n \n@@ -181,6 +222,51 @@ static void show_path(size_t pos, void *data)\n \tprintf(\"%s\\n\", paths[pos]);\n }\n \n+static size_t rle_each_bit(const unsigned char *in, size_t len,\n+\t\t\t   void (*fn)(size_t, void *), void *data)\n+{\n+\tint curval = 0; /* look for zeroes first, then ones, etc */\n+\tconst unsigned char *cur = in;\n+\tconst unsigned char *end = in + len;\n+\tsize_t pos;\n+\n+\t/* we always have a first run, even if it's 0 zeroes */\n+\tpos = decode_varint(&cur);\n+\n+\t/*\n+\t * ugh, varint does not seem to have a way to prevent reading past\n+\t * the end of the buffer. We'll do a length check after each one,\n+\t * so the worst case is bounded.\n+\t */\n+\tif (cur > end) {\n+\t\terror(\"input underflow in rle\");\n+\t\treturn len;\n+\t}\n+\n+\twhile (1) {\n+\t\tsize_t run = decode_varint(&cur);\n+\n+\t\tif (cur > end) {\n+\t\t\terror(\"input underflow in rle\");\n+\t\t\treturn len;\n+\t\t}\n+\n+\t\tif (!run)\n+\t\t\tbreak; /* empty run signals end */\n+\n+\t\tcurval = 1 - curval; /* flip 0/1 */\n+\t\tif (curval) {\n+\t\t\t/* we have a run of 1's; deliver them */\n+\t\t\tsize_t i;\n+\t\t\tfor (i = 0; i < run; i++)\n+\t\t\t\tfn(pos + i, data);\n+\t\t}\n+\t\tpos += run;\n+\t}\n+\n+\treturn cur - in;\n+}\n+\n static void do_dump(void)\n {\n \tstruct strbuf in = STRBUF_INIT;\n@@ -219,7 +305,6 @@ static void do_dump(void)\n \t/* read the bitmap for each commit */\n \twhile (remain) {\n \t\tstruct object_id oid;\n-\t\tstruct ewah_bitmap *ewah;\n \t\tssize_t len;\n \n \t\tif (remain < GIT_SHA1_RAWSZ) {\n@@ -230,17 +315,9 @@ static void do_dump(void)\n \t\tcur += GIT_SHA1_RAWSZ;\n \t\tremain -= GIT_SHA1_RAWSZ;\n \n-\t\tewah = ewah_new();\n-\t\tlen = ewah_read_mmap(ewah, cur, remain);\n-\t\tif (len < 0) {\n-\t\t\tewah_free(ewah);\n-\t\t\tgoto out;\n-\t\t}\n-\n \t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n-\t\tewah_each_bit(ewah, show_path, paths);\n+\t\tlen = rle_each_bit((const unsigned char *)cur, remain, show_path, paths);\n \n-\t\tewah_free(ewah);\n \t\tcur += len;\n \t\tremain -= len;\n \t}\n-- \n2.19.1.550.g7610f1eecb\n"},{"id":"359969","messageId":"xmqqy3b6occq.fsf@gitster-ct.c.googlers.com","threadId":"49479","inReplyTo":"20181009231405.GB23730@sigill.intra.peff.net","subject":"Re: [PoC -- do not apply 2/3] test-tree-bitmap: add \"dump\" mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-10T00:48:53Z","receivedAt":"2018-10-10T00:48:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> The one difference is the sort order: git's diff output is\n> in tree-sort order, so a subtree \"foo\" sorts like \"foo/\",\n> which is after \"foo.bar\". Whereas the bitmap path list has a\n> true byte sort, which puts \"foo.bar\" after \"foo\".\n\nIf we truly cared, it is easy enough to fix by having a custom\ncomparison function in 1/3 used in collect_paths() phase.\n\n> +\t/* dump it while we have the sorted order in memory */\n> +\tfor (i = 0; i < n; i++) {\n> +\t\tprintf(\"%s\", sorted[i]->path);\n> +\t\tputchar('\\0');\n> +\t}\n\nWith printf(\"%s%c\", sorted[i]->path, '\\0'); you can lose the braces.\n\n> +\tputchar('\\0');\n> +\n>  \tfree(sorted);\n>  }\n>  \n> @@ -142,6 +150,8 @@ static void generate_bitmap(struct diff_queue_struct *q,\n>  \n>  \tewah = bitmap_to_ewah(bitmap);\n>  \tewah_serialize_strbuf(ewah, &out);\n> +\n> +\tfwrite(data->commit->object.oid.hash, 1, GIT_SHA1_RAWSZ, stdout);\n>  \tfwrite(out.buf, 1, out.len, stdout);\n\nOK, so per commit, we have ewah bitmap that records the \"changed\npaths\" after the commit object name.  Makes sense.\n\nAnd the list of paths are based on the \"one\" side of the filepair.\nWhen we do an equivalent of \"git show X\", we see \"diff-tree X~1 X\"\nand by collecting the \"one\" side (i.e. subset of paths in the tree\nof X~1 that were modified when going to X) we say \"commit X changed\nthese paths\".  Makes sense, too.\n\n> -int cmd_main(int argc, const char **argv)\n> +static void do_gen(void)\n>  {\n>  \tstruct hashmap paths;\n> -\n\nLet's not lose this blank line.\n\n>  \tsetup_git_directory();\n>  \tcollect_paths(&paths);\n>  \n>  \twalk_paths(generate_bitmap, &paths);\n> +}\n> +\n> +static void do_dump(void)\n> +{\n> +\tstruct strbuf in = STRBUF_INIT;\n> +\tconst char *cur;\n> +\tsize_t remain;\n> +\n> +\tconst char **paths = NULL;\n> +\tsize_t alloc_paths = 0, nr_paths = 0;\n> +\n> +\t/* slurp stdin; in the real world we'd mmap all this */\n> +\tstrbuf_read(&in, 0, 0);\n> +\tcur = in.buf;\n> +\tremain = in.len;\n> +\n> +\t/* read path for each bit; in the real world this would be separate */\n> +\twhile (remain) {\n> +\t\tconst char *end = memchr(cur, '\\0', remain);\n> +\t\tif (!end) {\n> +\t\t\terror(\"truncated input while reading path\");\n> +\t\t\tgoto out;\n> +\t\t}\n> +\t\tif (end == cur) {\n> +\t\t\t/* empty field signals end of paths */\n> +\t\t\tcur++;\n> +\t\t\tremain--;\n> +\t\t\tbreak;\n> +\t\t}\n> +\n> +\t\tALLOC_GROW(paths, nr_paths + 1, alloc_paths);\n> +\t\tpaths[nr_paths++] = cur;\n> +\n> +\t\tremain -= end - cur + 1;\n> +\t\tcur = end + 1;\n> +\t}\n> +\n\nOK.\n\n> +\twhile (remain) {\n> +\t\tstruct object_id oid;\n> +\t\tstruct ewah_bitmap *ewah;\n> +\t\tssize_t len;\n> +\n> +\t\tif (remain < GIT_SHA1_RAWSZ) {\n> +\t\t\terror(\"truncated input reading oid\");\n> +\t\t\tgoto out;\n> +\t\t}\n> +\t\thashcpy(oid.hash, (const unsigned char *)cur);\n> +\t\tcur += GIT_SHA1_RAWSZ;\n> +\t\tremain -= GIT_SHA1_RAWSZ;\n> +\n> +\t\tewah = ewah_new();\n> +\t\tlen = ewah_read_mmap(ewah, cur, remain);\n> +\t\tif (len < 0) {\n> +\t\t\tewah_free(ewah);\n> +\t\t\tgoto out;\n> +\t\t}\n> +\n> +\t\tprintf(\"%s\\n\", oid_to_hex(&oid));\n> +\t\tewah_each_bit(ewah, show_path, paths);\n> +\n> +\t\tewah_free(ewah);\n> +\t\tcur += len;\n> +\t\tremain -= len;\n> +\t}\n\nMakes perfect sense.\n\n> +out:\n> +\tfree(paths);\n> +\tstrbuf_release(&in);\n> +}\n> +\n> +int cmd_main(int argc, const char **argv)\n> +{\n> +\tconst char *usage_msg = \"test-tree-bitmap <gen|dump>\";\n> +\n> +\tif (!argv[1])\n> +\t\tusage(usage_msg);\n> +\telse if (!strcmp(argv[1], \"gen\"))\n> +\t\tdo_gen();\n> +\telse if (!strcmp(argv[1], \"dump\"))\n> +\t\tdo_dump();\n> +\telse\n> +\t\tusage(usage_msg);\n>  \n>  \treturn 0;\n>  }\n"},{"id":"359970","messageId":"xmqqr2gyobw4.fsf@gitster-ct.c.googlers.com","threadId":"49479","inReplyTo":"20181009231441.GC23730@sigill.intra.peff.net","subject":"Re: [PoC -- do not apply 3/3] test-tree-bitmap: replace ewah with custom rle encoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-10T00:58:51Z","receivedAt":"2018-10-10T00:58:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> +static void strbuf_add_varint(struct strbuf *out, uintmax_t val)\n> +{\n> +\tsize_t len;\n> +\tstrbuf_grow(out, 16); /* enough for any varint */\n> +\tlen = encode_varint(val, (unsigned char *)out->buf + out->len);\n> +\tstrbuf_setlen(out, out->len + len);\n> +}\n> +\n> +static void bitmap_to_rle(struct strbuf *out, struct bitmap *bitmap)\n> +{\n> +\tint curval = 0; /* count zeroes, then ones, then zeroes, etc */\n> +\tsize_t run = 0;\n> +\tsize_t word;\n> +\tsize_t orig_len = out->len;\n> +\n> +\tfor (word = 0; word < bitmap->word_alloc; word++) {\n> +\t\tint bit;\n> +\n> +\t\tfor (bit = 0; bit < BITS_IN_EWORD; bit++) {\n> +\t\t\tint val = !!(bitmap->words[word] & (((eword_t)1) << bit));\n> +\t\t\tif (val == curval)\n> +\t\t\t\trun++;\n> +\t\t\telse {\n> +\t\t\t\tstrbuf_add_varint(out, run);\n> +\t\t\t\tcurval = 1 - curval; /* flip 0/1 */\n> +\t\t\t\trun = 1;\n> +\t\t\t}\n> +\t\t}\n\nOK.  I find it a bit disturbing to see that the loop knows a bit too\nmuch about how \"struct bitmap\" is implemented, but that is a complaint\nagainst the bitmap API, not this new user of the API.\n\nWe do not try to handle the case where bitmap has bits that is not\nmultiple of BITS_IN_EWORD and instead pretend that size of such a\nbitmap can be rounded up, because we ignore trailing 0-bit anyway,\nand we know the \"struct bitmap\" would pad with 0-bit at the tail?\n\n> +\t}\n> +\n> +\t/*\n> +\t * complete the run, but do not bother with trailing zeroes, unless we\n> +\t * failed to write even an initial run of 0's.\n> +\t */\n> +\tif (curval && run)\n> +\t\tstrbuf_add_varint(out, run);\n> +\telse if (orig_len == out->len)\n> +\t\tstrbuf_add_varint(out, 0);\n> +\n> +\t/* signal end-of-input with an empty run */\n> +\tstrbuf_add_varint(out, 0);\n> +}\n\nOK.\n\n> +static size_t rle_each_bit(const unsigned char *in, size_t len,\n> +\t\t\t   void (*fn)(size_t, void *), void *data)\n> +{\n> +\tint curval = 0; /* look for zeroes first, then ones, etc */\n> +\tconst unsigned char *cur = in;\n> +\tconst unsigned char *end = in + len;\n> +\tsize_t pos;\n> +\n> +\t/* we always have a first run, even if it's 0 zeroes */\n> +\tpos = decode_varint(&cur);\n> +\n> +\t/*\n> +\t * ugh, varint does not seem to have a way to prevent reading past\n> +\t * the end of the buffer. We'll do a length check after each one,\n> +\t * so the worst case is bounded.\n> +\t */\n\nSorry about that :-).\n\n> +\tif (cur > end) {\n> +\t\terror(\"input underflow in rle\");\n> +\t\treturn len;\n> +\t}\n> +\n> +\twhile (1) {\n> +\t\tsize_t run = decode_varint(&cur);\n> +\n> +\t\tif (cur > end) {\n> +\t\t\terror(\"input underflow in rle\");\n> +\t\t\treturn len;\n> +\t\t}\n> +\n> +\t\tif (!run)\n> +\t\t\tbreak; /* empty run signals end */\n> +\n> +\t\tcurval = 1 - curval; /* flip 0/1 */\n> +\t\tif (curval) {\n> +\t\t\t/* we have a run of 1's; deliver them */\n> +\t\t\tsize_t i;\n> +\t\t\tfor (i = 0; i < run; i++)\n> +\t\t\t\tfn(pos + i, data);\n> +\t\t}\n> +\t\tpos += run;\n> +\t}\n> +\n> +\treturn cur - in;\n> +}\n\nMakes sense.\n\n"},{"id":"360092","messageId":"20181010220739.GF23446@szeder.dev","threadId":"49479","inReplyTo":"xmqqpnwo3ow9.fsf@gitster-ct.c.googlers.com","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-10T22:07:39Z","receivedAt":"2018-10-10T22:07:46Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Oct 04, 2018 at 11:09:58PM -0700, Junio C Hamano wrote:\n> SZEDER Gábor <szeder.dev@gmail.com> writes:\n> \n> >>     git-gc - Cleanup unnecessary files and optimize the local repository\n> >> \n> >> Creating these indexes like the commit-graph falls under \"optimize the\n> >> local repository\",\n> >\n> > But it doesn't fall under \"cleanup unnecessary files\", which the\n> > commit-graph file is, since, strictly speaking, it's purely\n> > optimization.\n> \n> I won't be actively engaged in this discussion soon, but I must say\n> that \"git gc\" doing \"garbage collection\" is merely an implementation\n> detail of optimizing the repository for further use.  And from that\n> point of view, what needs to be updated is the synopsis of the\n> git-gc doc.  It states \"X and Y\" above, but it actually is \"Y by\n> doing X and other things\".\n\nWell, then perhaps the name of the command should be updated, too, to\nbetter reflect what it actually does...\n\n> I understand your \"by definition there is no garbage immediately\n> after clone\" position, and also I would understand if you find it\n> (perhaps philosophically) disturbing that \"git clone\" may give users\n> a suboptimal repository that immediately needs optimizing [*1*].\n> \n> But that bridge was crossed long time ago ever since pack transfer\n> was invented.  The data source sends only the pack data stream, and\n> the receiving end is responsible for spending cycles to build .idx\n> file.  Theoretically, .pack should be all that is needed---you\n> should be able to locate any necessary object by parsing the .pack\n> file every time you open it, and .idx is mere optimization.  You can\n> think of the .midx and graph files the same way.\n\nI don't think this is a valid comparison, because, practically, Git\njust didn't work after I deleted all pack index files.  So while they\ncan be easily (re)generated, they are essential to make pack files\nusable.  The commit-graph and .midx files, however, can be safely\ndeleted, and everything keeps working as before.\n\nOTOH, this is an excellent comparison, and I do think of the .midx and\ngraph files the same way as the pack index files.  During a clone, the\npack index file isn't generated by running a separate 'git gc\n(--auto)', but by clone (or fetch-pack?) running 'git index-pack'.\nThe way I see it that should be the case for these other files as well.\n\nAnd it is much simpler, shorter, and cleaner to either run 'git\ncommit-graph ...' or even to call write_commit_graph_reachable()\ndirectly from cmd_clone(), than to bolt on another option and config\nvariable on 'git gc' [1] to coax it into some kind of an \"after clone\"\nmode, that it shouldn't be doing in the first place.  At least for\nnow, so when we'll eventually get as far ...\n\n> I would not be surprised by a future in which the initial index-pack\n> that is responsible for receiving the incoming pack stream and\n> storing that in .pack file(s) while creating corresponding .idx\n> file(s) becomes also responsible for building .midx and graph files\n> in the same pass, or at least smaller number of passes.  Once we\n> gain experience and confidence with these new auxiliary files, that\n> ought to happen naturally.  And at that point, we won't be having\n> this discussion---we'd all happily run index-pack to receive the\n> pack data, because that is pretty much the fundamental requirement\n> to make use of the data.\n\n... that what you wrote here becomes a reality (and I fully agree that\nthis is what we should ultimately aim for), then we won't have that\noption and config variable still lying around and requiring\nmaintenance because of backwards compatibility.\n\n1 - https://public-inbox.org/git/87in2hgzin.fsf@evledraar.gmail.com/\n\n> [Footnote]\n> \n> *1* Even without considering these recent invention of auxiliary\n>     files, cloning from a sloppily packed server whose primary focus\n>     is to avoid spending cycles by not computing better deltas will\n>     give the cloner a suboptimal repository.  If we truly want to\n>     have an optimized repository ready to be used after cloning, we\n>     should run an equivalent of \"repack -a -d -f\" immediately after\n>     \"git clone\".\n\nI noticed a few times that I got surprisingly large packs from GitHub,\ne.g. there is over 70% size difference between --single-branch cloning\nv2.19.0 from GitHub and from my local clone or from kernel.org (~95MB\nvs. ~55MB vs ~52MB).  After running 'git repack -a -d -f' they all end\nup at ~65MB, which is a nice size reduction for the clone from GitHub,\nbut the others just gained 10-13 more MBs.\n\n"},{"id":"360106","messageId":"87lg75e78g.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"20181010220739.GF23446@szeder.dev","subject":"Re: We should add a \"git gc --auto\" after \"git clone\" due to commit graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-10T23:01:51Z","receivedAt":"2018-10-10T23:01:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 10 2018, SZEDER Gábor wrote:\n\n> On Thu, Oct 04, 2018 at 11:09:58PM -0700, Junio C Hamano wrote:\n>> SZEDER Gábor <szeder.dev@gmail.com> writes:\n>>\n>> >>     git-gc - Cleanup unnecessary files and optimize the local repository\n>> >>\n>> >> Creating these indexes like the commit-graph falls under \"optimize the\n>> >> local repository\",\n>> >\n>> > But it doesn't fall under \"cleanup unnecessary files\", which the\n>> > commit-graph file is, since, strictly speaking, it's purely\n>> > optimization.\n>>\n>> I won't be actively engaged in this discussion soon, but I must say\n>> that \"git gc\" doing \"garbage collection\" is merely an implementation\n>> detail of optimizing the repository for further use.  And from that\n>> point of view, what needs to be updated is the synopsis of the\n>> git-gc doc.  It states \"X and Y\" above, but it actually is \"Y by\n>> doing X and other things\".\n>\n> Well, then perhaps the name of the command should be updated, too, to\n> better reflect what it actually does...\n\nI don't disagree, but between \"git gc\" being a longstanding thing called\nnot just by us , but third parties expecting it to a be a general swiss\narmy knife for \"make repo better\" (so for a new tool they'd need to\nupdate their code), and general name bikeshedding I think it's best if\nwe just proceed for the sake of argument with the assumption that none\nof us find the name confusing in this context.\n\nAt least that's the discussion I'm interested in. I.e. whether it makes\nconceptual / structural sense for this stuff to live in the same place /\nfunction in the code. We can always argue about the name of the function\nas a separate topic.\n\n>> I understand your \"by definition there is no garbage immediately\n>> after clone\" position, and also I would understand if you find it\n>> (perhaps philosophically) disturbing that \"git clone\" may give users\n>> a suboptimal repository that immediately needs optimizing [*1*].\n>>\n>> But that bridge was crossed long time ago ever since pack transfer\n>> was invented.  The data source sends only the pack data stream, and\n>> the receiving end is responsible for spending cycles to build .idx\n>> file.  Theoretically, .pack should be all that is needed---you\n>> should be able to locate any necessary object by parsing the .pack\n>> file every time you open it, and .idx is mere optimization.  You can\n>> think of the .midx and graph files the same way.\n>\n> I don't think this is a valid comparison, because, practically, Git\n> just didn't work after I deleted all pack index files.  So while they\n> can be easily (re)generated, they are essential to make pack files\n> usable.  The commit-graph and .midx files, however, can be safely\n> deleted, and everything keeps working as before.\n\nFor \"things that would run in 20ms now run in 30 seconds\" (actual\nnumbers on a repo I have) values of \"keeps working\".\n\nSo I think this line gets blurred somewhat. In practice if you're\nexpecting the graph to be there to run the sort of commands that most\nbenefit from it it's essential that it be generated, not some nice\noptional extra.\n\n> OTOH, this is an excellent comparison, and I do think of the .midx and\n> graph files the same way as the pack index files.  During a clone, the\n> pack index file isn't generated by running a separate 'git gc\n> (--auto)', but by clone (or fetch-pack?) running 'git index-pack'.\n> The way I see it that should be the case for these other files as well.\n>\n> And it is much simpler, shorter, and cleaner to either run 'git\n> commit-graph ...' or even to call write_commit_graph_reachable()\n> directly from cmd_clone(), than to bolt on another option and config\n> variable on 'git gc' [1] to coax it into some kind of an \"after clone\"\n> mode, that it shouldn't be doing in the first place.  At least for\n> now, so when we'll eventually get as far ...\n>\n>> I would not be surprised by a future in which the initial index-pack\n>> that is responsible for receiving the incoming pack stream and\n>> storing that in .pack file(s) while creating corresponding .idx\n>> file(s) becomes also responsible for building .midx and graph files\n>> in the same pass, or at least smaller number of passes.  Once we\n>> gain experience and confidence with these new auxiliary files, that\n>> ought to happen naturally.  And at that point, we won't be having\n>> this discussion---we'd all happily run index-pack to receive the\n>> pack data, because that is pretty much the fundamental requirement\n>> to make use of the data.\n>\n> ... that what you wrote here becomes a reality (and I fully agree that\n> this is what we should ultimately aim for), then we won't have that\n> option and config variable still lying around and requiring\n> maintenance because of backwards compatibility.\n\nWe'll still have the use-case of wanting to turn on\ngc.writeCommitGraph=true or equivalent and wanting previously-cloned\nrepositories to \"catch up\" and get a commit graph ASAP (but not do a\nfull repack).\n\nThis is why my patch tries to unify those two codepaths, i.e. so I can\nturn this on and know that on the next \"git fetch\" we'll have the graph\n(without needing to also run a full repack if it's not needed).\n\n> 1 - https://public-inbox.org/git/87in2hgzin.fsf@evledraar.gmail.com/\n>\n>> [Footnote]\n>>\n>> *1* Even without considering these recent invention of auxiliary\n>>     files, cloning from a sloppily packed server whose primary focus\n>>     is to avoid spending cycles by not computing better deltas will\n>>     give the cloner a suboptimal repository.  If we truly want to\n>>     have an optimized repository ready to be used after cloning, we\n>>     should run an equivalent of \"repack -a -d -f\" immediately after\n>>     \"git clone\".\n>\n> I noticed a few times that I got surprisingly large packs from GitHub,\n> e.g. there is over 70% size difference between --single-branch cloning\n> v2.19.0 from GitHub and from my local clone or from kernel.org (~95MB\n> vs. ~55MB vs ~52MB).  After running 'git repack -a -d -f' they all end\n> up at ~65MB, which is a nice size reduction for the clone from GitHub,\n> but the others just gained 10-13 more MBs.\n\nTo me this sounds like even more reason for why we shouldn't be\nsplitting up the entry point for the \"does the repo look shitty? Fix\nit!\" function (currently called git-gc).\n\nWe might get such crappy packs from a clone, or from a fetch, or maybe\nin the future when e.g. \"git add\" or \"git commit\" generate a pack\ninstead of a bunch of loose objects we'd want to immediately kick off\nsomething in the background to optimize / consolidate them.\n\nSo instead of having clone/commit/add/whatever call some custom\nfunction(s) to just do the things they *think* they want, we just call\n\"gc --auto\", because that entry point needs to eventually know how to\n\"recover\" from any of those states without any prior knowledge or hints\nabout what just happened.\n\nThis is why I'm calling \"gc --auto\" from clone in this WIP series, with\nthe only special sauce being \"if you have stuff to do, don't background\nyourself\", because not having a commit graph at all after a clone is\njust one special case of many where we have no commit graph and want to\nhave one made ASAP (e.g. someone rm'd it, or the \"I want a commit graph\"\nconfig variable just got turned on).\n\nSo since we need all those smarts in some function anyway, let's just\nhave one entry point to that logic. Discarding what it doesn't need to\ndo (e.g. too_many_loose_objects() just after a clone) is a trivial cost,\nso if we don't care about that we have less complexity to worry about by\nnot having a proliferation of entry points into what are now subsets of\nthe GC code.\n"},{"id":"360123","messageId":"cover.1539219248.git.jonathantanmy@google.com","threadId":"49479","inReplyTo":"20181009193445.21908-1-szeder.dev@gmail.com","subject":"[PATCH 0/2] Per-commit filter proof of concept","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-10-11T01:21:19Z","receivedAt":"2018-10-11T01:21:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I did my own experiments on top of what Szeder provided - the first\npatch is to have one fixed-size bloom filter per commit, and the second\npatch makes that bloom filter apply to only the first parent of each\ncommit. The results are:\n\n  Original (szeder's)\n  $ GIT_USE_POC_BLOOM_FILTER=$((8*1024*1024*8)) time ./git commit-graph write\n  0:10.28\n  $ ls -l .git/objects/info/bloom\n  8388616\n  $ GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y time ./git -c \\\n    core.commitgraph=true rev-list --count --full-history HEAD -- \\\n    t/valgrind/valgrind.sh\n  886\n  bloom filter total queries: 66459 definitely not: 65276 maybe: 1183 false positives: 297 fp ratio: 0.004469\n  0:00.24\n\n  With patch 1\n  $ GIT_USE_POC_BLOOM_FILTER=256 time ./git commit-graph write\n  0:16.22\n  $ ls -l .git/objects/info/bloom\n  1832620\n  $ GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y time ./git -c \\\n    core.commitgraph=true rev-list --count --full-history HEAD -- \\\n    t/valgrind/valgrind.sh\n  886\n  bloom filter total queries: 66459 definitely not: 46637 maybe: 19822 false positives: 18936 fp ratio: 0.284928\n  0:01.53\n\n  With patch 2\n  $ GIT_USE_POC_BLOOM_FILTER=256 time ./git commit-graph write\n  0:06.70\n  $ ls -l .git/objects/info/bloom\n  1832620\n  $ GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y time ./git -c \\\n    core.commitgraph=true rev-list --count --full-history HEAD -- \\\n    t/valgrind/valgrind.sh\n  886\n  bloom filter total queries: 53096 definitely not: 52989 maybe: 107 false positives: 89 fp ratio: 0.001676\n  0:01.29\n\nFor comparison, a non-GIT_USE_POC_BLOOM_FILTER rev-list takes 3.517\nseconds.\n\nI haven't investigated why patch 1 takes longer than the original to\ncreate the bloom filter.\n\nUsing per-commit filters and restricting the bloom filter to a single\nparent increases the relative power of the filter in omitting tree\ninspections compared to the original (107/53096 vs 1183/66459), but the\nlack of coverage w.r.t. the non-first parents had a more significant\neffect than I thought (1.29s vs .24s). It might be best to have one\nfilter for each (commit, parent) pair (or, at least, the first two\nparents of each commit - we probably don't need to care that much about\noctopus merges) - this would take up more disk space than if we only\nstore filters for the first parent, but is still less than the original\nexample of storing information for all commits in one filter.\n\nThere are more possibilities like dynamic filter sizing, different\nhashing, and hashing to support wildcard matches, which I haven't looked\ninto.\n\nJonathan Tan (2):\n  One filter per commit\n  Only make bloom filter for first parent\n\n bloom-filter.c | 31 ++++++++++++++++++-------------\n bloom-filter.h | 12 ++++--------\n commit-graph.c | 30 ++++++++++++++----------------\n revision.c     | 29 +++++++++++++++--------------\n 4 files changed, 51 insertions(+), 51 deletions(-)\n\n-- \n2.19.0.271.gfe8321ec05.dirty\n\n"},{"id":"360124","messageId":"ebf0a811be047e9fd24b61fea3c4a164b3d18dc0.1539219248.git.jonathantanmy@google.com","threadId":"49479","inReplyTo":"cover.1539219248.git.jonathantanmy@google.com","subject":"[PATCH 1/2] One filter per commit","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-10-11T01:21:20Z","receivedAt":"2018-10-11T01:21:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n bloom-filter.c | 31 ++++++++++++++++++-------------\n bloom-filter.h | 12 ++++--------\n commit-graph.c | 26 ++++++++++++--------------\n revision.c     |  9 +++------\n 4 files changed, 37 insertions(+), 41 deletions(-)\n\ndiff --git a/bloom-filter.c b/bloom-filter.c\nindex 7dce0e35fa..39b453908f 100644\n--- a/bloom-filter.c\n+++ b/bloom-filter.c\n@@ -1,14 +1,17 @@\n #include \"cache.h\"\n #include \"bloom-filter.h\"\n \n-void bloom_filter_init(struct bloom_filter *bf, uint32_t bit_size)\n+void bloom_filter_init(struct bloom_filter *bf, uint32_t commit_nr, uint32_t bit_size)\n {\n \tif (bit_size % CHAR_BIT)\n \t\tBUG(\"invalid size for bloom filter\");\n+\tif (bit_size > 1024)\n+\t\tBUG(\"aborting: the bit size is per commit, not for the whole filter\");\n \n \tbf->nr_entries = 0;\n+\tbf->commit_nr = commit_nr;\n \tbf->bit_size = bit_size;\n-\tbf->bits = xmalloc(bit_size / CHAR_BIT);\n+\tbf->bits = xcalloc(1, commit_nr * bit_size / CHAR_BIT);\n }\n \n void bloom_filter_free(struct bloom_filter *bf)\n@@ -19,24 +22,24 @@ void bloom_filter_free(struct bloom_filter *bf)\n }\n \n \n-void bloom_filter_set_bits(struct bloom_filter *bf, const uint32_t *offsets,\n+static void bloom_filter_set_bits(struct bloom_filter *bf, uint32_t graph_pos, const uint32_t *offsets,\n \t\t\t   int nr_offsets, int nr_entries)\n {\n \tint i;\n \tfor (i = 0; i < nr_offsets; i++) {\n-\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size) / CHAR_BIT;\n+\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size + graph_pos * bf->bit_size) / CHAR_BIT;\n \t\tunsigned char mask = 1 << offsets[i] % CHAR_BIT;\n \t\tbf->bits[byte_offset] |= mask;\n \t}\n \tbf->nr_entries += nr_entries;\n }\n \n-int bloom_filter_check_bits(struct bloom_filter *bf, const uint32_t *offsets,\n+static int bloom_filter_check_bits(struct bloom_filter *bf, uint32_t graph_pos, const uint32_t *offsets,\n \t\t\t    int nr)\n {\n \tint i;\n \tfor (i = 0; i < nr; i++) {\n-\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size) / CHAR_BIT;\n+\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size + graph_pos * bf->bit_size) / CHAR_BIT;\n \t\tunsigned char mask = 1 << offsets[i] % CHAR_BIT;\n \t\tif (!(bf->bits[byte_offset] & mask))\n \t\t\treturn 0;\n@@ -45,19 +48,19 @@ int bloom_filter_check_bits(struct bloom_filter *bf, const uint32_t *offsets,\n }\n \n \n-void bloom_filter_add_hash(struct bloom_filter *bf, const unsigned char *hash)\n+void bloom_filter_add_hash(struct bloom_filter *bf, uint32_t graph_pos, const unsigned char *hash)\n {\n \tuint32_t offsets[GIT_MAX_RAWSZ / sizeof(uint32_t)];\n \thashcpy((unsigned char*)offsets, hash);\n-\tbloom_filter_set_bits(bf, offsets,\n+\tbloom_filter_set_bits(bf, graph_pos, offsets,\n \t\t\t     the_hash_algo->rawsz / sizeof(*offsets), 1);\n }\n \n-int bloom_filter_check_hash(struct bloom_filter *bf, const unsigned char *hash)\n+int bloom_filter_check_hash(struct bloom_filter *bf, uint32_t graph_pos, const unsigned char *hash)\n {\n \tuint32_t offsets[GIT_MAX_RAWSZ / sizeof(uint32_t)];\n \thashcpy((unsigned char*)offsets, hash);\n-\treturn bloom_filter_check_bits(bf, offsets,\n+\treturn bloom_filter_check_bits(bf, graph_pos, offsets,\n \t\t\tthe_hash_algo->rawsz / sizeof(*offsets));\n }\n \n@@ -80,11 +83,12 @@ int bloom_filter_load(struct bloom_filter *bf)\n \t\treturn -1;\n \n \tread_in_full(fd, &bf->nr_entries, sizeof(bf->nr_entries));\n+\tread_in_full(fd, &bf->commit_nr, sizeof(bf->commit_nr));\n \tread_in_full(fd, &bf->bit_size, sizeof(bf->bit_size));\n \tif (bf->bit_size % CHAR_BIT)\n \t\tBUG(\"invalid size for bloom filter\");\n-\tbf->bits = xmalloc(bf->bit_size / CHAR_BIT);\n-\tread_in_full(fd, bf->bits, bf->bit_size / CHAR_BIT);\n+\tbf->bits = xmalloc(bf->commit_nr * bf->bit_size / CHAR_BIT);\n+\tread_in_full(fd, bf->bits, bf->commit_nr * bf->bit_size / CHAR_BIT);\n \n \tclose(fd);\n \n@@ -96,8 +100,9 @@ void bloom_filter_write(struct bloom_filter *bf)\n \tint fd = xopen(git_path_bloom(), O_WRONLY | O_CREAT | O_TRUNC, 0666);\n \n \twrite_in_full(fd, &bf->nr_entries, sizeof(bf->nr_entries));\n+\twrite_in_full(fd, &bf->commit_nr, sizeof(bf->commit_nr));\n \twrite_in_full(fd, &bf->bit_size, sizeof(bf->bit_size));\n-\twrite_in_full(fd, bf->bits, bf->bit_size / CHAR_BIT);\n+\twrite_in_full(fd, bf->bits, bf->commit_nr * bf->bit_size / CHAR_BIT);\n \n \tclose(fd);\n }\ndiff --git a/bloom-filter.h b/bloom-filter.h\nindex 94d0af1708..607649b8db 100644\n--- a/bloom-filter.h\n+++ b/bloom-filter.h\n@@ -5,30 +5,26 @@\n \n struct bloom_filter {\n \tuint32_t nr_entries;\n+\tuint32_t commit_nr;\n \tuint32_t bit_size;\n \tunsigned char *bits;\n };\n \n \n-void bloom_filter_init(struct bloom_filter *bf, uint32_t bit_size);\n+void bloom_filter_init(struct bloom_filter *bf, uint32_t commit_nr, uint32_t bit_size);\n void bloom_filter_free(struct bloom_filter *bf);\n \n-void bloom_filter_set_bits(struct bloom_filter *bf, const uint32_t *offsets,\n-\t\t\t   int nr_offsets, int nr_enries);\n-int bloom_filter_check_bits(struct bloom_filter *bf, const uint32_t *offsets,\n-\t\t\t    int nr);\n-\n /*\n  * Turns the given (SHA1) hash into 5 unsigned ints, and sets the bits at\n  * those positions (modulo the bitmap's size) in the Bloom filter.\n  */\n-void bloom_filter_add_hash(struct bloom_filter *bf, const unsigned char *hash);\n+void bloom_filter_add_hash(struct bloom_filter *bf, uint32_t graph_pos, const unsigned char *hash);\n /*\n  * Turns the given (SHA1) hash into 5 unsigned ints, and checks the bits at\n  * those positions (modulo the bitmap's size) in the Bloom filter.\n  * Returns 1 if all those bits are set, 0 otherwise.\n  */\n-int bloom_filter_check_hash(struct bloom_filter *bf, const unsigned char *hash);\n+int bloom_filter_check_hash(struct bloom_filter *bf, uint32_t graph_pos, const unsigned char *hash);\n \n void hashxor(const unsigned char *hash1, const unsigned char *hash2,\n \t     unsigned char *out);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f415d3b41f..90b0b3df90 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -715,13 +715,11 @@ static int add_ref_to_list(const char *refname,\n static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n \t\t\t\t\tstruct commit *parent,\n \t\t\t\t\tstruct commit *commit,\n+\t\t\t\t\tint index,\n \t\t\t\t\tstruct diff_options *diffopt)\n {\n-\tunsigned char p_c_hash[GIT_MAX_RAWSZ];\n \tint i;\n \n-\thashxor(parent->object.oid.hash, commit->object.oid.hash, p_c_hash);\n-\n \tdiff_tree_oid(&parent->object.oid, &commit->object.oid, \"\", diffopt);\n \tdiffcore_std(diffopt);\n \n@@ -756,8 +754,8 @@ static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n \t\t\tthe_hash_algo->update_fn(&ctx, path, p - path);\n \t\t\tthe_hash_algo->final_fn(name_hash, &ctx);\n \n-\t\t\thashxor(name_hash, p_c_hash, hash);\n-\t\t\tbloom_filter_add_hash(bf, hash);\n+\t\t\thashxor(name_hash, parent->object.oid.hash, hash);\n+\t\t\tbloom_filter_add_hash(bf, index, hash);\n \t\t} while (*p);\n \n \t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n@@ -768,11 +766,10 @@ static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n }\n \n static void fill_bloom_filter(struct bloom_filter *bf,\n-\t\t\t\t    struct progress *progress)\n+\t\t\t\t    struct progress *progress, struct commit **commits, int commit_nr)\n {\n \tstruct rev_info revs;\n \tconst char *revs_argv[] = {NULL, \"--all\", NULL};\n-\tstruct commit *commit;\n \tint i = 0;\n \n \t/* We (re-)create the bloom filter from scratch every time for now. */\n@@ -783,18 +780,19 @@ static void fill_bloom_filter(struct bloom_filter *bf,\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed while preparing bloom filter\");\n \n-\twhile ((commit = get_revision(&revs))) {\n+\tfor (i = 0; i < commit_nr; i++) {\n+\t\tstruct commit *commit = commits[i];\n \t\tstruct commit_list *parent;\n \n \t\tfor (parent = commit->parents; parent; parent = parent->next)\n-\t\t\tadd_changes_to_bloom_filter(bf, parent->item, commit,\n+\t\t\tadd_changes_to_bloom_filter(bf, parent->item, commit, i,\n \t\t\t\t\t\t    &revs.diffopt);\n \n-\t\tdisplay_progress(progress, ++i);\n+\t\tdisplay_progress(progress, i);\n \t}\n }\n \n-static void write_bloom_filter(int report_progress, int commit_nr)\n+static void write_bloom_filter(int report_progress, struct commit **commits, int commit_nr)\n {\n \tstruct bloom_filter bf;\n \tstruct progress *progress = NULL;\n@@ -809,13 +807,13 @@ static void write_bloom_filter(int report_progress, int commit_nr)\n \tif (*end)\n \t\tdie(\"GIT_USE_POC_BLOOM_FILTER must specify the number of bits in the bloom filter (multiple of 8, n < 2^32)\");\n \n-\tbloom_filter_init(&bf, bitsize);\n+\tbloom_filter_init(&bf, commit_nr, bitsize);\n \n \tif (report_progress)\n \t\tprogress = start_progress(_(\"Computing bloom filter\"),\n \t\t\t\t\t  commit_nr);\n \n-\tfill_bloom_filter(&bf, progress);\n+\tfill_bloom_filter(&bf, progress, commits, commit_nr);\n \n \tbloom_filter_write(&bf);\n \tbloom_filter_free(&bf);\n@@ -1030,7 +1028,7 @@ void write_commit_graph(const char *obj_dir,\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n-\twrite_bloom_filter(report_progress, commits.nr);\n+\twrite_bloom_filter(report_progress, commits.list, commits.nr);\n \n \tfree(graph_name);\n \tfree(commits.list);\ndiff --git a/revision.c b/revision.c\nindex d5ba2b1598..c84a997928 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -490,7 +490,6 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\t\t\t\t\t struct commit *parent,\n \t\t\t\t\t\t struct commit *commit)\n {\n-\tunsigned char p_c_hash[GIT_MAX_RAWSZ];\n \tint i;\n \n \tif (!bf.bits)\n@@ -509,17 +508,15 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t */\n \tif (!the_repository->objects->commit_graph)\n \t\treturn -1;\n-\tif (commit->generation == GENERATION_NUMBER_INFINITY)\n+\tif (commit->graph_pos == COMMIT_NOT_FROM_GRAPH)\n \t\treturn -1;\n \n-\thashxor(parent->object.oid.hash, commit->object.oid.hash, p_c_hash);\n-\n \tfor (i = 0; i < revs->pruning.pathspec.nr; i++) {\n \t\tstruct pathspec_item *pi = &revs->pruning.pathspec.items[i];\n \t\tunsigned char hash[GIT_MAX_RAWSZ];\n \n-\t\thashxor(pi->name_hash, p_c_hash, hash);\n-\t\tif (bloom_filter_check_hash(&bf, hash)) {\n+\t\thashxor(pi->name_hash, parent->object.oid.hash, hash);\n+\t\tif (bloom_filter_check_hash(&bf, commit->graph_pos, hash)) {\n \t\t\t/*\n \t\t\t * At least one of the interesting pathspecs differs,\n \t\t\t * so we can return early and let the diff machinery\n-- \n2.19.0.271.gfe8321ec05.dirty\n\n"},{"id":"360125","messageId":"fb814c65d84f213778be21f49e62da2d959dd964.1539219248.git.jonathantanmy@google.com","threadId":"49479","inReplyTo":"cover.1539219248.git.jonathantanmy@google.com","subject":"[PATCH 2/2] Only make bloom filter for first parent","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-10-11T01:21:21Z","receivedAt":"2018-10-11T01:21:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n commit-graph.c |  4 ++--\n revision.c     | 20 ++++++++++++--------\n 2 files changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 90b0b3df90..d21d555611 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -782,9 +782,9 @@ static void fill_bloom_filter(struct bloom_filter *bf,\n \n \tfor (i = 0; i < commit_nr; i++) {\n \t\tstruct commit *commit = commits[i];\n-\t\tstruct commit_list *parent;\n+\t\tstruct commit_list *parent = commit->parents;\n \n-\t\tfor (parent = commit->parents; parent; parent = parent->next)\n+\t\tif (parent)\n \t\t\tadd_changes_to_bloom_filter(bf, parent->item, commit, i,\n \t\t\t\t\t\t    &revs.diffopt);\n \ndiff --git a/revision.c b/revision.c\nindex c84a997928..5a433a5878 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -539,11 +539,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n }\n \n static int rev_compare_tree(struct rev_info *revs,\n-\t\t\t    struct commit *parent, struct commit *commit)\n+\t\t\t    struct commit *parent, struct commit *commit, int nth_parent)\n {\n \tstruct tree *t1 = get_commit_tree(parent);\n \tstruct tree *t2 = get_commit_tree(commit);\n-\tint bloom_ret;\n+\tint bloom_ret = 1;\n \n \tif (!t1)\n \t\treturn REV_TREE_NEW;\n@@ -568,17 +568,21 @@ static int rev_compare_tree(struct rev_info *revs,\n \t\t\treturn REV_TREE_SAME;\n \t}\n \n-\tbloom_ret = check_maybe_different_in_bloom_filter(revs, parent, commit);\n-\tif (bloom_ret == 0)\n-\t\treturn REV_TREE_SAME;\n+\tif (!nth_parent) {\n+\t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, parent, commit);\n+\t\tif (bloom_ret == 0)\n+\t\t\treturn REV_TREE_SAME;\n+\t}\n \n \ttree_difference = REV_TREE_SAME;\n \trevs->pruning.flags.has_changes = 0;\n \tif (diff_tree_oid(&t1->object.oid, &t2->object.oid, \"\",\n \t\t\t   &revs->pruning) < 0)\n \t\treturn REV_TREE_DIFFERENT;\n-\tif (bloom_ret == 1 && tree_difference == REV_TREE_SAME)\n-\t\tbloom_filter_count_false_positive++;\n+\tif (!nth_parent) {\n+\t\tif (bloom_ret == 1 && tree_difference == REV_TREE_SAME)\n+\t\t\tbloom_filter_count_false_positive++;\n+\t}\n \treturn tree_difference;\n }\n \n@@ -776,7 +780,7 @@ static void try_to_simplify_commit(struct rev_info *revs, struct commit *commit)\n \t\t\tdie(\"cannot simplify commit %s (because of %s)\",\n \t\t\t    oid_to_hex(&commit->object.oid),\n \t\t\t    oid_to_hex(&p->object.oid));\n-\t\tswitch (rev_compare_tree(revs, p, commit)) {\n+\t\tswitch (rev_compare_tree(revs, p, commit, nth_parent)) {\n \t\tcase REV_TREE_SAME:\n \t\t\tif (!revs->simplify_history || !relevant_commit(p)) {\n \t\t\t\t/* Even if a merge with an uninteresting\n-- \n2.19.0.271.gfe8321ec05.dirty\n\n"},{"id":"360135","messageId":"20181011031303.GB25067@sigill.intra.peff.net","threadId":"49479","inReplyTo":"xmqqy3b6occq.fsf@gitster-ct.c.googlers.com","subject":"Re: [PoC -- do not apply 2/3] test-tree-bitmap: add \"dump\" mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T03:13:03Z","receivedAt":"2018-10-11T03:13:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 10, 2018 at 09:48:53AM +0900, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> > The one difference is the sort order: git's diff output is\n> > in tree-sort order, so a subtree \"foo\" sorts like \"foo/\",\n> > which is after \"foo.bar\". Whereas the bitmap path list has a\n> > true byte sort, which puts \"foo.bar\" after \"foo\".\n> \n> If we truly cared, it is easy enough to fix by having a custom\n> comparison function in 1/3 used in collect_paths() phase.\n\nYep. I thought about doing it just so I could drop this \"one difference\"\nnote, but I got lazy.\n\nRunning this on linux.git, I do see a few other differences. It looks\nlike my code does actually compute lists of touched paths for some\nmerges (presumably using \"-c\"). That wasn't intended, and it would\nactually make my timings less good, but my goal was just to get a rough\nidea on size here (but see below).\n\n> > +\t/* dump it while we have the sorted order in memory */\n> > +\tfor (i = 0; i < n; i++) {\n> > +\t\tprintf(\"%s\", sorted[i]->path);\n> > +\t\tputchar('\\0');\n> > +\t}\n> \n> With printf(\"%s%c\", sorted[i]->path, '\\0'); you can lose the braces.\n\nHeh, I didn't really expect review at that level. I'm not even sure this\nis a good direction to go versus something like the bloom filters (or\neven a more full --raw cache). But if it is, this code is mostly\nthrow-away anyway, as we'd want to integrate it with the actual diff\ncode.\n\nMy original goal had mostly been to get an idea of the size, and the\n\"dump\" half was there to verify that the results were roughly sane. But\nit actually works for rough timing, too. I can generate roughly the same\nresults as \"rev-list --all | diff-tree --stdin -t --name-only\" in about\n300ms, as opposed to 33s. So that's good.\n\nBut it's also a slight cheat, since I'm not actually traversing the\ncommits, but rather just opening up the bitmaps in the order we wrote\nthem. ;)\n\nActually walking the commits (and not looking at the trees) takes ~7s,\nso it would at least be more like 33s versus 7.3s. With core.commitgraph,\nit's more like 1.1s, so imagine 27s versus 1.4s, I guess.\n\nThat's also neglecting any load/lookup time for actual random-access to\nthe bitmaps. I doubt that's more than a few hundred ms, but that's just\na made-up number.\n\nSo I think the rough timings are favorable, but the real proof would\nactually be using it from a revision walk, which I haven't written.\n\n> > +\tputchar('\\0');\n> > +\n> >  \tfree(sorted);\n> >  }\n> >  \n> > @@ -142,6 +150,8 @@ static void generate_bitmap(struct diff_queue_struct *q,\n> >  \n> >  \tewah = bitmap_to_ewah(bitmap);\n> >  \tewah_serialize_strbuf(ewah, &out);\n> > +\n> > +\tfwrite(data->commit->object.oid.hash, 1, GIT_SHA1_RAWSZ, stdout);\n> >  \tfwrite(out.buf, 1, out.len, stdout);\n> \n> OK, so per commit, we have ewah bitmap that records the \"changed\n> paths\" after the commit object name.  Makes sense.\n\nYeah. This format, btw, is garbage. It was just the smallest and\nsimplest thing I could think of that would work for my case. We'd want\nrandom-access to the bitmaps for each commit, probably via an index\nblock in the commit-graph file.\n\n> And the list of paths are based on the \"one\" side of the filepair.\n> When we do an equivalent of \"git show X\", we see \"diff-tree X~1 X\"\n> and by collecting the \"one\" side (i.e. subset of paths in the tree\n> of X~1 that were modified when going to X) we say \"commit X changed\n> these paths\".  Makes sense, too.\n\nI didn't think too hard on whether we'd need to look at the \"two\" side\never. I turned off renames, so we'd see deletions via the \"one\". I feel\nlike we'd miss additions in that case, though, but from my results, we\ndo not seem to.\n\n-Peff\n"},{"id":"360137","messageId":"20181011032027.GC25067@sigill.intra.peff.net","threadId":"49479","inReplyTo":"xmqqr2gyobw4.fsf@gitster-ct.c.googlers.com","subject":"Re: [PoC -- do not apply 3/3] test-tree-bitmap: replace ewah with custom rle encoding","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T03:20:28Z","receivedAt":"2018-10-11T03:20:31Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 10, 2018 at 09:58:51AM +0900, Junio C Hamano wrote:\n\n> > +static void bitmap_to_rle(struct strbuf *out, struct bitmap *bitmap)\n> > +{\n> > +\tint curval = 0; /* count zeroes, then ones, then zeroes, etc */\n> > +\tsize_t run = 0;\n> > +\tsize_t word;\n> > +\tsize_t orig_len = out->len;\n> > +\n> > +\tfor (word = 0; word < bitmap->word_alloc; word++) {\n> > +\t\tint bit;\n> > +\n> > +\t\tfor (bit = 0; bit < BITS_IN_EWORD; bit++) {\n> > +\t\t\tint val = !!(bitmap->words[word] & (((eword_t)1) << bit));\n> > +\t\t\tif (val == curval)\n> > +\t\t\t\trun++;\n> > +\t\t\telse {\n> > +\t\t\t\tstrbuf_add_varint(out, run);\n> > +\t\t\t\tcurval = 1 - curval; /* flip 0/1 */\n> > +\t\t\t\trun = 1;\n> > +\t\t\t}\n> > +\t\t}\n> \n> OK.  I find it a bit disturbing to see that the loop knows a bit too\n> much about how \"struct bitmap\" is implemented, but that is a complaint\n> against the bitmap API, not this new user of the API.\n\nHeh, again, this is not really meant to be production code. I'm not at\nall happy about inventing a new compressed bitmap format here, and I'd\nwant to investigate the state of the art a bit more. In particular, the\nworst case here is quite bad, and I wonder if there are formats that can\nselect the best encoding when writing a bitmap (naive RLE when it's\ngood, something else other times).\n\nI also suspect part of why this does better is that other formats are\noptimized less for our case. We really don't care about setting or\nlooking at a few bits part way through a bitmap. Our bitmaps are small\nenough that we don't mind streaming through a whole one. It's just that\nwe have so _many_ of them that we want to be meticulous about wasted\nbytes.\n\nWhatever format we choose, I think it would become part of the bitmap.c\nfile, and internal details would be OK to access there. I just put it\nhere to keep the patch simple.\n\n> We do not try to handle the case where bitmap has bits that is not\n> multiple of BITS_IN_EWORD and instead pretend that size of such a\n> bitmap can be rounded up, because we ignore trailing 0-bit anyway,\n> and we know the \"struct bitmap\" would pad with 0-bit at the tail?\n\nRight. We do not know the \"real\" number of zero bits at all. It's just\nassumed that there are infinite zeroes trailing off the end (and this is\nhow \"struct bitmap\" works, since it is the one that does not bother to\nkeep a separate size pointer).\n\n> > +\t/*\n> > +\t * ugh, varint does not seem to have a way to prevent reading past\n> > +\t * the end of the buffer. We'll do a length check after each one,\n> > +\t * so the worst case is bounded.\n> > +\t */\n> \n> Sorry about that :-).\n\n:) We may want to address that. I know we did some hardening about\nreading off the end of .pack and .idx files. But it seems like any user\nof decode_varint() may read up to 16 bytes past the end of a buffer.\n\nWe seem to only use them for the $GIT_DIR/index, though. Anybody with a\n\"struct hashfile\" result at least has a 20-byte trailer we can\naccidentally read from. But I wouldn't be surprised if there's a way to\ntrick it in practice.\n\n-Peff\n"},{"id":"360144","messageId":"87k1mpdjcr.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"cover.1539219248.git.jonathantanmy@google.com","subject":"Re: [PATCH 0/2] Per-commit filter proof of concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-11T07:37:40Z","receivedAt":"2018-10-11T07:37:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 11 2018, Jonathan Tan wrote:\n\n> Using per-commit filters and restricting the bloom filter to a single\n> parent increases the relative power of the filter in omitting tree\n> inspections compared to the original (107/53096 vs 1183/66459), but the\n> lack of coverage w.r.t. the non-first parents had a more significant\n> effect than I thought (1.29s vs .24s). It might be best to have one\n> filter for each (commit, parent) pair (or, at least, the first two\n> parents of each commit - we probably don't need to care that much about\n> octopus merges) - this would take up more disk space than if we only\n> store filters for the first parent, but is still less than the original\n> example of storing information for all commits in one filter.\n>\n> There are more possibilities like dynamic filter sizing, different\n> hashing, and hashing to support wildcard matches, which I haven't looked\n> into.\n\nAnother way to deal with that is to havet the filter store change since\nthe merge base, from an E-Mail of mine back in May[1] when this was\ndiscussed:\n\n    From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n    Date: Fri, 04 May 2018 22:36:07 +0200\n    Message-ID: <87h8nnxio8.fsf@evledraar.gmail.com> (raw)\n\n    On Fri, May 04 2018, Jakub Narebski wrote:\n\n    (Just off-the cuff here and I'm surely about to be corrected by\n    Derrick...)\n\n    > * What to do about merge commits, and octopus merges in particular?\n    >   Should Bloom filter be stored for each of the parents?  How to ensure\n    >   fast access then (fixed-width records) - use large edge list?\n\n    You could still store it fixed with, you'd just say that if you\n    encounter a merge with N parents the filter wouldn't store files changed\n    in that commit, but rather whether any of the N (including the merge)\n    had changes to files as of the their common merge-base.\n\n    Then if they did you'd need to walk all sides of the merge where each\n    commit would also have the filter to figure out where the change(s)\n    was/were, but if they didn't you could skip straight to the merge base\n    and keep walking.\n    [...]\n\nIdeas are cheap and I don't have any code to back that up, just thought\nI'd mention it if someone found it interesting.\n\nThinking about this again I wonder if something like that could be\ngeneralized more, i.e. in the abstract the idea is really whether we can\nstore a filter for N commits so we can skip across N in the walk as an\noptimization, doing this for merges is just an implementation detail.\n\nSo what if the bloom filters were this sort of structure:\n\n    <commit_the_filter_is_for> = [<bloom bitmap>, <next commit with filter>]\n\nSo e.g. given a history like (\"-> \" = parent relationship)\n\n    A -> B\n    B -> C\n    C -> D\n    E -> F\n\nWe could store:\n\n    A -> B [<bloom bitmap for A..D>, D]\n    B -> C\n    C -> D\n    D -> E [<bloom bitmap for D..F>, F]\n    E -> F\n    F -> G [<bloom bitmap for F..G>, G]\n\nNote how the bitmaps aren't evenly spaced. That's because some algorithm\nwould have walked the graph and e.g. decided that from A..D we had few\nenough changes that the bitmap should apply for 4 commits, and then 3\nfor the next set etc. Whether some range was worth extending could just\nbe a configurable implementation detail.\n\n1. https://public-inbox.org/git/87h8nnxio8.fsf@evledraar.gmail.com/\n"},{"id":"360179","messageId":"bcf5586e-d0b0-e5e2-0dff-fb4606ff91c4@gmail.com","threadId":"49479","inReplyTo":"20181009231250.GA19342@sigill.intra.peff.net","subject":"Re: Bloom Filters","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-11T12:33:58Z","receivedAt":"2018-10-11T12:34:04Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/9/2018 7:12 PM, Jeff King wrote:\n> On Tue, Oct 09, 2018 at 05:14:50PM -0400, Jeff King wrote:\n>\n>> Hmph. It really sounds like we could do better with a custom RLE\n>> solution. But that makes me feel like I'm missing something, because\n>> surely I can't invent something better than the state of the art in a\n>> simple thought experiment, right?\n>>\n>> I know what I'm proposing would be quite bad for random access, but my\n>> impression is that EWAH is the same. For the scale of bitmaps we're\n>> talking about, I think linear/streaming access through the bitmap would\n>> be OK.\n> Thinking on it more, what I was missing is that for truly dense random\n> bitmaps, this will perform much worse. Because it will use a byte to say\n> \"there's one 1\", rather than a bit.\n>\n> But I think it does OK in practice for the very sparse bitmaps we tend\n> to see in this application.  I was able to generate a complete output\n> that can reproduce \"log --name-status -t\" for linux.git in 32MB. But:\n>\n>    - 15MB of that is commit sha1s, which will be stored elsewhere in a\n>      \"real\" system\n>\n>    - 5MB of that is path list (which should shrink by a factor of 10 with\n>      prefix compression, and is really a function of a tree size less\n>      than history depth)\n>\n> So the per-commit cost is not too bad. That's still not counting merges,\n> though, which would add another 10-15% (or maybe more; their bitmaps are\n> less sparse).\n>\n> I don't know if this is a fruitful path at all or not. I was mostly just\n> satisfying my own curiosity on the bitmap encoding question. But I'll\n> post the patches, just to show my work. The first one is the same\n> initial proof of concept I showed earlier.\n>\n>    [1/3]: initial tree-bitmap proof of concept\n>    [2/3]: test-tree-bitmap: add \"dump\" mode\n>    [3/3]: test-tree-bitmap: replace ewah with custom rle encoding\n>\n>   Makefile                    |   1 +\n>   t/helper/test-tree-bitmap.c | 344 ++++++++++++++++++++++++++++++++++++\n>   2 files changed, 345 insertions(+)\n>   create mode 100644 t/helper/test-tree-bitmap.c\nI'm trying to test this out myself, and am having trouble reverse \nengineering how I'm supposed to test it.\n\nLooks like running \"t/helper/test-tree-bitmap gen\" will output a lot of \nbinary data. Where should I store that? Does any file work?\n\nIs this series just for the storage costs, assuming that we would \nreplace all TREESAME checks with a query into this database? Or do you \nhave a way to test how much this would improve a \"git log -- <path>\" query?\n\nThanks,\n-Stolee\n"},{"id":"360180","messageId":"2f49b953-4e07-0eb6-05b8-90d2eb72994b@gmail.com","threadId":"49479","inReplyTo":"ebf0a811be047e9fd24b61fea3c4a164b3d18dc0.1539219248.git.jonathantanmy@google.com","subject":"Re: [PATCH 1/2] One filter per commit","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-11T12:49:24Z","receivedAt":"2018-10-11T12:49:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/10/2018 9:21 PM, Jonathan Tan wrote:\n> diff --git a/commit-graph.c b/commit-graph.c\n> index f415d3b41f..90b0b3df90 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -715,13 +715,11 @@ static int add_ref_to_list(const char *refname,\n>   static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n>   \t\t\t\t\tstruct commit *parent,\n>   \t\t\t\t\tstruct commit *commit,\n> +\t\t\t\t\tint index,\n>   \t\t\t\t\tstruct diff_options *diffopt)\n>   {\n> -\tunsigned char p_c_hash[GIT_MAX_RAWSZ];\n>   \tint i;\n>   \n> -\thashxor(parent->object.oid.hash, commit->object.oid.hash, p_c_hash);\n> -\n>   \tdiff_tree_oid(&parent->object.oid, &commit->object.oid, \"\", diffopt);\n>   \tdiffcore_std(diffopt);\n>   \n> @@ -756,8 +754,8 @@ static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n>   \t\t\tthe_hash_algo->update_fn(&ctx, path, p - path);\n>   \t\t\tthe_hash_algo->final_fn(name_hash, &ctx);\n>   \n> -\t\t\thashxor(name_hash, p_c_hash, hash);\n> -\t\t\tbloom_filter_add_hash(bf, hash);\n> +\t\t\thashxor(name_hash, parent->object.oid.hash, hash);\n> +\t\t\tbloom_filter_add_hash(bf, index, hash);\n>   \t\t} while (*p);\n>   \n>   \t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n[snip]\n> @@ -768,11 +766,10 @@ static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n>   }\n>   \n>   static void fill_bloom_filter(struct bloom_filter *bf,\n> -\t\t\t\t    struct progress *progress)\n> +\t\t\t\t    struct progress *progress, struct commit **commits, int commit_nr)\n>   {\n>   \tstruct rev_info revs;\n>   \tconst char *revs_argv[] = {NULL, \"--all\", NULL};\n> -\tstruct commit *commit;\n>   \tint i = 0;\n>   \n>   \t/* We (re-)create the bloom filter from scratch every time for now. */\n> @@ -783,18 +780,19 @@ static void fill_bloom_filter(struct bloom_filter *bf,\n>   \tif (prepare_revision_walk(&revs))\n>   \t\tdie(\"revision walk setup failed while preparing bloom filter\");\n>   \n> -\twhile ((commit = get_revision(&revs))) {\n> +\tfor (i = 0; i < commit_nr; i++) {\n> +\t\tstruct commit *commit = commits[i];\n>   \t\tstruct commit_list *parent;\n>   \n>   \t\tfor (parent = commit->parents; parent; parent = parent->next)\n> -\t\t\tadd_changes_to_bloom_filter(bf, parent->item, commit,\n> +\t\t\tadd_changes_to_bloom_filter(bf, parent->item, commit, i,\n>   \t\t\t\t\t\t    &revs.diffopt);\n>   \n[snip]\n>   \n> -\t\thashxor(pi->name_hash, p_c_hash, hash);\n> -\t\tif (bloom_filter_check_hash(&bf, hash)) {\n> +\t\thashxor(pi->name_hash, parent->object.oid.hash, hash);\n> +\t\tif (bloom_filter_check_hash(&bf, commit->graph_pos, hash)) {\n>   \t\t\t/*\n>   \t\t\t * At least one of the interesting pathspecs differs,\n>   \t\t\t * so we can return early and let the diff machinery\nOne main benefit of storing on Bloom filter per commit is to avoid \nrecomputing hashes at every commit. Currently, this patch only improves \nlocality when checking membership at the cost of taking up more space. \nDrop the dependence on the parent oid and then we can save the time \nspent hashing during history queries.\n\n-Stolee\n"},{"id":"360181","messageId":"20181011134359.GA26956@sigill.intra.peff.net","threadId":"49479","inReplyTo":"bcf5586e-d0b0-e5e2-0dff-fb4606ff91c4@gmail.com","subject":"Re: Bloom Filters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-11T13:43:59Z","receivedAt":"2018-10-11T13:44:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 11, 2018 at 08:33:58AM -0400, Derrick Stolee wrote:\n\n> > I don't know if this is a fruitful path at all or not. I was mostly just\n> > satisfying my own curiosity on the bitmap encoding question. But I'll\n> > post the patches, just to show my work. The first one is the same\n> > initial proof of concept I showed earlier.\n> > \n> >    [1/3]: initial tree-bitmap proof of concept\n> >    [2/3]: test-tree-bitmap: add \"dump\" mode\n> >    [3/3]: test-tree-bitmap: replace ewah with custom rle encoding\n> > \n> >   Makefile                    |   1 +\n> >   t/helper/test-tree-bitmap.c | 344 ++++++++++++++++++++++++++++++++++++\n> >   2 files changed, 345 insertions(+)\n> >   create mode 100644 t/helper/test-tree-bitmap.c\n> I'm trying to test this out myself, and am having trouble reverse\n> engineering how I'm supposed to test it.\n> \n> Looks like running \"t/helper/test-tree-bitmap gen\" will output a lot of\n> binary data. Where should I store that? Does any file work?\n\nYeah, you can do:\n\n  # optionally run with GIT_TRACE=1 to see some per-bitmap stats\n  test-tree-bitmap gen >out\n\n  # this should be roughly the same as:\n  #  git rev-list --all |\n  #  git diff-tree --stdin -t --name-only\n  test-tree-bitmap dump <out\n\n> Is this series just for the storage costs, assuming that we would replace\n> all TREESAME checks with a query into this database? Or do you have a way to\n> test how much this would improve a \"git log -- <path>\" query?\n\nRight, I was just looking at storage cost here. It's not integrated with\nthe diff code at all. I left some hypothetical numbers elsewhere in the\nthread.\n\n-Peff\n"},{"id":"360201","messageId":"20181011191131.154425-1-jonathantanmy@google.com","threadId":"49479","inReplyTo":"2f49b953-4e07-0eb6-05b8-90d2eb72994b@gmail.com","subject":"[PATCH] Per-commit and per-parent filters for 2 parents","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-10-11T19:11:31Z","receivedAt":"2018-10-11T19:11:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n> One main benefit of storing on Bloom filter per commit is to avoid\n> recomputing hashes at every commit. Currently, this patch only improves\n> locality when checking membership at the cost of taking up more space.\n> Drop the dependence on the parent oid and then we can save the time\n> spent hashing during history queries.\n\nI've removed the hashing of the parent OID here and tried having\nper-parent and per-commit hashes for the first 2 parents of each commit\ninstead of only 1, thus doubling the filter size. The results are not\nmuch of an improvement though:\n\nbloom filter total queries: 66409 definitely not: 56424 maybe: 9985 false positives: 9099 fp ratio: 0.137015\n0:01.17\n---\n bloom-filter.c | 25 ++++++++++++-------------\n bloom-filter.h |  4 ++--\n commit-graph.c | 13 ++++++++-----\n revision.c     | 11 +++++------\n 4 files changed, 27 insertions(+), 26 deletions(-)\n\ndiff --git a/bloom-filter.c b/bloom-filter.c\nindex 39b453908f..10c73c45ae 100644\n--- a/bloom-filter.c\n+++ b/bloom-filter.c\n@@ -11,7 +11,7 @@ void bloom_filter_init(struct bloom_filter *bf, uint32_t commit_nr, uint32_t bit\n \tbf->nr_entries = 0;\n \tbf->commit_nr = commit_nr;\n \tbf->bit_size = bit_size;\n-\tbf->bits = xcalloc(1, commit_nr * bit_size / CHAR_BIT);\n+\tbf->bits = xcalloc(1, 2 * commit_nr * bit_size / CHAR_BIT);\n }\n \n void bloom_filter_free(struct bloom_filter *bf)\n@@ -22,24 +22,24 @@ void bloom_filter_free(struct bloom_filter *bf)\n }\n \n \n-static void bloom_filter_set_bits(struct bloom_filter *bf, uint32_t graph_pos, const uint32_t *offsets,\n+static void bloom_filter_set_bits(struct bloom_filter *bf, uint32_t graph_pos, int parent_index, const uint32_t *offsets,\n \t\t\t   int nr_offsets, int nr_entries)\n {\n \tint i;\n \tfor (i = 0; i < nr_offsets; i++) {\n-\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size + graph_pos * bf->bit_size) / CHAR_BIT;\n+\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size + (2 * graph_pos + parent_index) * bf->bit_size) / CHAR_BIT;\n \t\tunsigned char mask = 1 << offsets[i] % CHAR_BIT;\n \t\tbf->bits[byte_offset] |= mask;\n \t}\n \tbf->nr_entries += nr_entries;\n }\n \n-static int bloom_filter_check_bits(struct bloom_filter *bf, uint32_t graph_pos, const uint32_t *offsets,\n+static int bloom_filter_check_bits(struct bloom_filter *bf, uint32_t graph_pos, int parent_index, const uint32_t *offsets,\n \t\t\t    int nr)\n {\n \tint i;\n \tfor (i = 0; i < nr; i++) {\n-\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size + graph_pos * bf->bit_size) / CHAR_BIT;\n+\t\tuint32_t byte_offset = (offsets[i] % bf->bit_size + (2 * graph_pos + parent_index) * bf->bit_size) / CHAR_BIT;\n \t\tunsigned char mask = 1 << offsets[i] % CHAR_BIT;\n \t\tif (!(bf->bits[byte_offset] & mask))\n \t\t\treturn 0;\n@@ -48,19 +48,18 @@ static int bloom_filter_check_bits(struct bloom_filter *bf, uint32_t graph_pos,\n }\n \n \n-void bloom_filter_add_hash(struct bloom_filter *bf, uint32_t graph_pos, const unsigned char *hash)\n+void bloom_filter_add_hash(struct bloom_filter *bf, uint32_t graph_pos, int parent_index, const unsigned char *hash)\n {\n \tuint32_t offsets[GIT_MAX_RAWSZ / sizeof(uint32_t)];\n \thashcpy((unsigned char*)offsets, hash);\n-\tbloom_filter_set_bits(bf, graph_pos, offsets,\n-\t\t\t     the_hash_algo->rawsz / sizeof(*offsets), 1);\n+\tbloom_filter_set_bits(bf, graph_pos, parent_index, offsets, the_hash_algo->rawsz / sizeof(*offsets), 1);\n }\n \n-int bloom_filter_check_hash(struct bloom_filter *bf, uint32_t graph_pos, const unsigned char *hash)\n+int bloom_filter_check_hash(struct bloom_filter *bf, uint32_t graph_pos, int parent_index, const unsigned char *hash)\n {\n \tuint32_t offsets[GIT_MAX_RAWSZ / sizeof(uint32_t)];\n \thashcpy((unsigned char*)offsets, hash);\n-\treturn bloom_filter_check_bits(bf, graph_pos, offsets,\n+\treturn bloom_filter_check_bits(bf, graph_pos, parent_index, offsets,\n \t\t\tthe_hash_algo->rawsz / sizeof(*offsets));\n }\n \n@@ -87,8 +86,8 @@ int bloom_filter_load(struct bloom_filter *bf)\n \tread_in_full(fd, &bf->bit_size, sizeof(bf->bit_size));\n \tif (bf->bit_size % CHAR_BIT)\n \t\tBUG(\"invalid size for bloom filter\");\n-\tbf->bits = xmalloc(bf->commit_nr * bf->bit_size / CHAR_BIT);\n-\tread_in_full(fd, bf->bits, bf->commit_nr * bf->bit_size / CHAR_BIT);\n+\tbf->bits = xmalloc(2 * bf->commit_nr * bf->bit_size / CHAR_BIT);\n+\tread_in_full(fd, bf->bits, 2 * bf->commit_nr * bf->bit_size / CHAR_BIT);\n \n \tclose(fd);\n \n@@ -102,7 +101,7 @@ void bloom_filter_write(struct bloom_filter *bf)\n \twrite_in_full(fd, &bf->nr_entries, sizeof(bf->nr_entries));\n \twrite_in_full(fd, &bf->commit_nr, sizeof(bf->commit_nr));\n \twrite_in_full(fd, &bf->bit_size, sizeof(bf->bit_size));\n-\twrite_in_full(fd, bf->bits, bf->commit_nr * bf->bit_size / CHAR_BIT);\n+\twrite_in_full(fd, bf->bits, 2 * bf->commit_nr * bf->bit_size / CHAR_BIT);\n \n \tclose(fd);\n }\ndiff --git a/bloom-filter.h b/bloom-filter.h\nindex 607649b8db..20e0527451 100644\n--- a/bloom-filter.h\n+++ b/bloom-filter.h\n@@ -18,13 +18,13 @@ void bloom_filter_free(struct bloom_filter *bf);\n  * Turns the given (SHA1) hash into 5 unsigned ints, and sets the bits at\n  * those positions (modulo the bitmap's size) in the Bloom filter.\n  */\n-void bloom_filter_add_hash(struct bloom_filter *bf, uint32_t graph_pos, const unsigned char *hash);\n+void bloom_filter_add_hash(struct bloom_filter *bf, uint32_t graph_pos, int parent_index, const unsigned char *hash);\n /*\n  * Turns the given (SHA1) hash into 5 unsigned ints, and checks the bits at\n  * those positions (modulo the bitmap's size) in the Bloom filter.\n  * Returns 1 if all those bits are set, 0 otherwise.\n  */\n-int bloom_filter_check_hash(struct bloom_filter *bf, uint32_t graph_pos, const unsigned char *hash);\n+int bloom_filter_check_hash(struct bloom_filter *bf, uint32_t graph_pos, int parent_index, const unsigned char *hash);\n \n void hashxor(const unsigned char *hash1, const unsigned char *hash2,\n \t     unsigned char *out);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d21d555611..ca869c10e1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -716,6 +716,7 @@ static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n \t\t\t\t\tstruct commit *parent,\n \t\t\t\t\tstruct commit *commit,\n \t\t\t\t\tint index,\n+\t\t\t\t\tint parent_index,\n \t\t\t\t\tstruct diff_options *diffopt)\n {\n \tint i;\n@@ -738,7 +739,6 @@ static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n \t\tdo {\n \t\t\tgit_hash_ctx ctx;\n \t\t\tunsigned char name_hash[GIT_MAX_RAWSZ];\n-\t\t\tunsigned char hash[GIT_MAX_RAWSZ];\n \n \t\t\tp = strchrnul(p + 1, '/');\n \n@@ -754,8 +754,7 @@ static void add_changes_to_bloom_filter(struct bloom_filter *bf,\n \t\t\tthe_hash_algo->update_fn(&ctx, path, p - path);\n \t\t\tthe_hash_algo->final_fn(name_hash, &ctx);\n \n-\t\t\thashxor(name_hash, parent->object.oid.hash, hash);\n-\t\t\tbloom_filter_add_hash(bf, index, hash);\n+\t\t\tbloom_filter_add_hash(bf, index, parent_index, name_hash);\n \t\t} while (*p);\n \n \t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n@@ -784,9 +783,13 @@ static void fill_bloom_filter(struct bloom_filter *bf,\n \t\tstruct commit *commit = commits[i];\n \t\tstruct commit_list *parent = commit->parents;\n \n-\t\tif (parent)\n-\t\t\tadd_changes_to_bloom_filter(bf, parent->item, commit, i,\n+\t\tif (parent) {\n+\t\t\tadd_changes_to_bloom_filter(bf, parent->item, commit, i, 0,\n \t\t\t\t\t\t    &revs.diffopt);\n+\t\t\tif (parent->next)\n+\t\t\t\tadd_changes_to_bloom_filter(bf, parent->next->item, commit, i, 1,\n+\t\t\t\t\t\t\t    &revs.diffopt);\n+\t\t}\n \n \t\tdisplay_progress(progress, i);\n \t}\ndiff --git a/revision.c b/revision.c\nindex 5a433a5878..5478d08344 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -488,6 +488,7 @@ static void print_bloom_filter_stats_atexit(void)\n \n static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\t\t\t\t\t struct commit *parent,\n+\t\t\t\t\t\t int parent_index,\n \t\t\t\t\t\t struct commit *commit)\n {\n \tint i;\n@@ -513,10 +514,8 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \n \tfor (i = 0; i < revs->pruning.pathspec.nr; i++) {\n \t\tstruct pathspec_item *pi = &revs->pruning.pathspec.items[i];\n-\t\tunsigned char hash[GIT_MAX_RAWSZ];\n \n-\t\thashxor(pi->name_hash, parent->object.oid.hash, hash);\n-\t\tif (bloom_filter_check_hash(&bf, commit->graph_pos, hash)) {\n+\t\tif (bloom_filter_check_hash(&bf, commit->graph_pos, parent_index, pi->name_hash)) {\n \t\t\t/*\n \t\t\t * At least one of the interesting pathspecs differs,\n \t\t\t * so we can return early and let the diff machinery\n@@ -568,8 +567,8 @@ static int rev_compare_tree(struct rev_info *revs,\n \t\t\treturn REV_TREE_SAME;\n \t}\n \n-\tif (!nth_parent) {\n-\t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, parent, commit);\n+\tif (nth_parent <= 1) {\n+\t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, parent, nth_parent, commit);\n \t\tif (bloom_ret == 0)\n \t\t\treturn REV_TREE_SAME;\n \t}\n@@ -579,7 +578,7 @@ static int rev_compare_tree(struct rev_info *revs,\n \tif (diff_tree_oid(&t1->object.oid, &t2->object.oid, \"\",\n \t\t\t   &revs->pruning) < 0)\n \t\treturn REV_TREE_DIFFERENT;\n-\tif (!nth_parent) {\n+\tif (nth_parent <= 1) {\n \t\tif (bloom_ret == 1 && tree_difference == REV_TREE_SAME)\n \t\t\tbloom_filter_count_false_positive++;\n \t}\n-- \n2.19.0.271.gfe8321ec05.dirty\n\n"},{"id":"360527","messageId":"61559c5b-546e-d61b-d2e1-68de692f5972@gmail.com","threadId":"49479","inReplyTo":"20181009193445.21908-1-szeder.dev@gmail.com","subject":"Re: [PATCH 0/4] Bloom filter experiment","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-15T14:39:45Z","receivedAt":"2018-10-15T14:39:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/9/2018 3:34 PM, SZEDER Gábor wrote:\n> To keep the ball rolling, here is my proof of concept in a somewhat\n> cleaned-up form, with still plenty of rough edges.\n\nPeff, Szeder, and Jonathan,\n\nThanks for giving me the kick in the pants to finally write a proof of \nconcept for my personal take on how this should work. My implementation \nborrows things from both Szeder and Jonathan's series. You can find my \ncommits for all of the versions on GitHub (it's a bit too messy to share \nas a patch series right now, I think):\n\nRepo: https://github.com/derrickstolee/git\nBranches: bloom/* (includes bloom/stolee, bloom/peff, bloom/szeder, and \nbloom/tan for the respective implementations, and bloom/base as the \ncommon ancestor)\n\nMy implementation uses the following scheme:\n\n1. Bloom filters are computed and stored on a commit-by-commit basis.\n\n2. The filters are sized according to the number of changes in each \ncommit, with a minimum of one 64-bit word.\n\n3. The filters are stored in the commit-graph using two new optional \nchunks: one stores a single 32-bit integer for each commit that provides \nthe end of its Bloom filter in the second \"data\" chunk. The data chunk \nalso stores the magic constants (hash version, num hash keys, and num \nbits per entry).\n\n4. We fill the Bloom filters as (const char *data, int len) pairs as \n\"struct bloom_filter\"s in a commit slab.\n\n5. In order to evaluate containment, we need the struct bloom_filter, \nbut also struct bloom_settings (stores the magic constants in one \nplace), and struct bloom_key (stores the _k_ hash values). This allows \nus to hash a path once and test the same path against many Bloom filters.\n\n6. When we compute the Bloom filters, we don't store a filter for \ncommits whose first-parent diff has more than 512 paths.\n\n7. When we compute the commit-graph, we can re-use the pre-existing \nfilters without needing to recompute diffs. (Caveat: the current \nimplementation will re-compute diffs for the commits with diffs that \nwere too large.)\n\nYou can build the Bloom filters in my implementation this way:\n\nGIT_TEST_BLOOM_FILTERS=1 ./git commit-graph write --reachable\n\n> You can play around with it like this:\n>\n>    $ GIT_USE_POC_BLOOM_FILTER=$((8*1024*1024*8)) git commit-graph write\n>    Computing commit graph generation numbers: 100% (52801/52801), done.\n>    Computing bloom filter: 100% (52801/52801), done.\n>    # Yeah, I even added progress indicator! :)\n>    $ GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y git rev-list --count --full-history HEAD -- t/valgrind/valgrind.sh\n>    886\n>    20:40:24.783699 revision.c:486          bloom filter total queries: 66095 definitely not: 64953 maybe: 1142 false positives: 256 fp ratio: 0.003873\n\nJonathan used this same test, so will I. Here is a summary table:\n\n| Implementation | Queries | Maybe | FP # | FP %  |\n|----------------|---------|-------|------|-------|\n| Szeder         | 66095   | 1142  | 256  | 0.38% |\n| Jonathan       | 66459   | 107   | 89   | 0.16% |\n| Stolee         | 53025   | 492   | 479  | 0.90% |\n\n(Note that we must have used different starting points, which is why my \n\"Queries\" is so much smaller.)\n\nThe increase in false-positive percentage is expected in my \nimplementation. I'm using the correct filter sizes to hit a <1% FP \nratio. This could be lowered by changing the settings, and the size \nwould dynamically grow. For my Git repo (which contains \ngit-for-windows/git and microsoft/git) this implementation grows the \ncommmit-graph file from 5.8 MB to 7.3 MB (1.5 MB total, compared to \nSzeder's 8MB filter). For 105,260 commits, that rounds out to less than \n20 bytes per commit (compared to Jonathan's 256 bytes per commit).\n\nRelated stats for my Linux repo: 781,756 commits, commit-graph grows \nfrom 43.8 to 55.6 MB (~12 MB additional, ~16 bytes per commit).\n\nI haven't done a side-by-side performance test for these \nimplementations, but it would be interesting to do so.\n\nDespite writing a lot of code in a short amount of time, there is a lot \nof work to be done before this is submittable:\n\n1. There are three different environment variables right now. It would \nbe better to have one GIT_TEST_ variable and rely on existing tracing \nfor logs (trace2 values would be good here).\n\n2. We need config values for writing and consuming bloom filters, but \nalso to override the default settings.\n\n3. My bloom.c/bloom.h is too coupled to the commit-graph. I want to \nharden that interface to let Bloom filters live as their own thing, but \nthat the commit-graph could load a bloom filter from the file instead of \nfrom the slab.\n\n4. Tests, tests, and more tests.\n\nWe'll see how much time I have to do this polish, but I think the \nbenefit is proven.\n\nThanks,\n-Stolee\n"},{"id":"360577","messageId":"xmqqlg6y1ovh.fsf@gitster-ct.c.googlers.com","threadId":"49479","inReplyTo":"61559c5b-546e-d61b-d2e1-68de692f5972@gmail.com","subject":"Re: [PATCH 0/4] Bloom filter experiment","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-16T04:45:06Z","receivedAt":"2018-10-16T04:45:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n> 2. The filters are sized according to the number of changes in each\n> commit, with a minimum of one 64-bit word.\n> ...\n> 6. When we compute the Bloom filters, we don't store a filter for\n> commits whose first-parent diff has more than 512 paths.\n\nJust being curious but was 512 taken out of thin air or is there\nsome math behind it, e.g. to limit false positive rate down to\ncertain threshold?  With a wide-enough bitset, you could store\narbitrary large number of paths with low enough false positive, I\nguess, but is there a point where there is too many paths in the\nchange that gives us diminishing returns and not worth having a\nfilter in the first place?\n\nIn a normal source-code-control context, the set of paths modified\nby any single commit ought to be a small subset of the entire paths,\nand whole-tree changes ought to be fairly rare.  In a project for\nwhich that assumption does not hold, it might help to have a\nnegative bloom filter (i.e. \"git log -- A\" asks \"does the commit\nmodify A?\" and the filter would say \"we know it does not, because we\nthrew all the paths that are not touched to the bloom filter\"), but\nI think that would optimize for a wrong case.\n\n"},{"id":"360609","messageId":"8c66cbe3-6830-05cb-f3bb-be2e4902e8f5@gmail.com","threadId":"49479","inReplyTo":"xmqqlg6y1ovh.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 0/4] Bloom filter experiment","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-16T11:13:03Z","receivedAt":"2018-10-16T11:13:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/16/2018 12:45 AM, Junio C Hamano wrote:\n> Derrick Stolee <stolee@gmail.com> writes:\n>\n>> 2. The filters are sized according to the number of changes in each\n>> commit, with a minimum of one 64-bit word.\n>> ...\n>> 6. When we compute the Bloom filters, we don't store a filter for\n>> commits whose first-parent diff has more than 512 paths.\n> Just being curious but was 512 taken out of thin air or is there\n> some math behind it, e.g. to limit false positive rate down to\n> certain threshold?  With a wide-enough bitset, you could store\n> arbitrary large number of paths with low enough false positive, I\n> guess, but is there a point where there is too many paths in the\n> change that gives us diminishing returns and not worth having a\n> filter in the first place?\n512 is somewhat arbitrary, but having a maximum size is not.\n> In a normal source-code-control context, the set of paths modified\n> by any single commit ought to be a small subset of the entire paths,\n> and whole-tree changes ought to be fairly rare.  In a project for\n> which that assumption does not hold, it might help to have a\n> negative bloom filter (i.e. \"git log -- A\" asks \"does the commit\n> modify A?\" and the filter would say \"we know it does not, because we\n> threw all the paths that are not touched to the bloom filter\"), but\n> I think that would optimize for a wrong case.\n\nA commit with many changed paths is very rare. The 512 I picked above is \nenough to cover 99% of commits in each of the repos I sampled when first \ninvestigating Bloom filters.\n\nWhen a Bloom filter response says \"maybe yes\" (in our case, \"maybe not \nTREESAME\"), then we need to verify that it is correct. In the extreme \ncase that every path is changed, then the Bloom filter does nothing but \nadd extra work.\n\nThese extreme cases are also not unprecedented: in our Azure Repos \ncodebase, we were using core.autocrlf  to smudge CRLFs to LFs, but when \nit was time to dogfood VFS for Git, we needed to turn off the smudge \nfilter. So, there is one commit that converts every LF to a CRLF in \nevery text file. Storing a Bloom filter for those ~250,000 entries would \ntake ~256KB for essentially no value. By not storing a filter for this \ncommit, we go immediately to the regular TREESAME check, which would \nhappen for most pathspecs.\n\nThis is all to say: having a maximum size is good. 512 is big enough to \ncover _most_ commits, but not so big that we may store _really_ big filters.\n\nThanks,\n-Stolee\n\n"},{"id":"360619","messageId":"87zhvecamd.fsf@evledraar.gmail.com","threadId":"49479","inReplyTo":"8c66cbe3-6830-05cb-f3bb-be2e4902e8f5@gmail.com","subject":"Re: [PATCH 0/4] Bloom filter experiment","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-16T12:57:30Z","receivedAt":"2018-10-16T12:57:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Oct 16 2018, Derrick Stolee wrote:\n\n> On 10/16/2018 12:45 AM, Junio C Hamano wrote:\n>> Derrick Stolee <stolee@gmail.com> writes:\n>>\n>>> 2. The filters are sized according to the number of changes in each\n>>> commit, with a minimum of one 64-bit word.\n>>> ...\n>>> 6. When we compute the Bloom filters, we don't store a filter for\n>>> commits whose first-parent diff has more than 512 paths.\n>> Just being curious but was 512 taken out of thin air or is there\n>> some math behind it, e.g. to limit false positive rate down to\n>> certain threshold?  With a wide-enough bitset, you could store\n>> arbitrary large number of paths with low enough false positive, I\n>> guess, but is there a point where there is too many paths in the\n>> change that gives us diminishing returns and not worth having a\n>> filter in the first place?\n> 512 is somewhat arbitrary, but having a maximum size is not.\n>> In a normal source-code-control context, the set of paths modified\n>> by any single commit ought to be a small subset of the entire paths,\n>> and whole-tree changes ought to be fairly rare.  In a project for\n>> which that assumption does not hold, it might help to have a\n>> negative bloom filter (i.e. \"git log -- A\" asks \"does the commit\n>> modify A?\" and the filter would say \"we know it does not, because we\n>> threw all the paths that are not touched to the bloom filter\"), but\n>> I think that would optimize for a wrong case.\n>\n> A commit with many changed paths is very rare. The 512 I picked above\n> is enough to cover 99% of commits in each of the repos I sampled when\n> first investigating Bloom filters.\n>\n> When a Bloom filter response says \"maybe yes\" (in our case, \"maybe not\n> TREESAME\"), then we need to verify that it is correct. In the extreme\n> case that every path is changed, then the Bloom filter does nothing\n> but add extra work.\n>\n> These extreme cases are also not unprecedented: in our Azure Repos\n> codebase, we were using core.autocrlf to smudge CRLFs to LFs, but\n> when it was time to dogfood VFS for Git, we needed to turn off the\n> smudge filter. So, there is one commit that converts every LF to a\n> CRLF in every text file. Storing a Bloom filter for those ~250,000\n> entries would take ~256KB for essentially no value. By not storing a\n> filter for this commit, we go immediately to the regular TREESAME\n> check, which would happen for most pathspecs.\n>\n> This is all to say: having a maximum size is good. 512 is big enough\n> to cover _most_ commits, but not so big that we may store _really_ big\n> filters.\n\nMakes sense. 512 is good enough to hardcode initially, but I couldn't\ntell from briefly skimming the patches if it was possible to make this\nsize dynamic per-repo when the graph/filter is written.\n\nI.e. we might later add some discovery step where we look at N number of\ncommits at random, until we're satisfied that we've come up with some\naverage/median number of total (recursive) tree entries & how many tend\nto be changed per-commit.\n\nI.e. I can imagine repositories (with some automated changes) where we\nhave 10k files and tend to change 1k per commit, or ones with 10k files\nwhere we tend to change just 1-10 per commit, which would mean a\nlarger/smaller filter would be needed / would do.\n"},{"id":"360621","messageId":"ec496cf6-1d54-e9af-6269-65350ab35a47@gmail.com","threadId":"49479","inReplyTo":"87zhvecamd.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/4] Bloom filter experiment","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-16T13:03:14Z","receivedAt":"2018-10-16T13:03:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/16/2018 8:57 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Tue, Oct 16 2018, Derrick Stolee wrote:\n>\n>> On 10/16/2018 12:45 AM, Junio C Hamano wrote:\n>>> Derrick Stolee <stolee@gmail.com> writes:\n>>>\n>>>> 2. The filters are sized according to the number of changes in each\n>>>> commit, with a minimum of one 64-bit word.\n>>>> ...\n>>>> 6. When we compute the Bloom filters, we don't store a filter for\n>>>> commits whose first-parent diff has more than 512 paths.\n>>> Just being curious but was 512 taken out of thin air or is there\n>>> some math behind it, e.g. to limit false positive rate down to\n>>> certain threshold?  With a wide-enough bitset, you could store\n>>> arbitrary large number of paths with low enough false positive, I\n>>> guess, but is there a point where there is too many paths in the\n>>> change that gives us diminishing returns and not worth having a\n>>> filter in the first place?\n>> 512 is somewhat arbitrary, but having a maximum size is not.\n>>> In a normal source-code-control context, the set of paths modified\n>>> by any single commit ought to be a small subset of the entire paths,\n>>> and whole-tree changes ought to be fairly rare.  In a project for\n>>> which that assumption does not hold, it might help to have a\n>>> negative bloom filter (i.e. \"git log -- A\" asks \"does the commit\n>>> modify A?\" and the filter would say \"we know it does not, because we\n>>> threw all the paths that are not touched to the bloom filter\"), but\n>>> I think that would optimize for a wrong case.\n>> A commit with many changed paths is very rare. The 512 I picked above\n>> is enough to cover 99% of commits in each of the repos I sampled when\n>> first investigating Bloom filters.\n>>\n>> When a Bloom filter response says \"maybe yes\" (in our case, \"maybe not\n>> TREESAME\"), then we need to verify that it is correct. In the extreme\n>> case that every path is changed, then the Bloom filter does nothing\n>> but add extra work.\n>>\n>> These extreme cases are also not unprecedented: in our Azure Repos\n>> codebase, we were using core.autocrlf to smudge CRLFs to LFs, but\n>> when it was time to dogfood VFS for Git, we needed to turn off the\n>> smudge filter. So, there is one commit that converts every LF to a\n>> CRLF in every text file. Storing a Bloom filter for those ~250,000\n>> entries would take ~256KB for essentially no value. By not storing a\n>> filter for this commit, we go immediately to the regular TREESAME\n>> check, which would happen for most pathspecs.\n>>\n>> This is all to say: having a maximum size is good. 512 is big enough\n>> to cover _most_ commits, but not so big that we may store _really_ big\n>> filters.\n> Makes sense. 512 is good enough to hardcode initially, but I couldn't\n> tell from briefly skimming the patches if it was possible to make this\n> size dynamic per-repo when the graph/filter is written.\nMy proof-of-concept has it as a constant, but part of my plan is to make \nthese all config options, as of this item in my message:\n\n >>> 2. We need config values for writing and consuming bloom filters, \nbut also to override the default settings.\n\n> I.e. we might later add some discovery step where we look at N number of\n> commits at random, until we're satisfied that we've come up with some\n> average/median number of total (recursive) tree entries & how many tend\n> to be changed per-commit.\n>\n> I.e. I can imagine repositories (with some automated changes) where we\n> have 10k files and tend to change 1k per commit, or ones with 10k files\n> where we tend to change just 1-10 per commit, which would mean a\n> larger/smaller filter would be needed / would do.\nI'm not sure a dynamic approach would be worth the effort, but I'm open \nto hearing the results of an experiment.\n\nThanks,\n-Stolee\n"},{"id":"360719","messageId":"20181016234129.194138-1-jonathantanmy@google.com","threadId":"49479","inReplyTo":"61559c5b-546e-d61b-d2e1-68de692f5972@gmail.com","subject":"Re: [PATCH 0/4] Bloom filter experiment","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-10-16T23:41:29Z","receivedAt":"2018-10-16T23:41:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> | Implementation | Queries | Maybe | FP # | FP %  |\n> |----------------|---------|-------|------|-------|\n> | Szeder         | 66095   | 1142  | 256  | 0.38% |\n> | Jonathan       | 66459   | 107   | 89   | 0.16% |\n> | Stolee         | 53025   | 492   | 479  | 0.90% |\n> \n> (Note that we must have used different starting points, which is why my \n> \"Queries\" is so much smaller.)\n\nI suspect it's because your bloom filter implementation covers only the\nfirst parent (if I'm understanding get_bloom_filter() correctly). When I\nonly covered the first parent in my initial test (see patch 2 of [1]), I\ngot (following the columns in the table above):\n\n  53096 107 89 0.001676\n\nAlso, I think that the rejecting power (Queries - Maybe)/(Total tree\ncomparisons if no bloom filters were used) needs to be in the evaluation\ncriteria somewhere, as that indicates how many tree comparisons we\nmanaged to avoid.\n\nAlso, we probably should also test on a file that changes more\nfrequently :-)\n\n[1] https://public-inbox.org/git/cover.1539219248.git.jonathantanmy@google.com/\n\n> The increase in false-positive percentage is expected in my \n> implementation. I'm using the correct filter sizes to hit a <1% FP \n> ratio. This could be lowered by changing the settings, and the size \n> would dynamically grow. For my Git repo (which contains \n> git-for-windows/git and microsoft/git) this implementation grows the \n> commmit-graph file from 5.8 MB to 7.3 MB (1.5 MB total, compared to \n> Szeder's 8MB filter). For 105,260 commits, that rounds out to less than \n> 20 bytes per commit (compared to Jonathan's 256 bytes per commit).\n\nMine has 256 bits per commit, which is 32 bytes per commit (still more\nthan yours).\n\nHaving said all that, thanks for writing up your version - in\nparticular, variable sized filters (like in yours) seem to be the way to\ngo.\n\n> We'll see how much time I have to do this polish, but I think the \n> benefit is proven.\n\nAgreed.\n"},{"id":"360803","messageId":"xmqqsh14vws5.fsf@gitster-ct.c.googlers.com","threadId":"49479","inReplyTo":"87zhvecamd.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/4] Bloom filter experiment","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-18T02:00:42Z","receivedAt":"2018-10-18T02:01:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n>> This is all to say: having a maximum size is good. 512 is big enough\n>> to cover _most_ commits, but not so big that we may store _really_ big\n>> filters.\n>\n> Makes sense. 512 is good enough to hardcode initially, but I couldn't\n> tell from briefly skimming the patches if it was possible to make this\n> size dynamic per-repo when the graph/filter is written.\n>\n> I.e. we might later add some discovery step where we look at N number of\n> commits at random, until we're satisfied that we've come up with some\n> average/median number of total (recursive) tree entries & how many tend\n> to be changed per-commit.\n>\n> I.e. I can imagine repositories (with some automated changes) where we\n> have 10k files and tend to change 1k per commit, or ones with 10k files\n> where we tend to change just 1-10 per commit, which would mean a\n> larger/smaller filter would be needed / would do.\n\nI was more interested to find out what the advice our docs should\ngive to the end users to tune the value once such a knob is\ninvented, and what you gave in the above paragraphs may lead us to a\nnice auto-tuning heuristics.\n\n"}]}