{"thread":{"id":"45819","subject":"30min Script in git 2.7.4 takes 22+ hrs in git 2.9.3","startedAt":"2017-04-27T16:37:02Z","lastAt":"2017-04-27T20:42:27Z","messageCount":3,"participants":["Robert Stryker","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"318071","messageId":"CA+Up40iusByn-R55=2=2Ae8KH1mkj4hGF_E9dX3vn1vboyMwMw@mail.gmail.com","threadId":"45819","inReplyTo":null,"subject":"30min Script in git 2.7.4 takes 22+ hrs in git 2.9.3","fromName":"Robert Stryker","fromEmail":"rstryker@redhat.com","sentAt":"2017-04-27T16:36:54Z","receivedAt":"2017-04-27T16:37:02Z","isPatch":false,"sender":{"key":"rstryker@redhat.com","avatar":null},"body":"Hi all:\n\nThe following script attempts to merge 4 git repos into one,\nmaintaining tag and branch content (but not SHAs). Each original repo\nbasically gets its own subfolder in the new one. Original repos are\nfirst rewritten to have their history think they always belonged in\nthe target subfolder.\n\nThe problem:  the script takes 30 minutes for one environment\nincluding git 2.7.4, and generates a repo of about 30mb.   When run by\na coworker using git 2.9.3, it takes 22+ hours and generates a 10gb\nrepo.\n\nClearly something here is very wrong. Either there's a pretty horrible\nregression or my idea is a pretty bad one ;)\n\nGeneral process for the script:\n  - check out 4 repos\n  - rewrite their history so they always thought they were in a subfolder\n  - copy these 4 rewritten folders to a temporary location\n  - get a list of branches and tags for each of the 4 repos\n  - initialize a new repo with a readme.md\n  - for each unique tag\n       - check the 4 rewritten / backed up repos for the tag\n       - for each of the 4 rewritten repos:\n            - if the tag exists in that repo, merge it into the new\nrepo in a test branch\n           -  git pull --no-edit ../intermediate/oneRewrittenRepo    (SLOW PART)\n        - save the tag\n   - for each unique branch (same logic)\n\nSo... yeah... 30mb + 30 minutes -> 11gb + 22 hours somewhere between\nthese two versions of git?\n\nAccording to coworker:\n\nduring each pass of the Tags' loop it's sitting for a long time on:\n\n git pull --no-edit ../intermediate/webtools.common\n\nwhich runs in its turn\n\ngit fetch --update-head-ok ../intermediate/webtools.common\n\nwhich in its turn runs\n\ngit-upload-pack ../intermediate/webtools.common\n\n\nAny ideas here are much appreciated =/\n\nThe Script in question is here:\n   https://gist.github.com/robstryker/4854fc86ab3714a5e1af353b98cbc768\n"},{"id":"318086","messageId":"20170427200956.ssubj74fkxxi6wjk@sigill.intra.peff.net","threadId":"45819","inReplyTo":"CA+Up40iusByn-R55=2=2Ae8KH1mkj4hGF_E9dX3vn1vboyMwMw@mail.gmail.com","subject":"Re: 30min Script in git 2.7.4 takes 22+ hrs in git 2.9.3","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-04-27T20:09:57Z","receivedAt":"2017-04-27T20:10:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 27, 2017 at 12:36:54PM -0400, Robert Stryker wrote:\n\n> The problem:  the script takes 30 minutes for one environment\n> including git 2.7.4, and generates a repo of about 30mb.   When run by\n> a coworker using git 2.9.3, it takes 22+ hours and generates a 10gb\n> repo.\n> \n> Clearly something here is very wrong. Either there's a pretty horrible\n> regression or my idea is a pretty bad one ;)\n\nThe large size makes me think that you're getting an auto-gc in the\nmiddle that is exploding the unreachable objects into loose storage.\nThis can happen when objects are ready to be pruned, but Git holds on to\nthem for a grace periods (2 weeks by default) as a precaution against\nsimultaneous use.\n\nTry doing:\n\n  git config gc.auto 0\n\nin the repositories before the slow step. Or alternatively, try:\n\n  git config gc.pruneExpire now\n\nwhich will continue to do the auto-gc, but throw away unreachable\nobjects immediately.\n\nOr alternatively, we're failing to run gc at all and just getting tons\nof loose objects that need packed. What does running \"git gc --auto\" say\nif you run it in the slow repository? Does it improve the disk space\nproblem?\n\nEven if one of those helps, I'd still like to know why the gc behavior\nchanged between the two versions. The best way to do that is via\ngit-bisect.\n\nYou should be able to do:\n\n  # make sure you can compile git from source\n  git clone git://git.kernel.org/pub/scm/git/git.git\n  cd git\n  make\n\n  git bisect start\n  git bisect good v2.7.4\n  git bisect bad v2.9.3\n\n  # for each commit bisect dumps you at, run your test. The bin-wrappers\n  # part is important, because it sets up the environment to run\n  # sub-programs from the built version. And as pull is a shell script,\n  # the problem is likely in a sub-program.\n  /path/to/git/bin-wrappers/git pull ...\n\n  # And then mark whether it was fast or slow. You obviously don't need\n  # to run the program to completion; just enough to decide if it's fast\n  # or slow (which might be better done by observing disk space rather\n  # than timing).\n  git bisect good ;# or \"bad\" if it was slow\n\nIt's going to be tedious even if it takes 30 minutes per iteration. It\nmight be worth trying to adjust the test case for smaller repos. :)\n\nIt may also be worth trying the test with the latest tip of \"master\".\nv2.9.3 is several versions behind, and it's possible that something may\nhave been fixed since then (nothing comes immediately to mind, but it's\nworth a shot).\n\n-Peff\n"},{"id":"318089","messageId":"20170427204218.fywkvvr65ydljb7j@sigill.intra.peff.net","threadId":"45819","inReplyTo":"20170427200956.ssubj74fkxxi6wjk@sigill.intra.peff.net","subject":"Re: 30min Script in git 2.7.4 takes 22+ hrs in git 2.9.3","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-04-27T20:42:19Z","receivedAt":"2017-04-27T20:42:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 27, 2017 at 04:09:56PM -0400, Jeff King wrote:\n\n> On Thu, Apr 27, 2017 at 12:36:54PM -0400, Robert Stryker wrote:\n> \n> > The problem:  the script takes 30 minutes for one environment\n> > including git 2.7.4, and generates a repo of about 30mb.   When run by\n> > a coworker using git 2.9.3, it takes 22+ hours and generates a 10gb\n> > repo.\n> > \n> > Clearly something here is very wrong. Either there's a pretty horrible\n> > regression or my idea is a pretty bad one ;)\n> \n> The large size makes me think that you're getting an auto-gc in the\n> middle that is exploding the unreachable objects into loose storage.\n> This can happen when objects are ready to be pruned, but Git holds on to\n> them for a grace periods (2 weeks by default) as a precaution against\n> simultaneous use.\n> \n> Try doing:\n> \n>   git config gc.auto 0\n> \n> in the repositories before the slow step. Or alternatively, try:\n> \n>   git config gc.pruneExpire now\n> \n> which will continue to do the auto-gc, but throw away unreachable\n> objects immediately.\n> \n> Or alternatively, we're failing to run gc at all and just getting tons\n> of loose objects that need packed. What does running \"git gc --auto\" say\n> if you run it in the slow repository? Does it improve the disk space\n> problem?\n\nFiddling with your script a bit, I have a suspect. Between your two\nversions of git, we started disallowing merge of unrelated histories by\ndefault[1]. Which is exactly what your script is doing:\n\n  echo \"Merge in the four rewritten projects, with generic commit messages\"\n  git pull --no-edit webtools.common.fproj     \n  git pull --no-edit webtools.common           \n  git pull --no-edit webtools.common.tests     \n  git pull --no-edit webtools.common.snippets\n\nIf you run under \"set -e\", or just put \"|| exit 1\" after those, you'll\nsee that they fail with v2.9.3 and newer.\n\nSo what I think is happening is that we never create that shared\nhistory, and then your per-tag work is building further on a nonsense\nfake history. That has two implications:\n\n  - as the divergent history in the shared repo gets bigger and bigger,\n    the fetches have to do more and more work to try to find a common\n    ancestor (but of course they'll never find one, because the two\n    histories aren't related)\n\n  - the divergent history racks up tons of unreachable objects, which\n    auto-gc won't pack. After a while of the script running, you can see\n    that auto-gc fails with \"There are too many unreachable loose\n    objects\" after the pack. Due to the way background gc works these\n    days, that blocks further auto-gc from running until the situation\n    is resolved. And you just rack up tons of loose objects, which\n    explains the disk usage.\n\nTry adding \"--allow-unrelated-histories\" to your git-pull invocation.\n\n-Peff\n\n[1] See e379fdf34 (merge: refuse to create too cool a merge by default, 2016-03-18)\n"}]}