{"thread":{"id":"24382","subject":"VERY slow git format-patch (tens on minutes) during rebase and rev-list during rebase -i","startedAt":"2010-07-13T06:56:45Z","lastAt":"2010-10-13T07:56:53Z","messageCount":7,"participants":["Marat Radchenko","Michael J Gruber","Erik Faye-Lund"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"145431","messageId":"loom.20100713T082913-327@post.gmane.org","threadId":"24382","inReplyTo":null,"subject":"VERY slow git format-patch (tens on minutes) during rebase and rev-list during rebase -i","fromName":"Marat Radchenko","fromEmail":"marat@slonopotamus.org","sentAt":"2010-07-13T06:56:45Z","receivedAt":"2010-07-13T06:56:45Z","isPatch":false,"sender":{"key":"marat@slonopotamus.org","avatar":"https://avatars.githubusercontent.com/u/92637?v=4"},"body":"Hi.\n\nMy setup:\n0. Quad-code machine with 8GB of ram, 10K RPM hdd.\n1. SVN repo that i periodically fetch into origin/trunk branch. Has ~200 \ncommits/day.\n2. My local branch with 1-5 commits which i often rebase against trunk.\n3. I haven't rebased for 2 days, so i'm rebasing 3 (three) commits in my branch \nover 453 commits in trunk using \"git rebase trunk\".\n4. trunk does contain \"bad\" from diff POV files (big & binary).\n5. Sadly, data in repo is confidential.\n\nExpected: rebase takes some reasonable amount of time (< 1 min?).\n\nActual: rebase takes 20 mins.\n\nAlmost all of that time was spent doing `git format-patch -k --stdout --full-\nindex --ignore-if-in-upstream \n80bb0dfe3d86f3cc9095ea616d9d1b1530fbe7b8..d3fde4ae7497981a6fe61b0366b105477896cf\n52` (that's three commits from my branch) at 100% of one CPU core.\n\nAdditional info:\n\nAnother similar rebase but over 4.5k of commits took 2 hours.\n\nRunning without --ignore-if-in-upstream:\n$ time git format-patch -k --stdout --full-index \n80bb0dfe3d86f3cc9095ea616d9d1b1530fbe7b8..d3fde4ae7497981a6fe61b0366b105477896cf\n5 | wc -l\n25823\nIs it \nreal\t0m0.163s\nuser\t0m0.140s\nsys\t0m0.020s\n\nProof there are only three commits:\n\n$ git rev-list \n80bb0dfe3d86f3cc9095ea616d9d1b1530fbe7b8..d3fde4ae7497981a6fe61b0366b105477896cf\n52d3fde4ae7497981a6fe61b0366b105477896cf52\ne18069258806bda6a6165822003f5e9fd958f906\nc8c2f2e157e615b73d0baab1d793a22991c9ba71\n\nQuestions:\n1. Is it expected behavior (branch you rebase onto has binary files -> no \nperformance for you)?\n2. If [1] is yes, is it possible to prevent rebase from running --ignore-if-in-\nupstream?\n3. If [1] is no, should i run some kind of profiler (how?) to determine what \nexactly causes such performance drop?\n"},{"id":"145434","messageId":"4C3C1FE9.40605@drmicha.warpmail.net","threadId":"24382","inReplyTo":"loom.20100713T082913-327@post.gmane.org","subject":"Re: VERY slow git format-patch (tens on minutes) during rebase and rev-list during rebase -i","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2010-07-13T08:12:25Z","receivedAt":"2010-07-13T08:12:25Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Marat Radchenko venit, vidit, dixit 13.07.2010 08:56:\n> Hi.\n> \n> My setup:\n> 0. Quad-code machine with 8GB of ram, 10K RPM hdd.\n> 1. SVN repo that i periodically fetch into origin/trunk branch. Has ~200 \n> commits/day.\n> 2. My local branch with 1-5 commits which i often rebase against trunk.\n> 3. I haven't rebased for 2 days, so i'm rebasing 3 (three) commits in my branch \n> over 453 commits in trunk using \"git rebase trunk\".\n> 4. trunk does contain \"bad\" from diff POV files (big & binary).\n> 5. Sadly, data in repo is confidential.\n> \n> Expected: rebase takes some reasonable amount of time (< 1 min?).\n> \n> Actual: rebase takes 20 mins.\n> \n> Almost all of that time was spent doing `git format-patch -k --stdout --full-\n> index --ignore-if-in-upstream \n> 80bb0dfe3d86f3cc9095ea616d9d1b1530fbe7b8..d3fde4ae7497981a6fe61b0366b105477896cf\n> 52` (that's three commits from my branch) at 100% of one CPU core.\n> \n> Additional info:\n> \n> Another similar rebase but over 4.5k of commits took 2 hours.\n> \n> Running without --ignore-if-in-upstream:\n> $ time git format-patch -k --stdout --full-index \n> 80bb0dfe3d86f3cc9095ea616d9d1b1530fbe7b8..d3fde4ae7497981a6fe61b0366b105477896cf\n> 5 | wc -l\n> 25823\n> Is it \n> real\t0m0.163s\n> user\t0m0.140s\n> sys\t0m0.020s\n> \n> Proof there are only three commits:\n> \n> $ git rev-list \n> 80bb0dfe3d86f3cc9095ea616d9d1b1530fbe7b8..d3fde4ae7497981a6fe61b0366b105477896cf\n> 52d3fde4ae7497981a6fe61b0366b105477896cf52\n> e18069258806bda6a6165822003f5e9fd958f906\n> c8c2f2e157e615b73d0baab1d793a22991c9ba71\n> \n> Questions:\n> 1. Is it expected behavior (branch you rebase onto has binary files -> no \n> performance for you)?\n\nWell, with \"ignore-if-in-upstream\" git has to compute a patch-id for\nevery upstream patch (merge-base..upstream) and compare to the ids of\nthe commits in mb..HEAD.\n\n> 2. If [1] is yes, is it possible to prevent rebase from running --ignore-if-in-\n> upstream?\n\nNot currently, but with my upcoming patch ;)\n\nThis has the (side-) effect of not ignoring patches which have been\napplied (with different sha1) upstream, of course.\n\n> 3. If [1] is no, should i run some kind of profiler (how?) to determine what \n> exactly causes such performance drop?\n\nIt is the calculation of the patch-ids. Git first creates a \"binary\ndiff\" and then computes the patch-id (sha1) of that diff. I am sure we\ncould optimize the calculation of patch-ids for binary diffs, which may\nbe useful in addition to shutting off \"cherry\" with rebase.\n\nMichael\n"},{"id":"145435","messageId":"645d8a9bf671937c1a6962b49cf1c512e0af0d82.1279008702.git.git@drmicha.warpmail.net","threadId":"24382","inReplyTo":"4C3C1FE9.40605@drmicha.warpmail.net","subject":"[RFC/PATCH] rebase: Allow to turn of ignore-if-in-upstream","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2010-07-13T08:13:24Z","receivedAt":"2010-07-13T08:13:24Z","isPatch":true,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"git-rebase uses \"format-patch --ignore-if-in-upstream\" do determine\nwhich commits to apply. This may or may not be desired: a user may want\nto transplant all commits, or may opt to avoid the possibly time\nconsuming calculation of patch-ids.\n\nTherefore, introduce rebase.cherry (defaulting to true) and --cherry and\n--no-cherry options (to override the config), where --cherry means the\ncurrent behavior and --no-cherry avoids \"--ignore-if-in-upstream\".\n\nSigned-off-by: Michael J Gruber <git@drmicha.warpmail.net>\n---\nRFC for obvious reasons (doc, tests).\n\n git-rebase.sh |   16 +++++++++++++++-\n 1 files changed, 15 insertions(+), 1 deletions(-)\n\ndiff --git a/git-rebase.sh b/git-rebase.sh\nindex ab4afa7..1eb6ad1 100755\n--- a/git-rebase.sh\n+++ b/git-rebase.sh\n@@ -53,6 +53,7 @@ git_am_opt=\n rebase_root=\n force_rebase=\n allow_rerere_autoupdate=\n+cherry=$(git config --bool rebase.cherry)\n \n continue_merge () {\n \ttest -n \"$prev_head\" || die \"prev_head must be defined\"\n@@ -307,6 +308,12 @@ do\n \t\tesac\n \t\tdo_merge=t\n \t\t;;\n+\t--cherry)\n+\t\tcherry=true\n+\t\t;;\n+\t--no-cherry)\n+\t\tcherry=false\n+\t\t;;\n \t-n|--no-stat)\n \t\tdiffstat=\n \t\t;;\n@@ -540,9 +547,16 @@ else\n \trevisions=\"$upstream..$orig_head\"\n fi\n \n+if test \"x$cherry\" = \"xfalse\"\n+then\n+\tcherry_opt=\"\"\n+else\n+\tcherry_opt=\"--ignore-if-in-upstream\"\n+fi\n+\n if test -z \"$do_merge\"\n then\n-\tgit format-patch -k --stdout --full-index --ignore-if-in-upstream \\\n+\tgit format-patch -k --stdout --full-index $cherry_opt \\\n \t\t$root_flag \"$revisions\" |\n \tgit am $git_am_opt --rebasing --resolvemsg=\"$RESOLVEMSG\" &&\n \tmove_to_original_branch\n-- \n1.7.2.rc1.212.g850a\n"},{"id":"145455","messageId":"AANLkTikjTgc2OVuyFTHdnHNDq9II-lGSvmin-4EWELPg@mail.gmail.com","threadId":"24382","inReplyTo":"645d8a9bf671937c1a6962b49cf1c512e0af0d82.1279008702.git.git@drmicha.warpmail.net","subject":"Re: [RFC/PATCH] rebase: Allow to turn of ignore-if-in-upstream","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@googlemail.com","sentAt":"2010-07-13T19:33:14Z","receivedAt":"2010-07-13T19:33:14Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"s/of/off/ in the subject ;)\n\nOn Tue, Jul 13, 2010 at 10:13 AM, Michael J Gruber\n<git@drmicha.warpmail.net> wrote:\n> git-rebase uses \"format-patch --ignore-if-in-upstream\" do determine\n> which commits to apply. This may or may not be desired: a user may want\n> to transplant all commits, or may opt to avoid the possibly time\n> consuming calculation of patch-ids.\n>\n> Therefore, introduce rebase.cherry (defaulting to true) and --cherry and\n> --no-cherry options (to override the config), where --cherry means the\n> current behavior and --no-cherry avoids \"--ignore-if-in-upstream\".\n>\n> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>\n> ---\n> RFC for obvious reasons (doc, tests).\n\n-- \nErik \"kusma\" Faye-Lund\n"},{"id":"150044","messageId":"4C825FBA.8070605@drmicha.warpmail.net","threadId":"24382","inReplyTo":"AANLkTikjTgc2OVuyFTHdnHNDq9II-lGSvmin-4EWELPg@mail.gmail.com","subject":"Re: [RFC/PATCH] rebase: Allow to turn of ignore-if-in-upstream","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2010-09-04T15:03:22Z","receivedAt":"2010-09-04T15:03:22Z","isPatch":true,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Erik Faye-Lund venit, vidit, dixit 13.07.2010 21:33:\n> s/of/off/ in the subject ;)\n> \n> On Tue, Jul 13, 2010 at 10:13 AM, Michael J Gruber\n> <git@drmicha.warpmail.net> wrote:\n>> git-rebase uses \"format-patch --ignore-if-in-upstream\" do determine\n>> which commits to apply. This may or may not be desired: a user may want\n>> to transplant all commits, or may opt to avoid the possibly time\n>> consuming calculation of patch-ids.\n>>\n>> Therefore, introduce rebase.cherry (defaulting to true) and --cherry and\n>> --no-cherry options (to override the config), where --cherry means the\n>> current behavior and --no-cherry avoids \"--ignore-if-in-upstream\".\n>>\n>> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>\n>> ---\n>> RFC for obvious reasons (doc, tests).\n> \n\nPinging this one. Is there any interest? Erik is right, off course ;)\n\nMichael\n"},{"id":"150361","messageId":"1284019559.12549.2.camel@n900.home.ru","threadId":"24382","inReplyTo":"4C825FBA.8070605@drmicha.warpmail.net","subject":"Re: [RFC/PATCH] rebase: Allow to turn of ignore-if-in-upstream","fromName":"Marat Radchenko","fromEmail":"marat@slonopotamus.org","sentAt":"2010-09-09T08:05:59Z","receivedAt":"2010-09-09T08:05:59Z","isPatch":true,"sender":{"key":"marat@slonopotamus.org","avatar":"https://avatars.githubusercontent.com/u/92637?v=4"},"body":"> Pinging this one. Is there any interest? Erik is right, off course ;)\n\nThere definitely is. Since [1] rebasing became much faster (minutes instead of tens of minutes), though still it takes more than I'd like it to.\n\n[1]: http://repo.or.cz/w/git.git/commit/34597c1f5a77c710dae33092cb8a7cb01c6b21c1\n"},{"id":"153377","messageId":"1286956613749-5629926.post@n2.nabble.com","threadId":"24382","inReplyTo":"loom.20100713T082913-327@post.gmane.org","subject":"[FEATURE REQUEST] allow enabling patience diff algorithm by default","fromName":"Marat Radchenko","fromEmail":"marat@slonopotamus.org","sentAt":"2010-10-13T07:56:53Z","receivedAt":"2010-10-13T07:56:53Z","isPatch":false,"sender":{"key":"marat@slonopotamus.org","avatar":"https://avatars.githubusercontent.com/u/92637?v=4"},"body":"\nI observe patience algorithm being several times faster than standard diff on\nsome big (1MB<size<10MB) text files (and, actually, it produces smaller\ndiffs). So using patience diff is likely to improve git-rev-list\nperformance.\n\nSuggested way: add option to ~/.gitconfig to enable patience diff by\ndefault. Additionally, smth like--no-patience may be added to commands that\naccept --patience now so it is possible to override setting if needed.\n\n-- \nView this message in context: http://git.661346.n2.nabble.com/VERY-slow-git-format-patch-tens-on-minutes-during-rebase-and-rev-list-during-rebase-i-tp5286226p5629926.html\nSent from the git mailing list archive at Nabble.com.\n"}]}