{"thread":{"id":"38635","subject":"Salvaging borked project history","startedAt":"2015-02-23T17:05:27Z","lastAt":"2015-03-04T14:49:25Z","messageCount":9,"participants":["Mason","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"256522","messageId":"54EB5DD7.7050202@free.fr","threadId":"38635","inReplyTo":null,"subject":"Salvaging borked project history","fromName":"Mason","fromEmail":"slash.tmp@free.fr","sentAt":"2015-02-23T17:05:27Z","receivedAt":"2015-02-23T17:05:27Z","isPatch":false,"sender":{"key":"slash.tmp@free.fr","avatar":null},"body":"Hello everyone,\n\nHere's the situation:\n\nBack in 2012, we cloned a MIPS repo, which was itself a clone of\na 3.4.2 kernel with ~40 MIPS-specific patches applied.\n\nThen the devs started pushing patches; and once in a while, the\nmaintainer would \"sync\" with the mainline kernel. I don't know\nwhat tool he used to sync, but he discarded the original patches\ndoing so, and git blame now shows him as the author of every\nmainline change since the clone.\n\nNow that we are moving to a more recent kernel, I'm trying to\n\"fix\" this situation, i.e. I'd like kernel code to have proper\nattribution in git blame, while keeping intact the information\nfor \"internal\" commits.\n\nWhile I'm at it, I want to minimize history by ignoring patches\nthat are now irrelevant:\n\n1) we don't support MIPS anymore, I want to ignore any internal\nchange we made in the arch/mips directory\n\n2) for some reason, there was a large amount of create/delete churn\nin arch/arm/configs; so I want to ignore our changes, and I'll commit\nan acceptable config when the process is complete.\n\n3) ignore some internal platform-specific patches\n\n(NB: All these \"ignores\" are for internal patches, not mainline patches)\n\nSooo, using git log and grep, I came up with a list of ~300 patches I'd\nlike to apply on top of the latest 3.14 kernel. For testing purposes,\nI used git format-patch on the oldest of my 300 patches.\n\nI then made a shiny new clone of linux-stable.git, and tried using git am\nto apply that patch to the tip of linux-stable, expecting having to solve\na few dozens conflicts, and move on (hoping to automate it at some point).\n\nBut 'git am' just gave up. Small sample of errors:\n\nerror: patch failed: drivers/block/Kconfig:411\nerror: drivers/block/Kconfig: patch does not apply\nerror: drivers/tty/serial/8250/8250.c: does not exist in index\nerror: patch failed: drivers/tty/serial/8250/8250_early.c:121\nerror: drivers/tty/serial/8250/8250_early.c: patch does not apply\nerror: patch failed: drivers/video/Kconfig:101\nerror: drivers/video/Kconfig: patch does not apply\n...\nPatch failed at 0001 Commit for supporting Sigma Designs' SoCs.\nThe copy of the patch that failed is found in:\n    .git/rebase-apply/patch\nWhen you have resolved this problem, run \"git am --continue\".\nIf you prefer to skip this patch, run \"git am --skip\" instead.\nTo restore the original branch and stop patching, run \"git am --abort\".\n\nHow do I \"resolve this problem\".\n\nI suppose I am doing this the wrong way.\nWhat is the correct way to do it?\n(I'm using git 2.3)\n\nShould I use git am -3 in the original repo?\n\nRegards.\n"},{"id":"256538","messageId":"xmqq7fv8to7e.fsf@gitster.dls.corp.google.com","threadId":"38635","inReplyTo":"54EB5DD7.7050202@free.fr","subject":"Re: Salvaging borked project history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-23T19:36:21Z","receivedAt":"2015-02-23T19:36:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mason <slash.tmp@free.fr> writes:\n\n> But 'git am' just gave up. Small sample of errors:\n>\n> error: patch failed: drivers/block/Kconfig:411\n> error: drivers/block/Kconfig: patch does not apply\n> error: drivers/tty/serial/8250/8250.c: does not exist in index\n> error: patch failed: drivers/tty/serial/8250/8250_early.c:121\n> error: drivers/tty/serial/8250/8250_early.c: patch does not apply\n> error: patch failed: drivers/video/Kconfig:101\n> error: drivers/video/Kconfig: patch does not apply\n> ...\n> Patch failed at 0001 Commit for supporting Sigma Designs' SoCs.\n> The copy of the patch that failed is found in:\n>    .git/rebase-apply/patch\n> When you have resolved this problem, run \"git am --continue\".\n> If you prefer to skip this patch, run \"git am --skip\" instead.\n> To restore the original branch and stop patching, run \"git am --abort\".\n>\n> How do I \"resolve this problem\".\n\nYou look at .git/rebase-apply/patch and drivers/block/Kconfig in\nyour editor, and edit the former to make it apply (i.e. adjust the\ncommon context lines that begin with SP and preimage lines that\nbegin with '-' to match what you see in the drivers/block/Kconfig\nwhere the change wants to go).  You do so for all the other files.\n\nAnd then run \"git am\" again without any argument, until the patch\napplies.\n\nOne way to \"edit .git/rebase-apply/patch and make it apply\" that is\noften necessary when you are applying to codebase that is way more\nahead than where you took the original patches is to hand-edit the\nworking tree files (e.g. drivers/block/Kconfig) that the patch wants\nto touch and change it in the way the patch wants to, while viewing\nthe original .git/rebase-apply/patch in your pager (you do not\ndirectly edit this original patch---you only use it as a guide to\ndiscern what the patch wants to do).  Once the working tree files\nare in a state that the patch would have wanted to make if it were\ntaken based on the new codebase, overwrite .git/rebase-apply/patch\nwith the output from \"git diff HEAD\".  After that (i.e. you have\n\"edited\" .git/rebase-apply/patch, with the help with your editor,\nthe working tree files and \"git diff\"), do \"git reset --hard\".\n\nWhich brings you to the same state just after \"git am\" gave up and\nyou correctly edited \".git/rebase-apply/patch:\" to make it apply.\n\nSo you run \"git am\" again without any argument.  This time the patch\nwould apply and continue.\n\nBut I personally think \"git am -3\" may be easier to handle.\n"},{"id":"256680","messageId":"54EF098B.4080803@free.fr","threadId":"38635","inReplyTo":"xmqq7fv8to7e.fsf@gitster.dls.corp.google.com","subject":"Re: Salvaging borked project history","fromName":"Mason","fromEmail":"slash.tmp@free.fr","sentAt":"2015-02-26T11:54:51Z","receivedAt":"2015-02-26T11:54:51Z","isPatch":false,"sender":{"key":"slash.tmp@free.fr","avatar":null},"body":"Junio C Hamano wrote:\n\n> But I personally think \"git am -3\" may be easier to handle.\n\nThanks! At least now, I see the light at the end of the tunnel.\n\nI fetched linux-stable.git inside our repo.\nI created ~300 patches using git format-patch -1 in a loop.\nI can now run 'git am --3way $IGNORE *.patch'\n\nIGNORE is used to --exclude the directories I'm not interested in.\n\nNote: it seems --exclude=arch/mips and --exclude=arch/mips/ are\nnot sufficient, I need to write --exclude=arch/mips/* for git-apply\nto ignore changes to files inside arch/mips.\n\nIs that expected behavior?\n\nAnother nit: if a patch contains only changes to files inside arch/mips\nthen git-apply will create an \"empty commit\" (one with no diff). Is there\nan option to say \"skip empty patches\"?\n\nOne more thing: \"regular\" diff -q returns 0 when the files are identical,\nand 1 when they differ. It seems git diff -s does not have that behavior.\nIs that by design?\n\nIf there is no option to skip empty patches, I'm thinking I can script\na fixup step to squash all empty commits. What do you think?\n\nRegards.\n"},{"id":"256697","messageId":"xmqqh9u8mrha.fsf@gitster.dls.corp.google.com","threadId":"38635","inReplyTo":"54EF098B.4080803@free.fr","subject":"Re: Salvaging borked project history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-26T18:56:33Z","receivedAt":"2015-02-26T18:56:33Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mason <slash.tmp@free.fr> writes:\n\n> Thanks! At least now, I see the light at the end of the tunnel.\n>\n> I fetched linux-stable.git inside our repo.\n> I created ~300 patches using git format-patch -1 in a loop.\n> I can now run 'git am --3way $IGNORE *.patch'\n>\n> IGNORE is used to --exclude the directories I'm not interested in.\n>\n> Note: it seems --exclude=arch/mips and --exclude=arch/mips/ are\n> not sufficient, I need to write --exclude=arch/mips/* for git-apply\n> to ignore changes to files inside arch/mips.\n>\n> Is that expected behavior?\n\nI have no idea; at least to me, \"--exclude\" option to \"git apply\"\nwas invented to name individual paths, not patterns, and I wouldn't\nbe surprised if glob working were merely by accident not by design.\n\n> Another nit: if a patch contains only changes to files inside arch/mips\n> then git-apply will create an \"empty commit\" (one with no diff). Is there\n> an option to say \"skip empty patches\"?\n\n\"git am --skip\" perhaps?\n\n\"git am\" may pass the \"--exclude/--include\" options to \"git apply\"\nbut I wouldn't be surprised if that support was added without\nthinking.  Perhaps the reason why you discovered that it needed a\nlot more thinking to properly integrate these options to \"git am\"\nonly now is because hardly anybody uses it ;-).  Not just passing\nthese options, the code in \"git am\" to react to the result of patch\napplication to avoid the issue you observed when these options are\npassed need to be adjusted by changes that started passing them, but\nI do not think they did, cf. 77e9e496 (am: pass exclude down to\napply, 2011-08-03).\n\n> One more thing: \"regular\" diff -q returns 0 when the files are identical,\n> and 1 when they differ. It seems git diff -s does not have that behavior.\n> Is that by design?\n\n\"diff -s\" may be accepted but it is an idiotic thing for a user to\nsay.  The \"-s\" option is to squelch output from \"log\" and friends,\nand it is exposed to \"diff\" only because these two families of\ncommands share the command line parser.\n"},{"id":"256702","messageId":"54EF814D.1020105@free.fr","threadId":"38635","inReplyTo":"xmqqh9u8mrha.fsf@gitster.dls.corp.google.com","subject":"Re: Salvaging borked project history","fromName":"Mason","fromEmail":"slash.tmp@free.fr","sentAt":"2015-02-26T20:25:49Z","receivedAt":"2015-02-26T20:25:49Z","isPatch":false,"sender":{"key":"slash.tmp@free.fr","avatar":null},"body":"Junio C Hamano wrote:\n\n> Mason wrote:\n> \n>> I fetched linux-stable.git inside our repo.\n>> I created ~300 patches using git format-patch -1 in a loop.\n>> I can now run 'git am --3way $IGNORE *.patch'\n>>\n>> IGNORE is used to --exclude the directories I'm not interested in.\n>>\n>> Note: it seems --exclude=arch/mips and --exclude=arch/mips/ are\n>> not sufficient, I need to write --exclude=arch/mips/* for git-apply\n>> to ignore changes to files inside arch/mips.\n>>\n>> Is that expected behavior?\n> \n> I have no idea; at least to me, \"--exclude\" option to \"git apply\"\n> was invented to name individual paths, not patterns, and I wouldn't\n> be surprised if glob working were merely by accident not by design.\n\nAccording to the docs, IIUC globbing is by design:\nhttp://git-scm.com/docs/git-apply\n\n--exclude=<path-pattern>\n  Don't apply changes to files matching the given path pattern.\n  This can be useful when importing patchsets, where you want to\n  exclude certain files or directories.\n\nThis lead me to believe that --exclude=arch/mips should work.\n\n>> Another nit: if a patch contains only changes to files inside arch/mips\n>> then git-apply will create an \"empty commit\" (one with no diff). Is there\n>> an option to say \"skip empty patches\"?\n> \n> \"git am --skip\" perhaps?\n\nNah, as the doc states, --skip \"is only meaningful when restarting\nan aborted patch.\"\n\n> \"git am\" may pass the \"--exclude/--include\" options to \"git apply\"\n> but I wouldn't be surprised if that support was added without\n> thinking.  Perhaps the reason why you discovered that it needed a\n> lot more thinking to properly integrate these options to \"git am\"\n> only now is because hardly anybody uses it ;-).\n\nI do have a rather obscure use-case (and if I do it right, I will\nnever have to do it again).\n\n> Not just passing\n> these options, the code in \"git am\" to react to the result of patch\n> application to avoid the issue you observed when these options are\n> passed need to be adjusted by changes that started passing them, but\n> I do not think they did, cf. 77e9e496 (am: pass exclude down to\n> apply, 2011-08-03).\n\nSorry, I could not parse that paragraph :-)\n\n>> One more thing: \"regular\" diff -q returns 0 when the files are identical,\n>> and 1 when they differ. It seems git diff -s does not have that behavior.\n>> Is that by design?\n> \n> \"diff -s\" may be accepted but it is an idiotic thing for a user to\n> say.  The \"-s\" option is to squelch output from \"log\" and friends,\n> and it is exposed to \"diff\" only because these two families of\n> commands share the command line parser.\n\nHere is the use-case:\n\nif diff -q A B; then do_X; else do_Y; fi\n\nIt makes sense to prevent diff from writing to stdout.\n\nI was planning to write 'git diff -q commit^ commit'\nto test for empty commits. Looks like I'll be needing\n'git diff commit^ commit | wc -l' instead?\n\nRegards.\n"},{"id":"256711","messageId":"xmqqioeol5v4.fsf@gitster.dls.corp.google.com","threadId":"38635","inReplyTo":"54EF814D.1020105@free.fr","subject":"Re: Salvaging borked project history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-26T21:28:47Z","receivedAt":"2015-02-26T21:28:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mason <slash.tmp@free.fr> writes:\n\n>> Not just passing\n>> these options, the code in \"git am\" to react to the result of patch\n>> application to avoid the issue you observed when these options are\n>> passed need to be adjusted by changes that started passing them, but\n>> I do not think they did, cf. 77e9e496 (am: pass exclude down to\n>> apply, 2011-08-03).\n>\n> Sorry, I could not parse that paragraph :-)\n\nHeh, paraphrasing.\n\n77e9e496 and others tried to teach --exclude/--include to \"git am\".\nFor \"git am\" to be able to claim that these options are properly\nsupported, you need two things:\n\n - The options can be given from the command line and they are\n   passed to the underlying \"git apply\", instead of complaining with\n   \"no such option\".\n\n - After calling \"git mailsplit\", \"git mailinfo\", and \"git apply\"\n   and then these helper programs return, \"git am\" needs to inspect\n   what happened and react.  The patch application may have failed\n   due to conflicts, in which case it may have to ask the user to\n   resolve.  A patch application may have resulted in no-change,\n   which often is a sign that there is something wrong and \"am\"\n   would want to stop and ask the user for confirmation.  If use of\n   --include/--exclude introduces a new failure mode (e.g. mailsplit\n   and mailinfo may find a patch text and happy without complaint,\n   but passing --exclude to apply may cause the remaining patch to\n   become essentially empty, which never happens before \"am\" started\n   accepting these options), that codepath needs to be updated to\n   cope with the new failure mode it has introduced.\n\nAnd I think 77e9e496 and the other one that added --include only did\nthe former without doing the latter.\n\n> Here is the use-case:\n>\n> if diff -q A B; then do_X; else do_Y; fi\n> ...\n> I was planning to write 'git diff -q commit^ commit'\n> to test for empty commits.\n\ns/-q/--quiet/ and all is well, no?\n\n\"git diff --quiet\" doesn't abbreviate down to \"git diff -q\"\nprimarily because \"-q\" for the underlying \"git diff-files\" means\nsomething different from \"--quiet\".\n"},{"id":"256727","messageId":"54EFB0AE.6000600@free.fr","threadId":"38635","inReplyTo":"xmqqioeol5v4.fsf@gitster.dls.corp.google.com","subject":"Re: Salvaging borked project history","fromName":"Mason","fromEmail":"slash.tmp@free.fr","sentAt":"2015-02-26T23:47:58Z","receivedAt":"2015-02-26T23:47:58Z","isPatch":false,"sender":{"key":"slash.tmp@free.fr","avatar":null},"body":"Junio C Hamano wrote:\n\n> Mason wrote:\n> \n>> I was planning to write 'git diff -q commit^ commit'\n>> to test for empty commits.\n> \n> s/-q/--quiet/ and all is well, no?\n\nDoh! I've no idea how I missed these...\n\n--exit-code\n  Make the program exit with codes similar to diff(1). That is, it\n  exits with 1 if there were differences and 0 means no differences.\n\n--quiet\n  Disable all output of the program. Implies --exit-code.\n\nThanks for walking me through it.\n\nRegards.\n"},{"id":"256729","messageId":"xmqqlhjkjkkc.fsf@gitster.dls.corp.google.com","threadId":"38635","inReplyTo":"54EFB0AE.6000600@free.fr","subject":"Re: Salvaging borked project history","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-02-26T23:54:11Z","receivedAt":"2015-02-26T23:54:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mason <slash.tmp@free.fr> writes:\n\n>> Mason wrote:\n>> \n>>> I was planning to write 'git diff -q commit^ commit'\n>>> to test for empty commits.\n>> \n>> s/-q/--quiet/ and all is well, no?\n>\n> Doh! I've no idea how I missed these...\n\nYeah, this is one of the unfortunate corners of Git that I can\napologize but cannot do much more than that about (in other words,\nmaking \"git diff -q\" as a short-hand for \"git diff --quiet\" is not\nacceptable) due to backward compatibility and consistency concerns.\n"},{"id":"257002","messageId":"54F71B75.7000104@free.fr","threadId":"38635","inReplyTo":"54EF814D.1020105@free.fr","subject":"Re: Salvaging borked project history","fromName":"Mason","fromEmail":"slash.tmp@free.fr","sentAt":"2015-03-04T14:49:25Z","receivedAt":"2015-03-04T14:49:25Z","isPatch":false,"sender":{"key":"slash.tmp@free.fr","avatar":null},"body":"Mason wrote:\n\n> I was planning to write 'git diff -q commit^ commit'\n> to test for empty commits. Looks like I'll be needing\n> 'git diff commit^ commit | wc -l' instead?\n\nI wrote a script to generate the list of empty commits.\n\ngit log --format=\"%h\" --reverse 413cb08.. | while read H\ndo\n   if git diff --quiet $H^ $H; then echo $H; fi\ndone >empty_commits\n\nBut it turns out all this is unnecessary, as git-rebase\nwill automatically filter empty commits! :-)\nUnless given the --keep-empty option, I presume.\n\nThanks to everyone involved in making this great tool.\n\nRegards.\n"}]}