{"thread":{"id":"34106","subject":"Tracking vendor release with Git","startedAt":"2013-06-11T17:06:50Z","lastAt":"2013-06-12T08:17:18Z","messageCount":5,"participants":["Yann Droneaud","Greg Troxel","Johannes Sixt","Philip Oakley","Carsten Fuchs"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"220462","messageId":"1370970410-7935-1-git-send-email-ydroneaud@opteya.com","threadId":"34106","inReplyTo":null,"subject":"Tracking vendor release with Git","fromName":"Yann Droneaud","fromEmail":"ydroneaud@opteya.com","sentAt":"2013-06-11T17:06:50Z","receivedAt":"2013-06-11T17:06:50Z","isPatch":false,"sender":{"key":"ydroneaud@opteya.com","avatar":"https://avatars.githubusercontent.com/u/881377?v=4"},"body":"Hi,\n\nI'm trying to setup a workflow to track vendor releases (upstream).\nEach new release are provided as an archive of source code, data,\ndocumentation, etc.\n\nFor each vendor releases, fixes need to be applied before making them\navailable to users (downstream).\n\nSeems to be a rather common use case, applied by most Linux distribution\nfor decades.\n\nIn my case, on top of each releases, a common set of patches will be applied,\nthe biggest, the most intrusive one, being converting CRLF to LF using dos2unix,\nthe others being small portability fixes. In this case, fixes are not going to\nbe applied by upstream.\n\nI'm trying to \"design\" (copy ;) a workflow with following properties,\nin order of importance:\n\n1- I wish to keep a branch with each new vendor release as a commit.\n This branch's history is only about vendor releases,\n so it's easy to read the \"changelog\" of the vendor releases\n with command such as git log \n\n2- I'd like to ease the process of applying our patches on top\n of each new vendor release, eg. reduces the likeliness of conflicts.\n\n3- I wish to keep a branch with each new fixed vendor release as a commit.\n Just like the upstream , only one commit\n per release, so it's easy to read the \"changelog\" of the vendor releases\n with command such as git log \n \n(4- I wish nobody is going to think about ponies ...)\n\nThe workflow, as I tried to implement it is, can be described like this:\nFor each release, the new sources are imported in the upstream branch,\nthen the previous branch holding fixes is rebased against the updated\nupstream branch, and finally, the updated fixes branch is merged in\nthe downstream branch.\n\nPlease find a diagram trying to explain the following workflow:\n\n Input Output\n\n . *(1) .\n . / \\ .\n . / \\ .\n . / \\ .\n . / \\ .\n ./ \\.\n / \\\nUpstream release 1 | |\n \\ v |\n --------->* |\n |\\ |\n | \\ |\n | | |\n | v |\nUpstream release 2 | * fix 1 |\n \\ | \\ |\n \\ | \\ v\n \\ | ----->* Downstream release 1\n \\ v |\n ------>* |\n |\\ |\n | \\ |\n | | |\n | v |\n | * fix 1' |\n | | |\nUpstream release 3 | v |\n \\ | * fix 2' |\n \\ | \\ |\n \\ | \\ v\n \\ | ----->* Downstream release 2\n \\ | |\n \\ v |\n ---->* |\n |\\ |\n . \\ |\n | | |\n . v |\n . * fix 1\" |\n | | |\n . v |\n . * fix 2\" |\n . | |\n | v |\n * fix 3\" |\n \\ |\n \\ v\n ----->* Downstream release 3\n |\n .\n |\n .\n .\n\n(1) empty root commit\n\nI hope someone would come with an easier workflow, more sustainable.\nAt least, please help me find flaws in this workflow, tell me where\nI can found the documentation of others workflows achieving the same\nresults.\n\nBTW while testing the workflow, I tried \"git merge -s theirs\"\nand found it doesn't exist. I thought it would be available for\nsuch a common use case. For each merge to the of the fixes branch\nto the downstream branch, something like a whole new content is dropped\nto the branch. The previous content is still needed but its only for history.\nSo I prefer to overwrite the content of the downstream branch, instead\nof fixing each conflicts \"manually\" when using \"git merge\" with default\nstrategy and option.\n\nIn some article[1], I've found an explanation of the lack of\n\"git merge -s theirs\", quoting maintainer opinion[2] regarding it.\n\n[1] \n[2] \n\nI was able to use \"git merge --strategy=recursive --strategy-option=theirs\" aka.\n\"git merge -X theirs\". So I'm only frustrated to see my workflow not being\nconsidered. But happy to be able to do it with \"uncommon\" options.\n\n\nAppended here a script to reproduce the workflow described previously:\n\n#! /bin/sh\n\nset -e\n\nGIT_AUTHOR_DATE=`date -R`\nGIT_COMMITTER_DATE=\"$GIT_AUTHOR_DATE\"\n\nexport GIT_AUTHOR_DATE\nexport GIT_COMMITTER_DATE\n\n# cleanup\nrm -rf .git main.c\n\n# prepare\ngit init\ngit commit --allow-empty -m \"empty root commit\"\ngit branch downstream master\ngit branch upstream master\n\n#\n# import first upstream release: v1\ngit checkout upstream\ncat > main.c < main.c < main.c < main.c <\n\n/* version string */\n#define VERSION \"2\"\n\nint\nmain(void)\n{\n printf(\"version \" VERSION \"\\n\");\n}\nEOF\ngit add main.c\ngit commit -m \"Fix 2\"\n\n# make the \"fixed\" release available\n# merge the integration branch\ngit checkout downstream\ngit merge --no-edit --no-ff upstream-2-fix\ngit tag downstream-default-2\n\n#\n# new upstream release: v3\ngit checkout upstream\ncat > main.c < main.c <\n\n/* version string */\n#define VERSION \"3\"\n#define PACKAGE \"foo\"\n\nint\nmain(void)\n{\n printf(\"version \" VERSION \"\\n\");\n\n return 0;\n}\nEOF\ngit add main.c\ngit commit -m \"Fix 3\"\n\n# make the \"fixed\" release available\n# try to merge the integration branch\ngit checkout downstream\ngit merge --no-edit --no-ff upstream-3-fix || { \necho \"Merge failed, \\\"manually\\\" resolve conflict ...\"\ncat > main.c <\n\n/* version string */\n#define VERSION \"3\"\n#define PACKAGE \"foo\"\n\nint\nmain(void)\n{\n printf(\"version \" VERSION \"\\n\");\n\n return 0;\n}\nEOF\ngit add main.c\ngit commit --no-edit\n}\ngit tag downstream-default-3\n\n# now, try a different merge strategy\ngit checkout master\ngit branch -D downstream\ngit branch downstream master\n\ngit checkout downstream\n\ngit merge --no-edit --no-ff --strategy-option theirs upstream-1-fix\ngit tag downstream-theirs-1\ngit merge --no-edit --no-ff --strategy-option theirs upstream-2-fix\ngit tag downstream-theirs-2\ngit merge --no-edit --no-ff --strategy-option theirs upstream-3-fix\ngit tag downstream-theirs-3\n\ngit checkout master\n\n# \"default\" and \"theirs\" should match\n# but have a different commit message:\n# no conflict reported for \"theirs\"\ngit diff downstream-default-3 downstream-theirs-3\n\n\n# Regards !\n\n-- \nYann Droneaud \nOPTEYA\n"},{"id":"220469","messageId":"rmi38so1rq7.fsf@fnord.ir.bbn.com","threadId":"34106","inReplyTo":"1370970410-7935-1-git-send-email-ydroneaud@opteya.com","subject":"Re: Tracking vendor release with Git","fromName":"Greg Troxel","fromEmail":"gdt@ir.bbn.com","sentAt":"2013-06-11T17:29:04Z","receivedAt":"2013-06-11T17:29:04Z","isPatch":false,"sender":{"key":"gdt@ir.bbn.com","avatar":null},"body":"\n I'm trying to setup a workflow to track vendor releases (upstream).\n Each new release are provided as an archive of source code, data,\n documentation, etc.\n\nI've been doing more or less this. A few comments:\n\n I suggest that you not view CRLF->LF as a \"patch\". I would do EOL\n hygiene as a preprocessing script, with a checked-in script, after\n unpacking the tarball or whatever, and before 'import'. Otherwise\n it's just going to be too messy.\n\n I use \"vendor.foo\" as the branch name.\n\n If your repo is only for this program, you can ignore this, but\n otherwise you way want to use subtree merge so that vendor.foo: maps\n to master:foo (putting foo in a subtree in master). This lets you\n have multiple upstreams in one repo, which is useful for system\n building more than maintaining.\n\n I would avoid rebase. You are essentially merging someone else's\n branch (that they aren't putting in git, but you are with the\n vendor.foo) into your master. With regular merge, you can still\n diff, but the natural history will be right. With rebase each \"local\n version\" as you call it will have different commits that will not have\n clear ancestry.\n"},{"id":"220482","messageId":"51B76C14.3060907@kdbg.org","threadId":"34106","inReplyTo":"1370970410-7935-1-git-send-email-ydroneaud@opteya.com","subject":"Re: Tracking vendor release with Git","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2013-06-11T18:27:32Z","receivedAt":"2013-06-11T18:27:32Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 11.06.2013 19:06, schrieb Yann Droneaud:\n> Hi,\n> \n> I'm trying to setup a workflow to track vendor releases (upstream).\n> Each new release are provided as an archive of source code, data,\n> documentation, etc.\n> \n> For each vendor releases, fixes need to be applied before making them\n> available to users (downstream).\n> \n> Seems to be a rather common use case, applied by most Linux distribution\n> for decades.\n> \n> In my case, on top of each releases, a common set of patches will be applied,\n> the biggest, the most intrusive one, being converting CRLF to LF using dos2unix,\n> the others being small portability fixes. In this case, fixes are not going to\n> be applied by upstream.\n> \n> I'm trying to \"design\" (copy ;) a workflow with following properties,\n> in order of importance:\n> \n> 1- I wish to keep a branch with each new vendor release as a commit.\n> This branch's history is only about vendor releases,\n> so it's easy to read the \"changelog\" of the vendor releases\n> with command such as git log \n> \n> 2- I'd like to ease the process of applying our patches on top\n> of each new vendor release, eg. reduces the likeliness of conflicts.\n> \n> 3- I wish to keep a branch with each new fixed vendor release as a commit.\n> Just like the upstream , only one commit\n> per release, so it's easy to read the \"changelog\" of the vendor releases\n> with command such as git log \n\nI suggest you aim for the following history (time flows from left to right):\n\n U---V-----W <-- upstream branch\n \\ \\ \\\n C---D-----E <-- CRLF conversion branch\n \\ \\ \\\n K---L--M--N--O <-- downstream branch\n\nU, V, W are the upstream releases.\n\nC is the initial CRLF->LF conversion. D merges the second upstream\nrelease into the CRLF branch, E the third upstream release. These merges\nvery likely create tons of conflicts. But that does not matter, because\nyou know that the only change in \"our\" side is CRLF conversion. The\ncommits on this branch can easily be automated. That's the primary\nmotivation for this scheme.\n\nK is your first small bugfix and also your first downstream release.\n\nAfter merging L, the second, CRLF-converted, upstream release, you make\nyour second small change, M, which is also your second downstream release.\n\nRinse and repeat with N and O for the third release.\n\n-- Hannes\n"},{"id":"220488","messageId":"32BB10B036234E0F9A6AFEF05C25880F@PhilipOakley","threadId":"34106","inReplyTo":"1370970410-7935-1-git-send-email-ydroneaud@opteya.com","subject":"Re: Tracking vendor release with Git","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-06-11T18:43:11Z","receivedAt":"2013-06-11T18:43:11Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Yann Droneaud\" \nSent: Tuesday, June 11, 2013 6:06 PM\n> Hi,\n>\n> I'm trying to setup a workflow to track vendor releases (upstream).\n> Each new release are provided as an archive of source code, data,\n> documentation, etc.\n>\n> For each vendor releases, fixes need to be applied before making them\n> available to users (downstream).\n>\n> Seems to be a rather common use case, ...\n\nHave you looked at the msysgit process that has to cope with upstream \ngit ;-)\n\ne.g. \nhttps://code.google.com/p/msysgit/source/browse/share/msysGit/merging-rebase.sh?name=python\nhttps://github.com/msysgit/msysgit/blob/master/share/msysGit/merging-rebase.sh\nand\nhttps://github.com/msysgit/msysgit/blob/master/share/msysGit/rebasing-merge.sh\n\nwhereby the guys re-apply all the patches that haven't been accepted \nupstream, along with local fixups to get each new release working.\n\nPhilip \n"},{"id":"220589","messageId":"51B82E8E.8010402@cafu.de","threadId":"34106","inReplyTo":"1370970410-7935-1-git-send-email-ydroneaud@opteya.com","subject":"Re: Tracking vendor release with Git","fromName":"Carsten Fuchs","fromEmail":"carsten.fuchs@cafu.de","sentAt":"2013-06-12T08:17:18Z","receivedAt":"2013-06-12T08:17:18Z","isPatch":false,"sender":{"key":"carsten.fuchs@cafu.de","avatar":"https://gravatar.com/avatar/bd0d6585c2cec9529584fc310bbcdee3ffd1f06f3fb88644fdb87fd880fc75a6?d=mp&s=160"},"body":"Hi Yann,\n\nAm 2013-06-11 19:06, schrieb Yann Droneaud:\n> I'm trying to setup a workflow to track vendor releases (upstream).\n> Each new release are provided as an archive of source code, data,\n> documentation, etc.\n>\n> For each vendor releases, fixes need to be applied before making them\n> available to users (downstream).\n>\n> Seems to be a rather common use case, applied by most Linux distribution\n> for decades.\n>\n> In my case, on top of each releases, a common set of patches will be applied,\n> the biggest, the most intrusive one, being converting CRLF to LF using dos2unix,\n> the others being small portability fixes. In this case, fixes are not going to\n> be applied by upstream.\n\n\nIf you did the end-of-line conversion via .gitattributes rather than explicitly as a \npatch, maybe the strategy described at \nhttp://happygiraffe.net/blog/2008/02/07/vendor-branches-in-git/ is what you're looking for?\n\nIf besides the branch you need another branch, this \nshould be extensible, inserting another \"layer\" into the middle.\nCopying and modifying Johannes' graph:\n\n U---V-----W <-- upstream branch (pristine vendor)\n \\ \\ \\\n C---D-----E <-- patched vendor\n \\ \\ \\\n K---L--M--N--O <-- downstream branch (\"master\" in above linked text)\n\n\nBest regards,\nCarsten\n\n\n\n-- \nDipl.-Inf. Carsten Fuchs\n\nCarsten Fuchs Software\nIndustriegebiet 3, c/o Rofu, 55768 Hoppstädten-Weiersbach, Germany\nInternet: http://www.cafu.de | E-Mail: info@cafu.de\n\nCafu - the open-source game and graphics engine for multiplayer 3D action\n"}]}