{"thread":{"id":"26163","subject":"rebase parents, or tracking upstream but removing non-distributable bits","startedAt":"2010-12-30T17:54:29Z","lastAt":"2011-01-05T11:44:15Z","messageCount":7,"participants":["Alexandre Oliva","Jonathan Nieder","Yann Dirson","Alexandre Erwin Ittner","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"158758","messageId":"ord3ojb0yy.fsf@livre.localdomain","threadId":"26163","inReplyTo":null,"subject":"rebase parents, or tracking upstream but removing non-distributable bits","fromName":"Alexandre Oliva","fromEmail":"lxoliva@fsfla.org","sentAt":"2010-12-30T17:54:29Z","receivedAt":"2010-12-30T17:54:29Z","isPatch":false,"sender":{"key":"lxoliva@fsfla.org","avatar":"https://gravatar.com/avatar/52d009770eaeed6dfb2bbe70d9ee9be17bd674e7f3338213a87332fb518d7665?d=mp&s=160"},"body":"Say the git repository of a project I use (with changes) on another\nprojet I work on contains portions that I oughtn't distribute.  Say,\nportions that are illegal, immoral or too risky in my jurisdiction:\npatented stuff that lawyers say I should not distribute in anyway,\nunauthorized or otherwise copyright-infringing bits, text or pictures\nthat are offensive or even illegal to publish, i.e., stuff that I must\nnot be caught distributing and that, ideally, I could arrange to not\neven possess.\n\nIf you guessed that my primary reason to want this is the non-Free\nSoftware in the Linux git repository, you got it right :-)  Anyhow,\nregardless of your opinion as to my stance in this matter, I hope you'll\nagree that the scenarios above are relevant and desirable.  Heck, even a\nbusiness that decides to remove all traces from a feature that was\nplanned for a certain release, but that is pushed back to a later\nrelease, could benefit from this.\n\nNote that simply reverting/removing these bits from the head of a branch\nwouldn't be enough: since the repository carries the entire history,\npushing the head of the branch to my public repository would amount to\npublishing the bits I must not publish.\n\nI need to be able to maintain and publish a modified repository, that\nfilters out the unwanted portions, but still be able to pull changes\nfrom the upstream repository.  Desirable, but not strictly necessary, is\nthe possibility of letting upstream pull my improvements, without\nbringing in the changes I made to remove the bits I'm not supposed to\ndistribute.\n\n\nGiven this problem statement, I started looking for solutions that\ndidn't require modifying git.\n\nI first looked into rewriting history, removing the unwanted bits and\nreplaying subsequent changes, but quickly discarded it, for it would\nmake my local repository incompatible with upstream both ways: I\nwouldn't be able to pull from it; upstream wouldn't be able to pull to\nit; third parties would run into ugly situations trying to carry patches\nfrom either one to the other.\n\nNow, it looks like I might be able to pull from upstream if I maintain\nmanually a graft file that named each upstream commit as an additional\nparent of the corresponding local rebase commit that brought it into my\nrewritten tree.  Workable, maybe, but this wouldn't help third parties\nthat used my public repository.\n\nBesides, I'm concerned that pushing from the local repository (with the\ngraft file) to the public repository would end up publishing the changes\nI'm not supposed to distribute, because they'd be taken as parents of\nthe local commits.\n\n\nAre there any other ways to support the desired features with git as-is?\nAFAICT, there isn't, so I've been thinking of how to introduce this.  I\nsuppose the simplest way to accomplish this is to introduce the notion\nof a “weak parent”: one that is taken into account for purposes of\nchecking whether a commit is present in a branch being merged- or\nrebased into, but that is not transmitted over pushes, and that is not\nretained over purges, and not complained about when missing.\n\nI'm under the impression that this could not just work, but also make\nrebasing in general (especially the hard case) far less problematic, for\ngit would be able to relate a rebased commit with an original commit.\n\n\nNow, assuming I'm correct in this assessment, there are two questions\nthat remain:\n\n- how to represent this?\n\nI thought of changing the commit blob format so as to somehow mark the\nweak parents, say, with an additional character on the same line:\n\nparent f00ba5... W\n\nan alternate header:\n\nwparent f00ba5...\n\nor even an additional line:\n\nparent f00ba5...\n...\nweak f00ba5...\n\nFor some backward compatibility, it looks like only the last form would\nas much as stand a chance of being properly parsed, if the weak notes\nare added at the end of the blob.\n\nAnother possibility is to create another kind of object, that named an\noriginal and rebased commit and that, like a tag object, would be\n(optionally?) transmitted when the (rebased) commit it named was\ntransmitted.  This could be more interesting, in that it might enable\nall traces of a rebase to be eventually removed.  A (named?) object that\nnames multiple such pairs of commits might make even more sense to this\nend.\n\n\nAm I on the right track?  Any thoughts, preferences, suggestions,\nconcerns, recommendations, advice, pointers or gotchas to watch out for\nbefore I start implementing any of these possibilities?\n\n\nI realize that, although this option could make “git pull --rebase” work\nto track upstream in the rebased branch, and would enable me to publish\nthe repository with the rebased branch without the pieces I shouldn't\ndistribute, I'm not sure this would enable upstream to easily integrate\nmy changes.  Or would it?\n\n\nThanks in advance,\n\n\nI'm not subscribed, but I'm going to look for replies in the archives.\nThat said, I'd appreciate if you'd explicitly copy me in any follow ups.\n(Mail-Followup-To: set accordingly)\n\n\nLast but not least: Happy GNU Year! :-)\n\n-- \nAlexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/\nYou must be the change you wish to see in the world. -- Gandhi\nBe Free! -- http://FSFLA.org/   FSF Latin America board member\nFree Software Evangelist      Red Hat Brazil Compiler Engineer\n"},{"id":"158761","messageId":"20101230205847.GA29012@burratino","threadId":"26163","inReplyTo":"ord3ojb0yy.fsf@livre.localdomain","subject":"Re: rebase parents, or tracking upstream but removing non-distributable bits","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-30T20:58:47Z","receivedAt":"2010-12-30T20:58:47Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Alexandre Oliva wrote:\n\n> Now, it looks like I might be able to pull from upstream if I maintain\n> manually a graft file that named each upstream commit as an additional\n> parent of the corresponding local rebase commit that brought it into my\n> rewritten tree.  Workable, maybe, but this wouldn't help third parties\n> that used my public repository.\n\nHave you looked into \"git replace\"?\n"},{"id":"158768","messageId":"or4o9uc2ny.fsf@livre.localdomain","threadId":"26163","inReplyTo":"20101230205847.GA29012@burratino","subject":"Re: rebase parents, or tracking upstream but removing non-distributable bits","fromName":"Alexandre Oliva","fromEmail":"lxoliva@fsfla.org","sentAt":"2010-12-30T22:32:33Z","receivedAt":"2010-12-30T22:32:33Z","isPatch":false,"sender":{"key":"lxoliva@fsfla.org","avatar":"https://gravatar.com/avatar/52d009770eaeed6dfb2bbe70d9ee9be17bd674e7f3338213a87332fb518d7665?d=mp&s=160"},"body":"On Dec 30, 2010, Jonathan Nieder <jrnieder@gmail.com> wrote:\n\n> Alexandre Oliva wrote:\n>> Now, it looks like I might be able to pull from upstream if I maintain\n>> manually a graft file that named each upstream commit as an additional\n>> parent of the corresponding local rebase commit that brought it into my\n>> rewritten tree.  Workable, maybe, but this wouldn't help third parties\n>> that used my public repository.\n\n> Have you looked into \"git replace\"?\n\nAs far as I could tell, it solves a complementary problem.  IIUC, it\nwould enable me to replace objects (say files, trees or commits) in my\nlocal repository so as to remove objectionable stuff, but when I pushed\na branch out of it, it would go out with the very stuff I'm not supposed\nto publish.  This is because AFAICT replace objects are not sent over\nthe wire.\n\nEven if they were, I still don't think it would be appropriate to use\nthem, for I'm speaking of really different trees.  Publishing a commit\nreplacement would, for anyone who had both my public repository and my\nupstream, affect not just the branches I published, but also those in\nupstream, which would be surprising and undesirable.\n\nFinally, it wouldn't be a complete solution.  Consider, for example, an\nobjectionable file or tree from an early commit, that I replaced with\nsomething I can live with.  A later commit that changed that tree, or\nany of those files, would AFAICT *silently* override my replacement,\nrequiring constant monitoring and new replacements for every such\nchange.\n\nWith the rewrite/rebase model I have in mind, changes to modified files\nwould conflict, prompting an immediate fix, without any risk of\npublishing modified versions of unwanted files.  (Of course, in my\nparticular case I'd still have to monitor for newly-introduced\nobjectionable stuff, but that's to be expected.)\n\n\nDid I make any mistakes in my analysis of the “replace” feature?  It\nwould be lovely if I could use it, but, in a way, it appears to be the\ndual of what I need: I need to fix a problem in what I provide to\nothers, while replace would fix the problem in what I see myself.\n\n\nAnyhow, thanks for the pointer, appreciated!\n\n-- \nAlexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/\nYou must be the change you wish to see in the world. -- Gandhi\nBe Free! -- http://FSFLA.org/   FSF Latin America board member\nFree Software Evangelist      Red Hat Brazil Compiler Engineer\n"},{"id":"158769","messageId":"20101230225234.GC6639@home.lan","threadId":"26163","inReplyTo":"ord3ojb0yy.fsf@livre.localdomain","subject":"Re: rebase parents, or tracking upstream but removing non-distributable bits","fromName":"Yann Dirson","fromEmail":"ydirson@free.fr","sentAt":"2010-12-30T22:52:34Z","receivedAt":"2010-12-30T22:52:34Z","isPatch":false,"sender":{"key":"ydirson@free.fr","avatar":null},"body":"On Thu, Dec 30, 2010 at 03:54:29PM -0200, Alexandre Oliva wrote:\n> Given this problem statement, I started looking for solutions that\n> didn't require modifying git.\n\nThis is a problem I have come to think a bit about already.  Although\nI do not have a proper solution either, let's share those ideas.\n\n> I first looked into rewriting history, removing the unwanted bits and\n> replaying subsequent changes, but quickly discarded it, for it would\n> make my local repository incompatible with upstream both ways: I\n> wouldn't be able to pull from it; upstream wouldn't be able to pull to\n> it; third parties would run into ugly situations trying to carry patches\n> from either one to the other.\n> \n> Now, it looks like I might be able to pull from upstream if I maintain\n> manually a graft file that named each upstream commit as an additional\n> parent of the corresponding local rebase commit that brought it into my\n> rewritten tree.  Workable, maybe, but this wouldn't help third parties\n> that used my public repository.\n\nAs a side note: I fear grafts won't scale very nicely performance-wise\nif you graft every commit.\n\nMy use-case was similar to yours, about \"how could Debian distribute a\ngit tree of the kernel ?\", but my focus was on distributing\nwell-defined snapshots, and I ended up with the idea of grafting only\nthe set of successive revisions that get published - and not\nnecessarily have a mirror of all upstream revisions in the filtered\ntree: such grafts do represent points in history where we would merge\nfrom upstream, and without the grafts the filtered tree would look\nlike successive imports of well-defined revisions (notably makes it\nrealistic to adhere to the \"all commits have been tested\" paradigm).\n\n\n> Besides, I'm concerned that pushing from the local repository (with the\n> graft file) to the public repository would end up publishing the changes\n> I'm not supposed to distribute, because they'd be taken as parents of\n> the local commits.\n\nGrafts cannot be pushed/pulled.  For this, as Jonathan suggests in\nanother reply, \"git replace\" looks like a better choice, esp. in that\nby default the replacement commits don't get pulled: you need to\nexplicitely request fetching refs/replace/commits (note: you may need\nto handle merges there when needed).  OTOH, those replacement commits\nwould pull the whole upstream history, so they cannot be part of the\nfiltered repository: it must come from an unfiltered kernel repo, that\nmay be a real problem if you cannot redistribute some upstream parts.\n\nOTOH, a grafts file can be distributed out-of-band, and would only\npull the problematic contents when put in place, so it might indeed be\nmore adequate.\n\n\n> Are there any other ways to support the desired features with git as-is?\n> AFAICT, there isn't, so I've been thinking of how to introduce this.  I\n> suppose the simplest way to accomplish this is to introduce the notion\n> of a ???weak parent???: one that is taken into account for purposes of\n> checking whether a commit is present in a branch being merged- or\n> rebased into, but that is not transmitted over pushes, and that is not\n> retained over purges, and not complained about when missing.\n\nThat sounds like heavy surgery with too many implications I can think\nof.\n\nWhen it comes to \"modifying commit metadata\", it rings \"git notes\" in\nmy ear - but then, you cannot add/change commit metadata from a note,\nmaybe that could be a better direction to dig into, so we can add\nparents when a specific notes namespace is activated ?  But then we\nwould get back to the problem I mentionned for refs/replace/\n\n\n> I'm under the impression that this could not just work, but also make\n> rebasing in general (especially the hard case) far less problematic, for\n> git would be able to relate a rebased commit with an original commit.\n\nI suppose that by \"hard case\" you mean forking off a branch that gets\nrebased later ?  Note that \"git pull\" seems to be able to cope with\nthis using reflogs already (although I did not test that feature).  A\nless volatile place than reflogs could again using notes, without a\nneed for core changes.\n\n\n> I'm not sure this would enable upstream to easily integrate\n> my changes.  Or would it?\n\nThis problem suggests a more generic one: how to \"merge back\" most\nchanges from a branch while still not merging some specific changes ?\nIt would also help when a maint branch gets some short-term\nworkarounds that we don't want in the master branch, but the very idea\nhas a serious flaw: it implies that the \"merge back\" commit contains\nalso the commits we don't want (here, the \"filtering commits\").  So I\nguess cherry-pick will be the way here.  Anyway, I doubt Linus would\nlike the idea of merging from such a filtered repo, sending patches is\nprobably prefered.\n\nBest regards,\n-- \nYann\n"},{"id":"158770","messageId":"20101230205840.bb5d3370.listas@ittner.com.br","threadId":"26163","inReplyTo":"ord3ojb0yy.fsf@livre.localdomain","subject":"Re: rebase parents, or tracking upstream but removing non-distributable bits","fromName":"Alexandre Erwin Ittner","fromEmail":"listas@ittner.com.br","sentAt":"2010-12-30T22:58:40Z","receivedAt":"2010-12-30T22:58:40Z","isPatch":false,"sender":{"key":"listas@ittner.com.br","avatar":null},"body":"Alexandre Oliva <lxoliva@fsfla.org> wrote\n\n> I need to be able to maintain and publish a modified repository, that\n> filters out the unwanted portions, but still be able to pull changes\n> from the upstream repository.\n\nHave you tried something with \"git filter-branch\"? I have never tried\nsomething like this but I think it is possible to automate a process to\n(1) pull the changes from the origin into a complete clone, (2) branch\nfrom the HEAD and run \"filter-branch\" with a customized script to create\na temporary sanitized branch, (3) merge this temporary branch into a\ncomplete sanitized branch, (4) record all the branchpoints, sparing the\nnext iteration from running through all the history again, and, (5) push\nthe sanitized branch somewhere.\n\nOf course, this approach creates a complete nightmare with the\nintegration, testing, and code attribution: it changes the SHA1s,\ninvalidates signed tags, forces the use of patches instead of pull\nrequests to the upstream contributions, may taint the validity of the\ncommit messages and authorship, etc.  Publishing edited commits seems a\ndelicate subject for me -- your \"filter-branch\" script should mark their\ncommit messages as such.\n\n\nAtt.\n\n-- \nAlexandre Erwin Ittner - alexandre@ittner.com.br\nOpenPGP pubkey 0x0041A1FB @ http://pgp.mit.edu\n"},{"id":"158771","messageId":"m37heqdfck.fsf@localhost.localdomain","threadId":"26163","inReplyTo":"or4o9uc2ny.fsf@livre.localdomain","subject":"Re: rebase parents, or tracking upstream but removing non-distributable bits","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-30T23:14:07Z","receivedAt":"2010-12-30T23:14:07Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Alexandre Oliva <lxoliva@fsfla.org> writes:\n\n> On Dec 30, 2010, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> \n> > Alexandre Oliva wrote:\n> >> Now, it looks like I might be able to pull from upstream if I maintain\n> >> manually a graft file that named each upstream commit as an additional\n> >> parent of the corresponding local rebase commit that brought it into my\n> >> rewritten tree.  Workable, maybe, but this wouldn't help third parties\n> >> that used my public repository.\n> \n> > Have you looked into \"git replace\"?\n> \n> As far as I could tell, it solves a complementary problem.  IIUC, it\n> would enable me to replace objects (say files, trees or commits) in my\n> local repository so as to remove objectionable stuff, but when I pushed\n> a branch out of it, it would go out with the very stuff I'm not supposed\n> to publish.  This is because AFAICT replace objects are not sent over\n> the wire.\n\nThey are not sent by default, but they (refs/replace/*) can be send as\nany other ref.\n\n> \n> Even if they were, I still don't think it would be appropriate to use\n> them, for I'm speaking of really different trees.  Publishing a commit\n> replacement would, for anyone who had both my public repository and my\n> upstream, affect not just the branches I published, but also those in\n> upstream, which would be surprising and undesirable.\n[...]\n\nI guess what Jonathan had in mind was something like that:\n* you have two branches, 'clean' and 'contaminated'\n* you want to merge 'contaminated' into 'clean', but you don't\n  want people to see history of 'contaminated'\n* in your private repository you merge 'contaminated' into 'clear'\n  (with --no-ff, just in case), save merge commit, then rewrite\n  top commit to be ordinary commit not a merge commit; it would\n  bring [redacted] changes but not history\n* you replace merge-turned-ordinary commit with a proper merge\n  commit\n* you don't distribute replacement refs to public repository\n\nThough I think that better solution would be feature-branch based\nworkflow.  Each feature is developed in seperate feature branch.  If\ngiven feature is sutable for 'clean', you merge it into both 'clean'\nand in 'contaminated'.  If it is not, you merge it only into\n'contaminated'.\n\nHopefully that would help develop workflow for you.\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"158956","messageId":"orwrmj378w.fsf@livre.localdomain","threadId":"26163","inReplyTo":"m37heqdfck.fsf@localhost.localdomain","subject":"Re: rebase parents, or tracking upstream but removing non-distributable bits","fromName":"Alexandre Oliva","fromEmail":"lxoliva@fsfla.org","sentAt":"2011-01-05T11:44:15Z","receivedAt":"2011-01-05T11:44:15Z","isPatch":false,"sender":{"key":"lxoliva@fsfla.org","avatar":"https://gravatar.com/avatar/52d009770eaeed6dfb2bbe70d9ee9be17bd674e7f3338213a87332fb518d7665?d=mp&s=160"},"body":"On Dec 30, 2010, Jakub Narebski <jnareb@gmail.com> wrote:\n\n> They are not sent by default, but they (refs/replace/*) can be send as\n> any other ref.\n\nOh, doh, I was modeling them after grafts, but indeed the replace refs,\nunlike grafts, can be sent out.  Which doesn't really help, since they'd\nbe sent out in addition to the objectionable stuff.\n\nUnless the idea is to replace the other way round, i.e., instead of\ncleaned-up commit replacing contaminated commit, mark the contaminated\ncommit as replacing the cleaned-up one.  I haven't explored this\npossibility, for it dids't seem to make much sense at first.\n\n> * you replace merge-turned-ordinary commit with a proper merge\n>   commit\n\nAah...  and this would presumably enable further merges onto my local\ntree, but I'd public commits that lost history and relationship with\ntheir upstream commits.\n\nI'm aiming at something better than this, something more like the result\nof filter-branch, but with improvements for git pull/merge that (i) use\nsome ref/original mapping (that provides nearly equivalent info to that\nof the weak parent idea I proposed before) to tell where we are, what we\nhave and what needs rewriting, and (ii) perform rewriting of each\nbrought in commit, keeping local history isomorphic to that of upstream,\nand updating the remapping.  Ideally, (iii) have means for merge to use\nthe remapping backwards, so that one could merge from the cleaned-up\nbranch to the contaminated branch, or even to publish the remapping as\nequivalences rather than unidirectional mappings.  Perhaps storing them\nas trees (or some other format) rather than as long lists of refs would\nmake them more efficient to deal with, especially after packing.\n\nMore details about what we're after in the thread containing:\nhttp://www.mail-archive.com/gnu-linux-libre@nongnu.org/msg00903.html\n\n\nAs for the rewriting itself (which I regard as a solved problem, it's\ncompatibility between rewritten branches that I'm trying to adress), I'm\nthinking of making manual changes to the trees whose commits introduced\nundesirable content, taking note of the contaminated and clean objects,\nand then writing a script to remap with git filter-branch the contents\nof the index for each commit, replacing contaminated with clean file, or\nremoving fully-contaminated file.\n\n> Though I think that better solution would be feature-branch based\n> workflow.\n\nWe are not in a position to influence how upstream does their\ndevelopment, and I suppose this would be the case in many (but not all)\nof the situations I described as motivators.\n\n\nOn Dec 30, 2010, Yann Dirson <ydirson@free.fr> wrote:\n\n>> I'm under the impression that this could not just work, but also make\n>> rebasing in general (especially the hard case) far less problematic, for\n>> git would be able to relate a rebased commit with an original commit.\n\n> I suppose that by \"hard case\" you mean forking off a branch that gets\n> rebased later ?\n\nI meant the case described as “hard case” in the git-rebase man page:\n\nhttp://www.kernel.org/pub/software/scm/git/docs/git-rebase.html\n\nHard case: The changes are not the same.\n\n    This happens if the subsystem rebase had conflicts, or used\n    --interactive to omit, edit, squash, or fixup commits; or if the\n    upstream used one of commit --amend, reset, or filter-branch.\n\n> This problem suggests a more generic one: how to \"merge back\" most\n> changes from a branch while still not merging some specific changes ?\n\nThanks for the suggestion.  That made me think that, more than a\nparent/child relationship, the original and rewritten commits should be\nperceived as siblings as far as merges are concerned, when a\ncorrespondence/equivalence table is given.  Hopefully this wouldn't be\ntoo much of a change to merge and rebase.\n\n\nAm I making sense?  Does this seem generally useful, say, for someone\ntrying to do participate in the development of unencumbered portions of\na (patent|copyright|contractually|restriction)-encumbered project?\n\n-- \nAlexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/\nYou must be the change you wish to see in the world. -- Gandhi\nBe Free! -- http://FSFLA.org/   FSF Latin America board member\nFree Software Evangelist      Red Hat Brazil Compiler Engineer\n"}]}