Re: rebase parents, or tracking upstream but removing non-distributable bits

From: Yann Dirson <ydirson@free.fr>
Date: Dec 30, 2010, 22:52 UTC
Message-ID: <20101230225234.GC6639@home.lan>
In-Reply-To: <ord3ojb0yy.fsf@livre.localdomain>

On Thu, Dec 30, 2010 at 03:54:29PM -0200, Alexandre Oliva wrote:

> Given this problem statement, I started looking for solutions that
> didn't require modifying git.

This is a problem I have come to think a bit about already. Although I do not have a proper solution either, let's share those ideas.

Show 12 quoted lines

> I first looked into rewriting history, removing the unwanted bits and
> replaying subsequent changes, but quickly discarded it, for it would
> make my local repository incompatible with upstream both ways: I
> wouldn't be able to pull from it; upstream wouldn't be able to pull to
> it; third parties would run into ugly situations trying to carry patches
> from either one to the other.
> 
> Now, it looks like I might be able to pull from upstream if I maintain
> manually a graft file that named each upstream commit as an additional
> parent of the corresponding local rebase commit that brought it into my
> rewritten tree.  Workable, maybe, but this wouldn't help third parties
> that used my public repository.

As a side note: I fear grafts won't scale very nicely performance-wise if you graft every commit.

My use-case was similar to yours, about "how could Debian distribute a git tree of the kernel ?", but my focus was on distributing well-defined snapshots, and I ended up with the idea of grafting only the set of successive revisions that get published - and not necessarily have a mirror of all upstream revisions in the filtered tree: such grafts do represent points in history where we would merge from upstream, and without the grafts the filtered tree would look like successive imports of well-defined revisions (notably makes it realistic to adhere to the "all commits have been tested" paradigm).

> Besides, I'm concerned that pushing from the local repository (with the
> graft file) to the public repository would end up publishing the changes
> I'm not supposed to distribute, because they'd be taken as parents of
> the local commits.

Grafts cannot be pushed/pulled. For this, as Jonathan suggests in another reply, "git replace" looks like a better choice, esp. in that by default the replacement commits don't get pulled: you need to explicitely request fetching refs/replace/commits (note: you may need to handle merges there when needed). OTOH, those replacement commits would pull the whole upstream history, so they cannot be part of the filtered repository: it must come from an unfiltered kernel repo, that may be a real problem if you cannot redistribute some upstream parts.

OTOH, a grafts file can be distributed out-of-band, and would only pull the problematic contents when put in place, so it might indeed be more adequate.

Show 7 quoted lines

> Are there any other ways to support the desired features with git as-is?
> AFAICT, there isn't, so I've been thinking of how to introduce this.  I
> suppose the simplest way to accomplish this is to introduce the notion
> of a ???weak parent???: one that is taken into account for purposes of
> checking whether a commit is present in a branch being merged- or
> rebased into, but that is not transmitted over pushes, and that is not
> retained over purges, and not complained about when missing.

That sounds like heavy surgery with too many implications I can think of.

When it comes to "modifying commit metadata", it rings "git notes" in my ear - but then, you cannot add/change commit metadata from a note, maybe that could be a better direction to dig into, so we can add parents when a specific notes namespace is activated ? But then we would get back to the problem I mentionned for refs/replace/

> I'm under the impression that this could not just work, but also make
> rebasing in general (especially the hard case) far less problematic, for
> git would be able to relate a rebased commit with an original commit.

I suppose that by "hard case" you mean forking off a branch that gets rebased later ? Note that "git pull" seems to be able to cope with this using reflogs already (although I did not test that feature). A less volatile place than reflogs could again using notes, without a need for core changes.

> I'm not sure this would enable upstream to easily integrate
> my changes.  Or would it?

This problem suggests a more generic one: how to "merge back" most changes from a branch while still not merging some specific changes ? It would also help when a maint branch gets some short-term workarounds that we don't want in the master branch, but the very idea has a serious flaw: it implies that the "merge back" commit contains also the commits we don't want (here, the "filtering commits"). So I guess cherry-pick will be the way here. Anyway, I doubt Linus would like the idea of merging from such a filtered repo, sending patches is probably prefered.

Best regards,

-- 
Yann