{"thread":{"id":"40962","subject":"publish from certain commit onward, keeping earlier history private, but provable","startedAt":"2015-12-09T13:45:44Z","lastAt":"2015-12-09T22:50:44Z","messageCount":6,"participants":["Jörn Hees","Johannes Löthberg","Jeff King","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"274206","messageId":"35583CFA-3BEE-4844-9F85-ED73A412A97F@joernhees.de","threadId":"40962","inReplyTo":null,"subject":"publish from certain commit onward, keeping earlier history private, but provable","fromName":"Jörn Hees","fromEmail":"dev@joernhees.de","sentAt":"2015-12-09T13:45:44Z","receivedAt":"2015-12-09T13:45:44Z","isPatch":false,"sender":{"key":"dev@joernhees.de","avatar":"https://gravatar.com/avatar/590cc6f9e7423070747b155451ff7227c749cc3d3621359f2f3eddce0099a8f4?d=mp&s=160"},"body":"Hi,\n\nI've been hacking away on a library for quite some time and have a lot of commits in my private repository:\n\nA -> B -> C -> D -> E\n\nFinally, I'm nearing completion of a first version, and want to publish it to a remote called public from D onward keeping A..C to myself, so public should afterwards look like this:\n\nD -> E\n\nMy main motivation is that i don't really want to put ridiculously first trials online, but still (on demand) I'd like to be able to prove how i arrived at D (think of copyright claims, etc).\n\nAs (at the moment) it's pretty much impossible to reverse-engineer the hashes of commits in the chain with times and changesets, i thought just keeping D's parent pointer to C would be one of the genius advantages of git. Sadly i can't find a way to actually make this work.\n\nCan i somehow push D -> E to public making it a fully functional public repository with all the necessary objects included to checkout D or E and D still pointing to C as parent? If not, why is that?\n\nWhat doesn't seem to work:\n\n- push with range\n  \n  git push public D..E:master\n  error: src refspec D..E does not match any.\n  error: failed to push some refs to '<public>'\n\n- any form of squashing / history rewriting\n  \n  As far as i know squashing A..D would introduce a new commit removing the parent pointer to C and thereby removing provability of the existence of A..C. (Simple example: say C reversed B, then you'd never be able to prove B was in there at some point.)\n  \n  I could obviously manually note the hash of C in the description of the squash commit, but there already is a parent pointer field, why not use it?\n  \n  Also in order to contribute further changes to public I'd have to rebase my private branches on top of this new squashed commit, which just seems as wrong...\n\n- push from local clone with limited depth\n  \n  I thought i found a solution to this by first creating a local clone local_public with the desired depth before pushing that clone to public like this:\n  \n  git clone --depth 2 file:///<abspath_private> local_public\n  \n  With\n  \n  git log --pretty=raw\n  \n  I can verify that local_public only contains D -> E and that the commit, tree and parent hashes are the same, which is exactly what i want.\n  \n  The problem is that when i try to push to an added public remote from local_public i get an error like this:\n  \n  ! [remote rejected] master -> master (shallow update not allowed)\n\n\nAny ideas how to make this work?\n\nCheers,\nJörn\n"},{"id":"274210","messageId":"20151209175431.GA18000@zorg.kyriasis.com","threadId":"40962","inReplyTo":"35583CFA-3BEE-4844-9F85-ED73A412A97F@joernhees.de","subject":"Re: publish from certain commit onward, keeping earlier history private, but provable","fromName":"Johannes Löthberg","fromEmail":"johannes@kyriasis.com","sentAt":"2015-12-09T17:54:31Z","receivedAt":"2015-12-09T17:54:31Z","isPatch":false,"sender":{"key":"johannes@kyriasis.com","avatar":"https://gravatar.com/avatar/af2dea1b1759403329a1b7eaa28a07b5c5d1901e9f55723103d6324ffb7737ae?d=mp&s=160"},"body":"On 09/12, Jörn Hees wrote:\n>Hi,\n>\n>I've been hacking away on a library for quite some time and have a lot \n>of commits in my private repository:\n>\n>A -> B -> C -> D -> E\n>\n>Finally, I'm nearing completion of a first version, and want to publish \n>it to a remote called public from D onward keeping A..C to myself, so \n>public should afterwards look like this:\n>\n>D -> E\n>\n>My main motivation is that i don't really want to put ridiculously \n>first trials online, but still (on demand) I'd like to be able to prove \n>how i arrived at D (think of copyright claims, etc).\n>\n>As (at the moment) it's pretty much impossible to reverse-engineer the \n>hashes of commits in the chain with times and changesets, i thought \n>just keeping D's parent pointer to C would be one of the genius \n>advantages of git. Sadly i can't find a way to actually make this work.\n>\n>Can i somehow push D -> E to public making it a fully functional public \n>repository with all the necessary objects included to checkout D or E \n>and D still pointing to C as parent? If not, why is that?\n>\n\nTake a look at git-replace[0][1].\n\n[0]: https://git-scm.com/2010/03/17/replace.html\n[1]: https://www.kernel.org/pub/software/scm/git/docs/git-replace.html\n\n-- \nSincerely,\n  Johannes Löthberg\n  PGP Key ID: 0x50FB9B273A9D0BB5\n  https://theos.kyriasis.com/~kyrias/\n"},{"id":"274215","messageId":"20151209222041.GB21751@sigill.intra.peff.net","threadId":"40962","inReplyTo":"35583CFA-3BEE-4844-9F85-ED73A412A97F@joernhees.de","subject":"Re: publish from certain commit onward, keeping earlier history private, but provable","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-09T22:20:41Z","receivedAt":"2015-12-09T22:20:41Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 09, 2015 at 02:45:44PM +0100, Jörn Hees wrote:\n\n> I've been hacking away on a library for quite some time and have a lot of commits in my private repository:\n> \n> A -> B -> C -> D -> E\n> \n> Finally, I'm nearing completion of a first version, and want to\n> publish it to a remote called public from D onward keeping A..C to\n> myself, so public should afterwards look like this:\n> \n> D -> E\n\nThe short answer is that you cannot do this without changing the names\n(i.e., sha1 commit ids) of D and E.\n\nOne of the fundamental assumptions git makes is that if a repository has\nan object X, it also has all of the objects reachable from it (past\ncommits, their trees, subtrees, and blobs). This is what makes the\npush/fetch object transfer efficient (one side says only \"I have X\" and\nthe other side knows \"Ah, that is a whole chunk of objects I do not have\nto bother sending\", without the names of those objects going over the\nwire).\n\nThe exception, of course, is shallow clones, where one side tells the\nother \"I am shallow at cutoff point Y; don't assume I have anything\nbelow there\". This does work, but there are some downsides (for\ninstance, we cannot apply some of the same reachability optimizations\nfor serving fetches).\n\n>   I can verify that local_public only contains D -> E and that the\n>   commit, tree and parent hashes are the same, which is exactly what i\n>   want.\n>   \n>   The problem is that when i try to push to an added public remote\n>   from local_public i get an error like this:\n>   \n>   ! [remote rejected] master -> master (shallow update not allowed)\n\nRight. The receiver must be explicitly configured to accept a shallow\npush (I do not recall offhand whether clients fetching from you would\nalso need an explicit config to accept a shallow history).\n\nSo the usual path here is to rewrite D and E (with the same trees, but\nthey will get new commit ids). If you want to retain the older history\n(commits A-C), you can distribute it separately and use git-replace to\n\"graft\" it onto the newer history at run-time.\n\nYou can do that with:\n\n  # set up a run-time replacement view so that D appears to have\n  # no parents; this doesn't impact the objects themselves, but\n  # rather git will use our parent-less \"replacement\" D anytime\n  # somebody mentions the original\n  git replace --graft D\n\n  # verify that the history is what you want; if you have a non-linear\n  # history you may have to make several such \"cuts\" in the graph\n  git log\n\n  # now cement it into place by rewriting\n  git filter-branch\n\nOf course that is a bitter pill to swallow if you have reasons for\nwanting to use the old sha1s. E.g., you have internal development\nproceeding against the old tree and want to share a truncated version\nwith the public.  In that case I still think the least painful thing is\nto rewrite the truncated history, have _everyone_, internal and public\nwork against that, and let internal folks graft the old history on for\ntheir own use. They can do that with:\n\n  git replace --graft the-rewritten-D the-original-C\n\n-Peff\n"},{"id":"274216","messageId":"20151209222412.GC21751@sigill.intra.peff.net","threadId":"40962","inReplyTo":"20151209222041.GB21751@sigill.intra.peff.net","subject":"Re: publish from certain commit onward, keeping earlier history private, but provable","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-09T22:24:12Z","receivedAt":"2015-12-09T22:24:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 09, 2015 at 05:20:41PM -0500, Jeff King wrote:\n\n> Of course that is a bitter pill to swallow if you have reasons for\n> wanting to use the old sha1s. E.g., you have internal development\n> proceeding against the old tree and want to share a truncated version\n> with the public.\n\nAfter re-reading your email, it looks like your use case is just to be\nable to later prove the existence of the original history. You could\nthat by mentioning the original \"C\" in your truncated \"D\", but in a way\nthat git does not traverse reachability. For instance, amend D's commit\nmessage to say:\n\n  This is based on earlier, unpublished work going up to commit C.\n\nThen retain C for yourself, and show it only to those you want to prove\nits contents to.\n\n-Peff\n"},{"id":"274218","messageId":"CAGZ79kYv0qJ==n3TCxTeNkenzKn5msRtR1jAiKEy1mECwUaAuA@mail.gmail.com","threadId":"40962","inReplyTo":"20151209222412.GC21751@sigill.intra.peff.net","subject":"Re: publish from certain commit onward, keeping earlier history private, but provable","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-12-09T22:29:12Z","receivedAt":"2015-12-09T22:29:12Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Dec 9, 2015 at 2:24 PM, Jeff King <peff@peff.net> wrote:\n> On Wed, Dec 09, 2015 at 05:20:41PM -0500, Jeff King wrote:\n>\n>> Of course that is a bitter pill to swallow if you have reasons for\n>> wanting to use the old sha1s. E.g., you have internal development\n>> proceeding against the old tree and want to share a truncated version\n>> with the public.\n>\n> After re-reading your email, it looks like your use case is just to be\n> able to later prove the existence of the original history. You could\n> that by mentioning the original \"C\" in your truncated \"D\", but in a way\n> that git does not traverse reachability. For instance, amend D's commit\n> message to say:\n>\n>   This is based on earlier, unpublished work going up to commit C.\n>\n> Then retain C for yourself, and show it only to those you want to prove\n> its contents to.\n\nI'd rather keep D for yourself and create a D' which is D just without\nparent and\nthe note above, such that the tree of D and parts of the commit message\nis obvious by looking at D'. All that is secret is Ds parent and the commit\ninformation such as exact date. (committer could be guessed easily)\n\n>\n> -Peff\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"274221","messageId":"20151209225043.GC32104@sigill.intra.peff.net","threadId":"40962","inReplyTo":"CAGZ79kYv0qJ==n3TCxTeNkenzKn5msRtR1jAiKEy1mECwUaAuA@mail.gmail.com","subject":"Re: publish from certain commit onward, keeping earlier history private, but provable","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-09T22:50:44Z","receivedAt":"2015-12-09T22:50:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 09, 2015 at 02:29:12PM -0800, Stefan Beller wrote:\n\n> On Wed, Dec 9, 2015 at 2:24 PM, Jeff King <peff@peff.net> wrote:\n> > On Wed, Dec 09, 2015 at 05:20:41PM -0500, Jeff King wrote:\n> >\n> >> Of course that is a bitter pill to swallow if you have reasons for\n> >> wanting to use the old sha1s. E.g., you have internal development\n> >> proceeding against the old tree and want to share a truncated version\n> >> with the public.\n> >\n> > After re-reading your email, it looks like your use case is just to be\n> > able to later prove the existence of the original history. You could\n> > that by mentioning the original \"C\" in your truncated \"D\", but in a way\n> > that git does not traverse reachability. For instance, amend D's commit\n> > message to say:\n> >\n> >   This is based on earlier, unpublished work going up to commit C.\n> >\n> > Then retain C for yourself, and show it only to those you want to prove\n> > its contents to.\n> \n> I'd rather keep D for yourself and create a D' which is D just without\n> parent and\n> the note above, such that the tree of D and parts of the commit message\n> is obvious by looking at D'. All that is secret is Ds parent and the commit\n> information such as exact date. (committer could be guessed easily)\n\nI think the point is that all of this is happening at time t (let's say\n2015), and the proof may be needed at time t+N (let's say 2020).\n\nShowing the original D (or C, or whatever) at that point proves nothing,\nas you could have created a fake history in 2020 that \"ends up\" at the\nD' tree. You need to publish _something_ in 2015 that says \"I know this\nthing, but I am not willing to show it to you yet\".\n\nThe classic way of doing this is to take out a small ad in the\nclassified section of a print newspaper with a hash of your data.\nLibraries keep archives of the paper, so later you can prove that you\nhave the data that matches the hash, and its timestamp is certified by\nthe library archives.\n\nHere we're abusing Git as the notary. If everyone spends the years from\n2015-2020 building on top of D', then they can all reasonably agree that\nthe content of D' was written in 2015, and any commit hash it mentions\nhad to have existed then. Revealing C (or the original D, or whatever\nhash you want to mention) proves the data.\n\n-Peff\n"}]}