{"thread":{"id":"6525","subject":"grafts+repack+prune = history at danger","startedAt":"2007-01-25T17:17:18Z","lastAt":"2007-01-27T00:56:41Z","messageCount":15,"participants":["Johannes Sixt","Junio C Hamano","Mark Wooding","Jakub Narebski","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"32625","messageId":"45B8E61E.C9C5E6C6@eudaptics.com","threadId":"6525","inReplyTo":null,"subject":"grafts+repack+prune = history at danger","fromName":"Johannes Sixt","fromEmail":"j.sixt@eudaptics.com","sentAt":"2007-01-25T17:17:18Z","receivedAt":"2007-01-25T17:17:18Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Isn't there a major hole in the logic how repack works when grafts are\nin effect?\n\nI did this (details follow):\n\n1. specify grafts\n2. repack\n3. prune\n4. clone\n\nResult: Broken history in the clone; info/grafts was not copied.\nThis is with git version 1.5.0.rc2.g18af.\n\n\n1. I imported a cvs repository into git and \"fixed\" the history using\ngrafts. In particular:\n\n o--B--X <== this commit is should be skipped\n \\ \\\ngraft => ---A--o\n\nI specified in .git/info/grafts that the parent of A should be B. Of\ncourse, commit A has still recorded X as its parent.\n\n2. Then I repacked the repo. But this did not erase all objects:\n\n$ git repack -a -d\n$ git count-objects -v\ncount: 5\nsize: 28\nin-pack: 3392\npacks: 1\nprune-packable: 0\ngarbage: 0\n$ git fsck-objects\ndangling commit bb828bfbd213a97817a95506bab4eeaa70538e2e\n\nThis commit bb828... is X.\n\n3. Now git prune happily removes the 5 objects.\n\n4. 'git clone First Second' clones the repository without problems.\n\nBut now in the clone the history is kaputt. Because commit X is not in\nthe cloned pack. Nor is there any info/grafts file. The original history\nis still OK as long as the info/grafts file is present; but if it is\nremoved, the original repo is also damaged.\n\nIMHO, this is a very serious issue. I think that repack should not walk\nthe grafted history. Alternatively, the info/grafts file must be copied\nby the clone and respected by fsck-objects.\n\n-- Hannes\n"},{"id":"32657","messageId":"7vireu7lj0.fsf@assigned-by-dhcp.cox.net","threadId":"6525","inReplyTo":"45B8E61E.C9C5E6C6@eudaptics.com","subject":"Re: grafts+repack+prune = history at danger","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-25T23:07:47Z","receivedAt":"2007-01-25T23:07:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Isn't there a major hole in the logic how repack works when grafts are\n> in effect?\n>\n> I did this (details follow):\n>\n> 1. specify grafts\n> 2. repack\n> 3. prune\n> 4. clone\n>\n> Result: Broken history in the clone; info/grafts was not copied.\n\nThat is expected.\n\nIf you had problem in the original repository (i.e. the one with\ngrafts) that lost objects after step 3., that would be serious\nand needs to be fixed, but otherwise the rule of thumb has\nalways been not to expose repositories with grafts without\ntelling unsuspecting downstream people for cloning or fetching.\nIt will give objects they did not even ask for.\n\ngrafts are local matter for archaeologist's convenience to glue\ntwo independent histories together, and not much more. For\nexample, the history that starts at v2.6.12-rc2 can be grafted\non top of old bkcvs history, but people who clone from you may\nnot expect to get anything beyond the true origin of the history\nat v2.6.12-rc2 (after all that commit object records it as a\nparentless commit).\n\nI suspect you could extend fetch-pack protocol to give existing\ngrafts from upload-pack to trivially fix 'clone', but I do not\nknow offhand what the ramifications of it are for normal\n'fetch'. You would need to merge potentially conflicting graft\ninformation you obtained from where you fetched from and what\nyou already had before starting to fetch.\n"},{"id":"32691","messageId":"45B9B80E.E2534F97@eudaptics.com","threadId":"6525","inReplyTo":"7vireu7lj0.fsf@assigned-by-dhcp.cox.net","subject":"Re: grafts+repack+prune = history at danger","fromName":"Johannes Sixt","fromEmail":"j.sixt@eudaptics.com","sentAt":"2007-01-26T08:13:02Z","receivedAt":"2007-01-26T08:13:02Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Junio C Hamano wrote:\n> \n> Johannes Sixt writes:\n> \n> > Isn't there a major hole in the logic how repack works when grafts are\n> > in effect?\n> >\n> > I did this (details follow):\n> >\n> > 1. specify grafts\n> > 2. repack\n> > 3. prune\n> > 4. clone\n> >\n> > Result: Broken history in the clone; info/grafts was not copied.\n> \n> That is expected.\n> \n> If you had problem in the original repository (i.e. the one with\n> grafts) that lost objects after step 3., that would be serious\n> and needs to be fixed,\n\nOh, the original repo *does* loose the object after step 3, but you\nwould not notice it until you remove the grafts file.\n\n> grafts are local matter for archaeologist's convenience to glue\n> two independent histories together, and not much more.\n\nAgreed. Then grafts must be disregarded by (almost) all plumbing, most\nnotably fsck-objects, prune, pack-objects, but also\n{fetch,upload,send,receive}-pack. They should be obeyed only by the log\nand diff families and certainly also rev-list on request.\n\n-- Hannes\n"},{"id":"32694","messageId":"7vr6ti183o.fsf@assigned-by-dhcp.cox.net","threadId":"6525","inReplyTo":"45B9B80E.E2534F97@eudaptics.com","subject":"Re: grafts+repack+prune = history at danger","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-26T08:54:19Z","receivedAt":"2007-01-26T08:54:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Oh, the original repo *does* loose the object after step 3, but you\n> would not notice it until you remove the grafts file.\n\nThat is Ok -- you did want to lose that object B because you\ntold git you want to pretend A's parent is not B in _your_ local\nrepository.\n\n>> grafts are local matter for archaeologist's convenience to glue\n>> two independent histories together, and not much more.\n>\n> Agreed. Then grafts must be disregarded by (almost) all plumbing, most\n> notably fsck-objects, prune, pack-objects,...\n\nYou are not agreeing.\n\nGraft is a local matter, but that does not mean it should\nintroduce inconsistencies. It is a way to _locally_ change the\nworld view, and to give the consistent world view locally, not\nonly the commands you listed (fsck, prune, pack-objects) but\nalso log, rev-list and friends all should take grafts into\naccount, which is why losing B is the right thing to do if you\nrepack or prune. In your altered world, B is not part of any\nremaining history.\n\nThe problem you noticed is a limitation of fetch/clone.\nExposing the locally modified world view to the other end so\nthat a cloned repository has the exactly the same view by\ncopying the grafts file would be trivial [*1*].\n\nHowever, it is rather tricky if you try to extend it to fetching\ninto an existing repository. Which may have its own grafts and\ndefine an altered world view in its own way. And that altered\nworld view may conflict (e.g. it may already say the parent of A\nis not B but not X as in the repository you are cloning from but\nsome other commit Y).\n\nThat's why traditionally we just punt the whole issue by saying\ndon't exchange objects between repositories that have grafts\nwithout thinking (primarily because we haven't thought things\nthrough -- we are lazy bastards).\n\nOne thing you could do is to take the local-ness of grafts more\nliterally and enforce it more strictly by dropping grafts while\nfetch-pack and receive-pack exchange common objects and spawn\npack-objects to come up with objects needed to be sent. But\nbecause we currently punt, we do not even do that.\n\nIf we were to spend the effort to do that temporary dropping of\ngrafts (which I would expect to be quite ugly code), I suspect\nwe are better off thinking things through to define the desired\nsemantics, what should happen when objects are exchanged between\ntwo repositories that have their world views altered with their\ngrafts. The end result would most likely update the info/grafts\nin your repository when you fetch from a repository with grafts,\nand probably update info/grafts at the remote when you push from\na repository with grafts.\n\n[*1*] It does require fetch-pack protocol update, though, so it\nis some work. It is still trivial in the sense that it is clear\nwhat is needed to realize exactly the same the world view -- the\ncopy should have the exact copy of info/grafts file.\n"},{"id":"32701","messageId":"slrnerjhlu.7v0.mdw@metalzone.distorted.org.uk","threadId":"6525","inReplyTo":"7vireu7lj0.fsf@assigned-by-dhcp.cox.net","subject":"Re: grafts+repack+prune = history at danger","fromName":"Mark Wooding","fromEmail":"mdw@distorted.org.uk","sentAt":"2007-01-26T09:15:42Z","receivedAt":"2007-01-26T09:15:42Z","isPatch":false,"sender":{"key":"mdw@distorted.org.uk","avatar":null},"body":"Junio C Hamano wrote:\n\n> grafts are local matter for archaeologist's convenience to glue\n> two independent histories together, and not much more. \n\nI've found them useful for doing imports from CVS. You run\ngit-cvsimport, and then manually find the places where merges happened\nand record them as grafts. gitk then correctly displays the history,\nwhich is nice.\n\nWhat you then do is run cg-admin-rewritehist, which magically transforms\nthe history into one with the grafts etched in. All that remains is to\ntranslate the refs, which you can do with a sed script you got\ncg-admin-rewritehist to write for you.\n\n-- [mdw]\n"},{"id":"32702","messageId":"45B9C836.728F31EC@eudaptics.com","threadId":"6525","inReplyTo":"7vr6ti183o.fsf@assigned-by-dhcp.cox.net","subject":"Re: grafts+repack+prune = history at danger","fromName":"Johannes Sixt","fromEmail":"j.sixt@eudaptics.com","sentAt":"2007-01-26T09:21:58Z","receivedAt":"2007-01-26T09:21:58Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Junio C Hamano wrote:\n> Graft is a local matter, but that does not mean it should\n> introduce inconsistencies. It is a way to _locally_ change the\n> world view, and to give the consistent world view locally, not\n> only the commands you listed (fsck, prune, pack-objects) but\n> also log, rev-list and friends all should take grafts into\n> account, which is why losing B is the right thing to do if you\n> repack or prune. In your altered world, B is not part of any\n> remaining history.\n\nHere's my stance on it. Grafts should be a local matter. And they alter\nthe world view, with a pronounciation on *view*. That's why I proposed\nthat only log familiy of commands obey them[*]. And probably rev-list so\nthat gitk et.al. have a way to obey them. And also the ref parser (so\nthat master~20 is what it looks it is). Everything else should disregard\ngrafts: repack, prune, fetch, -pack, push etc. No nasty side\neffects anymore. No transfer of the grafts file needed. No clash when\nsomeone else has a different *view* of the world.\n\nThen the location of the file in .git/info/grafts is justified. If\ngrafts continue to have the radical influence that they have now, then\nthe grafts file is better located in .git/objects/info/grafts as part of\nthe objects database.\n\n[*] ok, I originally also proposed the diff family, but that's likely\nnot necessary.\n\n-- Hannes\n"},{"id":"32704","messageId":"7vzm86yw0q.fsf@assigned-by-dhcp.cox.net","threadId":"6525","inReplyTo":"45B9C836.728F31EC@eudaptics.com","subject":"Re: grafts+repack+prune = history at danger","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-26T09:31:17Z","receivedAt":"2007-01-26T09:31:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Here's my stance on it. Grafts should be a local matter. And they alter\n> the world view, with a pronounciation on *view*. That's why I proposed\n> that only log familiy of commands obey them[*]. And probably rev-list so\n> that gitk et.al. have a way to obey them. And also the ref parser (so\n> that master~20 is what it looks it is). Everything else should disregard\n> grafts: repack, prune, fetch, -pack, push etc. No nasty side\n> effects anymore.\n\nI said you are not agreeing, but I should have said you are not\nunderstanding.\n\ngrafts can bring otherwise disconnected commits into the\naltered history, so if you want your log to honor grafts, your\nprune and repack need to be aware of them lest you would not\nlose them.\n"},{"id":"32708","messageId":"45B9CE56.D16DFC81@eudaptics.com","threadId":"6525","inReplyTo":"7vzm86yw0q.fsf@assigned-by-dhcp.cox.net","subject":"Re: grafts+repack+prune = history at danger","fromName":"Johannes Sixt","fromEmail":"j.sixt@eudaptics.com","sentAt":"2007-01-26T09:48:06Z","receivedAt":"2007-01-26T09:48:06Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Junio C Hamano wrote:\n> \n> Johannes Sixt writes:\n> \n> > Here's my stance on it. Grafts should be a local matter. And they alter\n> > the world view, with a pronounciation on *view*. That's why I proposed\n> > that only log familiy of commands obey them[*]. And probably rev-list so\n> > that gitk et.al. have a way to obey them. And also the ref parser (so\n> > that master~20 is what it looks it is). Everything else should disregard\n> > grafts: repack, prune, fetch, -pack, push etc. No nasty side\n> > effects anymore.\n> \n> I said you are not agreeing, but I should have said you are not\n> understanding.\n\nOh, I think I understand very well. It may just be that I cannot express\nmyself that well ;)\n\nI propose that grafts are only about *view*, not database integrity.\n\nThere are no tools that manipulate grafts, that would stop the user to\nmake some blunder; the user has to edit the file *manually*. It is\nwrong, wrong, wrong to let such a file dictate database integrity.\n\n> grafts can bring otherwise disconnected commits into the\n> altered history, so if you want your log to honor grafts, your\n> prune and repack need to be aware of them lest you would not\n> lose them.\n\nSure, if I connect my linux repo with a graft to the historical BK tree,\nthen toss the ref that pointed to the historical tree, then git prune:\n- then currently it won't prune the historical tree\n- but under my proposal it would. Silly me. Why did I remove that ref?\n\n-- Hannes\n"},{"id":"32712","messageId":"7vmz46ytyy.fsf@assigned-by-dhcp.cox.net","threadId":"6525","inReplyTo":"45B9CE56.D16DFC81@eudaptics.com","subject":"Re: grafts+repack+prune = history at danger","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-26T10:15:33Z","receivedAt":"2007-01-26T10:15:33Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Sure, if I connect my linux repo with a graft to the historical BK tree,\n> then toss the ref that pointed to the historical tree, then git prune:\n> - then currently it won't prune the historical tree\n> - but under my proposal it would. Silly me. Why did I remove that ref?\n\nIf your version of git worked the way you describe, the only\nreason you removed that ref would be to make sure that your\naltered view would be destroyed (you won't be able to do \"git\nlog\" across that graft boundary anymore). Indeed that would be\na silly thing to do.\n\nThankfully, the real git does not behave that way. That is why\nfsck/prune _must_ honor grafts. That makes the locally altered\nview consistent. To the altered world view, what are stored in\nthe object database do not change, but your view of how they are\nconnected does. And if your altered view thinks commit\nv2.6.12-rc2 has one of the commits in the bkcvs history as its\nparent, you do not want to lose that history merely because you\nlost a ref to it -- as long as the commit tagged as v2.6.12-rc2\nis reachable, its (imaginary) parent should be as well.\n\nIf you want to switch out of an altered universe, you may need\nto do more than just remove grafts (objects that were hidden by\ngrafts were immaterial in the altered universe, but now you may\nneed to get them back, as in your fixed-up imported repository\nexample, and objects that existed only because grafts pulled\nthem in are now made unreachable and become prunable), but that\ngoes without saying.\n\nIf you want to make it easier to switch back and forth between\naltered reality and the real world, fsck/prune/repack may need\nto be taught to consider both real and grafted parents to be\nconnected, so that you do not have to lose objects that will\nbecome necessary when you come out of the altered world, but I\nam not sure if it is worth it. If you prune while in the real\nworld the \"both real and grafted\" safety would obviously not\nkick in when you run prune, so when you reinstall the grafts,\nsome of the necessary objects would be already gone.\n"},{"id":"32717","messageId":"45B9DAC8.C04C6D3F@eudaptics.com","threadId":"6525","inReplyTo":"7vmz46ytyy.fsf@assigned-by-dhcp.cox.net","subject":"Re: grafts+repack+prune = history at danger","fromName":"Johannes Sixt","fromEmail":"j.sixt@eudaptics.com","sentAt":"2007-01-26T10:41:12Z","receivedAt":"2007-01-26T10:41:12Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Junio C Hamano wrote:\n> \n> Johannes Sixt writes:\n> \n> > Sure, if I connect my linux repo with a graft to the historical BK tree,\n> > then toss the ref that pointed to the historical tree, then git prune:\n> > - then currently it won't prune the historical tree\n> > - but under my proposal it would. Silly me. Why did I remove that ref?\n> \n> [...]\n> \n> Thankfully, the real git does not behave that way. That is why\n> fsck/prune _must_ honor grafts. That makes the locally altered\n> view consistent. To the altered world view, what are stored in\n> the object database do not change, but your view of how they are\n> connected does. And if your altered view thinks commit\n> v2.6.12-rc2 has one of the commits in the bkcvs history as its\n> parent, you do not want to lose that history merely because you\n> lost a ref to it -- as long as the commit tagged as v2.6.12-rc2\n> is reachable, its (imaginary) parent should be as well.\n\n>From your argument I deduce that grafts are a very important thing (once\nthey exist in a repo). But the current implementation does not honor\nthis:\n\n- the grafts file is not part of the objects database\n- it is manipulated manually instead of by tools the check for errors\n- it is not transferred across clones/pulls/pushes (it's even possible\nto create an inconsistent clone)\n\nThe way out that I see is to make grafts much, much less important.\nNamely that they are obeyed _only_ by tools that _present_ the database\ncontents. All manipulators must disregard grafts.\n\nConsequently, if I install grafts, I must make sure that I don't prune\naway objects that the grafted history needs (i.e. avoid the silliness\nmentioned above). If I happen to make the grafted history inconsistent,\nI can make it consistent again by removing the grafts file (it was a\nlocal thingy anyway) - no harm done - just the _presentation_ was\naltered.\n\n-- Hannes\n"},{"id":"32722","messageId":"7vodomxbz0.fsf@assigned-by-dhcp.cox.net","threadId":"6525","inReplyTo":"45B9DAC8.C04C6D3F@eudaptics.com","subject":"Re: grafts+repack+prune = history at danger","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-26T11:29:39Z","receivedAt":"2007-01-26T11:29:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> - the grafts file is not part of the objects database\n\nThis is a very conscious design decision from an ancient times.\nIt used to be fashionable to share object store across different\nrepositories (you literally symlinked .git/objects), and grafts\nare local in the sense that they are per-repository, and that is\nthe reason it lives in .git/info. There is not much reason\neither way and if I were doing this from scratch I would\nprobably place it in .git/objects/info next to alternates.\n\n> - it is manipulated manually instead of by tools the check for errors\n\nYes, but that is only because nobody saw need for such a tool so\nfar. In reality, grafts have been pretty much \"install and\nforget\" thing. You graft 2.6.12-rc2 on top of the bkcvs tip\nonce, and then do not think about it after doing so.\n\nWhen somebody sees a need, you know what will happen ;-).\n\n> - it is not transferred across clones/pulls/pushes (it's even possible\n> to create an inconsistent clone)\n\nYes, as I already said that is where we punted and declared that\nthe grafts are local matter.\n\nEven though your resulting clone is inconsistent, I do not even\nhave to say \"tough\". You can just tell what the necessary graft\nfile should look like to the repository owner at the other end,\nand the life will be peachy again.\n\nI even outlined the issues you (or somebody else who may be\ninterested) would need to look into to make it more global. Do\nyou need anything more?\n\n> The way out that I see is to make grafts much, much less important.\n\nBreaking what already works does not sound like a way out. \n\nFor local-only, \"install and forget\" use, what the current setup\ndoes is consistent and works reasonably well. I would not say\nit is perfect, but I do not know of any outstanding bugs (and\nwhat you mentioned in these message are certainly not).\n"},{"id":"32733","messageId":"epcue5$vrv$3@sea.gmane.org","threadId":"6525","inReplyTo":"45B9C836.728F31EC@eudaptics.com","subject":"Re: grafts+repack+prune = history at danger","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-01-26T13:08:27Z","receivedAt":"2007-01-26T13:08:27Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[Cc: git@vger.kernel.org]\n\nJohannes Sixt wrote:\n\n> Junio C Hamano wrote:\n>> Graft is a local matter, but that does not mean it should\n>> introduce inconsistencies. It is a way to _locally_ change the\n>> world view, and to give the consistent world view locally, not\n>> only the commands you listed (fsck, prune, pack-objects) but\n>> also log, rev-list and friends all should take grafts into\n>> account, which is why losing B is the right thing to do if you\n>> repack or prune. In your altered world, B is not part of any\n>> remaining history.\n> \n> Here's my stance on it. Grafts should be a local matter. And they alter\n> the world view, with a pronounciation on *view*. That's why I proposed\n> that only log familiy of commands obey them[*]. And probably rev-list so\n> that gitk et.al. have a way to obey them. And also the ref parser (so\n> that master~20 is what it looks it is). Everything else should disregard\n> grafts: repack, prune, fetch, -pack, push etc. No nasty side\n> effects anymore. No transfer of the grafts file needed. No clash when\n> someone else has a different *view* of the world.\n\nIf I remember correctly there was some time ago discussion about this\ntopic, namely should connectivity (including prune, repack, etc.) take\nonly true parents, only grafts (local view), or both. IIRC there were\nno conclusion (besides perhaps that the option to choose should be\nconfigurable), and no code.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"32747","messageId":"Pine.LNX.4.64.0701260747110.25027@woody.linux-foundation.org","threadId":"6525","inReplyTo":"7vr6ti183o.fsf@assigned-by-dhcp.cox.net","subject":"Re: grafts+repack+prune = history at danger","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-01-26T15:55:49Z","receivedAt":"2007-01-26T15:55:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 26 Jan 2007, Junio C Hamano wrote:\n> \n> One thing you could do is to take the local-ness of grafts more\n> literally and enforce it more strictly by dropping grafts while\n> fetch-pack and receive-pack exchange common objects and spawn\n> pack-objects to come up with objects needed to be sent. But\n> because we currently punt, we do not even do that.\n\nOne option might be:\n\n - add a global flag (like the current \"save_commit_buffer\") that commands \n can set to specify whether they want to honor grafts or not.\n\n The \"please_follow_grafts\" flag defaults to 1.\n\n - \"git send-pack\" would explicitly set it to zero, and thus we'd always \n send a non-grafted result.\n\n - \"git prune\" would *also* explicitly set it to zero, but would also \n manually look at the grafts file, and mark anything that is set in the \n grafts file as being reachable (the same way it does for index entries \n etc).\n\nIt might also be an option to then do:\n\n - \"git repack\" should probably also set it to zero - I think we might be \n better off packing any grafted data separately.\n\nThe alternative, of course, is to try to transfer the grafts file for \nclones and fetches, but that is likely to be a *bad* idea. It's even a \npotential security issue: grafts can literally be used to short-circuit \nsome of the inherent safety in git, in that an attacker can make a graft \nthat makes history *look* fine, but hide part of it (you can't \"really\" \nhide history, but you can make normal git operations like \"git log\" \nbasically ignore it by judicious use of grafts).\n\n\t\t\tLinus\n"},{"id":"32784","messageId":"7vtzydtkqu.fsf@assigned-by-dhcp.cox.net","threadId":"6525","inReplyTo":"Pine.LNX.4.64.0701260747110.25027@woody.linux-foundation.org","subject":"Re: grafts+repack+prune = history at danger","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-26T23:46:01Z","receivedAt":"2007-01-26T23:46:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Fri, 26 Jan 2007, Junio C Hamano wrote:\n>> \n>> One thing you could do is to take the local-ness of grafts more\n>> literally and enforce it more strictly by dropping grafts while\n>> fetch-pack and receive-pack exchange common objects and spawn\n>> pack-objects to come up with objects needed to be sent. But\n>> because we currently punt, we do not even do that.\n>\n> One option might be:\n>\n> - add a global flag (like the current \"save_commit_buffer\") that commands \n> can set to specify whether they want to honor grafts or not.\n>\n> The \"please_follow_grafts\" flag defaults to 1.\n>\n> - \"git send-pack\" would explicitly set it to zero, and thus we'd always \n> send a non-grafted result.\n>\n> - \"git prune\" would *also* explicitly set it to zero, but would also \n> manually look at the grafts file, and mark anything that is set in the \n> grafts file as being reachable (the same way it does for index entries \n> etc).\n\nI am not sure why your \"git prune\" one does that, but will think\nabout it for some time first before I ask you to waste your time\nexplaining it me.\n\n> It might also be an option to then do:\n>\n> - \"git repack\" should probably also set it to zero - I think we might be \n> better off packing any grafted data separately.\n>\n> The alternative, of course, is to try to transfer the grafts file for \n> clones and fetches, but that is likely to be a *bad* idea. It's even a \n> potential security issue: grafts can literally be used to short-circuit \n> some of the inherent safety in git, in that an attacker can make a graft \n> that makes history *look* fine, but hide part of it (you can't \"really\" \n> hide history, but you can make normal git operations like \"git log\" \n> basically ignore it by judicious use of grafts).\n\nI agree that transferring, potentially merging, and\nautomatically installing grafts upon fetch has security\nimplications. Thanks for pointing it out [*1*].\n\nBut if you are cloning, it would be handy if send-pack followed\nthe altered world view and the result had identical grafts,\nwhich is why I am not 100% convinced about send-pack always\nsending a non-grafted result.\n\n\n[Footnote]\n\n*1* Running \"git fetch some-random-url\" is supposed to be a safe\noperation. The only thing it does is to download some objects\nthat are only reachable from .git/FETCH_HEAD, and it never\noverwrites objects that existed in your repository before the\nfetch, so after looking at what .git/FETCH_HEAD has, potentially\nmalicious contents will become cruft and you can gc them away.\n\nRunning \"git pull some-random-url ref\" (without storing refspec)\nto merge and then running \"git reset --hard ORIG_HEAD\" also is,\nexcept that the reflog entry for the current branch would refer\nto the merge commit and you could inject bad objects that will\nnot be immediately pruned in your object database that way.\n\nThe moral of the story is you should not pull from suspicious\nsource without thinking; fetching and immediately discarding\nshould always be safe.\n"},{"id":"32789","messageId":"Pine.LNX.4.64.0701261652440.25027@woody.linux-foundation.org","threadId":"6525","inReplyTo":"7vtzydtkqu.fsf@assigned-by-dhcp.cox.net","subject":"Re: grafts+repack+prune = history at danger","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-01-27T00:56:41Z","receivedAt":"2007-01-27T00:56:41Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 26 Jan 2007, Junio C Hamano wrote:\n\n> Linus Torvalds writes:\n> >\n> > - \"git prune\" would *also* explicitly set it to zero, but would also \n> > manually look at the grafts file, and mark anything that is set in the \n> > grafts file as being reachable (the same way it does for index entries \n> > etc).\n> \n> I am not sure why your \"git prune\" one does that, but will think\n> about it for some time first before I ask you to waste your time\n> explaining it me.\n\nSimple: the grafts may actually _hide_ history too - not just add it.\n\nSure, commonly, a graft is used to graft two complete trees together (eg, \nyou'd graft the old Linux history into the new Linux history tree). \n\nHowever, they _can_ also be used to \"fix\" history - say that you had one \ntree that has a rough history (with all the releases, but not the full \nhistory between them), and another \"fine-grained\" historical tree. You \ncould use a graft to replace the rough history version with the \nfine-grained one, so the graft may actually hide stuff that is there in \nthe rough history.\n\nSo in this case, we wouldn't necessarily want to prune stuff that \n\"exists\", but is hidden by a graft. So in my suggestion, pruning would \nbasically only use the grafts file to *add* refs, never to hide them.\n\n\t\tLinus\n"}]}