{"thread":{"id":"13137","subject":"Re: Re: Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","startedAt":"2008-04-16T11:48:13Z","lastAt":"2008-04-17T06:18:39Z","messageCount":10,"participants":["Brian Foster","Dmitry Potapov","Johannes Sixt","Petr Baudis","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"74540","messageId":"a537dd660804160448w5ba28f13m8c081ebc2a71db91@mail.gmail.com","threadId":"13137","inReplyTo":"200804161334.17748.brian.foster@innova-card.com","subject":"Re: Re: Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Brian Foster","fromEmail":"brian.foster@innova-card.com","sentAt":"2008-04-16T11:48:13Z","receivedAt":"2008-04-16T11:48:13Z","isPatch":false,"sender":{"key":"brian.foster@innova-card.com","avatar":null},"body":"Dmitry Potapov <dpotapov@gmail.com> correctly deduced:\n>  On Wed, Apr 16, 2008 at 11:45:39AM +0200, Brian Foster wrote:\n>  >  Brian Foster <brian.foster@innova-card.com> writes:\n>  >  >  I've recently inherited a bare git repository,\n>  >  >  which, as far as I can tell (I'm something of a\n>  >  >  newbie with git), seems Ok [ but clones are not ].\n>\n>  I suspect your original git repository has info/grafts\n\nhi Dmitry,\n\n bingo!  YES, it does:\n\n\t$ cat info/grafts\n\t8700854c41a40d333e90104971c3abbbcf082e57\n34757f56bd7cb007bbb48ce064a2cd7e67c32428\n\tc87c46fe892211f8aa4fd363ccff4f667a9aaf7d\nd560b7a27640cdf9ff1e6a30ae161dc6144618bd\n\tdd3f3c0636cfd50719c706b030db5473b0270add\ne6cfa744e0434f9b700cd8a3780dcf9235ed10e3\n\te5a60f1636cceac33777bb8098a0b7a4a136a56c\nb38536a9a67b8d6b94e51860195529481703286b\n\t0ff75b3afff6fb306bef221bf1823ccf5ffc568b\n01d4f94e5e5e65058234256241c01a6caf1c5feb\n\tfb5967688f7b464421cff28f266b64ad2a313a9e\n00fe1cce8a053ecf9d8ba160991f28574fb11f63\n\t2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4\n\t$\n\n now, showing that I am indeed essentially a git newbie,\n WTF is a \"graft\"?  I don't recall running into the term.\n I shall, of course, immediately start trying to find some\n info blah blah ..., but pointers/explanations would help.\n\n the goal is to put things into a sane state so any new\n clones are healthy.  there's only one(?) existing clone,\n which may or may not be(? become?) an issue.\n\ncheers!\n\t-blf-\n-- \n\"How many surrealists does it take to    |  Brian Foster\n change a lightbulb?  Three.  One calms  |  somewhere in south of France\n the warthog, and two fill the bathtub   |     Stop E$$o (ExxonMobil)!\n with brightly-coloured machine tools.\"  |       http://www.stopesso.com\n"},{"id":"74542","messageId":"20080416121424.GA3133@dpotapov.dyndns.org","threadId":"13137","inReplyTo":"a537dd660804160448w5ba28f13m8c081ebc2a71db91@mail.gmail.com","subject":"Re: Re: Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2008-04-16T12:14:24Z","receivedAt":"2008-04-16T12:14:24Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Wed, Apr 16, 2008 at 01:48:13PM +0200, Brian Foster wrote:\n> \n>  now, showing that I am indeed essentially a git newbie,\n>  WTF is a \"graft\"?  I don't recall running into the term.\n\ninfo/grafts::\n\tThis file records fake commit ancestry information, to\n\tpretend the set of parents a commit has is different\n\tfrom how the commit was actually created.  One record\n\tper line describes a commit and its fake parents by\n\tlisting their 40-byte hexadecimal object names separated\n\tby a space and terminated by a newline.\n\nBasically, you use it if you want to modify the history in your\nrepository. It is often used after importing to Git from some\nother repository if you want to make some adjustment to history.\nSo, you edit info/grafts, then use gitk to verify that you got\nwhat you want, and after that you run git-filter-branch to actually\noverwrite the history.\n\ninfo/grafts should never be used during normal work.\n\n\n>  the goal is to put things into a sane state so any new\n>  clones are healthy.  there's only one(?) existing clone,\n>  which may or may not be(? become?) an issue.\n\nIf you run git-filter-branch then you'll change \"fake\" history\nto the real one, and after that the repository can be cloned.\nPlease, make sure that you have a backup before any further changes.\n\nDmitry\n"},{"id":"74545","messageId":"4805FD7C.5000206@viscovery.net","threadId":"13137","inReplyTo":"a537dd660804160448w5ba28f13m8c081ebc2a71db91@mail.gmail.com","subject":"Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-16T13:22:04Z","receivedAt":"2008-04-16T13:22:04Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Brian Foster schrieb:\n> Dmitry Potapov <dpotapov@gmail.com> correctly deduced:\n>>  I suspect your original git repository has info/grafts\n> \n> hi Dmitry,\n> \n>  bingo!  YES, it does:\n...\n>  the goal is to put things into a sane state so any new\n>  clones are healthy.  there's only one(?) existing clone,\n>  which may or may not be(? become?) an issue.\n\nJust move info/grafts out of the way and you *may* be all set. Don't\ndelete it - there might be a reason that the file exists.\n\nHowever!\n\nYou might observe one or more of the following:\n\n- Parts of the history are missing. The reason is that the grafts were\nused to \"splice\" histories that are actually independent.\n\n- Previously unkown (to you) parts of the history are now visible. Then\nthe grafts were used to hide these parts.\n\n- The repository is broken. Then 'git repack -f -d -a' was done while the\ngrafts were in effect (and obviously the grafts had hidden parts of the\nhistory, which are now no longer available).\n\nYour best bet is to run 'git filter-branch --tag-name-filter cat -- --all'\non *a copy* of your bare repository (with info/grafts in place). (Read the\nwarning about --tag-name-filter in the man page first.)\n\n-- Hannes\n"},{"id":"74556","messageId":"20080416142551.GB3133@dpotapov.dyndns.org","threadId":"13137","inReplyTo":"4805FD7C.5000206@viscovery.net","subject":"Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2008-04-16T14:25:51Z","receivedAt":"2008-04-16T14:25:51Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Wed, Apr 16, 2008 at 03:22:04PM +0200, Johannes Sixt wrote:\n> Brian Foster schrieb:\n> > Dmitry Potapov <dpotapov@gmail.com> correctly deduced:\n> >>  I suspect your original git repository has info/grafts\n> > \n> > hi Dmitry,\n> > \n> >  bingo!  YES, it does:\n> ...\n> >  the goal is to put things into a sane state so any new\n> >  clones are healthy.  there's only one(?) existing clone,\n> >  which may or may not be(? become?) an issue.\n> \n> Just move info/grafts out of the way and you *may* be all set. Don't\n> delete it - there might be a reason that the file exists.\n\nI believe that parts of history hidden by info/grafts are removed now,\notherwise git clone would not have problems to clone this repository.\nSo the best option now is only to turn the \"fake\" history to the real\none using git filter-branch as you wrote.\n\n\n> - The repository is broken. Then 'git repack -f -d -a' was done while the\n> grafts were in effect (and obviously the grafts had hidden parts of the\n> history, which are now no longer available).\n\nActually, 'git repack -f -d -a' does not remove hidden parts of the\nhistory (because 'git prune-packed' does not remove hidden parts), but\n'git prune' does remove hidden parts. I believe 'git prune' relies on\n'git-fsck --unreachable', which outputs hidden parts as unreachable.\nPerhaps, git-fsck should be corrected, so it will not treat hidden as\nunreachable, because anything what is hidden is still reachable for\nanyone who is trying to clone the repository.\n\nDmitry\n"},{"id":"74558","messageId":"48061388.6090705@viscovery.net","threadId":"13137","inReplyTo":"20080416142551.GB3133@dpotapov.dyndns.org","subject":"Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-16T14:56:08Z","receivedAt":"2008-04-16T14:56:08Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Dmitry Potapov schrieb:\n> On Wed, Apr 16, 2008 at 03:22:04PM +0200, Johannes Sixt wrote:\n>> Brian Foster schrieb:\n>>> Dmitry Potapov <dpotapov@gmail.com> correctly deduced:\n>>>>  I suspect your original git repository has info/grafts\n>>> hi Dmitry,\n>>>\n>>>  bingo!  YES, it does:\n>> ...\n>>>  the goal is to put things into a sane state so any new\n>>>  clones are healthy.  there's only one(?) existing clone,\n>>>  which may or may not be(? become?) an issue.\n>> Just move info/grafts out of the way and you *may* be all set. Don't\n>> delete it - there might be a reason that the file exists.\n> \n> I believe that parts of history hidden by info/grafts are removed now,\n> otherwise git clone would not have problems to clone this repository.\n\nNo. The reason why git clone'd repositories have problems is that\ngit-upload-pack sends a pack that does not contain the hidden objects; but\nsince the cloned repository doesn't have the info/grafts, it tries to look\nup the hidden, now missing, objects, and fails.\n\n>> - The repository is broken. Then 'git repack -f -d -a' was done while the\n>> grafts were in effect (and obviously the grafts had hidden parts of the\n>> history, which are now no longer available).\n> \n> Actually, 'git repack -f -d -a' does not remove hidden parts of the\n> history (because 'git prune-packed' does not remove hidden parts), but\n> 'git prune' does remove hidden parts. I believe 'git prune' relies on\n> 'git-fsck --unreachable', which outputs hidden parts as unreachable.\n> Perhaps, git-fsck should be corrected, so it will not treat hidden as\n> unreachable, because anything what is hidden is still reachable for\n> anyone who is trying to clone the repository.\n\nActually, no. The trouble is that *all* tools obey grafts. Hence,\ngit-repack -f -d -a will remove the hidden objects. But a subsequent fsck\nwon't notice, because it *also* obeys the grafts. git prune will remove\nhidden objects only as long as they are loose; if they are already packed,\nthen only a repack -f will remove them.\n\nHere's a thread with a proposal by Linus how to improve the situation, but\nnobody came along and implemented it:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/37744/focus=37908\n\n-- Hannes\n"},{"id":"74559","messageId":"20080416145900.GA10777@machine.or.cz","threadId":"13137","inReplyTo":"20080416121424.GA3133@dpotapov.dyndns.org","subject":"Re: Re: Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-04-16T14:59:00Z","receivedAt":"2008-04-16T14:59:00Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Wed, Apr 16, 2008 at 04:14:24PM +0400, Dmitry Potapov wrote:\n> info/grafts should never be used during normal work.\n\nI don't really agree with this advice in general. Grafts can be very\nuseful especially when migrating to Git - a very good strategy might be\njust to start from scratch with your current state, then import your\nprevious history and graft it. The big advantage is that if you find out\nlater that the import is screwed up, you can fix that and reimport. The\nfact that you don't need to haul 10 years of CVS history around unless\nyou actually know you need it can be perceived as both advantage and\ndisadvantage.\n\nThe _BAD_ grafts are those that replace list of commit's parents instead\nof just appending (typically to an empty one). I didn't actually have\nany idea people use grafts in such a twisted and perverse way... ;-)\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nWhatever you can do, or dream you can, begin it.\nBoldness has genius, power, and magic in it.\t-- J. W. von Goethe\n"},{"id":"74563","messageId":"20080416161742.GC3133@dpotapov.dyndns.org","threadId":"13137","inReplyTo":"48061388.6090705@viscovery.net","subject":"Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2008-04-16T16:17:42Z","receivedAt":"2008-04-16T16:17:42Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:\n> \n> No. The reason why git clone'd repositories have problems is that\n> git-upload-pack sends a pack that does not contain the hidden objects; but\n> since the cloned repository doesn't have the info/grafts, it tries to look\n> up the hidden, now missing, objects, and fails.\n\nIn my experiment, there was no problem with hidden commits and objects\nas long as they presented in the original repository. Maybe, it is\nbecause I cloned it as locally using:\n\n   git clone PATH-TO-ORIG-REPO PATH-TO-DEST-REPO\n\nMaybe things would be as you say if I used git://\n\n> \n> Actually, no. The trouble is that *all* tools obey grafts. Hence,\n> git-repack -f -d -a will remove the hidden objects. But a subsequent fsck\n> won't notice, because it *also* obeys the grafts. git prune will remove\n> hidden objects only as long as they are loose; if they are already packed,\n> then only a repack -f will remove them.\n\nI use git 1.5.5 and I have a script that creates are repo with a hidden\ncommit and then it cleans the reflog cleaned (as it would be happen after\nexpiration of gc.reflogExpire). Then I run 'git-repack -f -d -a' and the\nhidden commit still presents, but when I run 'git prune' then the hidden\nobject disappear. You can try it for yourself. Here is the script, I used:\n\n===\n#!/bin/sh\n\nset -e\n\nmkdir t5\ncd t5\ngit init\n\necho 1 > foo\ngit add foo\ngit commit -m 'add foo'\nR1=$(git rev-parse HEAD)\n\necho 2 >> foo\ngit commit -m 'edit foo' -a\nR2=$(git rev-parse HEAD)\n\necho 3 >> foo\ngit commit -m 'edit foo again' -a\nR3=$(git rev-parse HEAD)\n\nset -x\ngit log\necho \"$R3 $R1\" > .git/info/grafts\ngit log\ngit-fsck --unreachable\n: > .git/logs/HEAD\n: > .git/logs/refs/heads/master\ngit-fsck --unreachable\ngit gc\ngit-fsck --unreachable\n\n\ngit-repack -f -d -a\ngit-fsck --unreachable\n\ngit prune\ngit-fsck --unreachable\n===\n\nDmitry\n"},{"id":"74564","messageId":"20080416162338.GD3133@dpotapov.dyndns.org","threadId":"13137","inReplyTo":"20080416145900.GA10777@machine.or.cz","subject":"Re: Re: Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2008-04-16T16:23:38Z","receivedAt":"2008-04-16T16:23:38Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Wed, Apr 16, 2008 at 04:59:00PM +0200, Petr Baudis wrote:\n> On Wed, Apr 16, 2008 at 04:14:24PM +0400, Dmitry Potapov wrote:\n> > info/grafts should never be used during normal work.\n> \n> I don't really agree with this advice in general. Grafts can be very\n> useful especially when migrating to Git\n\nI agree that they can be very useful in the process of migrating, but\nI would not recommend to leave them _after_ the migration process is\ncomplete.\n\n> The _BAD_ grafts are those that replace list of commit's parents instead\n> of just appending (typically to an empty one). I didn't actually have\n> any idea people use grafts in such a twisted and perverse way... ;-)\n\nThey are used to replace _BAD_ commits imported from CVS. History in\nCVS can be very strange and even with best conversation tools, you may\nend up with something not exactly what you want. So, you want to replace\nsome commits with corrected versions of them. So, you add those commits\nto Git repo, \"fix\" history, and then using gitk check that now you have\nexactly what you want. If now the history is okay then you can use\ngit filter-branch to make those changes real.\n\nDmitry\n"},{"id":"74566","messageId":"fu5ajm$eaq$1@ger.gmane.org","threadId":"13137","inReplyTo":"20080416161742.GC3133@dpotapov.dyndns.org","subject":"Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-04-16T16:47:58Z","receivedAt":"2008-04-16T16:47:58Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dmitry Potapov wrote:\n\n> On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:\n>> \n>> No. The reason why git clone'd repositories have problems is that\n>> git-upload-pack sends a pack that does not contain the hidden objects;\nbut\n>> since the cloned repository doesn't have the info/grafts, it tries to\nlook\n>> up the hidden, now missing, objects, and fails.\n> \n> In my experiment, there was no problem with hidden commits and objects\n> as long as they presented in the original repository. Maybe, it is\n> because I cloned it as locally using:\n> \n>    git clone PATH-TO-ORIG-REPO PATH-TO-DEST-REPO\n> \n> Maybe things would be as you say if I used git://\n\nfile:// should be enough.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"74617","messageId":"4806EBBF.9060906@viscovery.net","threadId":"13137","inReplyTo":"20080416161742.GC3133@dpotapov.dyndns.org","subject":"Re: fsck --full is Ok, but clones are not, \"missing commits\"?!","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-17T06:18:39Z","receivedAt":"2008-04-17T06:18:39Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Dmitry Potapov schrieb:\n> On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:\n>> Actually, no. The trouble is that *all* tools obey grafts. Hence,\n>> git-repack -f -d -a will remove the hidden objects. But a subsequent fsck\n>> won't notice, because it *also* obeys the grafts. git prune will remove\n>> hidden objects only as long as they are loose; if they are already packed,\n>> then only a repack -f will remove them.\n> \n> I use git 1.5.5 and I have a script that creates are repo with a hidden\n> commit and then it cleans the reflog cleaned (as it would be happen after\n> expiration of gc.reflogExpire). Then I run 'git-repack -f -d -a' and the\n> hidden commit still presents, but when I run 'git prune' then the hidden\n> object disappear. You can try it for yourself. Here is the script, I used:\n\nAnd the reason for this is...\n\n> \n> ===\n> #!/bin/sh\n> \n> set -e\n> \n> mkdir t5\n> cd t5\n> git init\n> \n> echo 1 > foo\n> git add foo\n> git commit -m 'add foo'\n> R1=$(git rev-parse HEAD)\n> \n> echo 2 >> foo\n> git commit -m 'edit foo' -a\n> R2=$(git rev-parse HEAD)\n> \n> echo 3 >> foo\n> git commit -m 'edit foo again' -a\n> R3=$(git rev-parse HEAD)\n> \n> set -x\n> git log\n> echo \"$R3 $R1\" > .git/info/grafts\n> git log\n> git-fsck --unreachable\n> : > .git/logs/HEAD\n> : > .git/logs/refs/heads/master\n> git-fsck --unreachable\n> git gc\n\n... at this point the hidden commit still exists as loose objects (because\nthe repack inside gc didn't see the commit), and therefore...\n\n> git-fsck --unreachable\n> \n> \n> git-repack -f -d -a\n\n... this does not make a difference at all.\n\n> git-fsck --unreachable\n> \n> git prune\n\nHere you remove the loose objects (because prune doesn't see the commit,\neither), and by doing so you corrupt the repository.\n\n> git-fsck --unreachable\n> ===\n\nA repack -a -d makes a difference if info/grafts is installed *after* the\nfirst repack/gc.\n\n-- Hannes\n"}]}