Re: Re: Re: fsck --full is Ok, but clones are not, "missing commits"?!

Dmitry Potapov <dpotapov@gmail.com> correctly deduced:

hi Dmitry,

 bingo!  YES, it does:

	$ cat info/grafts
	8700854c41a40d333e90104971c3abbbcf082e57
34757f56bd7cb007bbb48ce064a2cd7e67c32428
	c87c46fe892211f8aa4fd363ccff4f667a9aaf7d
d560b7a27640cdf9ff1e6a30ae161dc6144618bd
	dd3f3c0636cfd50719c706b030db5473b0270add
e6cfa744e0434f9b700cd8a3780dcf9235ed10e3
	e5a60f1636cceac33777bb8098a0b7a4a136a56c
b38536a9a67b8d6b94e51860195529481703286b
	0ff75b3afff6fb306bef221bf1823ccf5ffc568b
01d4f94e5e5e65058234256241c01a6caf1c5feb
	fb5967688f7b464421cff28f266b64ad2a313a9e
00fe1cce8a053ecf9d8ba160991f28574fb11f63
	2dcaaf2decd31ac9a21d616604c0a7c1fa65d5a4
	$

 now, showing that I am indeed essentially a git newbie,
 WTF is a "graft"?  I don't recall running into the term.
 I shall, of course, immediately start trying to find some
 info blah blah ..., but pointers/explanations would help.

 the goal is to put things into a sane state so any new
 clones are healthy.  there's only one(?) existing clone,
 which may or may not be(? become?) an issue.

cheers!
	-blf-

On Wed, Apr 16, 2008 at 01:48:13PM +0200, Brian Foster wrote:

info/grafts::
	This file records fake commit ancestry information, to
	pretend the set of parents a commit has is different
	from how the commit was actually created.  One record
	per line describes a commit and its fake parents by
	listing their 40-byte hexadecimal object names separated
	by a space and terminated by a newline.

Basically, you use it if you want to modify the history in your repository. It is often used after importing to Git from some other repository if you want to make some adjustment to history. So, you edit info/grafts, then use gitk to verify that you got what you want, and after that you run git-filter-branch to actually overwrite the history.

info/grafts should never be used during normal work.

If you run git-filter-branch then you'll change "fake" history to the real one, and after that the repository can be cloned. Please, make sure that you have a backup before any further changes.

Dmitry

On Wed, Apr 16, 2008 at 04:14:24PM +0400, Dmitry Potapov wrote:

I don't really agree with this advice in general. Grafts can be very useful especially when migrating to Git - a very good strategy might be just to start from scratch with your current state, then import your previous history and graft it. The big advantage is that if you find out later that the import is screwed up, you can fix that and reimport. The fact that you don't need to haul 10 years of CVS history around unless you actually know you need it can be perceived as both advantage and disadvantage.

The _BAD_ grafts are those that replace list of commit's parents instead of just appending (typically to an empty one). I didn't actually have any idea people use grafts in such a twisted and perverse way... ;-)

On Wed, Apr 16, 2008 at 04:59:00PM +0200, Petr Baudis wrote:

I agree that they can be very useful in the process of migrating, but I would not recommend to leave them _after_ the migration process is complete.

They are used to replace _BAD_ commits imported from CVS. History in CVS can be very strange and even with best conversation tools, you may end up with something not exactly what you want. So, you want to replace some commits with corrected versions of them. So, you add those commits to Git repo, "fix" history, and then using gitk check that now you have exactly what you want. If now the history is okay then you can use git filter-branch to make those changes real.

Dmitry

Brian Foster schrieb:

...

Just move info/grafts out of the way and you *may* be all set. Don't delete it - there might be a reason that the file exists.

However!

You might observe one or more of the following:

- Parts of the history are missing. The reason is that the grafts were
used to "splice" histories that are actually independent.

- Previously unkown (to you) parts of the history are now visible. Then
the grafts were used to hide these parts.

- The repository is broken. Then 'git repack -f -d -a' was done while the
grafts were in effect (and obviously the grafts had hidden parts of the
history, which are now no longer available).

Your best bet is to run 'git filter-branch --tag-name-filter cat -- --all' on *a copy* of your bare repository (with info/grafts in place). (Read the warning about --tag-name-filter in the man page first.)

-- Hannes

On Wed, Apr 16, 2008 at 03:22:04PM +0200, Johannes Sixt wrote:

I believe that parts of history hidden by info/grafts are removed now, otherwise git clone would not have problems to clone this repository. So the best option now is only to turn the "fake" history to the real one using git filter-branch as you wrote.

Actually, 'git repack -f -d -a' does not remove hidden parts of the history (because 'git prune-packed' does not remove hidden parts), but 'git prune' does remove hidden parts. I believe 'git prune' relies on 'git-fsck --unreachable', which outputs hidden parts as unreachable. Perhaps, git-fsck should be corrected, so it will not treat hidden as unreachable, because anything what is hidden is still reachable for anyone who is trying to clone the repository.

Dmitry

Dmitry Potapov schrieb:

No. The reason why git clone'd repositories have problems is that git-upload-pack sends a pack that does not contain the hidden objects; but since the cloned repository doesn't have the info/grafts, it tries to look up the hidden, now missing, objects, and fails.

Actually, no. The trouble is that *all* tools obey grafts. Hence, git-repack -f -d -a will remove the hidden objects. But a subsequent fsck won't notice, because it *also* obeys the grafts. git prune will remove hidden objects only as long as they are loose; if they are already packed, then only a repack -f will remove them.

Here's a thread with a proposal by Linus how to improve the situation, but nobody came along and implemented it:

http://thread.gmane.org/gmane.comp.version-control.git/37744/focus=37908

-- Hannes

On Wed, Apr 16, 2008 at 04:56:08PM +0200, Johannes Sixt wrote:

In my experiment, there was no problem with hidden commits and objects as long as they presented in the original repository. Maybe, it is because I cloned it as locally using:

   git clone PATH-TO-ORIG-REPO PATH-TO-DEST-REPO

Maybe things would be as you say if I used git://

I use git 1.5.5 and I have a script that creates are repo with a hidden commit and then it cleans the reflog cleaned (as it would be happen after expiration of gc.reflogExpire). Then I run 'git-repack -f -d -a' and the hidden commit still presents, but when I run 'git prune' then the hidden object disappear. You can try it for yourself. Here is the script, I used:

=== #!/bin/sh

set -e

mkdir t5 cd t5 git init

echo 1 > foo git add foo git commit -m 'add foo' R1=$(git rev-parse HEAD)

echo 2 >> foo git commit -m 'edit foo' -a R2=$(git rev-parse HEAD)

echo 3 >> foo git commit -m 'edit foo again' -a R3=$(git rev-parse HEAD)

set -x git log echo "$R3 $R1" > .git/info/grafts git log git-fsck --unreachable : > .git/logs/HEAD : > .git/logs/refs/heads/master git-fsck --unreachable git gc git-fsck --unreachable

git-repack -f -d -a git-fsck --unreachable

git prune git-fsck --unreachable ===

Dmitry

Dmitry Potapov wrote:

but

look

file:// should be enough.

Dmitry Potapov schrieb:

And the reason for this is...

... at this point the hidden commit still exists as loose objects (because the repack inside gc didn't see the commit), and therefore...

... this does not make a difference at all.

Here you remove the loose objects (because prune doesn't see the commit, either), and by doing so you corrupt the repository.

A repack -a -d makes a difference if info/grafts is installed *after* the first repack/gc.

-- Hannes