New repo quickly corrupted

I was working on a new repo created from a tar snapshot of an old project, and after about a half-hour and about 5 commits for hacking, I had fixed the problem I set out to take care of.

However, I tried to run git gc to clean it up and put it away and I was met with the following:

[sewall@hobo SWIFT++_1.2 (master)]$ git gc Counting objects: 227, done. error: corrupt loose object '680aba836639d14a36b81f9e29da52bd9af69770' fatal: object 680aba836639d14a36b81f9e29da52bd9af69770 cannot be read error: failed to run repack

I've followed discussions on the list about corrupted repos and did [sewall@hobo SWIFT++_1.2 (master)]$ git fsck error: corrupt loose object '24bbb41919ef906db6f40c3c80f246b7e0cdf9f4' error: 24bbb41919ef906db6f40c3c80f246b7e0cdf9f4: object corrupt or missing error: corrupt loose object '680aba836639d14a36b81f9e29da52bd9af69770' error: 680aba836639d14a36b81f9e29da52bd9af69770: object corrupt or missing error: 8de8382bdc3cc5f936670df414b33ee63927f3a4: object corrupt or missing error: corrupt loose object 'd186d23e87ddf344acc56f48e3cf2f61c7a47e16' error: d186d23e87ddf344acc56f48e3cf2f61c7a47e16: object corrupt or missing missing blob 680aba836639d14a36b81f9e29da52bd9af69770 missing blob 8de8382bdc3cc5f936670df414b33ee63927f3a4 missing blob d186d23e87ddf344acc56f48e3cf2f61c7a47e16 missing blob 24bbb41919ef906db6f40c3c80f246b7e0cdf9f4

And since these were all blobs, I looked for em with git-log: [sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 24bbb* [sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep d186* [sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 8de83* [sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 680ab*

So it would assume that these are dangling objects. Given the simple history of the repo, the only think I can think of that could have caused it would be that my first commit was of the original files, and I immediately remembered that I wanted to run dos2unix on the files so I did and did a commit -a --amend.

I untarred the original snapshot again and ran git-hash-object over all the files, and sure enough each of those missing hashes corresponds to a blob in the original (non-dos2unix'd) file.

I copied the corrupted repo, manually deleted those objects, then hash-object -w the appropriate originals back in. git-fsck was mollified.

Finally, I tried repeating the process and the same error occurs. In summary: unzip original.zip git init git add . git commit -m "Initial commit" find . -type f -exec dos2unix {} + git commit -a --amend [accept same message] git fsck [same error message as above]

I just built my git today, unmodified: git version 1.5.3.5.721.g039b

What's going on?

Jason

P.S. These files aren't particularly secret, so I'm happy to post them if need be. Its actually someone else's code (a fellow student who graduated some time before I arrived in grad school); I was asked to modernize it enough to compile on on the latest gcc.

"Jason Sewall" <jasonsewall@gmail.com> writes:

Carelessness is what is going on.

Notice how nicely your "find . -type f -exec dos2unix {}" goes down to .git/objects and eats your loose objects

You probably wanted to do

	git ls-files | xargs dos2unix

instead.

On Nov 15, 2007 4:55 PM, Junio C Hamano <gitster@pobox.com> wrote:

Sure enough. I'll chalk that one up to too many mind-numbing meetings today.

Thanks for the wake-up...

On Thu, 15 Nov 2007, Jason Sewall wrote:

Oops.

You just ran "dos2unix" on all the *git* files!

Which certainly explains why all your old objects got corrupted!

PEBKAC ;)

To avoid this error in the future, may I suggest using

	git ls-files -z | xargs -0 dos2unix

or similar. Please DO NOT write to the .git/objects directory.

		Linus

On Nov 15, 2007 5:01 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:

Yeah, Junio beat you to it. :)

I take comfort in the fact that others have suffered far worse consequences from find misuse....

On Thu, 15 Nov 2007, Linus Torvalds wrote:

Does "dos2unix" override file access bits? Because the object store is always made read-only.

Nicolas

On Thu, 15 Nov 2007, Nicolas Pitre wrote:

Almost all programs like that will entirely ignor the fact that something is read-only.

Why? Becuase you end up having to create a new file *anyway*. So nobody does modify-in-place, they literally end up doing

 - create temp-file
 - while (data) 
	read old file, write to tempfile
 - rename temp-file over oldfile

and unless you *explicitly* look at the permission bits you'll never even notice that the old file was read-only, because none of the steps above care at all!

So marking things read-only will give only limited protection. It will protect against most editors, and will protect against things that change files in-place and literally try to open the original file as read-write, but not much else.

			Linus

Le jeudi 15 novembre 2007, Linus Torvalds a écrit :

What if the .git/objects/ sudirectories were also read-only ?

Christian.

Christian Couder wrote:

Then git wouldn't be able to write to it without chmod()'ing it each time.

Le vendredi 16 novembre 2007, Andreas Ericsson a écrit :

Yes, but some (not manly enough) people might want the extra safety even if it means a performance penalty.

Christian.

lördag 17 november 2007 skrev Christian Couder:

Those do manly enough to do run find . -exec  ;)

-- robin