{"thread":{"id":"2144","subject":"cg-clone, tag objects and cg-push/git-push don't play nice","startedAt":"2005-10-19T06:38:16Z","lastAt":"2005-10-19T09:06:52Z","messageCount":5,"participants":["Martin Langhoff","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"10259","messageId":"46a038f90510182338k6d3d52fbyc2057e9b775d5b14@mail.gmail.com","threadId":"2144","inReplyTo":null,"subject":"cg-clone, tag objects and cg-push/git-push don't play nice","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-10-19T06:38:16Z","receivedAt":"2005-10-19T06:38:16Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"I am seeing very strange issues with cloning one head with cg-clone\nfrom a repo that holds many heads and tags. The session looks like\nthis:\n\n  cg-clone git+ssh://locke.catalyst.net.nz/var/git/moodle-test.git#mdl-topnz\ntestdir\n  cd testdir\n  echo \"sillychange\" >> version.php\n  cg-commit -m \"testing\" version.php\n  cg-push\n  updating 'refs/heads/mdl-topnz' using 'refs/heads/master'\n    from 06ca8b3c4826d60e8cf5850c6474e66f816ba5c7\n    to   482d4b88aa482dfea7f7549470902049a050020a\n  fatal: bad object 1b0efdd8f31e5b8c7d32c85d11492db122b62a0a\n  Packing 0 objects\n  Unpacking 0 objects\n\n  error: unpack should have generated\n482d4b88aa482dfea7f7549470902049a050020a, but I can't find it!\n\nI get the same error if I try git-push manually. Apparently, there's a\nset of tag objects pointing nowhere.\n\ngit-fsck-objects --full --strict 482d4b88aa482dfea7f7549470902049a050020a\nbad sha1 file: .git/objects/44/7472d455667e426a96acf116e27c2f1efe674e\n3ae8dc25c642d8c59f3c44c5ba48faa6a0e7a2ee\n2ddfec0dfd0cffd4892af9aaf48ee29c40c7ada3\nmissing commit 1b0efdd8f31e5b8c7d32c85d11492db122b62a0a\ndangling commit 3e9472b3ef980e667d00d5374ccfa741cfb93fbc\nbroken link from     tag 447472d455667e426a96acf116e27c2f1efe674e\n              to  commit 1b0efdd8f31e5b8c7d32c85d11492db122b62a0a\ndangling tag 447472d455667e426a96acf116e27c2f1efe674e\nmissing commit 5a0bdfb0f7af34d002c6ced40f96f977fd6471e2\ndangling commit 715b3b0a7c94cb31760e87a320f0f612b962e3c3\nmissing commit 961bf469c6b309b8fc10d064368fd14480d231be\nbroken link from     tag 984bed44df42422839705aa4bd6d8f00086b1307\n              to  commit 961bf469c6b309b8fc10d064368fd14480d231be\ndangling tag 984bed44df42422839705aa4bd6d8f00086b1307\nbroken link from     tag ce3602d6a60c648402048b0970657a2961e1ed54\n              to  commit dd8c6c172fbab9905dd306c17d83b8d21ea5bfda\ndangling tag ce3602d6a60c648402048b0970657a2961e1ed54\nmissing commit dd8c6c172fbab9905dd306c17d83b8d21ea5bfda\nbroken link from     tag ddb658d070ba6688541e5b91e963f629ddd63b6a\n              to  commit 5a0bdfb0f7af34d002c6ced40f96f977fd6471e2\ndangling tag ddb658d070ba6688541e5b91e963f629ddd63b6a\n\nNow, if I go to the repository, it _has_ all the tags with their\nmatching commits. So the problem is in the cloned repo. It looks like:\n\n + cg-clone (cg-fetch actually) has brought in all the tag refs and\nobjects, regardless of whether they are relevant to this branch, but\ndid not fetch the commits, trees, or any other related bits and\npieces.\n\n + git-push is trying to walk all the refs it knows about when it does\nthe \"what do I have that the repo doesn't\" part, and it breaks on\nthose incomplete tag fetches.\n\nRemoving  .git/refs/tags/* didn't help. If I actually rm all the\nproblematic tag objects from the object repository, I can then push\ncorrectly. Lucky it wasn't packed.\n\nI am going to write a quick'n'dirty script to fix our repos when this\nhappens, and I'll try my hand at a patch to cg-fetch. Can git-push be\ntaught to be smarter in these cases?\n\nregards,\n\n\n\nmartin\n"},{"id":"10260","messageId":"7vzmp6dlii.fsf@assigned-by-dhcp.cox.net","threadId":"2144","inReplyTo":"46a038f90510182338k6d3d52fbyc2057e9b775d5b14@mail.gmail.com","subject":"Re: cg-clone, tag objects and cg-push/git-push don't play nice","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-19T07:16:37Z","receivedAt":"2005-10-19T07:16:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Langhoff <martin.langhoff@gmail.com> writes:\n\n>  + git-push is trying to walk all the refs it knows about when it does\n> the \"what do I have that the repo doesn't\" part, and it breaks on\n> those incomplete tag fetches.\n\nThat is expected.\n\n> Removing  .git/refs/tags/* didn't help. If I actually rm all the\n> problematic tag objects from the object repository, I can then push\n> correctly. Lucky it wasn't packed.\n>\n> I am going to write a quick'n'dirty script to fix our repos when this\n> happens, and I'll try my hand at a patch to cg-fetch. Can git-push be\n> taught to be smarter in these cases?\n\nAlthough I do not follow Cogito development closely, I seem to\nrecall that it fetched tags without making them complete at some\npoint in the past; I hope it is now fixed but I am not sure.\n\nI cannot think offhand of a way how git-push could help to cope\nwith a repository broken that way.  However, we could probably\nhave a tool that tangles from each ref, find incomplete ones and\nremove them from .git/refs.  After running that maybe pull from\na know good copy would fix the broken repository.\n\nI do not understand why removing .git/refs/tags/* did not help,\nand that is the biggest thing that disturbs me in this whole\nproblem report.  We (meaning git-* transfer, not cg-* transfer\nwhose correctness I cannot vouch for) _should_ be relying only\non refs not object existence.  Maybe there are some other files\nunder .git/refs/ directory that had copies of them?\n"},{"id":"10262","messageId":"46a038f90510190110g53c90c5t419ad6065292269e@mail.gmail.com","threadId":"2144","inReplyTo":"7vzmp6dlii.fsf@assigned-by-dhcp.cox.net","subject":"Re: cg-clone, tag objects and cg-push/git-push don't play nice","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-10-19T08:10:26Z","receivedAt":"2005-10-19T08:10:26Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 10/19/05, Junio C Hamano <junkio@cox.net> wrote:\n> Martin Langhoff <martin.langhoff@gmail.com> writes:\n>\n> >  + git-push is trying to walk all the refs it knows about when it does\n> > the \"what do I have that the repo doesn't\" part, and it breaks on\n> > those incomplete tag fetches.\n>\n> That is expected.\n\nHmmm. I was under the impression that if I call git-push naming a\nparticular head, it could restrict itself to the stuff needed for that\nhead only. Just to clarify, I'm running\n\n  git-push locke.catalyst.net.nz:/var/git/moodle-test.git master:mdl-topnz-prod\n\n> Although I do not follow Cogito development closely, I seem to\n> recall that it fetched tags without making them complete at some\n> point in the past; I hope it is now fixed but I am not sure.\n\nIt isn't fixed, but I'm trying to address that one :-p\n\n> I do not understand why removing .git/refs/tags/* did not help,\n> and that is the biggest thing that disturbs me in this whole\n> problem report\n\nI can't understand that either, but I manually removed all the\nrefs/tags, and the only heads I have are origin and master. git-push\nwon't let me do it until the exact point where I have removed the\nobject from the repo. And that's only possible on unpacked repos.\n\n> Maybe there are some other files\n> under .git/refs/ directory that had copies of them?\n\nNo. I run a test again, to make sure. Removing .git/refs/tags is not\nenough, and the only refs available are origin and master.\n\nI've come up with this awful script to resolve it, while I try to fix cg-fetch:\n\n#usr/bin/perl -w\n\nuse strict;\n\nmy @refs = `ls .git/refs/tags`;\n\nforeach my $ref (@refs) {\n  chomp $ref;\n  print \"testing for a commit linked from $ref\\n\";\n  my $commit = `git-rev-parse --verify \"$ref\"^{commit} 2>/dev/null`;\n  if ($?) {\n    # this one didn't even have a tag ref pointing to a commit!\n    #unlink \".git/refs/tags/$ref\"\n    #  or die \"cannot remove .git/refs/tags/$ref\";\n    #next;\n  }\n\n  # thest that we actually have the commit object...\n  chomp $commit;\n  my $file = `git-cat-file commit $commit  2>/dev/null`;\n  if ($?) {\n    # could not find the commit object, we better get rid of the\n    # tagref and tagobj\n    my $tagsha = `git-rev-parse --verify \"$ref\"  2>/dev/null`;\n\n    chomp $tagsha;\n    if ($tagsha) {\n      # doublecheck it is a tag\n      my $type = `git-cat-file -t $tagsha  2>/dev/null`;\n      chomp $type;\n      if ($type eq 'tag') {\n        my $fileobj = \".git/objects/\" . substr($tagsha,0,2) . '/' .\nsubstr($tagsha,2);\n        print \" removing $fileobj for ref $ref tagsha $tagsha \\n\";\n        `rm -f $fileobj`;\n        unlink  \".git/refs/tags/$ref\";\n      }\n    }\n  }\n}\n\ncheers,\n\n\nm\n"},{"id":"10268","messageId":"7vbr1ldh30.fsf@assigned-by-dhcp.cox.net","threadId":"2144","inReplyTo":"46a038f90510190110g53c90c5t419ad6065292269e@mail.gmail.com","subject":"Re: cg-clone, tag objects and cg-push/git-push don't play nice","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-19T08:52:19Z","receivedAt":"2005-10-19T08:52:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Langhoff <martin.langhoff@gmail.com> writes:\n\n>> That is expected.\n>\n> Hmmm. I was under the impression that if I call git-push naming a\n> particular head, it could restrict itself to the stuff needed for that\n> head only. Just to clarify, I'm running\n>\n>   git-push locke.catalyst.net.nz:/var/git/moodle-test.git master:mdl-topnz-prod\n\nYeah.  But the problem is that the repository on the other end\nclaims that it has everything reachable from those incomplete\ntags.  Hearing that, git-push (the real name of it is send-pack)\ndecides not to send things that are already reachable from the\nrefs the other end claims to have.  So if an incomplete tag\nrefers to a commit that contains a blob that the remote actually\ndoes not have, and if that blob is part of the head you are\npushing, send-pack would not (and should not) send that blob to\nthe remote.\n\n>> I do not understand why removing .git/refs/tags/* did not help,\n>> and that is the biggest thing that disturbs me in this whole\n>> problem report\n>\n> I can't understand that either, but I manually removed all the\n> refs/tags, and the only heads I have are origin and master. git-push\n> won't let me do it until the exact point where I have removed the\n> object from the repo. And that's only possible on unpacked repos.\n\nHmph.  It worries me even more.\n\nThis error message:\n\n      error: unpack should have generated\n    482d4b88aa482dfea7f7549470902049a050020a, but I can't find it!\n\ncomes from receive-pack that runs on the other repo (i.e. the\none with incomplete tags you just removed), so it means\nsend-pack decided it does not need to send that object --\nmeaning the other end claimed it already has it.  What to send\nand what need not to be sent is determined solely based on what\nsend-pack hears from receive-pack in the initial handshake,\nwhich is in receive-pack.c::write_head_info().  It scans\neverything under \".git/refs\" directory (not just .git/refs/heads\nor .git/refs/tags; if you had \".git/refs/FOOBAR\", it will cause\nthe remote end to claim it has everything reachable from it --\nthe only exceptions are things that starts with a dot '.', which\nis probably why cg-fetch places a temporary heads in\nrefs/*/.$name-fetching) and sends them -- it does not look at\nthe object directory and magically claim it has something that\nis not recorded in its .git/refs/ directory.\n"},{"id":"10272","messageId":"46a038f90510190206j722faefbl4fe60ed0052b7104@mail.gmail.com","threadId":"2144","inReplyTo":"7vbr1ldh30.fsf@assigned-by-dhcp.cox.net","subject":"Re: cg-clone, tag objects and cg-push/git-push don't play nice","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-10-19T09:06:52Z","receivedAt":"2005-10-19T09:06:52Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 10/19/05, Junio C Hamano <junkio@cox.net> wrote:\n> Yeah.  But the problem is that the repository on the other end\n> claims that it has everything reachable from those incomplete\n> tags.\n\nIt definitely does.\n\n> Hearing that, git-push (the real name of it is send-pack)\n> decides not to send things that are already reachable from the\n> refs the other end claims to have.  So if an incomplete tag\n> refers to a commit that contains a blob that the remote actually\n> does not have, and if that blob is part of the head you are\n> pushing, send-pack would not (and should not) send that blob to\n> the remote.\n\n*shrug* it's a bit over my head, but I've figured out taht\ngit-fetch-pack is segfaulting. Perhaps that helps?\n\n> Hmph.  It worries me even more.\n>\n> This error message:\n>\n>       error: unpack should have generated\n>     482d4b88aa482dfea7f7549470902049a050020a, but I can't find it!\n>\n> comes from receive-pack that runs on the other repo (i.e. the\n> one with incomplete tags you just removed), so it means\n> send-pack decided it does not need to send that object --\n> meaning the other end claimed it already has it.  What to send\n> and what need not to be sent is determined solely based on what\n> send-pack hears from receive-pack in the initial handshake,\n> which is in receive-pack.c::write_head_info().  It scans\n> everything under \".git/refs\" directory (not just .git/refs/heads\n> or .git/refs/tags; if you had \".git/refs/FOOBAR\", it will cause\n> the remote end to claim it has everything reachable from it --\n> the only exceptions are things that starts with a dot '.', which\n> is probably why cg-fetch places a temporary heads in\n> refs/*/.$name-fetching) and sends them -- it does not look at\n> the object directory and magically claim it has something that\n> is not recorded in its .git/refs/ directory.\n\n*Very* strange. The remote repo is packed, but is working well for\neveryone else. I think we should follow the segfault and see where it\nleads...\n\ncheers,\n\n\nm\n"}]}