{"thread":{"id":"24795","subject":"Tags, Grafts, and Clones, oh my!","startedAt":"2010-08-20T00:54:29Z","lastAt":"2010-09-07T23:14:00Z","messageCount":7,"participants":["Stephen Bash","Ramkumar Ramachandra","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"148494","messageId":"15347445.117294.1282265669453.JavaMail.root@mail.hq.genarts.com","threadId":"24795","inReplyTo":"29380346.117285.1282264933599.JavaMail.root@mail.hq.genarts.com","subject":"Tags, Grafts, and Clones, oh my!","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2010-08-20T00:54:29Z","receivedAt":"2010-08-20T00:54:29Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"Hi all-\n\nI'm currently working on migrating my company's SVN repository to git.  Based on a conversation with Ram at the beginning of the summer, I'm using svn-fe plus a couple of my own scripts.  To create the git branches and tags from the svn-fe generated repo I clone a bunch of \"mini-repos\", which I subdirectory-filter, then git fetch the branches/tags from the mini-repos into a \"fusion\" repo where I graft everything back together, and finally one last filter-branch to permanently commit the grafts.\n\nUnfortunately I'm running into a problem with cloning the resulting repository.  Any git tags that are not associated with a live branch are declared invalid:\n\n   error: refs/tags/tagFoo does not point to a valid object!\n\nI've now reproduced this issue in micro (much easier to work with than the 20k commits in the real repo), and it does go away if I git checkout -b branchFoo tagFoo before cloning the repository.  I've examined the source repository, and the tag appears valid to me, as does the commit it points to.  The first error I see is during the clone.\n\nDoes this situation make sense to anyone?  If it's a potential bug (rather than user error), I can submit my testcase.  I'm currently working with git 1.7.2.1 on MacOS 10.6.4.\n\nAny ideas will be greatly appreciated!\n\nThanks,\nStephen\n"},{"id":"148502","messageId":"20100820061459.GB12794@kytes","threadId":"24795","inReplyTo":"15347445.117294.1282265669453.JavaMail.root@mail.hq.genarts.com","subject":"Re: Tags, Grafts, and Clones, oh my!","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-08-20T06:15:04Z","receivedAt":"2010-08-20T06:15:04Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Stephen,\n\nStephen Bash writes:\n> Unfortunately I'm running into a problem with cloning the resulting repository.  Any git tags that are not associated with a live branch are declared invalid:\n> \n>    error: refs/tags/tagFoo does not point to a valid object!\n> \n> I've now reproduced this issue in micro (much easier to work with than the 20k commits in the real repo), and it does go away if I git checkout -b branchFoo tagFoo before cloning the repository.  I've examined the source repository, and the tag appears valid to me, as does the commit it points to.  The first error I see is during the clone.\n> \n> Does this situation make sense to anyone?  If it's a potential bug (rather than user error), I can submit my testcase.  I'm currently working with git 1.7.2.1 on MacOS 10.6.4.\n\nIt seems too vague to me. Can you submit your testcase? Perhaps we can\nwork from there?\n\n-- Ram\n"},{"id":"148556","messageId":"5752362.117414.1282309220152.JavaMail.root@mail.hq.genarts.com","threadId":"24795","inReplyTo":"20100820061459.GB12794@kytes","subject":"Re: Tags, Grafts, and Clones, oh my!","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2010-08-20T13:00:20Z","receivedAt":"2010-08-20T13:00:20Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"> >\n> >    error: refs/tags/tagFoo does not point to a valid object!\n> >\n> \n> It seems too vague to me. Can you submit your testcase? Perhaps we can\n> work from there?\n\nSure:\n\nmkdir gitTagBug\ncd gitTagBug\n\ngit init foo\ncd foo\necho A >> foo.txt\ngit add foo.txt\ngit commit -m \"Created foo\"\nGRAFT_PARENT=`git rev-parse master`\necho B >> foo.txt\ngit commit -am \"Added B\"\ncd ..\n\ngit init bar\ncd bar\necho A >> foo.txt\necho C >> foo.txt\ngit add foo.txt\ngit commit -m \"Added C\"\nGRAFT_CHILD=`git rev-parse master`\ngit tag -am \"Tagging foo C\" tagFoo $GRAFT_CHILD\ncd ..\n\ncd foo\ngit fetch ../bar refs/tags/tagFoo:refs/tags/tagFoo\necho \"$GRAFT_CHILD $GRAFT_PARENT\" >> .git/info/grafts\ngit filter-branch --tag-name-filter cat tagFoo\ncd ..\n\n# Purposely use a file URL to prune any unreferenced objects\ngit clone file:///`pwd`/foo newFoo\n\nMy output from the clone:\n\n   Cloning into newFoo...\n   remote: Counting objects: 9, done.\n   remote: Compressing objects: 100% (3/3), done.\n   remote: Total 9 (delta 0), reused 0 (delta 0)\n   Receiving objects: 100% (9/9), done.\n   error: refs/tags/tagFoo does not point to a valid object!\n\nThanks,\nStephen\n"},{"id":"148564","messageId":"20100820133925.GE9448@kytes","threadId":"24795","inReplyTo":"5752362.117414.1282309220152.JavaMail.root@mail.hq.genarts.com","subject":"Re: Tags, Grafts, and Clones, oh my!","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-08-20T13:39:32Z","receivedAt":"2010-08-20T13:39:32Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Stephen,\n\nStephen Bash writes:\n> Sure:\n> \n> mkdir gitTagBug\n> cd gitTagBug\n> \n> git init foo\n> cd foo\n> echo A >> foo.txt\n> git add foo.txt\n> git commit -m \"Created foo\"\n> GRAFT_PARENT=`git rev-parse master`\n> echo B >> foo.txt\n> git commit -am \"Added B\"\n> cd ..\n> \n> git init bar\n> cd bar\n> echo A >> foo.txt\n> echo C >> foo.txt\n> git add foo.txt\n> git commit -m \"Added C\"\n> GRAFT_CHILD=`git rev-parse master`\n> git tag -am \"Tagging foo C\" tagFoo $GRAFT_CHILD\n> cd ..\n> \n> cd foo\n> git fetch ../bar refs/tags/tagFoo:refs/tags/tagFoo\n> echo \"$GRAFT_CHILD $GRAFT_PARENT\" >> .git/info/grafts\n> git filter-branch --tag-name-filter cat tagFoo\n> cd ..\n> \n> # Purposely use a file URL to prune any unreferenced objects\n> git clone file:///`pwd`/foo newFoo\n> \n> My output from the clone:\n> \n>    Cloning into newFoo...\n>    remote: Counting objects: 9, done.\n>    remote: Compressing objects: 100% (3/3), done.\n>    remote: Total 9 (delta 0), reused 0 (delta 0)\n>    Receiving objects: 100% (9/9), done.\n>    error: refs/tags/tagFoo does not point to a valid object!\n\nThanks for the testcase! Offhand, it definitely looks like a bug. I'm\ninvestigating to figure out which part of the chain is at fault.\n\n-- Ram\n"},{"id":"148572","messageId":"5054888.117464.1282315250235.JavaMail.root@mail.hq.genarts.com","threadId":"24795","inReplyTo":"20100820133925.GE9448@kytes","subject":"Re: Tags, Grafts, and Clones, oh my!","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2010-08-20T14:40:50Z","receivedAt":"2010-08-20T14:40:50Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"> Thanks for the testcase! Offhand, it definitely looks like a bug. I'm\n> investigating to figure out which part of the chain is at fault.\n\nNo problem.  I was very happily to isolate it outside the large repo I was working with...\n\nContrary to my first e-mail, 'git checkout -b branchFoo tagFoo' doesn't solve the problem...  My guess is I got lazy and did a \"normal\" local clone (no file URL) when I was testing.\n\nData from further testing:\n  - doing a \"normal\" local clone doesn't emit the error\n  - a remote clone over ssh does emit the error (so it's not just file:///)\n  - in a brand new repo (init'ed, not cloned) 'git fetch ../foo refs/tags/tagFoo:refs/tags/tagFoo' fails:\n      error: unable to find 28fffee... (sha of tag object)\n  - in a brand new repo 'git fetch ../foo refs/heads/branchFoo:ref/heads/branchFoo' succeeds, and correctly fetches tagFoo  (where branchFoo is created via 'git checkout -b branchFoo tagFoo')\n\n\nThanks,\nStephen\n"},{"id":"148595","messageId":"32751199.117741.1282331329553.JavaMail.root@mail.hq.genarts.com","threadId":"24795","inReplyTo":"5054888.117464.1282315250235.JavaMail.root@mail.hq.genarts.com","subject":"refs/original breaks git-clone for tags (was Re: Tags, Grafts, and Clones, oh my!)","fromName":"Stephen Bash","fromEmail":"bash@genarts.com","sentAt":"2010-08-20T19:08:49Z","receivedAt":"2010-08-20T19:08:49Z","isPatch":false,"sender":{"key":"bash@genarts.com","avatar":null},"body":"> > Thanks for the testcase! Offhand, it definitely looks like a bug.\n> > I'm\n> > investigating to figure out which part of the chain is at fault.\n> \n> No problem. I was very happily to isolate it outside the large repo I\n> was working with...\n> \n> Data from further testing:\n> - doing a \"normal\" local clone doesn't emit the error\n> - a remote clone over ssh does emit the error (so it's not just\n> file:///)\n> - in a brand new repo (init'ed, not cloned) 'git fetch ../foo\n> refs/tags/tagFoo:refs/tags/tagFoo' fails:\n> error: unable to find 28fffee... (sha of tag object)\n> - in a brand new repo 'git fetch ../foo\n> refs/heads/branchFoo:ref/heads/branchFoo' succeeds, and correctly\n> fetches tagFoo (where branchFoo is created via 'git checkout -b\n> branchFoo tagFoo')\n\nAfter a lot of guess and check, it appears the issue is somehow related to the refs/original directory created by filter-branch.  If that directory is moved out of refs/ or deleted the clone succeeds.  Digging further, a simple rename of refs/original/refs/tags/tagFoo to anything else also fixes the problem.\n\nA simplified test case is:\n\ngit init foo\ncd foo\necho A >> foo.txt\ngit add foo.txt\ngit commit -m \"Created foo\"\ngit tag -am \"Tagging foo\" tagFoo\ngit filter-branch --env-filter 'export GIT_AUTHOR_NAME=xyz123' --tag-name-filter cat -- --all\ncd ..\ngit clone file:///`pwd`/foo newFoo\n\ngit clone will \"succeed\" (exit 0), but throw the error \n\n   error: refs/tags/tagFoo does not point to a valid object!\n\nand the tagFoo will not exist in the new repo.\n\n(The env-filter is arbitrary, just need something that will force a commit rewrite)  For this bug to occur, the filter-branch must create refs/original/refs/tags/tagFoo, so if the filter-branch command is \n\ngit filter-branch --env-filter 'export GIT_AUTHOR_NAME=xyz123' --tag-name-filter cat master\n\nfilter-branch will happily rewrite the tag, but won't create the offending file, so the clone will succeed without error (and the tag will exist in the new repo).  \n\nRemoving refs/original is a pretty trivial work-around, so I'm going to modify my scripts and continue working on my SVN transition.  Let me know if I can be of any assistance tracking down the actual bug.\n\nThanks,\nStephen\n"},{"id":"150258","messageId":"20100907231400.GA2024@capella.cs.uchicago.edu","threadId":"24795","inReplyTo":"32751199.117741.1282331329553.JavaMail.root@mail.hq.genarts.com","subject":"Re: refs/original breaks git-clone for tags","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-09-07T23:14:00Z","receivedAt":"2010-09-07T23:14:00Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Stephen,\n\nStephen Bash wrote:\n\n> After a lot of guess and check, it appears the issue is somehow\n> related to the refs/original directory created by filter-branch.  If\n> that directory is moved out of refs/ or deleted the clone succeeds.\n> Digging further, a simple rename of refs/original/refs/tags/tagFoo\n> to anything else also fixes the problem.\n\nThanks for a clear report.  I backported your test case:\n\n-- 8< --\n#!/bin/sh\nmkdir foo &&\n(\n\tcd foo &&\n\tgit init &&\n\techo A >foo.txt &&\n\tgit add foo.txt &&\n\tgit commit -m \"Created foo\" &&\n\tgit tag -am \"Tagging foo\" tagFoo &&\n\tgit filter-branch --env-filter 'export GIT_AUTHOR_NAME=xyz123' \\\n\t\t\t\t--tag-name-filter cat -- --all &&\n\tgit show-ref\n) &&\ngit clone file:///`pwd`/foo newFoo &&\n(\n\tcd newFoo &&\n\tgit show-ref\n)\n-- >8 --\n\nwhich produces the result:\n\n ...\n 07fa3d4c3aba83b293e24a12a87faca363ad34e6 refs/heads/master\n 90a32c29df06b4f6e4218c1b82b35a3e49aed9f2 refs/original/refs/heads/master\n 90a32c29df06b4f6e4218c1b82b35a3e49aed9f2 refs/original/refs/tags/tagfoo\n ef92777a31663135426a9648d0abb0c06b448fbe refs/tags/tagfoo\n ...\n Resolving deltas: 100% (1/1), done.\n error: refs/tags/tagfoo does not point to a valid object!\n 07fa3d4c3aba83b293e24a12a87faca363ad34e6 refs/heads/master\n 07fa3d4c3aba83b293e24a12a87faca363ad34e6 refs/remotes/origin/HEAD\n 07fa3d4c3aba83b293e24a12a87faca363ad34e6 refs/remotes/origin/master\n $\n\nThe error message bisects to:\n\ncommit 5bdc32d3e50d8335c65e136e6b5234c5dd92a7a9 (tags/v1.6.5-rc3~20)\nAuthor: Nicolas Pitre <nico@fluxnic.net>\nDate:   Fri Sep 25 23:54:42 2009 -0400\n\n    make 'git clone' ask the remote only for objects it cares about\n    \n    Current behavior of 'git clone' when not using --mirror is to fetch\n    everything from the peer, and then filter out unwanted refs just before\n    writing them out to the cloned repository.  This may become highly\n    inefficient if the peer has an unusual ref namespace, or if it simply\n    has \"remotes\" refs of its own, and those locally unwanted refs are\n    connecting to a large set of objects which becomes unreferenced as soon\n    as they are fetched.\n    \n    Let's filter out those unwanted refs from the peer _before_ asking it\n    what refs we want to fetch instead, which is the most logical thing to\n    do anyway.\n    \n    Signed-off-by: Nicolas Pitre <nico@fluxnic.net>\n    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>\n\nThat is not too surprising.  Before then, \"git clone\" fetched\nall objects.\n\nThe error message is from refs.c::do_one_ref(), which notices the\nmissing object ef92777a31663135426a9648d0abb0c06b448fbe (i.e., ye\nold tag object).\n\nCall chain:\n\n cmd_clone() ->\n  write_remote_refs() ->\n   pack_refs() ->\n    do_for_each_ref() ->\n     do_one_ref()\n\nThe transport is done afaict by then.\n\nNico, Shawn: ideas?\n"}]}