threads / discuss / 27497

Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Subject: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

## tl;dr

15 messages between May 29, 2011 and May 30, 2011.

replies: 14people: 5as markdown or json

Jérémie NIKAES· May 29, 2011, 16:09 UTC · lore

Hi, I am currently working on the git-mediawiki gateway project. This project uses fast-import to get data from mediawiki. For each revision from mediawiki, we generate a chunk of fast-export stream looking like this :

commit refs/remotes/origin/master mark :<int> commiter <user> <address> <timestamp> +0000 data <sizeofcomment> <comment> M 644 inline <title>.wiki data <sizeoffile> <content>

At the end of a git clone mediawiki::http://ourwikitest, everything looks perfectly fine, including the git log, but a "Warning duplicated ref: refs/remotes/origin/master" is thrown. And indeed, it is duplicated in .git/packedrefs

A git gc solves everything, but I wondered if there was a way to fix this or if this was a known bug.

Thank you, Regards,

-- Jérémie Nikaes

Jonathan Nieder· May 29, 2011, 23:24 UTC · re: Jérémie NIKAES · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Hi Jérémie,
Jérémie NIKAES wrote:
> At the end of a git clone mediawiki::http://ourwikitest, everything
> looks perfectly fine
I just wanted to say, thank you for working on this[1].

This code answers some questions I had about the remote helper protocol, among other things.

> a "Warning duplicated
> ref: refs/remotes/origin/master" is thrown.

To end the suspense: I assume this was fixed by 9055039c (Bug fix for pulls with multiple revisions pulled, 2011-05-29).

Cheers, Jonathan

[1] git://github.com/Bibzball/Git-Mediawiki.git
Jérémie NIKAES· May 30, 2011, 14:44 UTC · re: Jonathan Nieder · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Jonathan,
> I just wanted to say, thank you for working on this[1].
>
> This code answers some questions I had about the remote helper
> protocol, among other things.
Glad we can help !
Show 6 quoted lines
>
>> a "Warning duplicated
>> ref: refs/remotes/origin/master" is thrown.
>
> To end the suspense: I assume this was fixed by 9055039c (Bug fix for
> pulls with multiple revisions pulled, 2011-05-29).

No, I actually still have the problem. The version commited on github is pulling revisions straight to refs/heads/master which is gross After discussing this matter with our teacher Matthieu Moy, I wanted to change this to refs/remotes/origin/master but then this warning gets thrown.

Does anyone have an idea ?

Thanks, Cheers,

-- 
Jérémie Nikaes
Jonathan Nieder· May 30, 2011, 14:52 UTC · re: Jérémie NIKAES · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Jérémie NIKAES wrote:
>> Jérémie NIKAES wrote:
>>> a "Warning duplicated
>>> ref: refs/remotes/origin/master" is thrown.
[...]
Show 5 quoted lines
> No, I actually still have the problem. The version commited on github
> is pulling revisions straight to refs/heads/master which is gross
> After discussing this matter with our teacher Matthieu Moy, I wanted
> to change this to refs/remotes/origin/master but then this warning
> gets thrown.

Could you push a branch exhibiting the problem? The reason I ask is that searching with

	git grep -F -e 'duplicated ref' origin/master

in git.git yields no hits for me, so I wanted to reproduce this to get the exact error message or figure out which program is printing it.

Thanks.
Jérémie NIKAES· May 30, 2011, 15:07 UTC · re: Jonathan Nieder · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Show 11 quoted lines
>
> Could you push a branch exhibiting the problem?  The reason I ask
> is that searching with
>
>        git grep -F -e 'duplicated ref' origin/master
>
> in git.git yields no hits for me, so I wanted to reproduce this to get
> the exact error message or figure out which program is printing it.
>
> Thanks.
>

Ok I just committed my changes to the master branch, feel free to pull and try it out.

Thanks for your help
-- 
Jérémie Nikaes
Jonathan Nieder· May 30, 2011, 15:52 UTC · re: Jérémie NIKAES · lore

[PATCH] Documentation: do not misinterpret refspecs as bold text (Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master)

Jérémie NIKAES wrote:
> Ok I just committed my changes to the master branch, feel free to pull
> and try it out.
Nice.  Now I get
| $ git fetch --all
[...]
| Fetching & writing export data...
| 1/2: Revision n°7 of New_page
| 2/2: Revision n°8 of New_page
| 
| error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
| From mediawiki::http://localhost/mediawiki
|  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
| error: Could not fetch origin

which means that the transport machinery thought the helper was going to be fetching directly to "master". I suspect you will want a 'refspec' capability like

	refspec refs/heads/*:refs/mediawiki/${remotename}/*
to fix this.

Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map private ref names into normal names, 2009-11-18). What namespace should a helper use when asked to fetch to FETCH_HEAD without a remote name, like

	git fetch mediawiki::testwiki
?
-- >8 --
Subject: Documentation: do not misinterpret refspecs as bold text

In v1.7.3.3~2 (Documentation: do not misinterpret pull refspec as bold text, 2010-12-03) many uses of asterisks in expressions like "refs/heads/*:refs/svn/origin/branches/*" were escaped as {asterisk} to avoid being treated as delimiters for bold text, but these two were missed.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-remote-helpers.txt |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 87cd11f..58f6ad4 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -181,11 +181,11 @@ CAPABILITIES
 	When using the import command, expect the source ref to have
 	been written to the destination ref. The earliest applicable
 	refspec takes precedence. For example
-	"refs/heads/*:refs/svn/origin/branches/*" means that, after an
-	"import refs/heads/name", the script has written to
+	"refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}" means
+	that, after an "import refs/heads/name", the script has written to
 	refs/svn/origin/branches/name. If this capability is used at
 	all, it must cover all refs reported by the list command; if
-	it is not used, it is effectively "*:*"
+	it is not used, it is effectively "{asterisk}:{asterisk}"
 
 REF LIST ATTRIBUTES
 -------------------
-- 
1.7.5.3
Jonathan Nieder· May 30, 2011, 16:40 UTC · re: Jonathan Nieder · lore

remote helpers: best practices for using the "refspec" capability

Jonathan Nieder wrote:
Show 21 quoted lines
> | error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
> | From mediawiki::http://localhost/mediawiki
> |  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
> | error: Could not fetch origin
>
> which means that the transport machinery thought the helper was going
> to be fetching directly to "master".  I suspect you will want a
> 'refspec' capability like
>
>	refspec refs/heads/*:refs/mediawiki/${remotename}/*
>
> to fix this.
>
> Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map
> private ref names into normal names, 2009-11-18).  What namespace
> should a helper use when asked to fetch to FETCH_HEAD without a remote
> name, like
>
>	git fetch mediawiki::testwiki
>
> ?
Actually cc-ing this time.  Sorry for the noise.
Daniel Barkalow· May 30, 2011, 17:43 UTC · re: Jonathan Nieder · lore

Re: remote helpers: best practices for using the "refspec" capability

On Mon, 30 May 2011, Jonathan Nieder wrote:
Show 23 quoted lines
> Jonathan Nieder wrote:
> 
> > | error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
> > | From mediawiki::http://localhost/mediawiki
> > |  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
> > | error: Could not fetch origin
> >
> > which means that the transport machinery thought the helper was going
> > to be fetching directly to "master".  I suspect you will want a
> > 'refspec' capability like
> >
> >	refspec refs/heads/*:refs/mediawiki/${remotename}/*
> >
> > to fix this.
> >
> > Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map
> > private ref names into normal names, 2009-11-18).  What namespace
> > should a helper use when asked to fetch to FETCH_HEAD without a remote
> > name, like
> >
> >	git fetch mediawiki::testwiki
> >
> > ?

The main purpose of having the private ref names is to support incremental imports, where you obviously need to identify the sha1 of the last thing you imported, and you may have included sufficient machine-readable information in your imported commit messages to know what the foreign state was that generated that sha1. (And you need a namespace because you may have multiple refs that you imported.) I'm not sure that it makes much sense to do incremental imports without a remote name, since that's generally an operation you're not planning do repeatedly.

But the purpose of the namespacing is to be able to continue the correct incremental import, so it would make sense to do some arbitrary transformation to make your url be a valid ref directory, and use that.

It would make sense to add support for a namespace where fast-import can write whatever it wants, and it'll get discarded after the fetch is done, if it's the case that anyone can stand not having incremental imports.

	-Daniel
*This .sig left intentionally blank*
Sverre Rabbelier· May 30, 2011, 18:04 UTC · re: Daniel Barkalow · lore

Re: remote helpers: best practices for using the "refspec" capability

Heya,
On Mon, May 30, 2011 at 12:43, Daniel Barkalow <barkalow@iabervon.org> wrote:
Show 7 quoted lines
> But the purpose of the namespacing is to be able to continue the correct
> incremental import, so it would make sense to do some arbitrary
> transformation to make your url be a valid ref directory, and use that.
>
> It would make sense to add support for a namespace where fast-import can
> write whatever it wants, and it'll get discarded after the fetch is done,
> if it's the case that anyone can stand not having incremental imports.

I remember a patch series by someone else a lot more that _disabled_ creating of refs during fast-import, and instead allowed for specifying what refs to update with which values at the end of the import.

-- 
Cheers,

Sverre Rabbelier
Jonathan Nieder· May 30, 2011, 17:35 UTC · re: Jérémie NIKAES · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Jérémie NIKAES wrote:
Show 5 quoted lines
> No, I actually still have the problem. The version commited on github
> is pulling revisions straight to refs/heads/master which is gross
> After discussing this matter with our teacher Matthieu Moy, I wanted
> to change this to refs/remotes/origin/master but then this warning
> gets thrown.
The following fixes it for me.
-- >8 --
Subject: use refs/mediawiki/<remote> namespace for private refs

Unless the "refspec" capability is used, the remote helper machinery expects remote helpers to write remote refs/heads/master ref to refs/heads/master. That's not very good since it scribbles on top of whatever local changes the user might have.

So specify a refspec that puts the refs elsewhere. For now this only works when fetching from a named remote, like so:

	git remote add origin mediawiki::testwiki
	git fetch origin
Attempting to fetch directly from a URL with a command like
"git fetch mediawiki::testwiki" produces an "Invalid refspec"
error.
---
 git-remote-mediawiki |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/git-remote-mediawiki b/git-remote-mediawiki
index af66b58..7a3fdd3 100755
--- a/git-remote-mediawiki
+++ b/git-remote-mediawiki
@@ -7,6 +7,7 @@ use Storable qw(freeze thaw);
 use DateTime::Format::ISO8601;
 use Encode qw(encode_utf8);
 
+my $remotename = $ARGV[0];
 my $url = $ARGV[1];
 
 print STDERR "$url\n";
@@ -62,6 +63,7 @@ while ($loop) {
 
 sub mw_capabilities {
 #	print STDOUT "fetch\n";
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
 	print STDOUT "import\n";
 	print STDOUT "list\n";
 	print STDOUT "option\n";
@@ -91,7 +93,7 @@ sub mw_import {
 	
 	sub get_last_revision {
 		# Get last commit sha1
-		my $commit_sha1 = `git rev-parse refs/remotes/origin/master 2>/dev/null`;
+		my $commit_sha1 = `git rev-parse refs/mediawiki/$remotename/master 2>/dev/null`;
 
 		# Get note regarding that commit
 		chomp($commit_sha1);
@@ -202,15 +204,15 @@ sub mw_import {
 		
 		# If it's a clone and the first revision, initalize this ref
 		if ($fetch_from == 1 && $n == 1) {
-			#print "reset refs/remotes/origin/master\n";
+			#print "reset refs/mediawiki/$remotename/master\n";
 		}
-		print "commit refs/remotes/origin/master\n";
+		print "commit refs/mediawiki/$remotename/master\n";
 		print "mark :$n\n";
 		print "committer $user <$user\@$wiki_name> ", $dt->epoch, " +0000\n";
 		print "data ", bytes::length(encode_utf8($comment)), "\n", encode_utf8($comment);
 		# If it's not a clone, needs to know where to start from
 		if ($fetch_from != 1 && $n == 1) {
-			print "from refs/remotes/origin/master^0\n";
+			print "from refs/mediawiki/$remotename/master^0\n";
 		}
 		print "M 644 inline $title.wiki\n";
 		print "data ", bytes::length(encode_utf8($content)), "\n", encode_utf8($content);
-- 
1.7.5.3
Matthieu Moy· May 30, 2011, 19:46 UTC · re: Jonathan Nieder · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Jonathan Nieder <jrnieder@gmail.com> writes:
Show 9 quoted lines
> Jérémie NIKAES wrote:
>
>> No, I actually still have the problem. The version commited on github
>> is pulling revisions straight to refs/heads/master which is gross
>> After discussing this matter with our teacher Matthieu Moy, I wanted
>> to change this to refs/remotes/origin/master but then this warning
>> gets thrown.
>
> The following fixes it for me.
It seems it does, thanks!
clone does not give the warning anymore, and incremental pull work.
>  sub mw_capabilities {
>  #	print STDOUT "fetch\n";
> +	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";

Is this "mediawiki" comming from the URL (mediawiki::...), or is it just a convention you've set?

We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without success.

Do I understand correctly that the "pull" is done in 3 stages:
1) import into refs/mediawiki/origin
2) fetch the imported ref into refs/remotes/origin
3) and merge as usual into the current branch
?

If so, I don't understand why the distinction between 1) and 2) is necessary, and why not to fetch directly into refs/remotes/origin/. IOW, in which case is refs/remotes/origin/master different from refs/mediawiki/origin/master for example?

Thanks,
-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
Jérémie NIKAES· May 30, 2011, 19:55 UTC · re: Matthieu Moy · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

2011/5/30 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:
Show 16 quoted lines
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
>> Jérémie NIKAES wrote:
>>
>>> No, I actually still have the problem. The version commited on github
>>> is pulling revisions straight to refs/heads/master which is gross
>>> After discussing this matter with our teacher Matthieu Moy, I wanted
>>> to change this to refs/remotes/origin/master but then this warning
>>> gets thrown.
>>
>> The following fixes it for me.
>
> It seems it does, thanks!
>
> clone does not give the warning anymore, and incremental pull work.
>
Yes ! It definitely does ! Works like a charm.
Show 26 quoted lines
>>  sub mw_capabilities {
>>  #    print STDOUT "fetch\n";
>> +     print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
>
> Is this "mediawiki" comming from the URL (mediawiki::...), or is it just
> a convention you've set?
>
> We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without
> success.
>
> Do I understand correctly that the "pull" is done in 3 stages:
>
> 1) import into refs/mediawiki/origin
>
> 2) fetch the imported ref into refs/remotes/origin
>
> 3) and merge as usual into the current branch
>
> ?
>
> If so, I don't understand why the distinction between 1) and 2) is
> necessary, and why not to fetch directly into refs/remotes/origin/. IOW,
> in which case is refs/remotes/origin/master different from
> refs/mediawiki/origin/master for example?
>
> Thanks,

We just committed this change to our master branch on github. We actually have the same questions as Matthieu. We don't really understand why we have to use another ref to make it work.

Huge thanks to you,
-- 
Jérémie Nikaes
Jonathan Nieder· May 30, 2011, 20:13 UTC · re: Matthieu Moy · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Hi,
Matthieu Moy wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
Show 9 quoted lines
>>  sub mw_capabilities {
>>  #	print STDOUT "fetch\n";
>> +	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
>
> Is this "mediawiki" comming from the URL (mediawiki::...), or is it just
> a convention you've set?
>
> We've tried with refspec refs/heads/*:refs/remotes/origin/*, but without
> success.

Yes, I stole the convention for the namespace name from the commit message to v1.7.0-rc0~62^2~19 (Allow helper to map private ref names into normal names, 2009-11-18).

Show 14 quoted lines
> Do I understand correctly that the "pull" is done in 3 stages:
>
> 1) import into refs/mediawiki/origin
>
> 2) fetch the imported ref into refs/remotes/origin
>
> 3) and merge as usual into the current branch
>
> ?
>
> If so, I don't understand why the distinction between 1) and 2) is
> necessary, and why not to fetch directly into refs/remotes/origin/. IOW,
> in which case is refs/remotes/origin/master different from
> refs/mediawiki/origin/master for example?

That's a good question. One answer is that fetching to elsewhere allows the transport machinery to take care of rejecting non-fast-forward fetches and pretty-printing the ref update information:

 From mediawiki::http://localhost/mediawiki
    479ee21..accd2c1  master     -> origin/master

I think in the long term, importing to no ref at all as Sverre suggested and teaching the transport-helper to do the ref update would make more sense.

Regards, Jonathan

Jérémie NIKAES· May 30, 2011, 20:32 UTC · re: Jonathan Nieder · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

> Matthieu Moy wrote:
>> Jonathan Nieder <jrnieder@gmail.com> writes:
Show 15 quoted lines
>> If so, I don't understand why the distinction between 1) and 2) is
>> necessary, and why not to fetch directly into refs/remotes/origin/. IOW,
>> in which case is refs/remotes/origin/master different from
>> refs/mediawiki/origin/master for example?
>
> That's a good question.  One answer is that fetching to elsewhere
> allows the transport machinery to take care of rejecting non-fast-forward
> fetches and pretty-printing the ref update information:
>
>  From mediawiki::http://localhost/mediawiki
>    479ee21..accd2c1  master     -> origin/master
>
> I think in the long term, importing to no ref at all as Sverre
> suggested and teaching the transport-helper to do the ref update would
> make more sense.

Well yes that would indeed make more sense. It feels weird to have to do it like this. Anyways, thanks a lot again for your help, a rfc patch should come relatively quickly with the project in its current state.

Regards,
-- 
Jérémie Nikaes
Sverre Rabbelier· May 30, 2011, 23:09 UTC · re: Jérémie NIKAES · lore

Re: Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Heya,
On Mon, May 30, 2011 at 15:32, Jérémie NIKAES <jeremie.nikaes@gmail.com> wrote:
> Well yes that would indeed make more sense. It feels weird to have to
> do it like this.
The patch series I was referring to can be found at [0].
[0] http://thread.gmane.org/gmane.comp.version-control.git/157860/focus=157864
-- 
Cheers,

Sverre Rabbelier

← back to recent threads