{"thread":{"id":"7863","subject":"[PATCH] [PATCH] git-mirror - exactly mirror another repository","startedAt":"2007-04-27T02:15:05Z","lastAt":"2007-04-27T18:00:12Z","messageCount":7,"participants":["Petr Baudis","A Large Angry SCM","Johannes Schindelin","Andy Whitcroft"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"40546","messageId":"20070427021505.1740.58136.stgit@rover","threadId":"7863","inReplyTo":null,"subject":"[PATCH] [PATCH] git-mirror - exactly mirror another repository","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2007-04-27T02:15:05Z","receivedAt":"2007-04-27T02:15:05Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Sometimes its handy to be able to efficiently backup or mirror one\nGit repository to another Git repository by employing the native\nGit object transfer protocol.  But when mirroring or backing up a\nrepository you really want:\n\n  1) Every object in the source to go to the mirror.\n  2) Every ref in the source to go to the mirror.\n  3) Any ref removed from the source to be removed from the mirror.\n\nand since git-fetch doesn't do 2 and 3, here's a tool that does.\n\nThis is based on Shawn Pearce's patch from 25 Sep 2006, updated to take\nJunio's and Sergey's review into account, to use few newer pieces of Git\ninfrastructure and with few trivial tweaks. The repacking part was dropped\nsince git-fetch does that on its own now.\n\nI actually still would kind of prefer this to be a git-fetch's feature but\nthe general mood seems to be to have this as a separate command and I can't\nsay I care at all.\n\nSigned-off-by: Petr Baudis <pasky@suse.cz>\n---\n\n .gitignore                   |    1 \n Documentation/config.txt     |    6 ++\n Documentation/git-mirror.txt |   54 +++++++++++++++++++++\n Makefile                     |    2 -\n git-mirror.perl              |  110 ++++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 172 insertions(+), 1 deletions(-)\n\ndiff --git a/.gitignore b/.gitignore\nindex 4dc0c39..d0b67da 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -80,6 +80,7 @@ git-merge-resolve\n git-merge-stupid\n git-merge-subtree\n git-mergetool\n+git-mirror\n git-mktag\n git-mktree\n git-name-rev\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex e0aff53..e05e4c5 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -513,6 +513,12 @@ log.showroot::\n \tTools like gitlink:git-log[1] or gitlink:git-whatchanged[1], which\n \tnormally hide the root commit will now show it. True by default.\n \n+mirror.allowed::\n+\tIf true, gitlink:git-mirror[1] will be allowed to run on the\n+\trepository.  Please see its documentation for all the implications.\n+\n+mirror.\n+\n merge.summary::\n \tWhether to include summaries of merged commits in newly created\n \tmerge commit messages. False by default.\ndiff --git a/Documentation/git-mirror.txt b/Documentation/git-mirror.txt\nnew file mode 100644\nindex 0000000..0909124\n--- /dev/null\n+++ b/Documentation/git-mirror.txt\n@@ -0,0 +1,54 @@\n+git-mirror(1)\n+============\n+\n+NAME\n+----\n+git-mirror - Exactly mirror another repository\n+\n+\n+SYNOPSIS\n+--------\n+'git-mirror' <repository>\n+\n+\n+DESCRIPTION\n+-----------\n+Completely mirrors another repository into the local repository.\n+\n+All heads and tags from the other repository are copied to the\n+local repository without any regard for merging.  This means\n+that all heads and tags will be FORCIBLY CHANGED in the local\n+repository to make them match the other repository.  Any local\n+ref or tags which has been deleted from the other repository\n+will also be deleted from the local repository.\n+\n+After mirroring is complete, in case the 'HEAD' symref does not\n+seem to match locally the remote one, `git-mirror` will attempt\n+to guess which head is now the 'HEAD'; if there are more candidates,\n+one will be chosen roughly randomly.\n+\n+\n+CONFIGURATION\n+-------------\n+\n+Prior to updating the local repository 'git-mirror' requires\n+that the user set 'mirror.allowed' to a true value in the local\n+repository's config file.  This is considered to be a safety\n+latch which is intended to prevent accidental overwriting of\n+the local repository.\n+\n+\n+SEE ALSO\n+--------\n+gitlink:git-fetch[1]\n+gitlink:git-prune[1]\n+gitlink:git-repack[1]\n+\n+\n+Author\n+------\n+Written by Shawn Pearce <spearce@spearce.org> and Petr Baudis <pasky@suse.cz>\n+\n+GIT\n+---\n+Part of the gitlink:git[7] suite\ndiff --git a/Makefile b/Makefile\nindex 60c41fd..5fa370f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -208,7 +208,7 @@ SCRIPT_PERL = \\\n \tgit-archimport.perl git-cvsimport.perl git-relink.perl \\\n \tgit-cvsserver.perl git-remote.perl \\\n \tgit-svnimport.perl git-cvsexportcommit.perl \\\n-\tgit-send-email.perl git-svn.perl\n+\tgit-send-email.perl git-svn.perl git-mirror.perl\n \n SCRIPT_PYTHON = \\\n \tgit-p4import.py\ndiff --git a/git-mirror.perl b/git-mirror.perl\nnew file mode 100644\nindex 0000000..22ea879\n--- /dev/null\n+++ b/git-mirror.perl\n@@ -0,0 +1,110 @@\n+#!/usr/bin/env perl\n+# Copyright (C) 2006, Shawn Pearce <spearce@spearce.org>\n+# This file is licensed under the GPL v2, or a later version\n+# at the discretion of Linus.\n+\n+use warnings;\n+use strict;\n+use Git;\n+\n+my $remote = shift || 'origin';\n+my $repo = Git->repository();\n+\n+# Verify its OK to execute in this repository.\n+#\n+unless ($repo->config_bool('mirror.allowed')) {\n+\tprint STDERR <<EOF;\n+Error: mirror.allowed is false.\n+Error:\n+Error: For safety reasons please set mirror.allowed in this repository's\n+Error: config before using this command.\n+Error:\n+Error: Unless you are using this repository ONLY for mirroring another\n+Error: repository you probably don't want to do this. Please see the\n+Error: manpage for more details.\n+EOF\n+\texit 1;\n+}\n+\n+# Build our list of refs.\n+#\n+my $remote_refs = ls_refs($repo, $remote);\n+my $local_refs = ls_refs($repo, $repo->repo_path());\n+my $remote_HEAD = $remote_refs->{'HEAD'};\n+my $local_HEAD = $local_refs->{'HEAD'};\n+delete $remote_refs->{'HEAD'};\n+delete $local_refs->{'HEAD'};\n+\n+# Execute the fetch for any refs which differ from our own.\n+# We don't worry about trying to optimize for rewinds or\n+# exact branch copies as they are rather uncommon.\n+#\n+my @to_fetch;\n+while (my ($ref, $hash) = each %$remote_refs) {\n+\tpush(@to_fetch, \"$ref:$ref\")\n+\t\tif (!$local_refs->{$ref} || $local_refs->{$ref} ne $hash);\n+}\n+if (@to_fetch) {\n+\tgit_cmd_try {\n+\t\t$repo->command_noisy('fetch',\n+\t\t\t'--force',\n+\t\t\t'--update-head-ok',\n+\t\t\t$remote, sort @to_fetch);\n+\t} '%s failed w/ code %d';\n+} else {\n+\tprint \"No changed refs.  Skipping fetch.\\n\";\n+}\n+\n+# See what the remote has HEAD pointing at and update our local\n+# HEAD to point at some ref which points at the same hash.\n+# Prefer to keep HEAD the same if possible to avoid HEAD drifting\n+# between different branches.\n+# Note that with dumb protocols, we don't get to *know* HEAD implicitly\n+# with git-ls-remote...\n+#\n+git_cmd_try {\n+\tmy $headref = $repo->command_oneline('symbolic-ref', 'HEAD');\n+\tmy $HEAD;\n+\tif (not $remote_refs->{$headref}) {\n+\t\t$HEAD = 'refs/heads/master';\n+\t\tprint \"Local HEAD branch disappeared, falling back to refs/heads/master\\n\";\n+\t} elsif ($remote_HEAD and $remote_refs->{$headref} ne $remote_HEAD) {\n+\t\tmy %by_hash = map {$remote_refs->{$_} => $_}\n+\t\t\tgrep {m,^refs/heads/,}\n+\t\t\tsort keys %$remote_refs;\n+\t\t$HEAD = $by_hash{$remote_HEAD};\n+\t\tif ($HEAD) {\n+\t\t\tprint \"Setting HEAD to $HEAD ($remote_HEAD)\\n\";\n+\t\t} else {\n+\t\t\tprint \"Remote HEAD ($remote_HEAD) does not match any remote branch\\n\";\n+\t\t}\n+\t}\n+\tif ($HEAD) {\n+\t\t$repo->command_noisy('symbolic-ref', 'HEAD', $HEAD);\n+\t}\n+} '%s failed w/ code %d';\n+\n+# Delete any local refs which the server no longer contains.\n+#\n+foreach my $ref (keys %$local_refs) {\n+\tnext if $remote_refs->{$ref};\n+\tprint \"Removing $ref\\n\";\n+\tgit_cmd_try {\n+\t\t$repo->command_noisy('update-ref', '-d', $ref, $local_refs->{$ref});\n+\t} '%s failed w/ code %d';\n+}\n+\n+sub ls_refs {\n+\tmy $repo = shift;\n+\tmy $name = shift;\n+\tmy $refs = $repo->remote_refs($name);\n+\tmy @interesting = grep {\n+\t\t$_ eq 'HEAD' or (/^refs\\// and not /\\.\\./ and not /\\^{}$/);\n+\t} keys %$refs;\n+\n+\tmy %refs2;\n+\t# This funky-looking expression puts @interesting-subset of %$refs\n+\t# to %refs2.\n+\t@refs2{@interesting} = @{$refs}{@interesting};\n+\t\\%refs2;\n+}\n"},{"id":"40547","messageId":"20070427021708.GT4489@pasky.or.cz","threadId":"7863","inReplyTo":"20070427021505.1740.58136.stgit@rover","subject":"Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2007-04-27T02:17:08Z","receivedAt":"2007-04-27T02:17:08Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"  Sorry for the dupe mail, accidentally slipped out. Full patch series\nshow arrive when vger's greylisting permits.\n\n\t\t\t\tPetr \"Pasky\" Baudis\n"},{"id":"40552","messageId":"4631642B.2010501@gmail.com","threadId":"7863","inReplyTo":"20070427021505.1740.58136.stgit@rover","subject":"Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2007-04-27T02:47:07Z","receivedAt":"2007-04-27T02:47:07Z","isPatch":true,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Petr Baudis wrote:\n> Sometimes its handy to be able to efficiently backup or mirror one\n> Git repository to another Git repository by employing the native\n> Git object transfer protocol.  But when mirroring or backing up a\n> repository you really want:\n> \n>   1) Every object in the source to go to the mirror.\n>   2) Every ref in the source to go to the mirror.\n>   3) Any ref removed from the source to be removed from the mirror.\n> \n> and since git-fetch doesn't do 2 and 3, here's a tool that does.\n> \n> This is based on Shawn Pearce's patch from 25 Sep 2006, updated to take\n> Junio's and Sergey's review into account, to use few newer pieces of Git\n> infrastructure and with few trivial tweaks. The repacking part was dropped\n> since git-fetch does that on its own now.\n> \n> I actually still would kind of prefer this to be a git-fetch's feature but\n> the general mood seems to be to have this as a separate command and I can't\n> say I care at all.\n> \n\nIt would be nice if this had an option to not mirror HEAD and another \noption to map the refs from (/ref/[^/])/(.*) to /ref/\\1/${repo_path}/\\2.\n"},{"id":"40563","messageId":"20070427043812.GX4489@pasky.or.cz","threadId":"7863","inReplyTo":"4631642B.2010501@gmail.com","subject":"Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2007-04-27T04:38:12Z","receivedAt":"2007-04-27T04:38:12Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Fri, Apr 27, 2007 at 04:47:07AM CEST, A Large Angry SCM wrote:\n> It would be nice if this had an option to not mirror HEAD and another \n> option to map the refs from (/ref/[^/])/(.*) to /ref/\\1/${repo_path}/\\2.\n\nWhen I try to guess why you want this, I think this is just saying \"it\nwould be nice if this was part of git-fetch and if git-fetch actually\ndid this automagically by default\". Because when you clone a given\nrepository with all its branches, you are naturally interested not only\nin the snapshot of the branch set at the point of the clone but more\nlikely just want \"anytime whatever branches are over there\". So when\nfetching a * refspec it's natural to look for new branches and fetch\nthem too; and if you do that, it should be pretty easy to also (probably\noptionally) remove stale branches.\n\nBut heck, maybe with all the nifty fetch--tools git-fetch is already\ndoing it. But there seems to be basically zero documentation, the code\nis not that easy to follow and it's too late for me to be decyphering it\nnow... \n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nEver try. Ever fail. No matter. // Try again. Fail again. Fail better.\n\t\t-- Samuel Beckett\n"},{"id":"40582","messageId":"Pine.LNX.4.64.0704271055140.12006@racer.site","threadId":"7863","inReplyTo":"20070427021505.1740.58136.stgit@rover","subject":"Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-04-27T08:56:33Z","receivedAt":"2007-04-27T08:56:33Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 27 Apr 2007, Petr Baudis wrote:\n\n> I actually still would kind of prefer this to be a git-fetch's feature \n> but the general mood seems to be to have this as a separate command and \n> I can't say I care at all.\n\nI have to admit that I share your preference. And I really would like to \nbe able to say \"git fetch --all <remote>\", because I use Git for backup \noperations, too.\n\nCiao,\nDscho\n"},{"id":"40583","messageId":"4631BB32.9080405@shadowen.org","threadId":"7863","inReplyTo":"20070427021505.1740.58136.stgit@rover","subject":"Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository","fromName":"Andy Whitcroft","fromEmail":"apw@shadowen.org","sentAt":"2007-04-27T08:58:26Z","receivedAt":"2007-04-27T08:58:26Z","isPatch":true,"sender":{"key":"apw@shadowen.org","avatar":"https://gravatar.com/avatar/d3088262854661a913ef35cc40fedcc270142d4461791142bc1ea0b2a4e2e147?d=mp&s=160"},"body":"Petr Baudis wrote:\n> Sometimes its handy to be able to efficiently backup or mirror one\n> Git repository to another Git repository by employing the native\n> Git object transfer protocol.  But when mirroring or backing up a\n> repository you really want:\n> \n>   1) Every object in the source to go to the mirror.\n>   2) Every ref in the source to go to the mirror.\n>   3) Any ref removed from the source to be removed from the mirror.\n> \n> and since git-fetch doesn't do 2 and 3, here's a tool that does.\n> \n> This is based on Shawn Pearce's patch from 25 Sep 2006, updated to take\n> Junio's and Sergey's review into account, to use few newer pieces of Git\n> infrastructure and with few trivial tweaks. The repacking part was dropped\n> since git-fetch does that on its own now.\n> \n> I actually still would kind of prefer this to be a git-fetch's feature but\n> the general mood seems to be to have this as a separate command and I can't\n> say I care at all.\n> \n> Signed-off-by: Petr Baudis <pasky@suse.cz>\n> ---\n> \n>  .gitignore                   |    1 \n>  Documentation/config.txt     |    6 ++\n>  Documentation/git-mirror.txt |   54 +++++++++++++++++++++\n>  Makefile                     |    2 -\n>  git-mirror.perl              |  110 ++++++++++++++++++++++++++++++++++++++++++\n>  5 files changed, 172 insertions(+), 1 deletions(-)\n> \n> diff --git a/.gitignore b/.gitignore\n> index 4dc0c39..d0b67da 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -80,6 +80,7 @@ git-merge-resolve\n>  git-merge-stupid\n>  git-merge-subtree\n>  git-mergetool\n> +git-mirror\n>  git-mktag\n>  git-mktree\n>  git-name-rev\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index e0aff53..e05e4c5 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -513,6 +513,12 @@ log.showroot::\n>  \tTools like gitlink:git-log[1] or gitlink:git-whatchanged[1], which\n>  \tnormally hide the root commit will now show it. True by default.\n>  \n> +mirror.allowed::\n> +\tIf true, gitlink:git-mirror[1] will be allowed to run on the\n> +\trepository.  Please see its documentation for all the implications.\n> +\n> +mirror.\n> +\n\nOk, I can see why you'd want to protect a user from going git-mirror\n<foo> and losing their local changes.  But you've only protected them\nfrom mirroring at all.  If they want to use this repo as a mirror of\nsomewhere X you have not stopped them from git-mirror Y'ing themselves.\n Now yes if you know its a mirror then I guess you can recover with a\ngit-mirror X but ...\n\nIt seems to me if you are having a config option that perhaps it should\nbe the URL from whence the mirror comes, and the command should just be\n'git mirror'.\n\n[mirror]\n\turl='somewhere'\n\n-apw\n"},{"id":"40614","messageId":"20070427180012.GA4489@pasky.or.cz","threadId":"7863","inReplyTo":"20070427043812.GX4489@pasky.or.cz","subject":"Re: [PATCH] [PATCH] git-mirror - exactly mirror another repository","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2007-04-27T18:00:12Z","receivedAt":"2007-04-27T18:00:12Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Fri, Apr 27, 2007 at 06:38:12AM CEST, Petr Baudis wrote:\n> But heck, maybe with all the nifty fetch--tools git-fetch is already\n> doing it. But there seems to be basically zero documentation, the code\n> is not that easy to follow and it's too late for me to be decyphering it\n> now... \n\nOk, so it seems that git-fetch already can do almost all of this; at\nleast it seems that I'm far from being alone in not realizing it. :-)\n\nWith the refspec +refs/heads/*:refs/heads/* and git remote prune\n(thanks, Shawn!), it seems that git-fetch can already do everything\ngit-mirror tries to do, except updating the HEAD, but especially with\nthe symref embedding in git-ls-remote that Junio proposed, even that\nshould be fixable easily, right?\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nEver try. Ever fail. No matter. // Try again. Fail again. Fail better.\n\t\t-- Samuel Beckett\n"}]}