{"thread":{"id":"5687","subject":"[PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","startedAt":"2006-09-25T04:46:41Z","lastAt":"2006-11-01T16:15:53Z","messageCount":10,"participants":["Shawn Pearce","Junio C Hamano","Sergey Vlasov","Jakub Narebski"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"27610","messageId":"20060925044641.GB15757@spearce.org","threadId":"5687","inReplyTo":null,"subject":"[PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-25T04:46:41Z","receivedAt":"2006-09-25T04:46:41Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sometimes its handy to be able to efficiently backup or mirror one\nGit repository to another Git repository by employing the native\nGit object transfer protocol.  But when mirroring or backing up a\nrepository you really want:\n\n  1) Every object in the source to go to the mirror.\n  2) Every ref in the source to go to the mirror.\n  3) Any ref removed from the source to be removed from the mirror.\n  4) Automatically repack and prune the mirror when necessary.\n\nand since git-fetch doesn't do 2, 3, and 4 here's a tool that does.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n Since 'git fetch --mirror-all' wasn't quite ready here's something\n slightly more polished.\n\n I'm hoping this gets accepted as I'm thinking it would be very\n useful for backing up a user's repository to another host on a\n regular basis, not to mention also useful for Pasky's mirroring\n service.  I'm planning on using it for backups, especially of\n repositories where branches are frequently being added and deleted.\n\n .gitignore                   |    1 \n Documentation/git-mirror.txt |   64 ++++++++++++++++++++++++\n Makefile                     |    2 -\n git-mirror.perl              |  111 ++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 177 insertions(+), 1 deletions(-)\n\ndiff --git a/.gitignore b/.gitignore\nindex 78a3a3d..5f99149 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -68,6 +68,7 @@ git-merge-recur\n git-merge-recursive\n git-merge-resolve\n git-merge-stupid\n+git-mirror\n git-mktag\n git-mktree\n git-name-rev\ndiff --git a/Documentation/git-mirror.txt b/Documentation/git-mirror.txt\nnew file mode 100644\nindex 0000000..27e5167\n--- /dev/null\n+++ b/Documentation/git-mirror.txt\n@@ -0,0 +1,64 @@\n+git-mirror(1)\n+============\n+\n+NAME\n+----\n+git-mirror - Exactly mirror another repository.\n+\n+\n+SYNOPSIS\n+--------\n+'git-mirror' <repository>\n+\n+\n+DESCRIPTION\n+-----------\n+Completely mirrors another repository into the local repository.\n+\n+All heads and tags from the other repository are copied to the\n+local repository without any regard for merging.  This means\n+that all heads and tags will be forcibly changed in the local\n+repository to make them match the other repository.  Any local\n+ref or tags which has been deleted from the other repository\n+will also be deleted from the local repository.\n+\n+After mirroring is complete the 'HEAD' symref will point at\n+any branch in 'refs/heads' which has the same SHA1 as the other\n+repository's 'HEAD' contained when 'git-mirror' started.  This\n+is simply a rough guess and may not always be accurate.\n+\n+This command will also invoke 'git-repack' and 'git-prune' if\n+the number of loose objects exceeds the configured threshold.\n+As a result this command may cause the local repository to lose\n+commits that have been removed from the other repository.\n+\n+\n+CONFIGURATION\n+-------------\n+\n+Prior to updating the local repository 'git-mirror' requires\n+that the user set 'mirror.allowed' to a true value in the local\n+repository's config file.  This is considered to be a safety\n+feature which is intended to prevent accidental overwriting of\n+the local repository.\n+\n+After updating the local repository 'git-mirror' will run\n+'git-repack -a -d' and 'git-prune' if the number of loose objects\n+exceeds the limit specified in the configuration file by\n+'mirror.maxlooseobjects' (default 100).\n+\n+\n+SEE ALSO\n+--------\n+gitlink:git-fetch[1]\n+gitlink:git-prune[1]\n+gitlink:git-repack[1]\n+\n+\n+Author\n+------\n+Written by Shawn Pearce <spearce@spearce.org>\n+\n+GIT\n+---\n+Part of the gitlink:git[7] suite\ndiff --git a/Makefile b/Makefile\nindex 9a1f23f..08fb714 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -181,7 +181,7 @@ SCRIPT_PERL = \\\n \tgit-shortlog.perl git-rerere.perl \\\n \tgit-annotate.perl git-cvsserver.perl \\\n \tgit-svnimport.perl git-cvsexportcommit.perl \\\n-\tgit-send-email.perl git-svn.perl\n+\tgit-send-email.perl git-svn.perl git-mirror.perl\n \n SCRIPT_PYTHON = \\\n \tgit-merge-recursive.py\ndiff --git a/git-mirror.perl b/git-mirror.perl\nnew file mode 100755\nindex 0000000..bff2003\n--- /dev/null\n+++ b/git-mirror.perl\n@@ -0,0 +1,111 @@\n+#!/usr/bin/env perl\n+# Copyright (C) 2006, Shawn Pearce <spearce@spearce.org>\n+# This file is licensed under the GPL v2, or a later version\n+# at the discretion of Linus.\n+\n+use warnings;\n+use strict;\n+use Git;\n+\n+sub ls_refs ($$);\n+\n+my $remote = shift || 'origin';\n+my $repo = Git->repository();\n+\n+# Verify its OK to execute in this repository.\n+#\n+my $mirror_ok = $repo->config('mirror.allowed') || 0;\n+unless ($mirror_ok =~ /^(?:true|t|yes|y|1)$/i) {\n+\tprint STDERR <<EOF;\n+error: mirror.allowed is false.\n+error:\n+error: For safety reasons please set mirror.allowed in this repository's\n+error: config before using this command.\n+error:\n+error: Unless you are using this repository ONLY for mirroring another\n+error: repository you probably don't want to do this.\n+EOF\n+\texit 1;\n+}\n+\n+# Build our list of refs.\n+#\n+my $remote_refs = ls_refs($repo, $remote);\n+my $local_refs = ls_refs($repo, $repo->repo_path());\n+my $remote_HEAD = $remote_refs->{'HEAD'};\n+delete $remote_refs->{'HEAD'};\n+delete $local_refs->{'HEAD'};\n+\n+# Delete any local refs which the server no longer contains.\n+#\n+foreach my $ref (keys %$local_refs) {\n+\tnext if $remote_refs->{$ref};\n+\tprint \"removing $ref\\n\";\n+\tmy $log = \"logs/$ref\";\n+\tunlink($repo->repo_path() . '/' . $ref);\n+\tunlink($repo->repo_path() . '/' . $log);\n+\trmdir($repo->repo_path() . '/' . $ref) while $ref =~ s,/[^/]*$,,;\n+\trmdir($repo->repo_path() . '/' . $log) while $log =~ s,/[^/]*$,,;\n+}\n+\n+# Execute the fetch for any refs which differ from our own.\n+# We don't worry about trying to optimize for rewinds or\n+# exact branch copies as they are rather uncommon.\n+#\n+my @to_fetch;\n+while (my ($ref, $hash) = each %$remote_refs) {\n+\tpush(@to_fetch, \"$ref:$ref\")\n+\t\tif (!$local_refs->{$ref} || $local_refs->{$ref} ne $hash);\n+}\n+if (@to_fetch) {\n+\tgit_cmd_try {\n+\t\t$repo->command_noisy('fetch',\n+\t\t\t'--force',\n+\t\t\t'--update-head-ok',\n+\t\t\t$remote, sort @to_fetch);\n+\t} '%s failed w/ code %d';\n+} else {\n+\tprint \"No changed refs.  Skipping fetch.\\n\";\n+}\n+\n+# See what the remote has HEAD pointing at and update our local\n+# HEAD to point at the any ref which points at the same hash.\n+#\n+my %by_hash = map {$remote_refs->{$_} => $_}\n+\tgrep {m,^refs/heads/,}\n+\tkeys %$remote_refs;\n+my $HEAD = $by_hash{$remote_HEAD} || 'refs/heads/master';\n+print \"Setting HEAD to $HEAD\\n\";\n+print \"                ($remote_HEAD)\\n\";\n+git_cmd_try {\n+\t$repo->command_noisy('symbolic-ref', 'HEAD', $HEAD);\n+} '%s failed w/ code %d';\n+\n+# Repack if we have a large number of loose objects.\n+#\n+if (@to_fetch) {\n+\tmy $count_output = $repo->command('count-objects');\n+\tmy ($cur_loose) = ($count_output =~ /^(\\d+) objects/);\n+\tmy $max_loose = $repo->config('mirror.maxlooseobjects') || 100;\n+\tif ($cur_loose >= $max_loose) {\n+\t\tgit_cmd_try {\n+\t\t\t$repo->command_noisy('repack', '-a', '-d');\n+\t\t\t$repo->command_noisy('prune');\n+\t\t} '%s failed w/ code %d';\n+\t}\n+}\n+\n+sub ls_refs ($$) {\n+\tmy $repo = shift;\n+\tmy $name = shift;\n+\tmy ($fh, $c) = $repo->command_output_pipe('ls-remote', $name);\n+\tmy %refs;\n+\twhile (<$fh>) {\n+\t\tchomp;\n+\t\tnext if /\\^{}$/;\n+\t\tmy ($hash, $ref) = split(/\\t/, $_, 2);\n+\t\t$refs{$ref} = $hash if ($ref eq 'HEAD' || $ref =~ m,^refs/,);\n+\t}\n+\t$repo->command_close_pipe($fh, $c);\n+\t\\%refs;\n+}\n-- \n1.4.2.1.gde2b2-dirty\n"},{"id":"27654","messageId":"7vpsdjtew4.fsf@assigned-by-dhcp.cox.net","threadId":"5687","inReplyTo":"20060925044641.GB15757@spearce.org","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-26T04:45:15Z","receivedAt":"2006-09-26T04:45:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shawn Pearce <spearce@spearce.org> writes:\n\n> Sometimes its handy to be able to efficiently backup or mirror one\n> Git repository to another Git repository by employing the native\n> Git object transfer protocol.  But when mirroring or backing up a\n> repository you really want:\n>\n>   1) Every object in the source to go to the mirror.\n>   2) Every ref in the source to go to the mirror.\n>   3) Any ref removed from the source to be removed from the mirror.\n>   4) Automatically repack and prune the mirror when necessary.\n>\n> and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.\n\nJust a note.  I usually use git-push the other way for backups,\nand I believe that is how Linus does it, too.\n\n> diff --git a/git-mirror.perl b/git-mirror.perl\n> new file mode 100755\n> index 0000000..bff2003\n> --- /dev/null\n> +++ b/git-mirror.perl\n> @@ -0,0 +1,111 @@\n> +#!/usr/bin/env perl\n\nPlease don't.  \"#!/usr/bin/env perl\" is a disease.\n\n> +# This file is licensed under the GPL v2, or a later version\n> +# at the discretion of Linus.\n\nHeh ;-).\n\n> +use warnings;\n> +use strict;\n> +use Git;\n> +\n> +sub ls_refs ($$);\n\nI wonder why people like line-noise prototypes.  Do you ever\ncall ls_refs with parameters that benefit from this?  Otherwise\nI prefer not to see them.\n\n> +my $remote = shift || 'origin';\n> +my $repo = Git->repository();\n> +\n> +# Verify its OK to execute in this repository.\n> +#\n> +my $mirror_ok = $repo->config('mirror.allowed') || 0;\n> +unless ($mirror_ok =~ /^(?:true|t|yes|y|1)$/i) {\n\nThis _is_ ugly.  Doesn't $repo->config() know how to drive\nunderlying \"git-repo-config\" with specific type argument?\n\n> +# Delete any local refs which the server no longer contains.\n> +#\n> +foreach my $ref (keys %$local_refs) {\n> +\tnext if $remote_refs->{$ref};\n> +\tprint \"removing $ref\\n\";\n> +\tmy $log = \"logs/$ref\";\n> +\tunlink($repo->repo_path() . '/' . $ref);\n> +\tunlink($repo->repo_path() . '/' . $log);\n> +\trmdir($repo->repo_path() . '/' . $ref) while $ref =~ s,/[^/]*$,,;\n> +\trmdir($repo->repo_path() . '/' . $log) while $log =~ s,/[^/]*$,,;\n> +}\n\nIf you do this upfront and then you lose connection while the\nreal fetch, next fetch may need to take a lot longer than needed\nbecause it cannot rely on the refs you are losing here.  Ref\nremoval is rather a rare event, so we may not care too much\nabout it, though.\n\n> +# Execute the fetch for any refs which differ from our own.\n> +# We don't worry about trying to optimize for rewinds or\n> +# exact branch copies as they are rather uncommon.\n\nIf we need to support only git-native protocols, all of this\noptimization is not needed at all.  It's kind of sad that we\nneed to support commit walkers...\n\n> +if (@to_fetch) {\n> +\tgit_cmd_try {\n> +\t\t$repo->command_noisy('fetch',\n> +\t\t\t'--force',\n> +\t\t\t'--update-head-ok',\n> +\t\t\t$remote, sort @to_fetch);\n> +\t} '%s failed w/ code %d';\n\nWhy sort (no objection, just curious)?\n\n> +# Repack if we have a large number of loose objects.\n> +#\n> +if (@to_fetch) {\n> +\tmy $count_output = $repo->command('count-objects');\n> +\tmy ($cur_loose) = ($count_output =~ /^(\\d+) objects/);\n> +\tmy $max_loose = $repo->config('mirror.maxlooseobjects') || 100;\n> +\tif ($cur_loose >= $max_loose) {\n> +\t\tgit_cmd_try {\n> +\t\t\t$repo->command_noisy('repack', '-a', '-d');\n> +\t\t\t$repo->command_noisy('prune');\n> +\t\t} '%s failed w/ code %d';\n> +\t}\n> +}\n\nIf we truly have a large number of objects (in pack and loose),\nyou do not want to do \"repack -a -d\", do you?\n"},{"id":"27711","messageId":"20060926215745.GC8177@spearce.org","threadId":"5687","inReplyTo":"7vpsdjtew4.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-26T21:57:45Z","receivedAt":"2006-09-26T21:57:45Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano <junkio@cox.net> wrote:\n> Shawn Pearce <spearce@spearce.org> writes:\n> \n> > Sometimes its handy to be able to efficiently backup or mirror one\n> > Git repository to another Git repository by employing the native\n> > Git object transfer protocol.  But when mirroring or backing up a\n> > repository you really want:\n> >\n> >   1) Every object in the source to go to the mirror.\n> >   2) Every ref in the source to go to the mirror.\n> >   3) Any ref removed from the source to be removed from the mirror.\n> >   4) Automatically repack and prune the mirror when necessary.\n> >\n> > and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.\n> \n> Just a note.  I usually use git-push the other way for backups,\n> and I believe that is how Linus does it, too.\n\nI do the same thing right now but it doesn't get all topic branches\nunless I name them on the push line or put them into my remotes\nfile, and when I delete a topic branch it doesn't remove it from\nthe remote during the next push...  :-)\n \n> > diff --git a/git-mirror.perl b/git-mirror.perl\n> > new file mode 100755\n> > index 0000000..bff2003\n> > --- /dev/null\n> > +++ b/git-mirror.perl\n> > @@ -0,0 +1,111 @@\n> > +#!/usr/bin/env perl\n> \n> Please don't.  \"#!/usr/bin/env perl\" is a disease.\n\nI entirely blame git-svn.perl.  I copied that line from there.\n \n> > +# This file is licensed under the GPL v2, or a later version\n> > +# at the discretion of Linus.\n> \n> Heh ;-).\n\nYea, I thought you'd get a kick out of that, especially after the\nrecent discussion. :)\n\n> > +use warnings;\n> > +use strict;\n> > +use Git;\n> > +\n> > +sub ls_refs ($$);\n> \n> I wonder why people like line-noise prototypes.  Do you ever\n> call ls_refs with parameters that benefit from this?  Otherwise\n> I prefer not to see them.\n\nPerl is line noise.  I've gotten into the habit of prototyping\nmost of my functions but clearly this one could be omitted without\nany problems.\n \n> > +my $remote = shift || 'origin';\n> > +my $repo = Git->repository();\n> > +\n> > +# Verify its OK to execute in this repository.\n> > +#\n> > +my $mirror_ok = $repo->config('mirror.allowed') || 0;\n> > +unless ($mirror_ok =~ /^(?:true|t|yes|y|1)$/i) {\n> \n> This _is_ ugly.  Doesn't $repo->config() know how to drive\n> underlying \"git-repo-config\" with specific type argument?\n\nAgreed.  No it doesn't according to Git.pm.  I probably should have\nfixed Git.pm first.\n \n> > +# Execute the fetch for any refs which differ from our own.\n> > +# We don't worry about trying to optimize for rewinds or\n> > +# exact branch copies as they are rather uncommon.\n> \n> If we need to support only git-native protocols, all of this\n> optimization is not needed at all.  It's kind of sad that we\n> need to support commit walkers...\n\nHmm.  I tested this by mirroring a local Git clone (\"../git\") and\nfound it was MUCH faster even when only one head was different.\nThe large number of tags really made it take a lot longer.  And that\nwas local/local though the native protocols.  I figured it was\nworth the few lines of Perl code.\n \n> > +if (@to_fetch) {\n> > +\tgit_cmd_try {\n> > +\t\t$repo->command_noisy('fetch',\n> > +\t\t\t'--force',\n> > +\t\t\t'--update-head-ok',\n> > +\t\t\t$remote, sort @to_fetch);\n> > +\t} '%s failed w/ code %d';\n> \n> Why sort (no objection, just curious)?\n\nI'm a freak.  I typically don't like things that have a \"randomness\"\nto them.  Since I'm pulling from keys %foo the order I'm getting refs\nback in is \"unknown\" (up to Perl's hash function).  Sorting them\nbefore using them cleans up that randomness.  Although its not\nreally random here as the hash function is deterministic.\n\nMaybe its because I work with SQL databases all of the time and\nyou pretty much can't rely on anything coming back in any sort of\norder unless you explicitly force it.  So I tend to do the same\nwith a lot of other systems.\n\n> > +# Repack if we have a large number of loose objects.\n> > +#\n> > +if (@to_fetch) {\n> > +\tmy $count_output = $repo->command('count-objects');\n> > +\tmy ($cur_loose) = ($count_output =~ /^(\\d+) objects/);\n> > +\tmy $max_loose = $repo->config('mirror.maxlooseobjects') || 100;\n> > +\tif ($cur_loose >= $max_loose) {\n> > +\t\tgit_cmd_try {\n> > +\t\t\t$repo->command_noisy('repack', '-a', '-d');\n> > +\t\t\t$repo->command_noisy('prune');\n> > +\t\t} '%s failed w/ code %d';\n> > +\t}\n> > +}\n> \n> If we truly have a large number of objects (in pack and loose),\n> you do not want to do \"repack -a -d\", do you?\n\nYes.  Because then I want to get your new --unpacked= option\ninto git-repack.sh.  The -a should then repack all active packs,\nomitting the archive packs.  Which are the larger packs that are\ncostly to repack.\n\nIn my opinion mirroring should be a no-brain require activity.\nThis script is designed for exactly tracking another repository\nwithout any additional intervention from the user.  Carrying a\nhuge number of loose objects is not ideal, except for maybe the\nHTTP commit walker.\n\n-- \nShawn.\n"},{"id":"294098","messageId":"20061031174225.3c7c1e77.vsu@altlinux.ru","threadId":"5687","inReplyTo":"20060925044641.GB15757@spearce.org","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2006-10-31T14:42:25Z","receivedAt":"2006-10-31T14:42:25Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Mon, 25 Sep 2006 00:46:41 -0400 Shawn Pearce wrote:\n\n> Sometimes its handy to be able to efficiently backup or mirror one\n> Git repository to another Git repository by employing the native\n> Git object transfer protocol.  But when mirroring or backing up a\n> repository you really want:\n>\n>   1) Every object in the source to go to the mirror.\n>   2) Every ref in the source to go to the mirror.\n>   3) Any ref removed from the source to be removed from the mirror.\n>   4) Automatically repack and prune the mirror when necessary.\n>\n> and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.\n\nIs this patch forgotten, abandoned or what?\n\n[...]\n> --- /dev/null\n> +++ b/git-mirror.perl\n> @@ -0,0 +1,111 @@\n> +#!/usr/bin/env perl\n> +# Copyright (C) 2006, Shawn Pearce <spearce@spearce.org>\n> +# This file is licensed under the GPL v2, or a later version\n> +# at the discretion of Linus.\n> +\n> +use warnings;\n> +use strict;\n> +use Git;\n> +\n> +sub ls_refs ($$);\n> +\n> +my $remote = shift || 'origin';\n> +my $repo = Git->repository();\n> +\n> +# Verify its OK to execute in this repository.\n> +#\n> +my $mirror_ok = $repo->config('mirror.allowed') || 0;\n> +unless ($mirror_ok =~ /^(?:true|t|yes|y|1)$/i) {\n> +\tprint STDERR <<EOF;\n> +error: mirror.allowed is false.\n> +error:\n> +error: For safety reasons please set mirror.allowed in this repository's\n> +error: config before using this command.\n> +error:\n> +error: Unless you are using this repository ONLY for mirroring another\n> +error: repository you probably don't want to do this.\n> +EOF\n> +\texit 1;\n> +}\n> +\n> +# Build our list of refs.\n> +#\n> +my $remote_refs = ls_refs($repo, $remote);\n> +my $local_refs = ls_refs($repo, $repo->repo_path());\n> +my $remote_HEAD = $remote_refs->{'HEAD'};\n> +delete $remote_refs->{'HEAD'};\n> +delete $local_refs->{'HEAD'};\n> +\n> +# Delete any local refs which the server no longer contains.\n> +#\n> +foreach my $ref (keys %$local_refs) {\n> +\tnext if $remote_refs->{$ref};\n> +\tprint \"removing $ref\\n\";\n> +\tmy $log = \"logs/$ref\";\n> +\tunlink($repo->repo_path() . '/' . $ref);\n> +\tunlink($repo->repo_path() . '/' . $log);\n> +\trmdir($repo->repo_path() . '/' . $ref) while $ref =~ s,/[^/]*$,,;\n> +\trmdir($repo->repo_path() . '/' . $log) while $log =~ s,/[^/]*$,,;\n\nOne more instance of direct ref manipulation...\n\nWhat is the current state of packed-refs implementation?  Would it be\npossible to merge at least the interface part (git-update-ref -d for\ndeleting refs) while keeping the current ref representation, so that\nother programs could use proper interfaces now?\n\n> +}\n\nIt would be better to remove old refs after the fetch, because if\nsomeone has renamed a branch, and you remove the old ref before the\nfetch, you will fetch all commits for this branch again, even if they\nwere already available under a different name.  However, this may create\nproblems during git-fetch due to file/directory conflicts.\n\nAnother option is to rename deleted refs to something which does not\ncollide with other refs - git-fetch will see them irrespective of the\nname.\n\nYet another option is to select temporary names for all fetched refs to\nbe passed to git-fetch, then read their SHA-1 values after git-fetch\n(they might differ from the previous result of git-ls-remote), update\nnormal refs and delete temporary refs.\n\nBTW, I was thinking about the possibility to save removed refs under,\ne.g., refs/old/`date -I`/; maybe even non-fast-forward refs could be\nsaved there - this will ensure that no object will ever disappear from\nthe mirror, no matter what is done on the master side.  Obviously, in\nthis case remote refs like refs/old/* should be filtered.\n\n> +\n> +# Execute the fetch for any refs which differ from our own.\n> +# We don't worry about trying to optimize for rewinds or\n> +# exact branch copies as they are rather uncommon.\n> +#\n> +my @to_fetch;\n> +while (my ($ref, $hash) = each %$remote_refs) {\n> +\tpush(@to_fetch, \"$ref:$ref\")\n> +\t\tif (!$local_refs->{$ref} || $local_refs->{$ref} ne $hash);\n> +}\n> +if (@to_fetch) {\n> +\tgit_cmd_try {\n> +\t\t$repo->command_noisy('fetch',\n> +\t\t\t'--force',\n> +\t\t\t'--update-head-ok',\n> +\t\t\t$remote, sort @to_fetch);\n\nThis will fail for repositories with large number of refs due to the\nlimit on command line size.\n\nI'm not sure how to figure out the limit for splitting this to multiple\ngit-fetch calls.  And using xargs fed from a pipe means that we cannot\nuse existing Git.pm interfaces.\n\n> +\t} '%s failed w/ code %d';\n> +} else {\n> +\tprint \"No changed refs.  Skipping fetch.\\n\";\n> +}\n> +\n> +# See what the remote has HEAD pointing at and update our local\n> +# HEAD to point at the any ref which points at the same hash.\n> +#\n> +my %by_hash = map {$remote_refs->{$_} => $_}\n> +\tgrep {m,^refs/heads/,}\n> +\tkeys %$remote_refs;\n> +my $HEAD = $by_hash{$remote_HEAD} || 'refs/heads/master';\n> +print \"Setting HEAD to $HEAD\\n\";\n> +print \"                ($remote_HEAD)\\n\";\n> +git_cmd_try {\n> +\t$repo->command_noisy('symbolic-ref', 'HEAD', $HEAD);\n> +} '%s failed w/ code %d';\n> +\n> +# Repack if we have a large number of loose objects.\n> +#\n> +if (@to_fetch) {\n> +\tmy $count_output = $repo->command('count-objects');\n> +\tmy ($cur_loose) = ($count_output =~ /^(\\d+) objects/);\n> +\tmy $max_loose = $repo->config('mirror.maxlooseobjects') || 100;\n> +\tif ($cur_loose >= $max_loose) {\n> +\t\tgit_cmd_try {\n> +\t\t\t$repo->command_noisy('repack', '-a', '-d');\n> +\t\t\t$repo->command_noisy('prune');\n\ngit-prune might be unsafe when used concurrently with other\noperations...  This is probably not a concern for a mirror repository,\nif something ensures that two mirroring processes for the same\nrepository cannot be started.\n\n> +\t\t} '%s failed w/ code %d';\n> +\t}\n> +}\n> +\n> +sub ls_refs ($$) {\n> +\tmy $repo = shift;\n> +\tmy $name = shift;\n> +\tmy ($fh, $c) = $repo->command_output_pipe('ls-remote', $name);\n> +\tmy %refs;\n> +\twhile (<$fh>) {\n> +\t\tchomp;\n> +\t\tnext if /\\^{}$/;\n> +\t\tmy ($hash, $ref) = split(/\\t/, $_, 2);\n> +\t\t$refs{$ref} = $hash if ($ref eq 'HEAD' || $ref =~ m,^refs/,);\n\nProbably more sanity checks would be good here (to make sure that the\nother end does not feed you bogus ref names or hashes).\n\n> +\t}\n> +\t$repo->command_close_pipe($fh, $c);\n> +\t\\%refs;\n> +}\n> --\n> 1.4.2.1.gde2b2-dirty\n>\n"},{"id":"296556","messageId":"20061031145554.GA23007@spearce.org","threadId":"5687","inReplyTo":"20061031174225.3c7c1e77.vsu@altlinux.ru","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-10-31T14:55:55Z","receivedAt":"2006-10-31T14:55:55Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sergey Vlasov <vsu@altlinux.ru> wrote:\n> On Mon, 25 Sep 2006 00:46:41 -0400 Shawn Pearce wrote:\n> \n> > Sometimes its handy to be able to efficiently backup or mirror one\n> > Git repository to another Git repository by employing the native\n> > Git object transfer protocol.  But when mirroring or backing up a\n> > repository you really want:\n> >\n> >   1) Every object in the source to go to the mirror.\n> >   2) Every ref in the source to go to the mirror.\n> >   3) Any ref removed from the source to be removed from the mirror.\n> >   4) Automatically repack and prune the mirror when necessary.\n> >\n> > and since git-fetch doesn't do 2, 3, and 4 here's a tool that does.\n> \n> Is this patch forgotten, abandoned or what?\n\nIts waiting around for someone to clean it up.  :-)\n\nJunio didn't accept it as there were a number of issues that he\nidentified in the patch.  I still have them in my inbox but have\nnot had time to go back and fix them.  Since I haven't actually had\na real need for git-mirror it has been low on my priority list of\n\"git things to do\".\n\nYour message pointed out a number of issues with the current version\nthat would be worth fixing before accepting it into git.git.  I agree\nwith many of them, especially about the direct ref manipulation.\ngit-mirror was written before the Linus packed ref stuff came\nalong, so git-update-ref didn't have a -d option at the time...\notherwise I would have used it.\n\n-- \n"},{"id":"296122","messageId":"7vac3covlf.fsf@assigned-by-dhcp.cox.net","threadId":"5687","inReplyTo":"20061031174225.3c7c1e77.vsu@altlinux.ru","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-01T00:32:12Z","receivedAt":"2006-11-01T00:32:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sergey Vlasov <vsu@altlinux.ru> writes:\n\n> What is the current state of packed-refs implementation?\n\nScheduled for \"master\" probably tomorrow, with some disclaimers.\n\n> BTW, I was thinking about the possibility to save removed refs under,\n> e.g., refs/old/`date -I`/; maybe even non-fast-forward refs could be\n> saved there - this will ensure that no object will ever disappear from\n> the mirror, no matter what is done on the master side.  Obviously, in\n> this case remote refs like refs/old/* should be filtered.\n\nI do not think that belongs to git-mirror.  It might be better to\nhave that option in update-ref machinery so that you would even\nbe protected from a `git branch -d` ran by mistake.\n"},{"id":"294488","messageId":"ei8s19$e04$1@sea.gmane.org","threadId":"5687","inReplyTo":"7vac3covlf.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-11-01T01:08:08Z","receivedAt":"2006-11-01T01:08:08Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> Sergey Vlasov <vsu@altlinux.ru> writes:\n>  \n>> BTW, I was thinking about the possibility to save removed refs under,\n>> e.g., refs/old/`date -I`/; maybe even non-fast-forward refs could be\n>> saved there - this will ensure that no object will ever disappear from\n>> the mirror, no matter what is done on the master side.  Obviously, in\n>> this case remote refs like refs/old/* should be filtered.\n> \n> I do not think that belongs to git-mirror.  It might be better to\n> have that option in update-ref machinery so that you would even\n> be protected from a `git branch -d` ran by mistake.\n\nPerhaps just don't remove log when branch is deleted. I tend for example\nto be careful when reordering commits on a branch, and use git branch -f\ninstead of deleting and recreating branch to not lose reflog.\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"298217","messageId":"20061101151859.0e984d3f.vsu@altlinux.ru","threadId":"5687","inReplyTo":"ei8s19$e04$1@sea.gmane.org","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2006-11-01T12:18:59Z","receivedAt":"2006-11-01T12:18:59Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Wed, 01 Nov 2006 02:08:08 +0100 Jakub Narebski wrote:\n\n> Junio C Hamano wrote:\n> > Sergey Vlasov <vsu@altlinux.ru> writes:\n> >> BTW, I was thinking about the possibility to save removed refs under,\n> >> e.g., refs/old/`date -I`/; maybe even non-fast-forward refs could be\n> >> saved there - this will ensure that no object will ever disappear from\n> >> the mirror, no matter what is done on the master side.  Obviously, in\n> >> this case remote refs like refs/old/* should be filtered.\n> >\n> > I do not think that belongs to git-mirror.  It might be better to\n> > have that option in update-ref machinery so that you would even\n> > be protected from a `git branch -d` ran by mistake.\n\nI agree - this would be even better.\n\nWhat name format should be used for such saved refs?  refs/old/`date -I`\nis not unique enough; probably `date --utc +%Y.%m.%d-%H.%M.%S`?  And it\nwould be good if multiple refs which were deleted or modified in a\nnon-fast-forward way during a single operation (like git-mirror) would\nbe saved together - which may be tricky if they are saved at the lower\nlevel (in update-ref).\n\nAdding the fast-forward check into update-ref also does not look nice,\nbut this check is required for full safety.\n\n> Perhaps just don't remove log when branch is deleted. I tend for example\n> to be careful when reordering commits on a branch, and use git branch -f\n> instead of deleting and recreating branch to not lose reflog.\n\nKeeping the reflog does not help - reflog entries are not refs, and do\nnot keep the referenced objects live in the repository.  Therefore old\nobjects will be lost when the repository is repacked and pruned.\n\nAlso, the reflog really must be deleted together with the ref itself -\notherwise it may cause a file/directory conflict when another ref is\ncreated.  Of course, if the ref is renamed instead of deleting, the\ncorresponding reflog may be renamed too.\n"},{"id":"298530","messageId":"eia3jk$n2f$1@sea.gmane.org","threadId":"5687","inReplyTo":"20061101151859.0e984d3f.vsu@altlinux.ru","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-11-01T12:23:33Z","receivedAt":"2006-11-01T12:23:33Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Sergey Vlasov wrote:\n\n>> Perhaps just don't remove log when branch is deleted. I tend for example\n>> to be careful when reordering commits on a branch, and use git branch -f\n>> instead of deleting and recreating branch to not lose reflog.\n> \n> Keeping the reflog does not help - reflog entries are not refs, and do\n> not keep the referenced objects live in the repository.  Therefore old\n> objects will be lost when the repository is repacked and pruned.\n\nWell, now to reorder and choose which commits on branch to save, and\noptionally also rebase, I use\n$ git branch tmp\n$ git branch -f <branch> <upstream>\n$ git checkout <branch>\n$ git cherry-pick tmp~<n>\n...\n$ git branch -D tmp\n\n> Also, the reflog really must be deleted together with the ref itself -\n> otherwise it may cause a file/directory conflict when another ref is\n> created.  Of course, if the ref is renamed instead of deleting, the\n> corresponding reflog may be renamed too.\n\nThere is currently no command to rename branch (and rename reflog).\nIt would be nice if git-branch could also rename branch (and reflog).\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"296326","messageId":"7vodrr2ldy.fsf@assigned-by-dhcp.cox.net","threadId":"5687","inReplyTo":"20061101151859.0e984d3f.vsu@altlinux.ru","subject":"Re: [PATCH] Introduce git-mirror, a tool for exactly mirroring another repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-01T16:15:53Z","receivedAt":"2006-11-01T16:15:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sergey Vlasov <vsu@altlinux.ru> writes:\n\n> What name format should be used for such saved refs?  refs/old/`date -I`\n> is not unique enough; probably `date --utc +%Y.%m.%d-%H.%M.%S`?  And it\n> would be good if multiple refs which were deleted or modified in a\n> non-fast-forward way during a single operation (like git-mirror) would\n> be saved together - which may be tricky if they are saved at the lower\n> level (in update-ref).\n>\n> Adding the fast-forward check into update-ref also does not look nice,\n> but this check is required for full safety.\n\nI wasn't going to suggest doing the check or even naming at such\na low level.  From the usability point of view, the caller\nshould decide if the discarded refs are even saved at all, and\nif so in which namespace.  That way commit_lock_file() can just\nnotice the old version of the locked file (it is always sitting\nnext to it) and hardlink the discarded one to the purgatory\nbefore renaming the newly created .lock file there.\n\n\n\n"}]}