threads / patch / 5612

patchAdded --mirror-all to git-fetch.

Subject: [PATCH] Added --mirror-all to git-fetch.

## tl;dr

14 messages between Sep 19, 2006 and Sep 20, 2006. Diffs are folded; open one to read it.

replies: 13people: 4as markdown or json

Shawn Pearce· Sep 19, 2006, 23:28 UTC · lore

The --mirror-all option to git-fetch can be used to obtain a copy of every available remote ref into the current repository. This can be a rather destructive update as the local repository will have its HEAD ref overwritten, as well as any ref which it shares in common with the remote repository. On the other hand it can be useful if all you want to do is have this repository track another repository, such as if you are providing Git repository hosting and mirroring source repositories on other systems.

Currently local refs are not deleted even if they do not exist in the remote repository. This may be taken as either a feature or a bug.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 pasky was asking about this on #git.  So here it is.
 Documentation/fetch-options.txt |    7 ++++++
 git-fetch.sh                    |   43 ++++++++++++++++++++++++++++++++++++---
 2 files changed, 47 insertions(+), 3 deletions(-)
Show changes to 2 files +47 −3

Documentation/fetch-options.txt, git-fetch.sh

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 13f34d3..5ed9b4f 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -30,6 +30,13 @@
 	flag lets all tags and their associated objects be
 	downloaded.
 
+\--mirror-all::
+	All refs and tags on the remote side are downloaded to
+	the local side.  This option is not intendend for user
+	repositories as it will overwrite every local ref,
+	including HEAD.  You probably don't mean to use this
+	option.
+
 -k, \--keep::
 	Keep downloaded pack.
 
diff --git a/git-fetch.sh b/git-fetch.sh
index 09a5d6c..d22c560 100755
--- a/git-fetch.sh
+++ b/git-fetch.sh
@@ -14,6 +14,7 @@ IFS="$LF"
 rloga=fetch
 no_tags=
 tags=
+mirror_all=
 append=
 force=
 verbose=
@@ -36,6 +37,9 @@ do
 	-f|--f|--fo|--for|--forc|--force)
 		force=t
 		;;
+	--mirror-all)
+		mirror_all=t
+		;;
 	-t|--t|--ta|--tag|--tags)
 		tags=t
 		;;
@@ -216,12 +220,45 @@ case "$update_head_ok" in
 	;;
 esac
 
-# If --tags (and later --heads or --all) is specified, then we are
-# not talking about defaults stored in Pull: line of remotes or
+# If --tags or -mirror_all (and later --heads) is specified, then we
+# are not talking about defaults stored in Pull: line of remotes or
 # branches file, and just fetch those and refspecs explicitly given.
 # Otherwise we do what we always did.
 
-reflist=$(get_remote_refs_for_fetch "$@")
+if test "$mirror_all"
+then
+	tags=0;  # tags are implied in all 
+	reflist=`IFS="	" &&
+		  (
+			git-ls-remote $upload_pack "$remote" ||
+			echo fail ouch
+		  ) |
+	          while read sha1 name
+		  do
+			case "$sha1" in
+			fail)
+				exit 1
+			esac
+			case "$name" in
+			*^*) continue ;;
+			esac
+		  	if git-check-ref-format "$name"
+			then
+			    echo ".+${name}:${name}"
+			else
+				if test "x$name" = xHEAD
+				then
+					echo ".+${name}:${name}"
+				else
+					echo >&2 "warning: malformed ref ${name} ignored"
+				fi
+			fi
+		  done` || exit
+else
+	# Not using all; do what we always have done.
+	reflist=$(get_remote_refs_for_fetch "$@")
+fi
+
 if test "$tags"
 then
 	taglist=`IFS="	" &&
-- 
1.4.2.1.g832e
Petr Baudis· Sep 19, 2006, 23:41 UTC · re: Shawn Pearce · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Dear diary, on Wed, Sep 20, 2006 at 01:28:51AM CEST, I got a letter where Shawn Pearce <spearce@spearce.org> said that...

> Currently local refs are not deleted even if they do not exist in the
> remote repository.  This may be taken as either a feature or a bug.
  It's a bug for my purposes, but a minor one.
>  pasky was asking about this on #git.  So here it is.
  Thanks!
Show 7 quoted lines
> @@ -216,12 +220,45 @@ case "$update_head_ok" in
>  	;;
>  esac
>  
> -# If --tags (and later --heads or --all) is specified, then we are
> -# not talking about defaults stored in Pull: line of remotes or
> +# If --tags or -mirror_all (and later --heads) is specified, then we
                  ^^ missing dash ;-)
Show 8 quoted lines
> +# are not talking about defaults stored in Pull: line of remotes or
>  # branches file, and just fetch those and refspecs explicitly given.
>  # Otherwise we do what we always did.
>  
> -reflist=$(get_remote_refs_for_fetch "$@")
> +if test "$mirror_all"
> +then
> +	tags=0;  # tags are implied in all 
  This should be tags=;
-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam
Junio C Hamano· Sep 20, 2006, 16:06 UTC · re: Shawn Pearce · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Shawn Pearce <spearce@spearce.org> writes:
Show 5 quoted lines
> The --mirror-all option to git-fetch can be used to obtain a copy of
> every available remote ref into the current repository.  This can be
> a rather destructive update as the local repository will have its
> HEAD ref overwritten, as well as any ref which it shares in common
> with the remote repository.

I can sort of see where something like this is very much useful, but it sounds like a tool quite different from git-fetch.

 (1) I really do not like rolling this kind of speciale purpose
     command into git-fetch, which is frequently used by end
     users, to avoid mistakes.
     If there is reluctance against adding yet another new
     command (and there certainly is), this feels more like a
     cousin of "git-clone --bare".
 (2) Although there is no inherent reason not allowing a working
     tree associated with the repository that is kept updated
     this way, the user will be utterly confused in a working
     tree whose current branch head is updated like this, until
     the working tree and the index is matched to the updated
     HEAD.  It might be reasonable to run checkout -f HEAD when
     a working tree is associated with the repository (the
     command is screwing over even the current branch HEAD, so
     losing what happened to be in the working tree is really
     not an issue), but as a much easier safety measure we might
     want to allow this mode of updating only on a bare
     repository (that is, .git/index should not exist).
 (3) This feels primarily meant for something like Pasky is
     trying to run --- mirrored distribution point of git
     repositories perhaps displayed with gitweb.  When updating
     such a repository, you would want to do things like running
     update-server-info and automatically repacking the object
     store.  Especially the latter would be an interesting topic
     (the archive vs active repacking strategy we talked about,
     combined with set of packs with staggered spans to help
     commit walkers Pasky talked about quite a while ago).
Show 6 quoted lines
> ... 
> such as if you are providing Git repository hosting and mirroring
> source repositories on other systems.
>
> Currently local refs are not deleted even if they do not exist in the
> remote repository.  This may be taken as either a feature or a bug.
For that purpose I would say that is definitely a bug.
Petr Baudis· Sep 20, 2006, 16:14 UTC · re: Junio C Hamano · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Dear diary, on Wed, Sep 20, 2006 at 06:06:11PM CEST, I got a letter where Junio C Hamano <junkio@cox.net> said that... ..snip..

>      If there is reluctance against adding yet another new
>      command (and there certainly is), this feels more like a
>      cousin of "git-clone --bare".

Certainly. But, what exactly are you proposing here? :-) (Besides possible change of the switch name.) Making this a git-clone option sounds even much worse.

..snip..
Show 6 quoted lines
>  (2) Although there is no inherent reason not allowing a working
>      tree associated with the repository that is kept updated
>      this way, the user will be utterly confused in a working
>      tree whose current branch head is updated like this, until
>      the working tree and the index is matched to the updated
>      HEAD.

git-fetch --mirror-all can still be very useful even with a working copy if you are keeping all the remote heads in .git/refs/remotes/. I'd venture to say that in that case, this is frequently what you actually want when fetching from the repository (given that you have already let git clone do that once).

..snip..
>      (the archive vs active repacking strategy we talked about,
Hmm, I think I've missed this, I must look that in the archive.
>      combined with set of packs with staggered spans to help
>      commit walkers Pasky talked about quite a while ago).

Yes, this is certainly one of things I would like to implement at repo.or.cz.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam
Shawn Pearce· Sep 20, 2006, 16:21 UTC · re: Petr Baudis · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Petr Baudis <pasky@suse.cz> wrote:
> Dear diary, on Wed, Sep 20, 2006 at 06:06:11PM CEST, I got a letter
> where Junio C Hamano <junkio@cox.net> said that...
[snip]
Show 12 quoted lines
> >  (2) Although there is no inherent reason not allowing a working
> >      tree associated with the repository that is kept updated
> >      this way, the user will be utterly confused in a working
> >      tree whose current branch head is updated like this, until
> >      the working tree and the index is matched to the updated
> >      HEAD.
> 
> git-fetch --mirror-all can still be very useful even with a working copy
> if you are keeping all the remote heads in .git/refs/remotes/. I'd
> venture to say that in that case, this is frequently what you actually
> want when fetching from the repository (given that you have already let
> git clone do that once).

I think that's more `git fetch --all`. You are pulling every remote head available at a given remote into a subdirectory of your own ref space. That's rather different than replacing your entire ref space with the one available on the remote, which is what --mirror-all is doing and what you wanted for the hosting you are offering.

> ..snip..
> >      (the archive vs active repacking strategy we talked about,
> 
> Hmm, I think I've missed this, I must look that in the archive.

Junio pushed the core code out but nobody has done the Porecelain for it. The basic idea is to prevent repacking every pack all of the time; there's probably no reason to repack a 100 MiB pack file every time you repack your loose objects so you might want to keep say a <5 MiB "active pack" holding your recent created objects and repack that frequently and a larger 100+ MiB "history pack" holding everything else. Maybe you repack everything on a longer time scale, such as once a year.

Show 5 quoted lines
> >      combined with set of packs with staggered spans to help
> >      commit walkers Pasky talked about quite a while ago).
> 
> Yes, this is certainly one of things I would like to implement at
> repo.or.cz.

Borrowing your line: Hmm, I think I've missed this, I must look that in the archive.

:-)
-- 
Shawn.
Junio C Hamano· Sep 20, 2006, 16:34 UTC · re: Shawn Pearce · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Shawn Pearce <spearce@spearce.org> writes:
Show 22 quoted lines
> Petr Baudis <pasky@suse.cz> wrote:
>> Dear diary, on Wed, Sep 20, 2006 at 06:06:11PM CEST, I got a letter
>> where Junio C Hamano <junkio@cox.net> said that...
> [snip]
>> >  (2) Although there is no inherent reason not allowing a working
>> >      tree associated with the repository that is kept updated
>> >      this way, the user will be utterly confused in a working
>> >      tree whose current branch head is updated like this, until
>> >      the working tree and the index is matched to the updated
>> >      HEAD.
>> 
>> git-fetch --mirror-all can still be very useful even with a working copy
>> if you are keeping all the remote heads in .git/refs/remotes/. I'd
>> venture to say that in that case, this is frequently what you actually
>> want when fetching from the repository (given that you have already let
>> git clone do that once).
>
> I think that's more `git fetch --all`.  You are pulling every remote
> head available at a given remote into a subdirectory of your own ref
> space.  That's rather different than replacing your entire ref space
> with the one available on the remote, which is what --mirror-all
> is doing and what you wanted for the hosting you are offering.

I realize I am going around in circles, but Pasky's "remotes/" argument made me realize that this mirroring is not much more than "fetch --force --all". I initially had an impression that this was for only strict mirroring where you do not even want your own "origin", but if you arrange the .git/remotes/origin file the right way, "fetch --force --all" (if you remembered to put '+' in front of the refspecs, even without --force) would what --mirror-all would do wouldn't it?

Having said that, I am not sure if "fetch --all" should delete branches that do not exist on the remote end. For mirroring, I definitely would want the tool to delete them, and that is one of the reason I feel this is much more similar to git-clone than git-fetch, unless it is a separate command (perhaps git-mirror).

Shawn Pearce· Sep 20, 2006, 16:49 UTC · re: Junio C Hamano · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Junio C Hamano <junkio@cox.net> wrote:
Show 8 quoted lines
> I realize I am going around in circles, but Pasky's "remotes/"
> argument made me realize that this mirroring is not much more
> than "fetch --force --all".  I initially had an impression that
> this was for only strict mirroring where you do not even want
> your own "origin", but if you arrange the .git/remotes/origin
> file the right way, "fetch --force --all" (if you remembered to
> put '+' in front of the refspecs, even without --force) would
> what --mirror-all would do wouldn't it?

I started this change with '--all' and realized that ideally you want '--all' to copy all available refs/heads/* from the remote to refs/remotes/<name>/* here. You want to create any new branches which the remote has introduced since your last fetch.

You probably don't want to force a non-fast forward unless there's a '+' in the corresponding Pull line of remotes/<name> or if --force is used. However you probably also want to delete any removed branches.

Which I think is quite different from a mirror. A mirror wants to replace the entire ref namespace with what's on the remote as it has no need for a local namespace of its own.

Originally I gave Pasky a one-liner on #git:
  git fetch --force origin $(git ls-remote origin \
      | awk '{if(!/\^{}$/){print $2":"$2}}')

but he expressed interest in it being a native feature of the core-Git fetch Porcelain. To be honest I disagreed with him but submitted the patch anyway.

I think --all copying into .git/refs/remotes/<name>/* makes perfect sense.

And I think this mirror thing may make more sense as a small wrapper around git-fetch. A wrapper that checks for:

	- its running in a bare repository;
	- it has a single remote named origin;
	- HEAD isn't a symlink or a symref (its a normal ref in its
	  own right);
	- git-mirror.permitted is true in the config file.
 
-- 
Shawn.
Junio C Hamano· Sep 20, 2006, 17:13 UTC · re: Shawn Pearce · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Shawn Pearce <spearce@spearce.org> writes:
Show 8 quoted lines
> I started this change with '--all' and realized that ideally you
> want '--all' to copy all available refs/heads/* from the remote to
> refs/remotes/<name>/* here.  You want to create any new branches
> which the remote has introduced since your last fetch.
>
> You probably don't want to force a non-fast forward unless there's a
> '+' in the corresponding Pull line of remotes/<name> or if --force is
> used.  However you probably also want to delete any removed branches.
I think we are in agreement here.
> Which I think is quite different from a mirror.  A mirror wants to
> replace the entire ref namespace with what's on the remote as it
> has no need for a local namespace of its own.

If you have remotes/origin that does not use the refs/remotes hierarchy, does not copy master->origin (iow, the assumption that "origin" branch is to be merged into local "master") but copies remote master to local master, and forces all refs by default, then your local ref namespace is effectively a mirror of the remote, so it is very tempting to think --mirror is a logical extension of "fetch --all".

Except that new refs are not created and old refs are not deleted. And these two are rather big exceptions.

If we are to have another command (git-mirror which is a thin wrapper around git-fetch as you propose later), these two exceptions would be cleanly handled there. Otherwise, we would need to give the command a hint (git-mirror.permitted) to allow them to be done in addition to the usual "fetch --all", but then we would end up doing the equivalent of your one-liner anyway, so there is not much point having it in git-fetch. So I am in favor of your thin-wrapper idea.

> Originally I gave Pasky a one-liner on #git:
>
>   git fetch --force origin $(git ls-remote origin \
>       | awk '{if(!/\^{}$/){print $2":"$2}}')

Which sounds a very sane thing to do, especially given git is designed to be scripted this way.

> And I think this mirror thing may make more sense as a small wrapper
> around git-fetch.  A wrapper that checks for:
>
> 	- its running in a bare repository;
Ok.
> 	- it has a single remote named origin;

Do you mean ".git/remotes must have only one file 'origin' and nothing else", or do you mean "There must be .git/remotes/foobla for 'git-mirror foobla' to work"?

> 	- HEAD isn't a symlink or a symref (its a normal ref in its
> 	  own right);

Why? Usually HEAD is a pointer to the active branch. I consider it is a bug in the upload-pack protocol not to convey that information (but that is a separate topic).

> 	- git-mirror.permitted is true in the config file.
Ok.
Shawn Pearce· Sep 20, 2006, 17:31 UTC · re: Junio C Hamano · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Junio C Hamano <junkio@cox.net> wrote:
Show 8 quoted lines
> Shawn Pearce <spearce@spearce.org> writes:
> > Originally I gave Pasky a one-liner on #git:
> >
> >   git fetch --force origin $(git ls-remote origin \
> >       | awk '{if(!/\^{}$/){print $2":"$2}}')
> 
> Which sounds a very sane thing to do, especially given git is
> designed to be scripted this way.
Exactly.  So I'm not sure why he was against it.  After all he must
have some script that's looping over all mirrored repositories and
invoking git fetch in them.  :-)
 
Show 5 quoted lines
> > 	- it has a single remote named origin;
> 
> Do you mean ".git/remotes must have only one file 'origin' and
> nothing else", or do you mean "There must be .git/remotes/foobla
> for 'git-mirror foobla' to work"?

I meant ".git/remotes must have only one file 'origin' and nothing else". Because what does it mean to mirror two different repositories at the same time into the same ref namespace? The most recently fetched would always win.

But now that I'm writing this I can see someone wanting multiple
remotes so they could store different URLs for the same repository
and call fetch on another if the first isn't responding.  So I
retract this entirely.  So there doesn't even need to be remotes
file; the URL could just be passed on the command line.  A remotes
file (like the default "origin" one) is just a handy place to store
the URL.
 
Show 6 quoted lines
> > 	- HEAD isn't a symlink or a symref (its a normal ref in its
> > 	  own right);
> 
> Why?  Usually HEAD is a pointer to the active branch.  I
> consider it is a bug in the upload-pack protocol not to convey
> that information (but that is a separate topic).

Because we don't want to update HEAD with the SHA1 that's on the remote if that's actually going to update some other ref which doesn't have the same value. :-)

I think its simple enough to verify HEAD isn't a symlink or symref (or just doesn't exist) early in the mirror script. If the user really wants to do a mirror and they just did a `git init-db --bare` they could also do a `rm HEAD` before starting `git-mirror`.

But I agree with you entirely about HEAD being a SHA1 and not its actual link target in the network protocol. An annoying bug.

-- 
Shawn.
Junio C Hamano· Sep 20, 2006, 17:50 UTC · re: Shawn Pearce · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Shawn Pearce <spearce@spearce.org> writes:
Show 12 quoted lines
> I meant ".git/remotes must have only one file 'origin' and
> nothing else".  Because what does it mean to mirror two different
> repositories at the same time into the same ref namespace?  The most
> recently fetched would always win.
>
> But now that I'm writing this I can see someone wanting multiple
> remotes so they could store different URLs for the same repository
> and call fetch on another if the first isn't responding.  So I
> retract this entirely.  So there doesn't even need to be remotes
> file; the URL could just be passed on the command line.  A remotes
> file (like the default "origin" one) is just a handy place to store
> the URL.

Another possibility. The mirror might want to do something like:

        .git/refs/remotes/davem/sparc-2.6.git/...
	.git/refs/remotes/torvalds/linux-2.6.git/...
	.git/refs/remotes/jgarzik/netdev-2.6.git/...

to be a super-mirror of related projects. This would give a convenient way for the visitors to browse them all at one place, and would save object store. There are minor details such as gitweb may not allow browsing refs/remotes and git-fetch may not let you fetch from refs/remotes (I haven't checked), but they are fixable as needed.

> Because we don't want to update HEAD with the SHA1 that's on the
> remote if that's actually going to update some other ref which
> doesn't have the same value.  :-)

What I thought you were going to do with HEAD would be to update it not by the object name but modifying where it points at. IOW, if HEAD is a symlink pointing at refs/heads/master and the remote HEAD now points at refs/heads/next, you do not:

	echo $the_commit_object_name_of_remote_next >.git/HEAD 
which obviously is insane, but you would do
	rm -f .git/HEAD && ln -s refs/heads/next .git/HEAD
> But I agree with you entirely about HEAD being a SHA1 and not its
> actual link target in the network protocol.  An annoying bug.
Which fortunately is fixable ;-).
A Large Angry SCM· Sep 20, 2006, 18:42 UTC · re: Junio C Hamano · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Junio C Hamano wrote: [...]

Show 15 quoted lines
> 
> Another possibility.  The mirror might want to do something
> like:
> 
>         .git/refs/remotes/davem/sparc-2.6.git/...
> 	.git/refs/remotes/torvalds/linux-2.6.git/...
> 	.git/refs/remotes/jgarzik/netdev-2.6.git/...
> 
> to be a super-mirror of related projects.  This would give a
> convenient way for the visitors to browse them all at one place,
> and would save object store.  There are minor details such as
> gitweb may not allow browsing refs/remotes and git-fetch may not
> let you fetch from refs/remotes (I haven't checked), but they
> are fixable as needed.
> 

This is what I do with the 2 attached, poorly named, scripts. Very convenient for watching some of my favorite projects. ;-)

export GIT_COMMITTER_NAME="@" export GIT_AUTHOR_NAME="@"

./fred.sh git://www.kernel.org/pub/scm/git/git.git git/git
./fred.sh git://www.kernel.org/pub/scm/gitk/gitk.git gitk/gitk

./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.11.y.git stable/linux-2.6.11.y ./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.12.y.git stable/linux-2.6.12.y ./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.13.y.git stable/linux-2.6.13.y ./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.14.y.git stable/linux-2.6.14.y ./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.15.y.git stable/linux-2.6.15.y ./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.16.y.git stable/linux-2.6.16.y ./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.17.y.git stable/linux-2.6.17.y

./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git tglx/history

./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git torvalds/linux-2.6 ./fred.sh git://www.kernel.org/pub/scm/linux/kernel/git/torvalds/sparse.git torvalds/sparse

Petr Baudis· Sep 20, 2006, 18:29 UTC · re: Shawn Pearce · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Dear diary, on Wed, Sep 20, 2006 at 06:21:45PM CEST, I got a letter where Shawn Pearce <spearce@spearce.org> said that...

Show 9 quoted lines
> > >      combined with set of packs with staggered spans to help
> > >      commit walkers Pasky talked about quite a while ago).
> > 
> > Yes, this is certainly one of things I would like to implement at
> > repo.or.cz.
> 
> Borrowing your line:
> Hmm, I think I've missed this, I must look that in the archive.
> :-)

I've proposed something long time ago in <20051112135947.GC30496@pasky.or.cz>.

Dear diary, on Wed, Sep 20, 2006 at 07:31:26PM CEST, I got a letter where Shawn Pearce <spearce@spearce.org> said that...

Show 13 quoted lines
> Junio C Hamano <junkio@cox.net> wrote:
> > Shawn Pearce <spearce@spearce.org> writes:
> > > Originally I gave Pasky a one-liner on #git:
> > >
> > >   git fetch --force origin $(git ls-remote origin \
> > >       | awk '{if(!/\^{}$/){print $2":"$2}}')
> > 
> > Which sounds a very sane thing to do, especially given git is
> > designed to be scripted this way.
> 
> Exactly.  So I'm not sure why he was against it.  After all he must
> have some script that's looping over all mirrored repositories and
> invoking git fetch in them.  :-)

Basically, I imagined that what I wanted would be very close or identical to git fetch --all (or whatever you call it, just doing it with .git/refs/remotes) which I think should certainly be supported without funny awk pipes. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
lK[d2%Sa2/d0$^Ixp"|dc`;s/\W//g;$_=pack('H*',/((..)*)$/)
Junio C Hamano· Sep 20, 2006, 21:36 UTC · re: Shawn Pearce · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Shawn Pearce <spearce@spearce.org> writes:
Show 12 quoted lines
>> >      (the archive vs active repacking strategy we talked about,
>> 
>> Hmm, I think I've missed this, I must look that in the archive.
>
> Junio pushed the core code out but nobody has done the Porecelain
> for it.  The basic idea is to prevent repacking every pack all of
> the time; there's probably no reason to repack a 100 MiB pack file
> every time you repack your loose objects so you might want to keep
> say a <5 MiB "active pack" holding your recent created objects
> and repack that frequently and a larger 100+ MiB "history pack"
> holding everything else.  Maybe you repack everything on a longer
> time scale, such as once a year.

And IIRC we were stuck on the convention to tell which ones are archives and which ones are actives (i.e. fair game for repacking). We were trying to figure out how to mark active ones (whose name changes every time you repack).

I realized that there is a very simple and obvious solution for it. Introduce $GIT_OBJECT_DIRECTORY/info/archived-packs, a flat text file that lists the names of the archive packs. Any pack not listed there are active one and are subject to repacking.

Shawn Pearce· Sep 20, 2006, 21:42 UTC · re: Junio C Hamano · lore

Re: [PATCH] Added --mirror-all to git-fetch.

Junio C Hamano <junkio@cox.net> wrote:
Show 19 quoted lines
> Shawn Pearce <spearce@spearce.org> writes:
> 
> >> >      (the archive vs active repacking strategy we talked about,
> >> 
> >> Hmm, I think I've missed this, I must look that in the archive.
> >
> > Junio pushed the core code out but nobody has done the Porecelain
> > for it.  The basic idea is to prevent repacking every pack all of
> > the time; there's probably no reason to repack a 100 MiB pack file
> > every time you repack your loose objects so you might want to keep
> > say a <5 MiB "active pack" holding your recent created objects
> > and repack that frequently and a larger 100+ MiB "history pack"
> > holding everything else.  Maybe you repack everything on a longer
> > time scale, such as once a year.
> 
> And IIRC we were stuck on the convention to tell which ones are
> archives and which ones are actives (i.e. fair game for
> repacking).  We were trying to figure out how to mark active
> ones (whose name changes every time you repack).
Yes.  I had pretty much given up on a size threshold concept and
was thus in agreement with you that we just needed a way to flag
each pack file...
 
> I realized that there is a very simple and obvious solution for
> it.  Introduce $GIT_OBJECT_DIRECTORY/info/archived-packs, a flat
> text file that lists the names of the archive packs.  Any pack
> not listed there are active one and are subject to repacking.
Like that.  :-)
-- 
Shawn.

← back to recent threads