How to stop sharing objects between repositories

Hello.

Situation: I used "git clone -s" to share objects between repositories. 
That creates .git/objects/info/alternates, which points to the other 
repository. Later I need to remove the dependency and make the dependent 
repository self-sufficient, i.e. it should have all the objects internal 
to itself.

I've looked around for a way to do that but haven't found either tools or instructions.

I came up with this manual way, copying over the unique remote objects and then removing the alternate repository pointer:

(cd /path/to/alternate/repo.git/objects && tar cp .) | (cd .git/objects && tar xvpk) # some objects will already exist and be skipped, leading to an error on exit, which is fine rm .git/objects/info/alternates # or if there's more than one and you're only removing one, edit the alternates file and remove only that pointer

(With GNU tar -C the copy is a little simpler.)

I posted this to the wiki:

http://git.or.cz/gitwiki/GitFaq#Howtostopsharingobjectsbetweenrepositories.3F

If there's a better or built-in way to do this with Git tools, I'd like to learn it, and I'd be happy to update the wiki accordingly.

Thanks, Jon

Hi,

On Sat, 15 Aug 2009, Jon Jensen wrote:

I think what you need is done by

	git repack -l

(I agree it is not well documented, and I'd welcome a documentation patch.)

Ciao, Dscho

On Sun, Aug 16, 2009 at 10:43:11AM +0200, Johannes Schindelin wrote:

I think it is the opposite; packing _without_ "-l" will create a pack with objects from the alternate; using "-l" suppresses them. Running "git repack -a" should do the trick, I believe (and you need the "-a" to ensure that objects already packed in the repo are re-packed).

-Peff

Hi,

On Sun, 16 Aug 2009, Jeff King wrote:

Hmm.  I really would like a documentation patch, then.

Ciao, Dscho

Johannes Schindelin wrote:

As another way to do it, I've used something along the lines from
    http://article.gmane.org/gmane.comp.version-control.git/62062
namely:

<script> gitdir=$(git rev-parse --git-dir) [ -n "$gitdir" ] || die "cannot find Git directory"

cd "$gitdir"
a=objects/info/alternates
if [ -f $a ]; then
  git rev-parse --all HEAD | git pack-objects --revs objects/pack/pack
  rm $a
fi
</script>

I was not sure HEAD would be included via --all (e.g. HEAD pointing to a dangling commit), so I added it explicitly.

The reverse operation (enabling sharing for a standalone repository) is 
described here
    
http://git.or.cz/gitwiki/GitFaq#Howtoshareobjectsbetweenexistingrepositories.3F

-- Daniel

Hi,

On Sun, 16 Aug 2009, Daniel Villeneuve wrote:

What does this have to do with my comment?

Besides, you are teaching general Git users how to use plumbing. That is asking for pain, and the pain will come back to the Git developers, not to you.

Ciao, Dscho

On Sun, Aug 16, 2009 at 02:30:15PM +0200, Johannes Schindelin wrote:

To what? From git-repack(1):

  -l
     Pass the --local option to git-pack-objects. See git-pack-objects(1).

  --local
      This flag is similar to --incremental; instead of ignoring all
      packed objects, it only ignores objects that are packed and/or not
      in the local object store (i.e. borrowed from an alternate).

So I think the documentation is correct, but for the original poster, it suffers from two problems:

  1. The impact of "-l" in this case is a bit subtle and confusing.
     I.e., it is not an obvious "use this flag to break the dependency
     on an alternate", but rather "don't use this flag so that you won't
     ignore non-local objects when packing". In fact, you don't need to
     know about it at all

  2. He has to know that "git repack" is the right place to look in the
     first place (_and_ he has to figure out that "-l" is what he cares
     about and follow the docs to git-pack-objects to find out what it
     does).

So I think the best thing is a "by the way, here is how you break this dependency" closer to where the concept of alternates is defined. I guess the best part would be under the "-s" flag of git-clone, since that is presumably how such a situation was created (unless the user is savvy enough to edit .git/objects/info/alternates themselves, in which case I think we have to assume they know what they are doing).

So maybe something like this would be enough:

-- >8 --
Subject: [PATCH] docs: mention how to break alternates dependency

A user who has created a repository dependency by using "git clone -s" does not necessarily know where to look to find out how to break that dependency. Let's mention it right under "-s", where they are most likely to find it.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-clone.txt |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

Jeff King <peff@peff.net> writes:

After reading this, two points come to my mind. They may or may not be issues.

 (1) Such a user does not necessarily know a casual "git repack -a" breaks
     the dependency, defeating the -s option s/he deliberately used in
     order to save disk space in the first place.  Perhaps we can reword
     this further to kill two penguins with a single stone?

	Note that the pack resulting from running `git repack -a` in the
	repository cloned with the `-s` option will include objects that
	are borrowed from the source repository.  It essentially breaks
	the dependency created by cloning with the `-s` option by copying
	the objects from the source repository.  To keep borrowing from
	the source repository to save disk space, do not use `repack -a`.

     We should suggest an alternative immediately after this sentence,
     e.g. "Instead, use `repack -l`" or something, but somebody should
     check if it is a valid/viable alternative.

 (2) IIRC, "git gc --auto" runs "repack -A".  What is its effect with
     respect to this dependency between object stores?  I suspect it would
     also break the dependency, but if so, is it a good thing?  Perhaps
     should we change it to use a version that keeps the dependency
     instead?

On Sun, 2009-08-16 at 12:16 -0700, Junio C Hamano wrote:

Perhaps a runtime warning that you're about to break it? This user may not even be the one who set the thing up, no?

	-T. Peanut Gallery

On Mon, Aug 17, 2009 at 04:21:22AM +0200, Mike Galbraith wrote:

I'm not really sure what such a setup would look like. If it is a big hosting site like kernel.org or repo.or.cz, then probably it wouldn't matter much. The admins there should probably be running "git repack -l -d -A" periodically to consolidate the object stores (which can happen from this sort of repacking, or from people just pushing the same commits to their repos).

That being said, I can see there being setups where such a warning might be useful. However, we don't really know if the user _wants_ that effect, or if it is an accident. So people following the recommnded "here is how you break the dependency" advice will also get the warning.

I'm torn on whether this is actually a good idea.

-- >8 --
Subject: [PATCH] repack: warn when "-l" is not used with alternates

Failing to use "-l" means that we will copy objects from the source repository, nullifying the usefulness of "-s". We don't want to make this an error, though, since "git repack -a" is used to intentionally break the dependency.

Signed-off-by: Jeff King <peff@peff.net>
---
Is "test -s" portable? It's in POSIX, but I have some lingering doubt in
the back of my mind.

A user seeing such a warning can perhaps ^C to abort the pack. However, should we also give instructions on how to undo the copying (which should be "git repack -d -l -A")?

 git-repack.sh |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

On Mon, 2009-08-17 at 02:48 -0400, Jeff King wrote:

Yeah. There are any number of ways to shoot oneself in the foot, and while on the one hand idiot proofing can be nice if you were about to screw up, "really really?" messages are most frequently annoying noise.

	-Mike

Jeff King <peff@peff.net> writes:

I would understand if you were torn if the proposed change were to refuse to run without -l in a repository with alternates when --force is not given, or something of that nature.

But I can tell you that this "just warn" cannot be a good idea for a very simple reason: breaking and then warning is useless---it is too late for the user to do anything about it.

On Mon, Aug 17, 2009 at 12:24:22AM -0700, Junio C Hamano wrote:

Did you miss the part where I asked "should we include instructions to the user on how to fix this"?

-Peff

Jeff King <peff@peff.net> writes:

Actually, I didn't. It is very hard to lose data once you put it in git; "by following recovery insn the user can redo" is often trivially correct thanks to it.

But it is not a very good option to cause the damage and then give recovery insn. The user might have ran out of quota, and even if he didn't, he wasted needless cycles for the unwanted sort of repacking.

On Mon, Aug 17, 2009 at 12:35:53AM -0700, Junio C Hamano wrote:

OK, let's forget the warning, then. Hopefully between the note under "clone -s" and the fact that most people should be using "git gc" these days, it won't be a big issue.

-Peff

On Sun, Aug 16, 2009 at 12:16:22PM -0700, Junio C Hamano wrote:

Good point, but I don't think this wording is quite right. You can also cause such an inefficiency by simply running "git repack", if the source has loose objects. In other words:

  1. "git repack -a" is sufficient to break dependency, as it copies
     both packed and loose objects

  2. "git repack" _may_ break the dependency, if there are no packs, as
     it copies only loose objects. It _may_ introduce inefficiency, but
     only if there are loose objects.

  3. "git repack -l" always keeps the dependency and current efficiency.

     As an aside, making this list makes me realize there is no easy
     "keep the dependency and increase efficiency". In other words, pack
     everything that is not available otherwise, and then prune the
     remaining packs.

Modified patch is below.

It does work. From a user's perspective, I think "-l" would probably be a more sane default. But I think it is off for historical reasons, and these days we try to steer users towards "git gc", anyway, which does use "-l" by default.

-- >8 --
Subject: [PATCH] docs: describe impact of repack on "clone -s"

The effects of repacking on a repository with alternates are a bit subtle. The two main things users will want are:

  1. Not to waste disk space by accidentally copying objects
     which could be shared.

  2. Copying all objects explicitly to break the dependency
     on the source repo.

This patch describes both under the "clone -s" documentation. It makes sense to put it there rather than in git-repack.txt for both cases. For (1), we are warning the user who is using "clone -s" about what _not_ to do, so we need to get their attention when reading about "clone -s". For (2), we are telling them how git-repack can be used to accomplish a task, but until they know that git-repack is the right tool, they have no reason to look at the repack documentation.

Signed-off-by: Jeff King <peff@peff.net>
---

The extra deleted lines in the patch below are just cleaning up some excess whitespace.

 Documentation/git-clone.txt |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)

On Mon, Aug 17, 2009 at 02:19:16AM -0400, Jeff King wrote:

OK, I take it back. "git repack -d -l -A" will get rid of any packs that are redundant with your alternate.

-Peff

On Sun, Aug 16, 2009 at 12:16:22PM -0700, Junio C Hamano wrote:

No, it actually runs "repack -d -l -A", which behaves fine. I even tested it to make sure.

BTW, the "gc.auto" setting is really annoying at low levels (I set gc.auto to 1 for testing). For efficiency, it looks at only one hashed object directory, and then assumes the other 255 contain roughly the same number of objects. But you get bad sampling error when you have fewer than 256 objects. I don't think it is worth caring about, though. It doesn't seem very sane to set gc.auto to something so low.

-Peff