Re: GC of alternate object store (was: Bringing a bit more sanity to $GIT_DIR/objects/info/alternates?)

From: Oswald Buddenhagen <ossi@kde.org>
Date: Aug 29, 2012, 07:42 UTC
Message-ID: <20120829074249.GA14408@ugly.local>
In-Reply-To: <hbf.20120828vnfp@bombur.uio.no>

On Tue, Aug 28, 2012 at 09:19:53PM +0200, Hallvard Breien Furuseth wrote:

Show 9 quoted lines

> Oswald Buddenhagen wrote:
> > (...)so the second approach is the "bare aggregator repo" which adds
> > all other repos as remotes, and the other repos link back via
> > alternates. problems:
> > 
> > - to actually share objects, one always needs to push to the aggregator
> 
> Run a cron job which frequently does that?
>

nope. i also have separate repos which share the same code, so when i develop it i need to pick between them "live". of course it's unlikely to get conflicts in this case, so the missing object sharing is not that bad (the objects are transferred via format-patch, as i'm rewriting paths anyway), but when it happens it's messy to get out again.

Show 7 quoted lines

> > - tags having a shared namespace doesn't actually work, because the
> > repos have the same tags on different commits (they are independent
> > repos, after all)
> 
> Junio's proposal partially fixes that: It pushes refs/* instead of
> refs/heads/*, to refs/remotes/<borrowing repo>/.  However...
>

i did exacty that. the tags are *still* not populated - git just tries very hard to treat them specially. and the "stash" file is also ignored, unfortunately.

Show 8 quoted lines

> > - one still cannot safely garbage-collect the aggregator, as the refs
> > don't include the stashes and the index, so rebasing may invalidate
> > these more transient objects.
> 
> Also if you copy a repo (e.g. making a backup) instead of cloning it,
> and then start using both, they'll push into the same namespace -
> overwriting each other's refs.
>

right. it's a clear user error, though - i wouldn't *expect* it to work. anyway, i don't have *that* problem, as my aggregator actually pulls, not the other way round.

anyway, the bottom line is that using alternates as-is for anything but sharing refs/remotes/origin/* (which i'm assuming to be ff-only) is a recipe for disaster.

anything which is supposed to be in any way safe must make the "donor" object store aware of the sharing, which at the very least means setting the proposed append-only flag _by the borrowing_ object store. which means that the info/alternates file should be obfuscated, so people can't edit it manually.

Show 9 quoted lines

> > i would re-propose hallvard's "volatile" alternates (at least i think that's
> > what he was talking about two weeks ago): they can be used to obtain
> > objects, but every object which is in any way referenced from the current
> > clone must be available locally (or from a "regular" alternate). that means
> > that diffing, etc.  would get objects only temporarily, while cherry-picking
> > would actually copy (some of) the objects. this would make it possible to
> > "cross-link" repositories, safely and without any "3rd parties".
> 
> I'm afraid that idea by itself won't work:-(

> Either you borrow from a store or not.
>

correct. from "regular" alternates you "borrow", in "volatile" ones you only "peek". so apparently our definitions are different after all.

> If Git uses an object from the volatile store, it can't always know if
> the caller needs the object to be copied.
>

it doesn't have to. the distinction comes when creating objects: if an object is only in a volatile alternate, it does not already exist for the purpose of object creation and is thus created locally.

regards