RFC patch, 3 parts--seed as an alias for --dissociate --reference

In a thread a few months ago[1], we discussed the idea that the "--dissociate --reference=foo" interface was somewhat awkward for somebody who just wants to optimize their clone. This is mostly due to the historical development of the features. The logical interface for somebody who just wants a faster clone is something like

   git clone --optimize-my-clone-from=foo.git git://example.com/bar.git

But we got stuck in that thread on coming up with a decent name for the option. Having just read through it, I think a succinct name for the idea is "seed". That is, we seed the clone with objects from another repository.

That thread also brought up the idea that we do not necessarily need to seed from a local repository; we could do something like:

  1. Fetch from the seed repo into refs/seed/*

  2. Fetch from the real clone source; the fetch is optimized by the
     presence of refs/seed/*.

  3. Delete refs/seed/*. Optionally repack to drop any objects needed
     only by the seed refs.

This is awkward with the "--reference" interface, because its implementation is publicly tied to the concept of alternates. Whereas "--seed" is about the end result you want; we can implement it using alternates or with a clone, depending on where the repo is located.

There are a few open issues with this series:

  1. Assuming that "seed" is a reasonable verb for this concept, is
     "--seed=<repo>" OK for the option?  Would "--seed-from=<repo>" be
     better? (Also, the response "bleh, seed is a terrible name" is
     fine, too, but only if accompanied by your own suggestion :) ).

  2. My main goal here is making the concept easier to explain to users.
     The documentation in the third patch explains "--seed" as an alias
     for the other options, which probably isn't helping much. It might
     make sense to have a patch 4/3 that explains "--seed" first, and
     then explains "--reference" as "like --seed, but keep the
     relationship after the clone". Or maybe they should just get their
     own descriptions entirely.

  3. We can't dissociate from a specific alternate, so using "--seed"
     implies that all "--reference" options get dissociated. In this
     series, I issue a warning in that case.  But that would be easily
     solved if "--seed" used the fetch strategy described above, even
     for local clones (which would probably still be quite fast if we
     took clone's usual hard-link shortcut instead of actually fetching
     from a local clone).

I don't have particular plans to implement generic "--seed" from remotes anytime soon. I think this takes us a step in the right direction interface-wise, and it does introduce a succinct concept and option. But the abstraction does leak (e.g., in that it implies "--dissociate"). So one response might be "yes, this is a good building block, and later we can extend --seed; how it works is an implementation detail". But equally valid would be "eh, I like the name and the concept, but this implementation is too hacky; let's wait for somebody to implement it for real". Hence the RFC label.

The patches are:

  [1/3]: clone: use OPT_STRING_LIST for --reference
  [2/3]: clone: reorder --dissociate and --reference options
  [3/3]: clone: add `--seed` shorthand

The third one is the interesting one, and the first two are nearby cleanups. Whether we pursue the third one or not, I think the first two are worth taking by themselves.

-Peff

[1] http://thread.gmane.org/gmane.comp.version-control.git/264178/focus=264234

Not only does this save us having to implement a custom callback, but it handles "--no-reference" in the usual way (to clear the list).

The generic callback does copy the string, which we don't technically need, but that should not hurt anything.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/clone.c | 13 ++-----------
 1 file changed, 2 insertions(+), 11 deletions(-)

These options are intimately related, so it makes sense to list them nearby in the "-h" output (they are already adjacent in the manpage).

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/clone.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

The safe way to use `--reference` is to add in the recent `--dissociate` option, which optimizes the initial clone, but does not create any obligation to avoid pruning or deleting the reference repository. However, it can be rather tricky to explain why two options are necessary, and why using `--reference` alone is unsafe.

This patch introduces a single option, `--seed`, which does the right thing; we can steer users towards it rather than explaining the complexities. It also provides a natural interface if we later want to allow seeding from non-local repositories.

Note that git-repack cannot selectively dissociate from particular alternates. Therefore using `--reference` and `--seed` together will dissociate from _all_ referenced repositories. We issue a warning to the user in this case.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-clone.txt |  3 +++
 builtin/clone.c             | 12 +++++++++++-
 t/t5700-clone-reference.sh  |  6 ++++--
 3 files changed, 18 insertions(+), 3 deletions(-)

Hi Peff,

On 2015-05-21 06:16, Jeff King wrote:

Since you want to advertise this as an easier way than `--dissociate --reference=<repository>`, it might make sense to avoid sending the reader that way, too.

Maybe something like

--seed <repository>::
    Fetch objects from <repository> instead of the clone URL when possible. This is useful when a (possibly partial) clone already exists locally, to avoid transferring the same objects again.

Ciao, Dscho

From: "Johannes Schindelin" <johannes.schindelin@gmx.de>

Would it be worth mentioning here that a bundle is a satisfactory alternative to repository?

+--seed <repository|bundle>::
+    Fetch objects from <repository> or <bundle> instead of the clone 
URL when possible. This is useful when a (possibly partial) clone 
already exists locally, to avoid transferring the same objects again.

I haven't checked if the invocation would accept a bundle filename, but I'm presuming it can in the same way that clone does.

-- Philip

Hi Philip,

On 2015-05-21 21:45, Philip Oakley wrote:

The proof would lie in the pudding ;-) Would you mind testing whether it works with bundles?

Ciao, Dscho

On Fri, May 22, 2015 at 08:37:56AM +0200, Johannes Schindelin wrote:

I can't imagine that it would with my patch. The implementation is based on --reference, which is going to try to set up the bundle as an alternate.

Having slept on it, I really think "--seed" should be "fetch from the seed into temp refs", and not what I posted earlier.

-Peff

Jeff King <peff@peff.net> writes:

Yeah, I think that is the right way to do it.

And it happens to mesh well with the (not so well advertised but not so well hidden) plan to allow loaded server side to advise --seed from bundle hosted elsewhere, e.g.

     * "clone" connects to upload-pack

     * upload-pack advertises --seed=http://cdn.github.com/project.bundle
       via capability

     * "clone" disconnects from upload-pack, and runs (resumable)
       wget to the seed to grab bundle.

     * "clone" then fetches refs/*:refs/remotes/origin/* from the bundle

     * "clone" then continues to fetch into +refs/remotes/origin/* as
       usual, but does an equivalent of using --prune for this fetch
       to drop anything extra/stale that the seed bundle may have
       had.

     * optionally "clone" can immediately "repack".

... which I wanted to see happen in a near future.

And that --seed thing that can name a local bundle file is a very good first step toward the direction, I think.

Thanks.

On Sun, May 24, 2015 at 12:07:53PM -0700, Junio C Hamano wrote:

In the meantime, do you want to pick up patches 1 and 2? I think they are cleanups that stand on their own, whether we do patch 3 or not.

-Peff

Jeff King <peff@peff.net> writes:

Thanks for reminding.  Let me take a look.

Thanks.

On Thu, May 21, 2015 at 06:05:31PM +0200, Johannes Schindelin wrote:

Yeah, I agree that is much better (it's the "4/3" I suggested in the cover letter).

-Peff

Jeff King <peff@peff.net> writes:

That's a nice name.

The seed may not even have to be a repository. A bundle file hosted on CDN that is reachable via (resumable) wget would be another good way to prime the well, and it would fit with the above framework nicely. Grab it, fetch from it into a temporary hierarchy and then run "fetch --prune" against the repository you originally wanted to clone from.

Yes.  I like the name, the concept and the general direction.

On Wed, May 20, 2015 at 10:01:49PM -0700, Junio C Hamano wrote:

Yeah, I was just looking over the list archives for the past few months, for things I had marked as "to read and think about later"[1]. That's how I recalled our prior discussion on --dissociate.

Anyway, I happened upon the "prime the clone from a bundle" concept being discussed again recently, and had the same thought. We already treat local bundles as a possible source for fetching/cloning. Once upon a time I had some patches that would let you clone straight from a bundle over http (it just spooled to disk, which is not the _most_ efficient way to do it, but trying to massage the bundle straight into a packfile[2] ends up every complex very quickly). I should resurrect those patches.

-Peff

[1] My "think about later" mailbox has ~5000 messages in it, some of
    which are from 2010. I think I may need to just declare bankruptcy.

[2] There's that word again.

Jeff King <peff@peff.net> writes:

Were you absolutely bored and didn't have anything better to do ;-)?

Thanks for sweeping the backlog. I usually do the same myself while the tree is more or less frozen first week of the new cycle waiting for brown paper bag bugs to be discovered, but since we somehow kept moving immediately after 2.4.0 release, I didn't have a chance to do that.

Thanks for working on this. I have one little bikeshedding comment...

On 05/21/2015 06:14 AM, Jeff King wrote:

I think "seed" is a pretty good name. The only downside is that "seed" suggests that the process injects just a few seeds that are much smaller than the whole. But in fact, (hopefully) this option causes the bulk of the needed objects to be pre-fetched.

I can't think of any name that is clearly better. Some brainstorming:

* prepare -- meh
* prime (as in "prime the pump"). But "prime" alone could have too many
meanings, so...
* prime-from
* pre-fetch or pre-fetch-from
* pre-load or pre-load-from

BTW I think "--seed-from=<repo>" is more self-explanatory than "--seed=<repo>".

Michael