RE: Re-Transmission of blobs?

From: Pyeron, Jason J CTR (US) <jason.j.pyeron.ctr@mail.mil>
Date: Sep 12, 2013, 12:45 UTC
Message-ID: <871B6C10EBEFE342A772D1159D132085571A7A1B@umechphj.easf.csd.disa.mil>
In-Reply-To: <20130912092339.GA30702@sigill.intra.peff.net>

Show 21 quoted lines

> -----Original Message-----
> From: Jeff King
> Sent: Thursday, September 12, 2013 5:24 AM
> 
> On Thu, Sep 12, 2013 at 09:42:41AM +0200, Josef Wolf wrote:
> 
> > > >> There are some work being done to optimize this further using
> > > >> various techniques, but they are not ready yet.
> > >
> > > And this still stands.
> >
> > Do you have a pointer or something? I'd like to check out whether I
> can
> > contribute to this work.
> 
> I think Junio is referring to the reachability bitmap work. We may know
> that the other side has commit "E" (and therefore every object
> reachable
> from it), but we do not walk the graph to find the complete set of
> reachable objects. Doing so requires a lot of CPU and I/O, and in most
> cases does not help much.

If the rules of engagement are change a bit, the server side can be release from most of its work (CPU/IO).

Client does the following, looping as needed:

Heads=server->heads(); KnownCommits=Local->AllCommits(); Missingblobs=[]; Foreach(commit:heads) if (!knownCommits->contains(commit)) MissingBlobs[]=commit; Foreach(commit:knownCommit) if (!commit->isValid()) MissingBlobs[]=commit->blobs(); If (missingBlobs->size()>0) server->FetchBlobs(missingBlobs);

This should work efficiently for the server if a) the client is empty b) the client is corrupt c) the client is up to date

Extending the server->fetchBlobs() to be more fancy, like taking patterns, such as between aaaaaa and dddddd exclusive is an exercise for someone else.