Re: Errors cloning large repo

From: Shawn O. Pearce <spearce@spearce.org>
Date: Mar 12, 2007, 14:24 UTC
Message-ID: <20070312142451.GC15150@spearce.org>
In-Reply-To: <200703121209.35052.jnareb@gmail.com>

Jakub Narebski <jnareb@gmail.com> wrote:

> But what would happen if server supporting concatenated packfiles
> sends such stream to the old client? So I think some kind of protocol
> extension, or at least new request / new feature is needed for that.

No, a protocol extension is not required. The packfile format is: 12 byte header, objects, 20 byte SHA-1 footer. When sending concatenated packfiles to a client the server just needs to:

  - figure out how many objects total will be sent;
  - send its own (new) header with that count;
  - initialize a SHA-1 context and update it with the header;
  - for each packfile to be sent:
    - strip the first 12 bytes of the packfile;
    - send the remaining bytes, except the last 20;
    - update the SHA-1 context with the packfile data;
  - send its own footer with the SHA-1 context.

Very simple.  Even the oldest Git clients (pre multi-ack extension)
would understand that.  That's what's great about the way the
packfile protocol and disk format is organized.  ;-)

> Wouldn't it be better to pack loose objects into separate pack
> (and perhaps save it, if some threshold is crossed, and we have
> writing rights to repo), by the way?

Perhaps.  Interesting food for thought, something nobody has tried
to experiment with.  Currently servers pack to update the fetching
client.  That means they may be sending a mixture of already-packed
(older) objects and loose (newer) objects.  But with the new kept
pack thing in receive-pack its more likely that things are already
packed on the server, and not loose.  (I suspect most public open
source users are pushing >100 objects when they do push to their
server.)

Show 5 quoted lines

> > The client could easily segment that into multiple packfiles
> > locally using two rules:
> > 
> >   - if the last object was not a OBJ_COMMIT and this object is
> >   an OBJ_COMMIT, start a new packfile with this object.

...

> 
> Without first rule, wouldn't client end with strange packfile?
> Or would it have to rewrite a pack?

Nope. We don't care about the order of the objects in a packfile. Never have. Never will. Even in pack v4 where we have special object types that should only appear once in a packfile, they can appear at any position within the packfile. MUCH simpler code.

-- 
Shawn.