Re: Using bitmaps to accelerate fetch and clone

From: Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date: Sep 28, 2012, 12:00 UTC
Message-ID: <CACsJy8AUdRyjSrAgM+ABzWet2NKz7N7M4re2QVoRPrrA=zfvvg@mail.gmail.com>
In-Reply-To: <CAJo=hJstK1tGrWhtBt3s+R1a6C0ge3wMtJnoo43Fjfg5A57eVw@mail.gmail.com>

On Thu, Sep 27, 2012 at 7:47 AM, Shawn Pearce <spearce@spearce.org> wrote:

> * https://git.eclipse.org/r/7939
>
>   Defines the new E003 index format and the bit set
>   implementation logic.

Quote from the patch's message:

"Currently, the new index format can only be used with pack files that contain a complete closure of the object graph e.g. the result of a garbage collection."

You mentioned this before in your idea mail a while back. I wonder if it's worth storing bitmaps for all packs, not just the self contained ones. We could have one leaf bitmap per pack to mark all leaves where we'll need to traverse outside the pack. Commit leaves are the best as we can potentially reuse commit bitmaps from other packs. Tree leaves will be followed in the normal/slow way.

For connectivity check, fewer trees/commits to deflate/parse means less time. And connectivity check is done on every git-fetch (I suspect the other end of a push also has the same check). It's not unusual for me to fetch some repos once every few months so these incomplete packs could be quite big and it'll take some time for gc --auto to kick in (of course we could adjust gc --auto to start based on the number of non-bitmapped objects, in additional to number of packs).

-- 
Duy