Re: [PATCH v2 4/4] bundle v3: the beginning

From: Duy Nguyen <pclouds@gmail.com>
Date: Jun 7, 2016, 10:22 UTC
Message-ID: <CACsJy8DSvS6DxBM-RUknOEe5DquF-qcRr-MK5FUZcRXYDpSgXg@mail.gmail.com>
In-Reply-To: <CAP8UFD2t=2wJ=1U1ctMYNuMSejBYLh2yeLU7ZfP5Q6KLxUApjQ@mail.gmail.com>

On Tue, Jun 7, 2016 at 3:46 PM, Christian Couder <christian.couder@gmail.com> wrote:

Show 8 quoted lines

>> Any thought on object streaming support?
>
> No I didn't think about this. In fact I am not sure what this means.
>
>> It could be a big deal (might
>> affect some design decisions).
>
> Could you elaborate on this?

Object streaming api is in streaming.h. Normally objects are small and we can inflate the whole thing in memory before doing anything with them. For really large objects (which I guess is one of the reasons for remote odb) we don't want to do that. It takes lots of memory and you could have objects larger than your physical memory. In some cases when can ignore those objects (e.g. mark them binary and choose not to diff). In some other cases (e.g. checkout), we use streaming interface to process an object while we're inflating it to keep memory usage down. It's easy to add a new streaming backend, once you settle on how remote odb streams stuff.

Show 7 quoted lines

>> I would also think about how pack v4
>> fits in this (e.g. how a tree walker can still walk fast, a big
>> promise of pack v4; I suppose if you still maintain "pack" concept
>> over external odb then it might work). Not that it really matters.
>> Pack v4 is the future, but the future can never be "today" :)
>
> Sorry I haven't really followed pack v4 and I forgot what it is about.

It's a new pack format (and practically vaporware at this point) that promises much faster access when you need to walk through trees and commits (think rev-list --objects --all, or git-blame). Because we are (or I am) still not sure if pack v4 will ever get to the state where it can be merged to git.git, I think it's ok for you to ignore it too if you want. You can read more about the format here [1] and go even further back to [2] when Nicolas teased us with the pack size (smaller, which is a nice side effect). The potential issue with pack v4 is, the tree walker (struct tree_desc and related funcs in walk-tree.h) needs to know about pack v4 in order to walk fast. Current tree walker does not care if an object is packed (using what format) at all. Remote odb for pack v4 must have some way that allows to read pack data directly, something close to "mmap", it's not just about an api to "get me the canonical content of this object".

[1] http://article.gmane.org/gmane.comp.version-control.git/234012 [2] http://article.gmane.org/gmane.comp.version-control.git/233038

-- 
Duy