Re: Balanced packing strategy
- From
- Josef Weidendorfer <josef.weidendorfer@gmx.de>
- Date
- Nov 13, 2005, 20:06 UTC
- Message-ID
- <200511132106.29841.Josef.Weidendorfer@gmx.de>
- In-Reply-To
- <20051112135947.GC30496@pasky.or.cz>
On Saturday 12 November 2005 14:59, Petr Baudis wrote:
Show 10 quoted lines
> The repacking should be done in such a way to minimize the overhead for > the dumb transport users. Ideal for this is some structure like (at the > end of october): > > year2003.pack > year2004.pack > ... > week42.pack > week43.pack > <individual objects for weeks 43, 44>
I am not sure if it is really beneficial, as packs have the requirement to be self contained, so you get a lot of objects undeltified which could be deltified in a better scheme (as eg. in git native protocol).
AFAICS, the git native protocol (which is nothing more than a pack itself for each transfer) even has this problem, too: If you are updating every day via git native, the sum of transfered bytes in a month will be a multiple of one git transfer for all the month's changes.
To keep the pack self-containment property, but work better with dumb transfers, we could introduce incremental packs:
Instead of fully repacking, create a new pack by only appendending new objects at the end of the pack. Thus, most objects will be appended in deltified form, making the incremental addition quite small. The outcome would be a totally new package.
Unfortunately, I do not know the package format in detail, and hope that this is possible at all.
For dumb protocols to take advantage of this, the information that the first part of a package is actually the same as another package has to be stored somewhere visible. If a client detects that it has the first part of a pack already locally, it would be enough to fetch only some the second part.
This is more or less the same as Pasky's solution, but by using incremental packs instead. I think that such incremental packing will not even take much more space that fully repacking.
Josef