Re: Resumable clone/Gittorrent (again)

>> Listing all those commits in linux-2.6.git takes 160k*20=3M (I suppose
>> compressing is useless because SHA-1 is random). A compressed listing
>> of those 46k paths takes 200k.
>
> Sure, Linux has only 4 times as much commits as paths, but the commits
> need 30 times more storage. What does it tell us?
>
> IMHO it speaks in favor of my proposal. Imagine a path changing with
> nearly every commit. The root directory is such a path and near top
> directories come close to (as may other files like todo-lists do). For
> each such file you need 3MB for storing the commits SHAs only. Of
> course, you can invent a schema making storing all the SHAs unnecessary,
> but this is another complication.
>
> OTOH, with the commits used as directory entries we get quite a large
> directory. Is this a problem you wanted me to get aware of?

>> The point is you need to fetch its parent commits first in order to
>> verify a commit. Fetching a whole commit is more expensive than a
>> file. So while you can fetch a few commit bases and request for packs
>> from those bases in parallel, the cost of initial commit bases will be
>> high.
>
> You've lost me. I assume you mean that something like that there may be
> very large commits (e.g., in a project not versioned from the very
> beginning). I'd suggest to split such commits in two parts by
> classifying the blobs (and trees) using a fixed bit of their SHAs. Of
> course, this can be repeated in order to get even smaller parts. Let's
> assume a commit X gets split into X0 and X1. As before, for compressing
> of X0 you may use the content any predecessor of X. For compressing of
> X0 you may additionally use the content of X0. This way the compression
> rate could stay close to optimal, IMHO.

>> They are interchangeable as a whole, yes. But you cannot fetch half
>> the pack from server A and the other half from server B. You can try
>> to recover as many deltas as possible in a broken pack, but how do you
>> request a server to send the rest of the pack to you?
>
> Indeed, it's not resumable. For most commits it's not needed since they
> are very small. Why? There are more commits than paths, so the commits
> are smaller than paths on the average. I expect my schema to allow for
> nearly as good compression as git usually does, especially I'd hope it's
> no worse than when packing paths.

> However, there may be very large commits in my schema (and maybe also
> very large "path-packs" in yours). Such large commits get split as I
> described above. Small commits get paired (possibly multiple times) as I
> described earlier. You end up with only reasonably sized pieces of data,
> let's say between 256 and 512 kB, so you don't need to resume.

> Actually, with a really bad connection, you could ask the very server
> from which you obtained an incomplete pack to resume from a given byte
> offset (similar to HTTP ranges). The server may or may not have it. This
> time it should try to keep it available for you in case you connections
> abort again. Don't get me wrong -- this is just an additional help for
> very badly connected people.