Re: git pack/unpack over bittorrent - works!

From: Luke Kenneth Casson Leighton <luke.leighton@gmail.com>
Date: Sep 2, 2010, 19:41 UTC
Message-ID: <AANLkTinFPxsY6frVnga8u15aovQarfWreBYJfri6ywoK@mail.gmail.com>
In-Reply-To: <alpine.LFD.2.00.1009021249510.19366@xanadu.home>

nicolas, thanks for responding: you'll see this some time in the future when you catch up, it's not a high priority, nothing new, just thinking out loud, for benefit of archives.

On Thu, Sep 2, 2010 at 6:21 PM, Nicolas Pitre <nico@fluxnic.net> wrote:

Show 6 quoted lines

>>  * what _can_ be guaranteed?
>
> You can guarantee that if the SHA1 name of different packs is the same
> then they contain the same set of objects.  Obviously their packed
> encoding will be different, and even the pack sizes might be quite
> different too.

 ack.  ok, so the idea of creating lots and lots of 2nd level
.torrents by name {ref}-{commitref}-{SHA-1}.torrent is about the only
way to get around that.

@begin lots of no, no and hell no...

> [...]

@end

 dang.  diffs, versions, threads and zlibs as well, all conspiring against me :)

>> * is it possible to _make_ the repository guaranteed to produce
>> identical pack objects?
>
> Sure, but performance will suck.

 that's fiiine :)  as i've learned on the pyjamas project, it's rare
that you have speed and interoperability at the same time...

> The only way to get a bit-for-bit reproducible pack one one specific
> system is to use 'git repack' with the -f switch, and limit it to only
> one thread.

 whew - a way out, at last.  you had me going, for a minute :)

Show 5 quoted lines

>> * is it a versioning issue?  is it because there are different
>> versions (2 and 3)?  if so, that's ok, you just force people to use
>> the same pack-object versions.
>
> Not at all.  FYI [....]

 appreciated.

> I'm sorry as this isn't going to help you much unfortunately.

 neeh, i'm flexible.  it looks like i'm going to need to deviate from
bittorrent, after all, start adding new commands over which the git
rev-list gets transferred, rather than as a VFS layer.  the reason is
that bittorrent depends on the files and the data in the files being
all the same, so that a hash can be taken of the whole lot and the
end-result verified.

 if the pack-objects are going to vary, then the VFS layer idea is
blown completely out the water, except for the absolute basic
meta-info such as "refs/heads/*".  so i might as well just use
"actual" bittorrent to transfer packs via
{ref}-{commitref}-{SHA-1}.torrent.

ho hum, drawing board we come...

l.