Re: Index format v5

From: Thomas Gummerer <t.gummerer@gmail.com>
Date: May 3, 2012, 20:36 UTC
Message-ID: <CALgYhfOLGFK92833nZp5ny25+uoRnsygkH2M4yYRzU4JBE3Evw@mail.gmail.com>
In-Reply-To: <87ehr15dnh.fsf@an-dro.info.enstb.org>

On Thu, May 3, 2012 at 8:21 PM, Ronan Keryell <Ronan.Keryell@hpc-project.com> wrote:

Show 15 quoted lines

>>>>>> On Thu, 3 May 2012 19:25:12 +0200, Thomas Gummerer <t.gummerer@gmail.com> said:
>
>    Thomas> I have been drafting the Version 5 of the index format over
>    Thomas> the past few days with the help of Thomas Rast, Michael
>    Thomas> Haggerty, cmn and barrbrain on IRC. It will save with prefix
>    Thomas> compression on the path, and using a crc32 over the stat
>    Thomas> data, instead of the full data, since it is only used for
>    Thomas> checking if the file is changed. (Thanks Michael Haggerty
>    Thomas> for this hint. Unless we are missing something this will
>    Thomas> save another ~4 MB on the Webkit index.
>
> Great!
>
> But I wonder whether it may not worth to investigate a 64-bit version for
> the offsets and so on, just in case...

64-bit versions of the offsets were taken into consideration, but currently the Webkit index (the largest I know) has a size of 26 Mb, which is reduced to about 15 MB or less with the v5 format. With 32-bit we can address 4GB, which is about 266 times the Webkit index. Therefore there probably is no use for 64-bit offsets in the years to come.