Re: Compressing packed-refs

From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Date: Jul 16, 2020, 22:54 UTC
Message-ID: <20200716225429.i7pb6xorkwdsf5fn@chatter.i7.local>
In-Reply-To: <xmqqsgdrf64c.fsf@gitster.c.googlers.com>

On Thu, Jul 16, 2020 at 03:27:15PM -0700, Junio C Hamano wrote:

Show 13 quoted lines

> I think the reftable is the longer term direction, but let's see if
> there is easy enough optimization opportunity that we can afford the
> development and maintenance cost for the short term.
> 
> My .git/packed-refs file begins like so:
> 
>     # pack-refs with: peeled fully-peeled sorted 
>     c3808ca6982b0ad7ee9b87eca9b50b9a24ec08b0 refs/heads/maint-2.10
>     3b9e3c2cede15057af3ff8076c45ad5f33829436 refs/heads/maint-2.11
>     584f8975d2d9530a34bd0b936ae774f82fe30fed refs/heads/master
>     2cccc8116438182c988c7f26d9559a1c22e78f1c refs/heads/next
>     8300349bc1f0a0e2623d5824266bd72c1f4b5f24 refs/notes/commits
>     ...

Let me offer a more special-case (but not crazy) example from git.kernel.org. The newer version of grokmirror that I'm working on is built to take advantage of the pack-islands feature that was added a while back. We fetch all linux forks into a single "object storage" repo, with each fork going into its own refs/virtual/[uniquename]/(heads|tags) place. This means there's lots of duplicates in packed-refs, as all the tags from torvalds/linux.git will end up duplicated in almost every fork.

So, after running git pack-refs --all, the packed-refs file is 50-ish MB in size, with a lot of same stuff like:

5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c refs/virtual/00018460b026/tags/v2.6.11 ^c39ae07f393806ccf406ef966e9a15afc43cc36a ... 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c refs/virtual/00bcef8138af/tags/v2.6.11 ^c39ae07f393806ccf406ef966e9a15afc43cc36a

etc, duplicated 600 times with each fork. It compresses decently well with gzip -9, and *amazingly* well with xz -9:

$ ls -ahl packed-refs -rw-r--r--. 1 mirror mirror 46M Jul 16 22:37 packed-refs $ ls -ahl packed-refs.gz -rw-r--r--. 1 mirror mirror 19M Jul 16 22:47 packed-refs.gz $ ls -ahl packed-refs.xz -rw-r--r--. 1 mirror mirror 2.3M Jul 16 22:47 packed-refs.xz

Which really just indicates how much duplicated data is in that file. If reftables will eventually replace refs entirely, then we probably shouldn't expend too much effort super-optimizing it, especially if I'm one of the very few people who would benefit from it. However, I'm curious if a different sorting strategy would help remove most of the duplication without requiring too much engineering time.

-K