Re: 1.3.0 creating bigger packs than 1.2.3

From: Nicolas Pitre <nico@cam.org>
Date: Apr 20, 2006, 21:02 UTC
Message-ID: <Pine.LNX.4.64.0604201630320.2215@localhost.localdomain>
In-Reply-To: <7v8xq0yteb.fsf@assigned-by-dhcp.cox.net>

On Thu, 20 Apr 2006, Junio C Hamano wrote:

Show 21 quoted lines

> Nicolas Pitre <nico@cam.org> writes:
> 
> > On Thu, 20 Apr 2006, Shawn Pearce wrote:
> >
> >> The more that I think about it the more it seems possible that the
> >> pathname hashing is what may be causing the problem.  Not only did
> >> bisect point to 1d6b38cc76c348e2477506ca9759fc241e3d0d46 but the
> >> directory which contains the bulk of the space has many files with
> >> the same name located in different directories:
> > [...]
> >
> > But the bad commit according to your bisection talks about "thin" packs 
> > which are not involved in your case.  So something looks fishy with that 
> > commit which should not have touched path hashing in the non-thin pack 
> > case...  I think...
> 
> I think this explains it.  The new code hashes full-path, but
> places bins for the paths with the same basename next to each
> other, so before Makefile and doc/Makefile and t/Makefile were
> all in the same bin, but now they are in three different bins
> next to each other.

That is fine. In fact I did try with a tweaked name_hash() that completely ignored all directory components and the resulting pack was even bigger, much bigger, when repacking Shawn's repo.

> I originally thought, with one single notable exception of
> Makefile, having the identically named file in many different
> directories is not common nor sane,

I'd tend to disagree with that but...

Show 5 quoted lines

> and the new code favors to
> delta with the exact same path for deeper history over wasting
> delta window for making delta with objects with the same name in
> different places in more recent history.  I think I benched this
> with kernel repository (git.git was too small for that).

This is obviously fine. And if a file in a given directory has few revisions then the delta window will consider objects for a file with the same name in other directories as well, which is also sensible. So if files of the same name are located in different directories they should delta well against each other if they're similar enough. This should cover Shawn's repo layout.

Show 5 quoted lines

> But I suspect we have a built-in "we sort bigger to smaller, and
> we cut off when we switch bins" somewhere in find_delta() loop,
> which I do not recall touching when I did that change, so that
> may be interfering and preventing 0-11-AdjLite.deg from all over
> the place to delta against each other.

I just cannot find something that would do that in the code. When --no-reuse-delta is specified, the only things that will break the loop in find_delta() is when try_delta() returns -1, and that happens only when changing object type or when the size difference is too big, but nothing looks at the name hash.

It is also hard to corelate it with commit 1d6b38cc which is the one that introduced the regression.

Nicolas