Re: [BUG] fast-import producing very deep tree deltas

From: Brian Downing <bdowning@lavos.net>
Date: Nov 13, 2007, 14:36 UTC
Message-ID: <20071113143559.GS6212@lavos.net>
In-Reply-To: <20071113092721.GD14735@spearce.org>

On Tue, Nov 13, 2007 at 04:27:21AM -0500, Shawn O. Pearce wrote:

Show 11 quoted lines

> Brian, does this fix it?
> 
> So if the tree we reloaded was already at the maximum depth we
> wouldn't know it and make the new tree a delta.  Multiply the
> number of times the branch cache has to swap out the tree times
> max_depth (10) and you get the maximum delta depth of a tree created
> by fast-import.  In Brian's case above the active branch cache had
> to swap the branch out 603/604 times during this import to produce
> a tree with a delta depth of 6035.
> 
> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

/Much/ better:

git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects:     140000
Total objects:       135970 (     62664 duplicates                  )
      blobs  :        42196 (     13695 duplicates      19898 deltas)
      trees  :        72143 (     48969 duplicates      62402 deltas)
      commits:        21631 (         0 duplicates          0 deltas)
      tags   :            0 (         0 duplicates          0 deltas)
Total branches:          10 (         1 loads     )
      marks:        1048576 (     63827 unique    )
      atoms:          18971
Memory total:          8329 KiB
       pools:          2860 KiB
     objects:          5468 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =     273071
pack_report: pack_mmap_calls          =      16855
pack_report: pack_open_windows        =         50 /        363
pack_report: pack_mapped              = 8529277175 / 8589933814
---------------------------------------------------------------------

depths: count 135970 total 380519 min 0 max 10 mean 2.80 median 1 std_dev 3.22

In addition, fast-import ran much (probably 10x) faster and with much less memory usage (last time it peaked around 1GB):

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
13098 bdowning  18   0 8223m  34m 6576 R   72  1.7   0:51.44 git-fast-import

Presumably not having to rebuild the root tree object from a hundreds-deep delta chain many hundreds of times sped things up a bit.

Acked-by: Brian Downing <bdowning@lavos.net>

-bcd