Re: fast-import's gfi_unpack_entry causes too many munmap/mmap cycles

From: Mike Hommey <mh@glandium.org>
Date: Apr 17, 2016, 00:54 UTC
Message-ID: <20160417005443.GA15847@glandium.org>
In-Reply-To: <20160416110403.GA19197@glandium.org>

On Sat, Apr 16, 2016 at 08:04:03PM +0900, Mike Hommey wrote:

Show 25 quoted lines

> So I think I got myself a workaround...
> 
> > A --- B
> >  \
> >   \-- C
> > 
> > I have:
> > - diff between null-tree and A
> > - diff between A and B
> > - diff between B and C
> 
> I should be able to do:
> 
> - start the commit command for A
> - before finishing it, `ls ""`
> - then apply the diff for B and `ls ""`
> - then apply the diff for C and `ls ""`
> - then `deleteall`
> - then `M 040000 sha1_from_first_ls ` and finally finish A
> - create the commit for B with `from
>   0000000000000000000000000000000000000000\nmerge :mark` and `M 040000
>   sha1_from_second_ls`
> - likewise for C
> 
> ... and avoid gfi_unpack_entry.

And it works... as an avoidance of gfi_unpack_entry... but it has its own problem: somehow the store_tree() that happens for each of those `ls ""` commands is storing *all* trees. Even the ones that haven't changed. In terms of a minimalistic fast-import script:

With:
  commit refs/FOO
  committer <foo@foo> 0 +0
  data 0

  M 644 inline a/a
  data 1
  a

  commit refs/FOO
  committer <foo@foo> 0 +0
  data 0

  M 644 inline b/b
  data 1
  b

store_tree is called for:
- b39954843ff6e09ec3aa2b942938c30c6bd1629e
- 2c3b59f77afa6fea6c1a380eeb0cb1eb292515b5
- 51e58bf6ce558dd384bbf9d493f9a376f3bcb089
- a97dda9f3a819113b3b239b9a62edece27136080

With:
  commit refs/FOO
  committer <foo@foo> 0 +0
  data 0

  M 644 inline a/a
  data 1
  a
  ls ""
  M 644 inline b/b
  data 1
  b

store_tree is called for:
- b39954843ff6e09ec3aa2b942938c30c6bd1629e
- 2c3b59f77afa6fea6c1a380eeb0cb1eb292515b5
- b39954843ff6e09ec3aa2b942938c30c6bd1629e
- 51e58bf6ce558dd384bbf9d493f9a376f3bcb089
- a97dda9f3a819113b3b239b9a62edece27136080

Note how b39954843ff6e09ec3aa2b942938c30c6bd1629e is being stored twice (it's the tree for a/).

So in the scenario I'm testing, which has many more trees, I'm trading 29k gfi_unpack_entry calls and 230k store_tree calls for 1.96M store_tree calls. On even larger trees, I'm not sure that wouldn't make things even worse than they already are with gfi_unpack_entry.

Mike