threads / discuss / 5585

git-repack: Outof memory

Subject: git-repack: Outof memory

## tl;dr

6 messages between Sep 17, 2006 and Sep 18, 2006.

replies: 5people: 4as markdown or json

Dongsheng Song· Sep 17, 2006, 01:31 UTC · lore

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all:

I'm import from subversion. The problem appears to be git-repack phase using too many memory:

$ git-repack -a -f -d --window=64 --depth=64
Generating pack...
Done counting 123497 objects.
Deltifying 123497 objects.
  24% (29677/123497) done

$ top

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3572 www-data  18   0 2591m 1.9g  528 R   13 94.8  81:48.98 git-pack-object

$ free
             total       used       free     shared    buffers     cached
Mem:       2076308    2029824      46484          0       2760       6800
- -/+ buffers/cache:    2020264      56044
Swap:      1028152     684032     344120

How to compute memory usage of git-repack ?

Thanks and regards,

Dongsheng

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFDKVh90pbDJCgbHoRAlweAJ45DhTXI+bb+nb2Y+JlbIBoFusK8wCgk0U2 XcUc9K/chYiUYq3ZLychzcU= =NzXf -----END PGP SIGNATURE-----

Shawn Pearce· Sep 17, 2006, 02:25 UTC · re: Dongsheng Song · lore

Re: git-repack: Outof memory

Dongsheng Song <dongsheng.song@gmail.com> wrote:

Show 12 quoted lines

> I'm import from subversion. The problem appears to be git-repack phase using too many memory:
> 
> $ git-repack -a -f -d --window=64 --depth=64
> Generating pack...
> Done counting 123497 objects.
> Deltifying 123497 objects.
>   24% (29677/123497) done
> 
> $ top
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  3572 www-data  18   0 2591m 1.9g  528 R   13 94.8  81:48.98 git-pack-object

*ouch* You are probably running out of address space if you are on a 32 bit architecture. A 2.5 GiB virtual address space is pretty close to the maximum allowed on most OSes.

The code that I'm sitting on but haven't yet completed rebasing onto current Git would probably help here.

Do you have any existing .pack files in .git/objects/pack? How big are they and their corresponding .idx files?

git-repack will need to mmap every .pack and .idx in
.git/objects/pack, plus it needs working memory for each object
(123,497 of 'em) but as I recall its pretty frugal on its per-object
allocation.  It can easily work with as many as 2 million objects on
a 32 bit system, assuming the .pack and .idx files aren't too large.

-- 
Shawn.

Nicolas Pitre· Sep 17, 2006, 21:09 UTC · re: Dongsheng Song · lore

Re: git-repack: Outof memory

On Sun, 17 Sep 2006, Dongsheng Song wrote:

Show 9 quoted lines

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi all:
> 
> 
> I'm import from subversion. The problem appears to be git-repack phase using too many memory:
> 
> $ git-repack -a -f -d --window=64 --depth=64

Try with that --window=64 argument removed.

Nicolas

Shawn Pearce· Sep 18, 2006, 00:23 UTC · lore

Re: git-repack: Outof memory

Dongsheng Song <dongsheng.song@gmail.com> wrote:

Show 10 quoted lines

> finished, thanks a lot.
> 
> $ ls -l .git/objects/pack/
> total 1675466
> -rw-r--r-- 1 www-data www-data    2964992 Sep 17 19:17
> pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.idx
> -rw-r--r-- 1 www-data www-data 1711037142 Sep 17 19:17
> pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.pack
> 
> If the pack files larger than 2.5g,  how can  I repack it on i686 ?

*youch* A 1.5 GiB pack file?

Your files apparently do not delta compress very well. I fear that you are going to bump up against address space limitations soon on 32 bit systems. Then you will bump up against the 4 GiB pack file size limit. Which means you will need to use several packs and avoid the '-a' flag when calling git-repack.

Nico was suggesting using the default window size (rather than --window=64) as larger windows requires more memory during repack. But you may also need the mmap window code I'm working on. I better hurry up and get that into Junio's testing branches. :-)

-- 
Shawn.

Junio C Hamano· Sep 18, 2006, 02:47 UTC · re: Shawn Pearce · lore

Re: git-repack: Outof memory

Shawn Pearce <spearce@spearce.org> writes:

Show 5 quoted lines

> ...  I fear that
> you are going to bump up against address space limitations soon
> on 32 bit systems.  Then you will bump up against the 4 GiB pack
> file size limit.  Which means you will need to use several packs
> and avoid the '-a' flag when calling git-repack.

I presume that you are hinting that we would need to update git-repack so that it is still useful without --all.

Which means "pack-objects --unpacked=active-pack" would need to be pushed out so that git-repack can be updated to do the "archived ones and repacking the active pack" we talked about earlier.

I think pack-objects --unpacked=active-pack is ready, so I'll push it out.

Shawn Pearce· Sep 18, 2006, 06:50 UTC · re: Junio C Hamano · lore

Re: git-repack: Outof memory

Junio C Hamano <junkio@cox.net> wrote:

Show 10 quoted lines

> Shawn Pearce <spearce@spearce.org> writes:
> 
> > ...  I fear that
> > you are going to bump up against address space limitations soon
> > on 32 bit systems.  Then you will bump up against the 4 GiB pack
> > file size limit.  Which means you will need to use several packs
> > and avoid the '-a' flag when calling git-repack.
> 
> I presume that you are hinting that we would need to update
> git-repack so that it is still useful without --all.

Yes.  :-)

Apparently the Porcelain level is still open for debate. Someone just needs to post a reasonable implementation to get discussion (or lack there of) going. I'm too tired right now to try to roll something up right, even though it looks to be rather simple.

As it is I'm having a difficult time navigating in vi to write this email and normally vi navigation is as easy as breathing for me...

> I think pack-objects --unpacked=active-pack is ready, so I'll
> push it out.

Thanks; I noticed that in your "What's in git.git" announcement.

-- 
Shawn.