# git-repack: Outof memory 6 messages from 2006-09-17 to 2006-09-18. Participants: Dongsheng Song, Shawn Pearce, Nicolas Pitre, Junio C Hamano. Thread: https://gitlist.dev/t/5585 ## Dongsheng Song, 2006-09-17 01:31 Subject: git-repack: Outof memory Message-ID: <450CA561.9030602@gmail.com> URL: https://gitlist.dev/e/450CA561.9030602%40gmail.com ``` -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all: I'm import from subversion. The problem appears to be git-repack phase using too many memory: $ git-repack -a -f -d --window=64 --depth=64 Generating pack... Done counting 123497 objects. Deltifying 123497 objects. 24% (29677/123497) done $ top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3572 www-data 18 0 2591m 1.9g 528 R 13 94.8 81:48.98 git-pack-object $ free total used free shared buffers cached Mem: 2076308 2029824 46484 0 2760 6800 - -/+ buffers/cache: 2020264 56044 Swap: 1028152 684032 344120 How to compute memory usage of git-repack ? Thanks and regards, Dongsheng -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (MingW32) iD8DBQFFDKVh90pbDJCgbHoRAlweAJ45DhTXI+bb+nb2Y+JlbIBoFusK8wCgk0U2 XcUc9K/chYiUYq3ZLychzcU= =NzXf -----END PGP SIGNATURE----- ``` ## Shawn Pearce, 2006-09-17 02:25 Subject: Re: git-repack: Outof memory Message-ID: <20060917022534.GB7512@spearce.org> URL: https://gitlist.dev/e/20060917022534.GB7512%40spearce.org In-Reply-To: <450CA561.9030602@gmail.com> ``` Dongsheng Song wrote: > I'm import from subversion. The problem appears to be git-repack phase using too many memory: > > $ git-repack -a -f -d --window=64 --depth=64 > Generating pack... > Done counting 123497 objects. > Deltifying 123497 objects. > 24% (29677/123497) done > > $ top > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 3572 www-data 18 0 2591m 1.9g 528 R 13 94.8 81:48.98 git-pack-object *ouch* You are probably running out of address space if you are on a 32 bit architecture. A 2.5 GiB virtual address space is pretty close to the maximum allowed on most OSes. The code that I'm sitting on but haven't yet completed rebasing onto current Git would probably help here. Do you have any existing .pack files in .git/objects/pack? How big are they and their corresponding .idx files? git-repack will need to mmap every .pack and .idx in .git/objects/pack, plus it needs working memory for each object (123,497 of 'em) but as I recall its pretty frugal on its per-object allocation. It can easily work with as many as 2 million objects on a 32 bit system, assuming the .pack and .idx files aren't too large. -- Shawn. ``` ## Nicolas Pitre, 2006-09-17 21:09 Subject: Re: git-repack: Outof memory Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0609171707430.2627%40xanadu.home In-Reply-To: <450CA561.9030602@gmail.com> ``` On Sun, 17 Sep 2006, Dongsheng Song wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi all: > > > I'm import from subversion. The problem appears to be git-repack phase using too many memory: > > $ git-repack -a -f -d --window=64 --depth=64 Try with that --window=64 argument removed. Nicolas ``` ## Shawn Pearce, 2006-09-18 00:23 Subject: Re: git-repack: Outof memory Message-ID: <20060918002357.GA19727@spearce.org> URL: https://gitlist.dev/e/20060918002357.GA19727%40spearce.org In-Reply-To: <4b3406f0609170543p68d96b9x9ba0c5a74d9e89e8@mail.gmail.com> ``` Dongsheng Song wrote: > finished, thanks a lot. > > $ ls -l .git/objects/pack/ > total 1675466 > -rw-r--r-- 1 www-data www-data 2964992 Sep 17 19:17 > pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.idx > -rw-r--r-- 1 www-data www-data 1711037142 Sep 17 19:17 > pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.pack > > If the pack files larger than 2.5g, how can I repack it on i686 ? *youch* A 1.5 GiB pack file? Your files apparently do not delta compress very well. I fear that you are going to bump up against address space limitations soon on 32 bit systems. Then you will bump up against the 4 GiB pack file size limit. Which means you will need to use several packs and avoid the '-a' flag when calling git-repack. Nico was suggesting using the default window size (rather than --window=64) as larger windows requires more memory during repack. But you may also need the mmap window code I'm working on. I better hurry up and get that into Junio's testing branches. :-) -- Shawn. ``` ## Junio C Hamano, 2006-09-18 02:47 Subject: Re: git-repack: Outof memory Message-ID: <7v64flyjpe.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7v64flyjpe.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <20060918002357.GA19727@spearce.org> ``` Shawn Pearce writes: > ... I fear that > you are going to bump up against address space limitations soon > on 32 bit systems. Then you will bump up against the 4 GiB pack > file size limit. Which means you will need to use several packs > and avoid the '-a' flag when calling git-repack. I presume that you are hinting that we would need to update git-repack so that it is still useful without --all. Which means "pack-objects --unpacked=active-pack" would need to be pushed out so that git-repack can be updated to do the "archived ones and repacking the active pack" we talked about earlier. I think pack-objects --unpacked=active-pack is ready, so I'll push it out. ``` ## Shawn Pearce, 2006-09-18 06:50 Subject: Re: git-repack: Outof memory Message-ID: <20060918065055.GC20660@spearce.org> URL: https://gitlist.dev/e/20060918065055.GC20660%40spearce.org In-Reply-To: <7v64flyjpe.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano wrote: > Shawn Pearce writes: > > > ... I fear that > > you are going to bump up against address space limitations soon > > on 32 bit systems. Then you will bump up against the 4 GiB pack > > file size limit. Which means you will need to use several packs > > and avoid the '-a' flag when calling git-repack. > > I presume that you are hinting that we would need to update > git-repack so that it is still useful without --all. Yes. :-) Apparently the Porcelain level is still open for debate. Someone just needs to post a reasonable implementation to get discussion (or lack there of) going. I'm too tired right now to try to roll something up right, even though it looks to be rather simple. As it is I'm having a difficult time navigating in vi to write this email and normally vi navigation is as easy as breathing for me... > I think pack-objects --unpacked=active-pack is ready, so I'll > push it out. Thanks; I noticed that in your "What's in git.git" announcement. -- Shawn. ```