{"thread":{"id":"5585","subject":"git-repack: Outof memory","startedAt":"2006-09-17T01:31:13Z","lastAt":"2006-09-18T06:50:55Z","messageCount":6,"participants":["Dongsheng Song","Shawn Pearce","Nicolas Pitre","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"27032","messageId":"450CA561.9030602@gmail.com","threadId":"5585","inReplyTo":null,"subject":"git-repack: Outof memory","fromName":"Dongsheng Song","fromEmail":"dongsheng.song@gmail.com","sentAt":"2006-09-17T01:31:13Z","receivedAt":"2006-09-17T01:31:13Z","isPatch":false,"sender":{"key":"dongsheng.song@gmail.com","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nHi all:\n\n\nI'm import from subversion. The problem appears to be git-repack phase using too many memory:\n\n$ git-repack -a -f -d --window=64 --depth=64\nGenerating pack...\nDone counting 123497 objects.\nDeltifying 123497 objects.\n 24% (29677/123497) done\n\n$ top\n\n PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND\n 3572 www-data 18 0 2591m 1.9g 528 R 13 94.8 81:48.98 git-pack-object\n\n\n$ free\n total used free shared buffers cached\nMem: 2076308 2029824 46484 0 2760 6800\n- -/+ buffers/cache: 2020264 56044\nSwap: 1028152 684032 344120\n\n\nHow to compute memory usage of git-repack ?\n\nThanks and regards,\n\nDongsheng\n\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.5 (MingW32)\n\niD8DBQFFDKVh90pbDJCgbHoRAlweAJ45DhTXI+bb+nb2Y+JlbIBoFusK8wCgk0U2\nXcUc9K/chYiUYq3ZLychzcU=\n=NzXf\n-----END PGP SIGNATURE-----\n"},{"id":"27039","messageId":"20060917022534.GB7512@spearce.org","threadId":"5585","inReplyTo":"450CA561.9030602@gmail.com","subject":"Re: git-repack: Outof memory","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-17T02:25:35Z","receivedAt":"2006-09-17T02:25:35Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Dongsheng Song wrote:\n> I'm import from subversion. The problem appears to be git-repack phase using too many memory:\n> \n> $ git-repack -a -f -d --window=64 --depth=64\n> Generating pack...\n> Done counting 123497 objects.\n> Deltifying 123497 objects.\n> 24% (29677/123497) done\n> \n> $ top\n> \n> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND\n> 3572 www-data 18 0 2591m 1.9g 528 R 13 94.8 81:48.98 git-pack-object\n\n*ouch* You are probably running out of address space if you are on\na 32 bit architecture. A 2.5 GiB virtual address space is pretty\nclose to the maximum allowed on most OSes.\n\nThe code that I'm sitting on but haven't yet completed rebasing\nonto current Git would probably help here.\n\nDo you have any existing .pack files in .git/objects/pack? How big\nare they and their corresponding .idx files?\n\ngit-repack will need to mmap every .pack and .idx in\n.git/objects/pack, plus it needs working memory for each object\n(123,497 of 'em) but as I recall its pretty frugal on its per-object\nallocation. It can easily work with as many as 2 million objects on\na 32 bit system, assuming the .pack and .idx files aren't too large.\n \n-- \nShawn.\n"},{"id":"27087","messageId":"Pine.LNX.4.64.0609171707430.2627@xanadu.home","threadId":"5585","inReplyTo":"450CA561.9030602@gmail.com","subject":"Re: git-repack: Outof memory","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-09-17T21:09:03Z","receivedAt":"2006-09-17T21:09:03Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 17 Sep 2006, Dongsheng Song wrote:\n\n> -----BEGIN PGP SIGNED MESSAGE-----\n> Hash: SHA1\n> \n> Hi all:\n> \n> \n> I'm import from subversion. The problem appears to be git-repack phase using too many memory:\n> \n> $ git-repack -a -f -d --window=64 --depth=64\n\nTry with that --window=64 argument removed.\n\n\nNicolas\n"},{"id":"27097","messageId":"20060918002357.GA19727@spearce.org","threadId":"5585","inReplyTo":"4b3406f0609170543p68d96b9x9ba0c5a74d9e89e8@mail.gmail.com","subject":"Re: git-repack: Outof memory","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-18T00:23:57Z","receivedAt":"2006-09-18T00:23:57Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Dongsheng Song wrote:\n> finished, thanks a lot.\n> \n> $ ls -l .git/objects/pack/\n> total 1675466\n> -rw-r--r-- 1 www-data www-data 2964992 Sep 17 19:17\n> pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.idx\n> -rw-r--r-- 1 www-data www-data 1711037142 Sep 17 19:17\n> pack-b09c57c25e4459f6365b5d27139abfd93bf1c86f.pack\n> \n> If the pack files larger than 2.5g, how can I repack it on i686 ?\n\n*youch* A 1.5 GiB pack file?\n\nYour files apparently do not delta compress very well. I fear that\nyou are going to bump up against address space limitations soon\non 32 bit systems. Then you will bump up against the 4 GiB pack\nfile size limit. Which means you will need to use several packs\nand avoid the '-a' flag when calling git-repack.\n\nNico was suggesting using the default window size (rather than\n--window=64) as larger windows requires more memory during repack.\nBut you may also need the mmap window code I'm working on. I better\nhurry up and get that into Junio's testing branches. :-)\n\n-- \nShawn.\n"},{"id":"27105","messageId":"7v64flyjpe.fsf@assigned-by-dhcp.cox.net","threadId":"5585","inReplyTo":"20060918002357.GA19727@spearce.org","subject":"Re: git-repack: Outof memory","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-18T02:47:25Z","receivedAt":"2006-09-18T02:47:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shawn Pearce writes:\n\n> ... I fear that\n> you are going to bump up against address space limitations soon\n> on 32 bit systems. Then you will bump up against the 4 GiB pack\n> file size limit. Which means you will need to use several packs\n> and avoid the '-a' flag when calling git-repack.\n\nI presume that you are hinting that we would need to update\ngit-repack so that it is still useful without --all.\n\nWhich means \"pack-objects --unpacked=active-pack\" would need to\nbe pushed out so that git-repack can be updated to do the\n\"archived ones and repacking the active pack\" we talked about\nearlier.\n\nI think pack-objects --unpacked=active-pack is ready, so I'll\npush it out.\n"},{"id":"27120","messageId":"20060918065055.GC20660@spearce.org","threadId":"5585","inReplyTo":"7v64flyjpe.fsf@assigned-by-dhcp.cox.net","subject":"Re: git-repack: Outof memory","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-18T06:50:55Z","receivedAt":"2006-09-18T06:50:55Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> Shawn Pearce writes:\n> \n> > ... I fear that\n> > you are going to bump up against address space limitations soon\n> > on 32 bit systems. Then you will bump up against the 4 GiB pack\n> > file size limit. Which means you will need to use several packs\n> > and avoid the '-a' flag when calling git-repack.\n> \n> I presume that you are hinting that we would need to update\n> git-repack so that it is still useful without --all.\n\nYes. :-)\n\nApparently the Porcelain level is still open for debate. Someone\njust needs to post a reasonable implementation to get discussion\n(or lack there of) going. I'm too tired right now to try to roll\nsomething up right, even though it looks to be rather simple.\n\nAs it is I'm having a difficult time navigating in vi to write this\nemail and normally vi navigation is as easy as breathing for me...\n\n> I think pack-objects --unpacked=active-pack is ready, so I'll\n> push it out.\n\nThanks; I noticed that in your \"What's in git.git\" announcement.\n\n-- \nShawn.\n"}]}