Memory issue with fast-import, why track branches?
- From
Felipe Contreras <felipe.contreras@gmail.com>
- Date
- Dec 21, 2008, 05:54 UTC
- Message-ID
- <94a0d4530812202154l26dfe0dfm49397c63dbfdfdf9@mail.gmail.com>
Hi,
I tracked down an issue I have when importing a big repository. For some reason memory usage keeps increasing until there is no more memory.
Here is what valgrind shows: ==21034== 471,080,280 bytes in 114,517 blocks are still reachable in loss record 8 of 8 ==21034== at 0x4004BA2: calloc (vg_replace_malloc.c:397) ==21034== by 0x806A340: xcalloc (wrapper.c:75) ==21034== by 0x8063BC1: use_pack (sha1_file.c:808) ==21034== by 0x8063DA9: unpack_object_header (sha1_file.c:1443) ==21034== by 0x8064F4F: unpack_entry (sha1_file.c:1736) ==21034== by 0x8065393: cache_or_unpack_entry (sha1_file.c:1606) ==21034== by 0x8065464: read_packed_sha1 (sha1_file.c:2000) ==21034== by 0x80655E5: read_object (sha1_file.c:2090) ==21034== by 0x8065677: read_sha1_file (sha1_file.c:2106) ==21034== by 0x8056AE9: parse_object (object.c:190) ==21034== by 0x805E90A: write_ref_sha1 (refs.c:1214) ==21034== by 0x804CC4F: update_branch (fast-import.c:1558)
After looking at the code my guess is that I have a humongous amount of branches.
Actually they are not really branches, but refs. For each git commit there's an original mtn ref that I store in 'refs/mtn/sha1', but since I'm using 'commit refs/mtn/sha1' to store it, a branch is created for every commit.
I guess there are many ways to fix the issue, but for starters I wonder why is fast-import keeping track of all the branches? In my case I would like fast-import to work exactly the same if I specify branches or not (I'll update them later).
Cheers.
-- Felipe Contreras