{"thread":{"id":"16817","subject":"Memory issue with fast-import, why track branches?","startedAt":"2008-12-21T05:54:57Z","lastAt":"2008-12-22T02:36:11Z","messageCount":5,"participants":["Felipe Contreras","John Chapman","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"98477","messageId":"94a0d4530812202154l26dfe0dfm49397c63dbfdfdf9@mail.gmail.com","threadId":"16817","inReplyTo":null,"subject":"Memory issue with fast-import, why track branches?","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2008-12-21T05:54:57Z","receivedAt":"2008-12-21T05:54:57Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"Hi,\n\nI tracked down an issue I have when importing a big repository. For\nsome reason memory usage keeps increasing until there is no more\nmemory.\n\nHere is what valgrind shows:\n==21034== 471,080,280 bytes in 114,517 blocks are still reachable in\nloss record 8 of 8\n==21034== at 0x4004BA2: calloc (vg_replace_malloc.c:397)\n==21034== by 0x806A340: xcalloc (wrapper.c:75)\n==21034== by 0x8063BC1: use_pack (sha1_file.c:808)\n==21034== by 0x8063DA9: unpack_object_header (sha1_file.c:1443)\n==21034== by 0x8064F4F: unpack_entry (sha1_file.c:1736)\n==21034== by 0x8065393: cache_or_unpack_entry (sha1_file.c:1606)\n==21034== by 0x8065464: read_packed_sha1 (sha1_file.c:2000)\n==21034== by 0x80655E5: read_object (sha1_file.c:2090)\n==21034== by 0x8065677: read_sha1_file (sha1_file.c:2106)\n==21034== by 0x8056AE9: parse_object (object.c:190)\n==21034== by 0x805E90A: write_ref_sha1 (refs.c:1214)\n==21034== by 0x804CC4F: update_branch (fast-import.c:1558)\n\nAfter looking at the code my guess is that I have a humongous amount\nof branches.\n\nActually they are not really branches, but refs. For each git commit\nthere's an original mtn ref that I store in 'refs/mtn/sha1', but since\nI'm using 'commit refs/mtn/sha1' to store it, a branch is created for\nevery commit.\n\nI guess there are many ways to fix the issue, but for starters I\nwonder why is fast-import keeping track of all the branches? In my\ncase I would like fast-import to work exactly the same if I specify\nbranches or not (I'll update them later).\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"98479","messageId":"1229847042.798.5.camel@therock.nsw.bigpond.net.au","threadId":"16817","inReplyTo":"94a0d4530812202154l26dfe0dfm49397c63dbfdfdf9@mail.gmail.com","subject":"Re: Memory issue with fast-import, why track branches?","fromName":"John Chapman","fromEmail":"thestar@fussycoder.id.au","sentAt":"2008-12-21T08:10:42Z","receivedAt":"2008-12-21T08:10:42Z","isPatch":false,"sender":{"key":"thestar@fussycoder.id.au","avatar":null},"body":"My first response was along the lines of \"Why the heck are you storing\nsha1's like that!?\", until I realised that you're not storing actual git\nsha1's, but mtn's hashes, which does make sense.\n\nI'm doing something very similar with my perforce scripts, however I am\ndoing a bit more magic instead of making so many branches.\n\nInstead of making branches, I make a tag instead, for each and every\nchangeset. Every time I make a new git commit, if I need to do it from\na tag, I first read the tag and determine the sha1 I should use, and use\nthat instead.\n\nAlternatively, you could choose to manage your mapping yourself, and\nwrite them to a .git/mtg-git-map file.\n\nOn Sun, 2008-12-21 at 07:54 +0200, Felipe Contreras wrote:\n> Hi,\n> \n> I tracked down an issue I have when importing a big repository. For\n> some reason memory usage keeps increasing until there is no more\n> memory.\n> \n> Here is what valgrind shows:\n> ==21034== 471,080,280 bytes in 114,517 blocks are still reachable in\n> loss record 8 of 8\n> ==21034== at 0x4004BA2: calloc (vg_replace_malloc.c:397)\n> ==21034== by 0x806A340: xcalloc (wrapper.c:75)\n> ==21034== by 0x8063BC1: use_pack (sha1_file.c:808)\n> ==21034== by 0x8063DA9: unpack_object_header (sha1_file.c:1443)\n> ==21034== by 0x8064F4F: unpack_entry (sha1_file.c:1736)\n> ==21034== by 0x8065393: cache_or_unpack_entry (sha1_file.c:1606)\n> ==21034== by 0x8065464: read_packed_sha1 (sha1_file.c:2000)\n> ==21034== by 0x80655E5: read_object (sha1_file.c:2090)\n> ==21034== by 0x8065677: read_sha1_file (sha1_file.c:2106)\n> ==21034== by 0x8056AE9: parse_object (object.c:190)\n> ==21034== by 0x805E90A: write_ref_sha1 (refs.c:1214)\n> ==21034== by 0x804CC4F: update_branch (fast-import.c:1558)\n> \n> After looking at the code my guess is that I have a humongous amount\n> of branches.\n> \n> Actually they are not really branches, but refs. For each git commit\n> there's an original mtn ref that I store in 'refs/mtn/sha1', but since\n> I'm using 'commit refs/mtn/sha1' to store it, a branch is created for\n> every commit.\n> \n> I guess there are many ways to fix the issue, but for starters I\n> wonder why is fast-import keeping track of all the branches? In my\n> case I would like fast-import to work exactly the same if I specify\n> branches or not (I'll update them later).\n> \n> Cheers.\n> \n"},{"id":"98486","messageId":"94a0d4530812210323q2ae392d5o2381fd990be708e8@mail.gmail.com","threadId":"16817","inReplyTo":"1229847042.798.5.camel@therock.nsw.bigpond.net.au","subject":"Re: Memory issue with fast-import, why track branches?","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2008-12-21T11:23:05Z","receivedAt":"2008-12-21T11:23:05Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Sun, Dec 21, 2008 at 10:10 AM, John Chapman wrote:\n> My first response was along the lines of \"Why the heck are you storing\n> sha1's like that!?\", until I realised that you're not storing actual git\n> sha1's, but mtn's hashes, which does make sense.\n\nYes :)\n\n> I'm doing something very similar with my perforce scripts, however I am\n> doing a bit more magic instead of making so many branches.\n>\n> Instead of making branches, I make a tag instead, for each and every\n> changeset. Every time I make a new git commit, if I need to do it from\n> a tag, I first read the tag and determine the sha1 I should use, and use\n> that instead.\n\nWell, simple tags and branches are exactly the same thing: refs. tags\nare in 'refs/tags' and branches in 'refs/heads'; 'refs/mtn' are not\nreally branches.\n\n> Alternatively, you could choose to manage your mapping yourself, and\n> write them to a .git/mtg-git-map file.\n\nThe advantage of my approach is that the git tools handle all the mtn\nsha1's almost as good as git sha1's, I just need to prepend 'mtn/'.\n\nAlso, git name-rev finds the mtn revision of a git commit. It' all so\nconvenient.\n\nThe only problem is that fast-import seems to be doing something wrong\nwith those \"branches\".\n\n-- \nFelipe Contreras\n"},{"id":"98510","messageId":"20081221221702.GC17355@spearce.org","threadId":"16817","inReplyTo":"94a0d4530812202154l26dfe0dfm49397c63dbfdfdf9@mail.gmail.com","subject":"Re: Memory issue with fast-import, why track branches?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-12-21T22:17:02Z","receivedAt":"2008-12-21T22:17:02Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Felipe Contreras wrote:\n> I tracked down an issue I have when importing a big repository. For\n> some reason memory usage keeps increasing until there is no more\n> memory.\n> \n> After looking at the code my guess is that I have a humongous amount\n> of branches.\n> \n> Actually they are not really branches, but refs. For each git commit\n> there's an original mtn ref that I store in 'refs/mtn/sha1', but since\n> I'm using 'commit refs/mtn/sha1' to store it, a branch is created for\n> every commit.\n> \n> I guess there are many ways to fix the issue, but for starters I\n> wonder why is fast-import keeping track of all the branches? In my\n> case I would like fast-import to work exactly the same if I specify\n> branches or not (I'll update them later).\n\nBecause fast-import has to buffer them until the pack file is done.\nThe objects aren't available to the repository until after a\ncheckpoint is sent or until the stream ends. Either way until\nthen fast-import has to buffer the refs so they don't get exposed\nto other git processes reading that same repository, because they\nwould point to objects that the process cannot find.\n\nI guess it could release the brnach memory after it dumps the\nbranches in a checkpoint, but its memory allocators work under an\nassumption that strings (like branch and file names) will be reused\nheavily by the frontend and thus they are poooled inside of a string\npool. The branch objects are also pooled inside of a common alloc\npool, to ammortize the cost of malloc's block headers out over the\ndata used.\n\nIOW, fast-import was designed for ~5k branches, not ~1 million\nunique branches.\n\n-- \nShawn.\n"},{"id":"98530","messageId":"94a0d4530812211836w16933c7dv52ffa099ca18f731@mail.gmail.com","threadId":"16817","inReplyTo":"20081221221702.GC17355@spearce.org","subject":"Re: Memory issue with fast-import, why track branches?","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2008-12-22T02:36:11Z","receivedAt":"2008-12-22T02:36:11Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Mon, Dec 22, 2008 at 12:17 AM, Shawn O. Pearce wrote:\n> Felipe Contreras wrote:\n>> I tracked down an issue I have when importing a big repository. For\n>> some reason memory usage keeps increasing until there is no more\n>> memory.\n>>\n>> After looking at the code my guess is that I have a humongous amount\n>> of branches.\n>>\n>> Actually they are not really branches, but refs. For each git commit\n>> there's an original mtn ref that I store in 'refs/mtn/sha1', but since\n>> I'm using 'commit refs/mtn/sha1' to store it, a branch is created for\n>> every commit.\n>>\n>> I guess there are many ways to fix the issue, but for starters I\n>> wonder why is fast-import keeping track of all the branches? In my\n>> case I would like fast-import to work exactly the same if I specify\n>> branches or not (I'll update them later).\n>\n> Because fast-import has to buffer them until the pack file is done.\n> The objects aren't available to the repository until after a\n> checkpoint is sent or until the stream ends. Either way until\n> then fast-import has to buffer the refs so they don't get exposed\n> to other git processes reading that same repository, because they\n> would point to objects that the process cannot find.\n>\n> I guess it could release the brnach memory after it dumps the\n> branches in a checkpoint, but its memory allocators work under an\n> assumption that strings (like branch and file names) will be reused\n> heavily by the frontend and thus they are poooled inside of a string\n> pool. The branch objects are also pooled inside of a common alloc\n> pool, to ammortize the cost of malloc's block headers out over the\n> data used.\n>\n> IOW, fast-import was designed for ~5k branches, not ~1 million\n> unique branches.\n\nMy point is: why is it not designed for 0 branches? In many places in\nthe code there's the assumption that the tree = branch, but that's not\nalways the case. You can specify a 'from sha1' and then the branch\nbecomes irrelevant.\n\nIn fact in monotone some commits are not part of any branch, and many\nare part of multiple branches. Those cases can't be handled by\nfast-import right now. Not to mention random refs like 'ref/mtn/foo'\nwhich would come in handy for my script.\n\nNow my question is: would it be possible to get rid of the notion of\nbranches on fast-import and go for refs instead?\n\nOn the other hand if branch memory is freed after a checkpoint then\nthere's no limit to how many 'branches' can be handled.\n\n-- \nFelipe Contreras\n"}]}