{"thread":{"id":"32438","subject":"Missing Refs after Garbage Collection","startedAt":"2012-12-22T01:41:43Z","lastAt":"2013-01-02T22:43:07Z","messageCount":4,"participants":["Earl Gresh","Dmitry Potapov","Jeff King","Martin Fick"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"205384","messageId":"C0A16EC8-D05A-41D0-BF2A-34BF3B1B839E@codeaurora.org","threadId":"32438","inReplyTo":null,"subject":"Missing Refs after Garbage Collection","fromName":"Earl Gresh","fromEmail":"egresh@codeaurora.org","sentAt":"2012-12-22T01:41:43Z","receivedAt":"2012-12-22T01:41:43Z","isPatch":false,"sender":{"key":"egresh@codeaurora.org","avatar":null},"body":"Hi-\n\nI have observed that after running GC, one particular git repository ended up with some missing refs in the refs/changes/* namespace the Gerrit uses for storing patch sets. The refs were valid and should not have been pruned. Concerned about loosing data, GC is still enabled but ref packing is turned off. Now the number of refs has grown to the point that it's causing performance problems when cloning the project.\n\nIs anyone familiar with git gc deleting valid references? I'm running git version 1.7.8. Have there been any patches in later git releases that might address this issue ( if it is a git problem )?\n\nThanks"},{"id":"205434","messageId":"CAHkcotjJ8nyPVEbDFcGVYqirY4HBNBPN-PNCMriLyeU8iBEKXg@mail.gmail.com","threadId":"32438","inReplyTo":"C0A16EC8-D05A-41D0-BF2A-34BF3B1B839E@codeaurora.org","subject":"Re: Missing Refs after Garbage Collection","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2012-12-22T22:26:53Z","receivedAt":"2012-12-22T22:26:53Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"Hi,\n\nOn Sat, Dec 22, 2012 at 5:41 AM, Earl Gresh wrote:\n>\n> Is anyone familiar with git gc deleting valid references? I'm running\n> git version 1.7.8. Have there been any patches in later git releases\n> that might address this issue ( if it is a git problem )?\n\nI have not seen any relevant changes in git. I have looked at the code,\nand what git-gc is running \"git pack-refs --all --prune\", which is very\ncareful in packing and fsyncing the new file with all packed references\nbefore deleting anything. Only those references that were packed can be\ndeleted. Also, it does not matter whether a reference is valid or not,\nor whether it is stored in refs/changes or in some other place, like\nrefs/heads. So if references were really lost as you described, I think\nother people would notice that by now.\n\nThe only plausible explanation that comes to my mind now is that file\ncreation using O_EXCL is not atomic on your system, then the lock did\nnot work and one process could overwrite packed references created by\nanother.\n\n\nDmitry\n"},{"id":"205436","messageId":"20121223010427.GA2878@sigill.intra.peff.net","threadId":"32438","inReplyTo":"C0A16EC8-D05A-41D0-BF2A-34BF3B1B839E@codeaurora.org","subject":"Re: Missing Refs after Garbage Collection","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-12-23T01:04:27Z","receivedAt":"2012-12-23T01:04:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 21, 2012 at 05:41:43PM -0800, Earl Gresh wrote:\n\n> I have observed that after running GC, one particular git repository\n> ended up with some missing refs in the refs/changes/* namespace the\n> Gerrit uses for storing patch sets. The refs were valid and should not\n> have been pruned. Concerned about loosing data, GC is still enabled\n> but ref packing is turned off. Now the number of refs has grown to the\n> point that it's causing performance problems when cloning the project.\n> \n> Is anyone familiar with git gc deleting valid references? I'm running\n> git version 1.7.8. Have there been any patches in later git releases\n> that might address this issue ( if it is a git problem )?\n\nI have never seen deletion, but I did recently find a race condition\nwith ref packing that caused rewinds, where:\n\n 1. Two processes simultaneously repack the refs.\n\n 2. At least one process is using an \"old\" version of the pack-refs\n file. That is, it cached the packed refs list earlier in the\n process and is now rewriting it based on that cached notion.\n\n 3. The first process takes the lock, packs refs, drops the\n lock, and then deletes the loose versions. The simultaneous packer\n then takes the lock, overwrites the packed-refs file with a stale\n copy from its memory, and then releases the lock. We're left with\n the stale copy in pack-refs, and deleted loose refs.\n\nIn my case, it looked like a rewind, because the stale, memory-cached\nrefs had the old version. But if you have a ref which was not previously\npacked, it would appear to have been deleted.\n\nThe tricky thing about triggering this race is that step (2) needs a\nprocess which has previously read and cached the packed-refs, and then\ndecided to pack the refs. The \"git pack-refs\" command does not do this,\nbecause it starts, packs the ref, and exists. But processes which delete\na ref need to rewrite the packed-refs file (omitting the deleted ref),\nand depending on the process, may have previously read and cached the\npacked refs file. The obvious candidate is \"receive-pack\".\n\nSo this may be your culprit if:\n\n 1. This is a repo people are pushing into via C git.\n\n 2. You simultaneously run \"git pack-refs\" (or \"git gc\") while people\n may be pushing.\n\nYou mentioned Gerrit, so I wonder if people are actually pushing via C\ngit (I thought it used JGit entirely). Or perhaps JGit has the same bug.\nMy fix (which is not yet released in any git version) is here:\n\n http://article.gmane.org/gmane.comp.version-control.git/211956\n\n-Peff\n"},{"id":"205891","messageId":"201301021543.08029.mfick@codeaurora.org","threadId":"32438","inReplyTo":"C0A16EC8-D05A-41D0-BF2A-34BF3B1B839E@codeaurora.org","subject":"Re: Missing Refs after Garbage Collection","fromName":"Martin Fick","fromEmail":"mfick@codeaurora.org","sentAt":"2013-01-02T22:43:07Z","receivedAt":"2013-01-02T22:43:07Z","isPatch":false,"sender":{"key":"mfick@codeaurora.org","avatar":null},"body":"On Friday, December 21, 2012 06:41:43 pm Earl Gresh wrote:\n> Hi-\n> \n> I have observed that after running GC, one particular git\n> repository ended up with some missing refs in the\n> refs/changes/* namespace the Gerrit uses for storing\n> patch sets. The refs were valid and should not have been\n> pruned. Concerned about loosing data, GC is still enabled\n> but ref packing is turned off. Now the number of refs has\n> grown to the point that it's causing performance problems\n> when cloning the project.\n> \n> Is anyone familiar with git gc deleting valid references?\n> I'm running git version 1.7.8. Have there been any\n> patches in later git releases that might address this\n> issue ( if it is a git problem )?\n\n\nWhen Earl was testing ref-packing a few months ago, that the \nrefs in question where reported invalid by git show-ref:\n\n git show-ref 2>&1 |grep refs/changes/45/129345/1\n error: refs/changes/45/129345/1 does not point to a valid \nobject!\n\nBut we could trace the refs manually to git show-object just \nfine. But oddly enough, when using git show-ref with the -v, \nthe error above would not be spit out.\n\nSo, my guess is that something during the repack was \nfollowing the same code path that git show-ref (without the -\nv) was following and determining that the ref was invalid and \ntherefor it was not able to add it to the new packfile, but \nyet perhaps it was still being added to the prune-list and \nthus getting pruned? Is this possible somehow? \n\nLooking at handle_one_ref() I can't see how. The fprintf() \nhappens before the ref is added to the prune list and is \nunconditional. I am grasping here, but what if the sha1 \npassed into handle_one_ref() somehow gets set incorrectly to \n000...? Would it then basically get written to the packed-\nref file as 000... (deleted), but then still get added to the \nprune list? You might say \"but then it wouldn't get pruned \nsince the loose ref doesn't match 000..., but if the logic \nwhich checks this matching makes the same error reading the \nsha1 and thinks it is 000... it might then get pruned?\n\n\n-Martin\n"}]}