{"thread":{"id":"35654","subject":"Diagnosing stray/stale .keep files -- explore what is in a pack?","startedAt":"2014-01-14T14:54:56Z","lastAt":"2014-01-21T05:19:56Z","messageCount":10,"participants":["Martin Langhoff","Martin Fick","Jeff King","Junio C Hamano","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"233090","messageId":"CACPiFCLa3X-Xt5GwrHHA-PFj-Bi9_sW+=y2xidZ7tDbFfM26rA@mail.gmail.com","threadId":"35654","inReplyTo":null,"subject":"Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2014-01-14T14:54:56Z","receivedAt":"2014-01-14T14:54:56Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"hi folks,\n\nI have a git server which gets pushes of data (not code) from a couple\nhundred VMs every hour. Every round of pushes leaves two stray .keep\nfiles, so I am guessing two clients are having problems completing the\npush. The contents being pushed are reports of a puppet run.\n\nIs there a handy way to list the blobs in a pack, so I can feed them\nto git-cat-file and see what's in there? I'm sure that'll help me\nnarrow down on the issue.\n\nAre there other ways to try diagnose this?\n\nDoes the server-side record anything if a push fails? There are a\nnumber of problems I am familiar with, and they always require\ncollaboration from the \"client\" side to spot and diagnose\n\n - a ref is not up to date and the server rejects non-ft\n - perms issues over objects or refs\n - ENOSPC\n - ... catchable signals (ETERM?)\n\nAFAIK, I think it doesn't, and maybe it should, even if it's as simple\nas trying to spawn a pipe to \"\"/usr/bin/logger -t git-server\" and\nattach it to stderr...\n\nThis has veered a bit off topic, but I think it's important for large\ngit server installations.\n\ncheers,\n\n\nm\n-- \n martin.langhoff@gmail.com\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n ~ http://docs.moodle.org/en/User:Martin_Langhoff\n"},{"id":"233093","messageId":"CACPiFCJVx0dkkPQ=LosbAAKq7CvK6_yQL5QDHMYr5oJAS6wb6Q@mail.gmail.com","threadId":"35654","inReplyTo":"CACPiFCLa3X-Xt5GwrHHA-PFj-Bi9_sW+=y2xidZ7tDbFfM26rA@mail.gmail.com","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2014-01-14T17:10:31Z","receivedAt":"2014-01-14T17:10:31Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Tue, Jan 14, 2014 at 9:54 AM, Martin Langhoff\n wrote:\n> Is there a handy way to list the blobs in a pack, so I can feed them\n> to git-cat-file and see what's in there? I'm sure that'll help me\n> narrow down on the issue.\n\ngit show-index <\n/var/lib/ppg/reports.git/objects/pack/pack-22748bcca7f50a3a49aa4aed61444bf9c4ced685.idx\n|\ncut -d\\ -f2 | xargs -iHASH git --git-dir /var/lib/ppg/reports.git/\nunpack-file HASH\n\nAfter a bit of looking at the output, clearly I have two clients, out\nof the many that connect here, that have the problem. I will be\nlooking into those clients to see what's the problem.\n\nIn my use case, clients push to their own head. Looking at refs/heads\nshows that there are stale .lock files there. Hmmm.\n\nThis is on git 1.7.1 (RHEL and CentOS clients).\n\ncheers,\n\n\nm\n-- \n martin.langhoff@gmail.com\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n ~ http://docs.moodle.org/en/User:Martin_Langhoff\n"},{"id":"233099","messageId":"201401141236.44393.mfick@codeaurora.org","threadId":"35654","inReplyTo":"CACPiFCJVx0dkkPQ=LosbAAKq7CvK6_yQL5QDHMYr5oJAS6wb6Q@mail.gmail.com","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Martin Fick","fromEmail":"mfick@codeaurora.org","sentAt":"2014-01-14T19:36:44Z","receivedAt":"2014-01-14T19:36:44Z","isPatch":false,"sender":{"key":"mfick@codeaurora.org","avatar":null},"body":"Perhaps the receiving process is dying hard and leaving \nstuff behind? Out-of-memory, out of disk space?\n\n-Martin\n\nOn Tuesday, January 14, 2014 10:10:31 am Martin Langhoff \nwrote:\n> On Tue, Jan 14, 2014 at 9:54 AM, Martin Langhoff\n> \n> wrote:\n> > Is there a handy way to list the blobs in a pack, so I\n> > can feed them to git-cat-file and see what's in there?\n> > I'm sure that'll help me narrow down on the issue.\n> \n> git show-index <\n> /var/lib/ppg/reports.git/objects/pack/pack-22748bcca7f50a\n> 3a49aa4aed61444bf9c4ced685.idx\n> \n> cut -d\\ -f2 | xargs -iHASH git --git-dir \n> /var/lib/ppg/reports.git/ unpack-file HASH\n> \n> After a bit of looking at the output, clearly I have two\n> clients, out of the many that connect here, that have\n> the problem. I will be looking into those clients to see\n> what's the problem.\n> \n> In my use case, clients push to their own head. Looking\n> at refs/heads shows that there are stale .lock files\n> there. Hmmm.\n> \n> This is on git 1.7.1 (RHEL and CentOS clients).\n> \n> cheers,\n> \n> \n> m\n\n-- \nThe Qualcomm Innovation Center, Inc. is a member of Code \nAurora Forum, hosted by The Linux Foundation\n \n"},{"id":"233101","messageId":"CACPiFCLxiCOqv=wLeq9LxisWn5T62hk8xDYwXmeFRNT05HY0iQ@mail.gmail.com","threadId":"35654","inReplyTo":"201401141236.44393.mfick@codeaurora.org","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2014-01-14T19:42:09Z","receivedAt":"2014-01-14T19:42:09Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":" On Tue, Jan 14, 2014 at 2:36 PM, Martin Fick wrote:\n> Perhaps the receiving process is dying hard and leaving\n> stuff behind? Out-of-memory, out of disk space?\n\nYes, that's my guess as well. This server had gc misconfigured, so it\nhit ENOSPC a few weeks ago.\n\nIt is likely that the .lock files were left behind back then, and\nsince then the clients pushing to these refs were transferring their\nwhole history and still failing to update the ref, leading to rapid\nrepo growth.\n\nSo my situation is diagnosed and solved; I am still unhappy that it\ntook so much work and expertise; mainly because git isn't logging\nanywhere. See my \"Error logging for git over ssh?\" message...\n\nthanks,\n\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n ~ http://docs.moodle.org/en/User:Martin_Langhoff\n"},{"id":"233136","messageId":"20140115091220.GB14335@sigill.intra.peff.net","threadId":"35654","inReplyTo":"CACPiFCLxiCOqv=wLeq9LxisWn5T62hk8xDYwXmeFRNT05HY0iQ@mail.gmail.com","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-01-15T09:12:20Z","receivedAt":"2014-01-15T09:12:20Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 14, 2014 at 02:42:09PM -0500, Martin Langhoff wrote:\n\n> On Tue, Jan 14, 2014 at 2:36 PM, Martin Fick wrote:\n> > Perhaps the receiving process is dying hard and leaving\n> > stuff behind? Out-of-memory, out of disk space?\n> \n> Yes, that's my guess as well. This server had gc misconfigured, so it\n> hit ENOSPC a few weeks ago.\n> \n> It is likely that the .lock files were left behind back then, and\n> since then the clients pushing to these refs were transferring their\n> whole history and still failing to update the ref, leading to rapid\n> repo growth.\n\nWe see these occasionally at GitHub, too. I haven't yet figured out a\ndefinite cause, though whatever it is, it's relatively rare.\n\nI think the \".keep\" files and the \".lock\" files are in two separate\nboats, though.\n\npack-objects creates the .keep files as a \"lock\" between the time it\nmoves them into place and when receive-pack updates the refs (so that a\nsimultaneous prune does not think they should be removed). Receive-pack\nthen updates the refs and removes the \".keep\" file. However, in the\ninterim code, we are just updating the refs, and are careful to return\nany errors rather than calling die() (so if ENOSPC prevented ref write,\nthat would not cause this). So for us to leave a .keep there, it is\nprobably one of:\n\n 1. A few generic library functions, like xmalloc, can cause us to die.\n This should be very rare, though.\n\n 2. We tried to unlink the keep-file, but couldn't (could ENOSPC\n prevent a deletion? I suspect it depends on the filesystem).\n\n 3. We were killed by signal (or system crash).\n\nFetch-pack also will create .keep files, and it is much less careful\nduring the time the file exists. However, busy servers tend to be\nreceiving pushes, not initiating fetches.\n\nActual \".lock\" files are added to a signal/atexit handle that cleans\nthem up automatically on program exit. So those really should be caused\nby system crash (or \"kill -9\"), and that has generally been our\nexperience at GitHub. But again, if ENOSPC could prevent deletion on\nyour filesystem, it could be related. But there is not much git can do\nto clean up if unlink() fails us.\n\n-Peff\n"},{"id":"233160","messageId":"CACPiFCKeOYHUb22d_Ea0PcbU-uAn=fVAn0QP1qbLAiNh1KEoqQ@mail.gmail.com","threadId":"35654","inReplyTo":"20140115091220.GB14335@sigill.intra.peff.net","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2014-01-15T13:42:26Z","receivedAt":"2014-01-15T13:42:26Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Wed, Jan 15, 2014 at 4:12 AM, Jeff King wrote:\n> We see these occasionally at GitHub, too. I haven't yet figured out a\n> definite cause, though whatever it is, it's relatively rare.\n\nDo you have a cleanup script to safely get rid of stale .keep and\n.lock files? I wonder what other stale bits merit a cleanup...\n\nWe could draft a 'git-repo-clean' that works akin to git clean (i.e.:\nonly reports by default), or add it to gc.\n\ncheers,\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n ~ http://docs.moodle.org/en/User:Martin_Langhoff\n"},{"id":"233164","messageId":"xmqqr4892l0u.fsf@gitster.dls.corp.google.com","threadId":"35654","inReplyTo":"CACPiFCKeOYHUb22d_Ea0PcbU-uAn=fVAn0QP1qbLAiNh1KEoqQ@mail.gmail.com","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-01-15T17:49:05Z","receivedAt":"2014-01-15T17:49:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Langhoff writes:\n\n> On Wed, Jan 15, 2014 at 4:12 AM, Jeff King wrote:\n>> We see these occasionally at GitHub, too. I haven't yet figured out a\n>> definite cause, though whatever it is, it's relatively rare.\n>\n> Do you have a cleanup script to safely get rid of stale .keep and\n> .lock files? I wonder what other stale bits merit a cleanup...\n\nAs long as we can reliably determine that it is safe to do so\nwithout risking races, automatically cleaning .lock files is a good\nthing to do.\n\nCleaning .keep files needs the same care and a bit more, though.\nYou of course have to be sure that no other concurrent process is in\nthe middle of doing something, but you also need to be sure that the\n\".keep\" file is not a marker created by the end user to say \"keep\nthis pack, do not subject its contents to repacking\" after a careful\nrepacking of the stable part of the history.\n"},{"id":"233181","messageId":"CACPiFC+koU1Fan+tbE2YgOstWGsDtDihpK-7CMOct7XAEpwJ2A@mail.gmail.com","threadId":"35654","inReplyTo":"xmqqr4892l0u.fsf@gitster.dls.corp.google.com","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2014-01-15T23:50:33Z","receivedAt":"2014-01-15T23:50:33Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Wed, Jan 15, 2014 at 12:49 PM, Junio C Hamano wrote:\n> As long as we can reliably determine that it is safe to do so\n> without risking races, automatically cleaning .lock files is a good\n> thing to do.\n\nIf the .lock file is a day old, it seems to me that it should be safe\nto call it stale.\n\nCan anyone \"take the lock\" if there is already a lock file?\n\n> Cleaning .keep files needs the same care and a bit more, though.\n> You of course have to be sure that no other concurrent process is in\n> the middle of doing something, but you also need to be sure that the\n> \".keep\" file is not a marker created by the end user to say \"keep\n> this pack, do not subject its contents to repacking\" after a careful\n> repacking of the stable part of the history.\n\nFor the keep files, I already drafted a script that looks inside the\nkeep file, if it reads 'receive-pack [pid] [host]' it checks whether\nthe hostname matches, and if so whether the pid matches a running\nprocess.\n\nOnly if the host matches and the pid is dead we call it stale.\n\nSeems fairly conservative to me. Are there scenarios where we think\nthis can misfire?\n\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n ~ http://docs.moodle.org/en/User:Martin_Langhoff\n"},{"id":"233188","messageId":"CACsJy8Bx4n7DSfA9JFZd_EcfN3baMtVtHHPVVKKq97jx=byigw@mail.gmail.com","threadId":"35654","inReplyTo":"CACPiFC+koU1Fan+tbE2YgOstWGsDtDihpK-7CMOct7XAEpwJ2A@mail.gmail.com","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-16T01:14:56Z","receivedAt":"2014-01-16T01:14:56Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 16, 2014 at 6:50 AM, Martin Langhoff\n wrote:\n> On Wed, Jan 15, 2014 at 12:49 PM, Junio C Hamano wrote:\n>> As long as we can reliably determine that it is safe to do so\n>> without risking races, automatically cleaning .lock files is a good\n>> thing to do.\n>\n> If the .lock file is a day old, it seems to me that it should be safe\n> to call it stale.\n\nPerhaps report those stale locks (and stale .keep files as well if you\ncan detct them) as garbage in count-objects too.\n-- \nDuy\n"},{"id":"233429","messageId":"20140121051956.GA5878@sigill.intra.peff.net","threadId":"35654","inReplyTo":"CACPiFC+koU1Fan+tbE2YgOstWGsDtDihpK-7CMOct7XAEpwJ2A@mail.gmail.com","subject":"Re: Diagnosing stray/stale .keep files -- explore what is in a pack?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-01-21T05:19:56Z","receivedAt":"2014-01-21T05:19:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 15, 2014 at 06:50:33PM -0500, Martin Langhoff wrote:\n\n> On Wed, Jan 15, 2014 at 12:49 PM, Junio C Hamano wrote:\n> > As long as we can reliably determine that it is safe to do so\n> > without risking races, automatically cleaning .lock files is a good\n> > thing to do.\n> \n> If the .lock file is a day old, it seems to me that it should be safe\n> to call it stale.\n\nProbably. The way our \"lease\" system works, nobody should be\nholding a ref lock for more than a few milliseconds.\n\nThat being said, we do lock other things, like the index. Generally I\nthink the index lock should be quick, too. And similar for config file\nrewrites, and shallow files. And rerere files, it looks like. My, \"git\ngrep commit_lock_file\" turns up a lot of hits. :)\n\nSo I think all of the existing uses are fine, and I suppose that most\nnew cases should be fine, too, because git processes tend not to last a\nlong time.\n\nYou asked earlier if I had a script for cleaning locks. No code worth\nsharing, but I'll give an outline of what we do at GitHub. We basically\ndo:\n\n find -name *.lock -mmin +60 | xargs rm\n\nI.e., we give only an hour. For keep files, we give a day (since things\nlike hooks may run for a while under the lock, though a day is probably\nexcessive). And we check that it begins with \"^receive-pack\".\n\nAs far as I know, neither of these has ever caused any problems. Of\ncourse, any problems might not be immediately obvious.\n\n> Can anyone \"take the lock\" if there is already a lock file?\n\nGit never takes an existing lock. It expects you to clean it up\nyourself.\n\n> For the keep files, I already drafted a script that looks inside the\n> keep file, if it reads 'receive-pack [pid] [host]' it checks whether\n> the hostname matches, and if so whether the pid matches a running\n> process.\n> \n> Only if the host matches and the pid is dead we call it stale.\n\nThat sounds reasonable.\n\n> Seems fairly conservative to me. Are there scenarios where we think\n> this can misfire?\n\nI cannot think of any.\n\n-Peff\n"}]}