{"thread":{"id":"60758","subject":"git fsck does not check the packed-refs file","startedAt":"2024-01-18T08:02:48Z","lastAt":"2024-01-20T01:00:56Z","messageCount":3,"participants":["R. Diez","Patrick Steinhardt","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"486959","messageId":"6cfee0e4-3285-4f18-91ff-d097da9de737@rd10.de","threadId":"60758","inReplyTo":null,"subject":"git fsck does not check the packed-refs file","fromName":"R. Diez","fromEmail":"rdiez-temp3@rd10.de","sentAt":"2024-01-18T08:02:30Z","receivedAt":"2024-01-18T08:02:48Z","isPatch":false,"sender":{"key":"rdiez-temp3@rd10.de","avatar":null},"body":"Hi all:\n\nI have been hit by an unfortunate system problem, and as a result, a few files in my Git repository got corrupted on my last git push. Some random blocks of bytes were overwritten with binary zeros, so I started getting weird unpacking errors etc.\n\nIt took a while to realise what the problem was. During my investigation, I ran \"git fsck\", which reported no problems, and then \"git push\" failed.\n\nOne of the very few corrupted files was packed-refs. This is a text file, so it was easy to compare it and see the corrupting binary zeros. But that made me wonder what \"git fsck\" checks.\n\nI am guessing that \"git fsck\" does not check file packed-refs at all. I mean, it does not even attempt to parse it, in order to check whether at least the format makes any sense. Only \"git push\" does it.\n\nWhat other parts of the repository does \"git fsck\" not check then?\n\nThe repository check is suspiciously fast. Is there a slow way to check that a repository is fine? I mean, something along the lines of checking whether every commit can be checked out without problems.\n\nBest regards,\n   rdiez\n"},{"id":"486968","messageId":"ZakIPEytlxHGCB9Y@tanuki","threadId":"60758","inReplyTo":"6cfee0e4-3285-4f18-91ff-d097da9de737@rd10.de","subject":"Re: git fsck does not check the packed-refs file","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-01-18T11:15:08Z","receivedAt":"2024-01-18T11:15:13Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jan 18, 2024 at 09:02:30AM +0100, R. Diez wrote:\n> Hi all:\n> \n> I have been hit by an unfortunate system problem, and as a result, a\n> few files in my Git repository got corrupted on my last git push. Some\n> random blocks of bytes were overwritten with binary zeros, so I\n> started getting weird unpacking errors etc.\n> \n> It took a while to realise what the problem was. During my\n> investigation, I ran \"git fsck\", which reported no problems, and then\n> \"git push\" failed.\n> \n> One of the very few corrupted files was packed-refs. This is a text\n> file, so it was easy to compare it and see the corrupting binary\n> zeros. But that made me wonder what \"git fsck\" checks.\n\nCan you maybe expand a bit on how you arrived at this bug? Was this a\nhard crash of the system that corrupted the repository or rather\nsomething like actual disk corruption?\n\nI'm mostly asking because I have been fixing some sources of refdb\ncorruption:\n\n  - bc22d845c4 (core.fsync: new option to harden references, 2022-03-11)\n    started to fsync loose refs to disk before renaming them into place,\n    released with Git v2.36.\n\n  - ce54672f9b (refs: fix corruption by not correctly syncing\n    packed-refs to disk, 2022-12-20) started to sync packed-refs to disk\n    before renaming them into place, released with Git v2.40 and\n    backported to Git v2.39.3.\n\nSo if:\n\n  - you use a journaling filesystem,\n\n  - you didn't disable `core.fsync`,\n\n  - you use Git v2.40 or newer,\n\nthen you should in theory not run into any refdb corruption anymore. At\nleast we didn't experience corruption anymore at GitLab.com, whereas\nbefore we encountered corruption every so often.\n\n> I am guessing that \"git fsck\" does not check file packed-refs at all.\n> I mean, it does not even attempt to parse it, in order to check\n> whether at least the format makes any sense. Only \"git push\" does it.\n\nIndeed it doesn't. While the issue is comparatively easy to spot by\nmanually inspecting the `packed-refs` file, I agree that it would be\ngreat if git-fsck(1) knew how to check the refdb for consistency. This\nproblem is only going to get worse once the upcoming reftable backend\nlands -- it is a binary format, and just opening it with a text editor\nto check whether it looks sane-ish stops being a viable option here.\n\nIn fact, I already planned to introduce such consistency checks for the\nrefdb soonish. Once the reftable backend is upstream I will focus more\non additional tooling to support it, and extending our consistency\nchecks is one of the first items on my todo list here.\n\n> What other parts of the repository does \"git fsck\" not check then?\n\nThere may be some metadata and cache-like data structures that we don't\ncheck, but the object database is checked by default.\n\n> The repository check is suspiciously fast. Is there a slow way to\n> check that a repository is fine? I mean, something along the lines of\n> checking whether every commit can be checked out without problems.\n\nOther than running `git fsck --full --strict`: not that I'm aware of.\nAnd `--full` isn't even needed because it's the default.\n\nPatrick\n"},{"id":"487122","messageId":"20240120010055.GC117170@coredump.intra.peff.net","threadId":"60758","inReplyTo":"ZakIPEytlxHGCB9Y@tanuki","subject":"Re: git fsck does not check the packed-refs file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-01-20T01:00:55Z","receivedAt":"2024-01-20T01:00:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 18, 2024 at 12:15:08PM +0100, Patrick Steinhardt wrote:\n\n> > I am guessing that \"git fsck\" does not check file packed-refs at all.\n> > I mean, it does not even attempt to parse it, in order to check\n> > whether at least the format makes any sense. Only \"git push\" does it.\n> \n> Indeed it doesn't. While the issue is comparatively easy to spot by\n> manually inspecting the `packed-refs` file, I agree that it would be\n> great if git-fsck(1) knew how to check the refdb for consistency. This\n> problem is only going to get worse once the upcoming reftable backend\n> lands -- it is a binary format, and just opening it with a text editor\n> to check whether it looks sane-ish stops being a viable option here.\n\nWe don't check the packed-refs file explicitly, but we do open and parse\nit to iterate over the refs it contains. E.g.:\n\n  $ git init\n  $ echo foo >.git/packed-refs\n  $ git fsck\n  Checking object directories: 100% (256/256), done.\n  fatal: unexpected line in .git/packed-refs: foo\n\nIt's quite possible that the reading code could be more careful. I'd\nhave to see the exact corruption that \"git fsck\" didn't complain about\nto say more.  If there's a page full of NUL bytes at the end of the\nfile, I wouldn't be surprised if the reading code gently ignores that,\nwhich obviously is not ideal.\n\nFundamentally we cannot catch all cases here; a simple truncation, for\nexample, might yield a valid file that is simply missing some entries.\nUnlike objects (which make promises about reachability and so on), there\nis no real \"consistency\" for the state of the refs. But probably warning\nif saw a bunch of garbage in the file is a good thing.\n\nI also agree that a specific refdb consistency check would be valuable.\nThere are some things that the regular reading code will not check, but\nwhich an fsck should (e.g., if the packed-refs file claims to have the\n\"sorted\" trait, we should confirm that).\n\n-Peff\n"}]}