{"thread":{"id":"54498","subject":"Re: safer git?","startedAt":"2020-10-24T21:39:37Z","lastAt":"2020-10-25T15:17:55Z","messageCount":6,"participants":["hv@crypt.org","Randall S. Becker","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"408299","messageId":"202010242021.09OKLPP13264@crypt.org","threadId":"54498","inReplyTo":"202010242019.09OKJTP13180@crypt.org","subject":"Re: safer git?","fromName":"","fromEmail":"hv@crypt.org","sentAt":"2020-10-24T20:21:25Z","receivedAt":"2020-10-24T21:39:37Z","isPatch":false,"sender":{"key":"hv@crypt.org","avatar":null},"body":"Sorry, I should have added: I'm using Ubuntu 18.04.5 (bionic) with\ngit version 2.17.1.\n\nHugo\n"},{"id":"408300","messageId":"202010242019.09OKJTP13180@crypt.org","threadId":"54498","inReplyTo":null,"subject":"safer git?","fromName":"","fromEmail":"hv@crypt.org","sentAt":"2020-10-24T20:19:29Z","receivedAt":"2020-10-24T21:39:42Z","isPatch":false,"sender":{"key":"hv@crypt.org","avatar":null},"body":"Hi, due to a hardware or firmware issue, my machine is currently crashing\nrather a lot. If that happens around the time I'm performing any git\nactivity, it seems quite likely to cause substantial corruption of the\ngit state.\n\nI've responded both by running sync(1) every 60 seconds and by pushing\nwork to github more frequently, but the latter isn't appropriate for\neverything; twice recently I've lost a fair bit of work as a result.\n\nQ: Is there a mode in which I can run git that would make it a bit more\nrobust against crashes, at the cost of being a bit slower?\n\n\nThe primary symptom is that files modified shortly before a crash show\nup existing but zero-length after the crash. For source files I mostly\nknow what to do in that situation, but `git fsck` shows a lot of files\nunder '.git/objects' that are empty, which seems to make things hard\nto recover:\n\n% git fsck\nerror: object file .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty\nerror: unable to mmap .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file or directory\nerror: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533\n[... a dozen similar entries ...]\nerror: object file .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty\nerror: unable to mmap .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or directory\nerror: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5\nChecking object directories: 100% (256/256), done.\nChecking objects: 100% (1577/1577), done.\nerror: refs/stash: invalid sha1 pointer 0000000000000000000000000000000000000000\nerror: bad ref for .git/logs/refs/stash\ndangling commit 1c0ea4e6159952501957012d2b9db7d68b52d107\n% \n\nLast time I checked out the previous state from github in a new\ndirectory and was able to find and copy over most of my work before\ncontinuing. On this occasion I did a `git stash save` shortly before\nthe crash, and I'm not sure how to get that back. I see René Scharfe's\nsuggestion of:\n  git fsck --unreachable |\n  grep commit | cut -d\\  -f3 |\n  xargs git log --merges --no-walk --grep=WIP\nfrom a recent message, but that is only showing me an older stash item.\n\nThanks in advance for any suggestions,\n\nHugo van der Sanden\n"},{"id":"408301","messageId":"016001d6aa52$b1cbc510$15634f30$@nexbridge.com","threadId":"54498","inReplyTo":"202010242019.09OKJTP13180@crypt.org","subject":"RE: safer git?","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2020-10-24T22:11:54Z","receivedAt":"2020-10-24T22:12:10Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On October 24, 2020 4:19 PM, hv@crypt.org wrote:\n> Hi, due to a hardware or firmware issue, my machine is currently crashing\n> rather a lot. If that happens around the time I'm performing any git\nactivity,\n> it seems quite likely to cause substantial corruption of the git state.\n> \n> I've responded both by running sync(1) every 60 seconds and by pushing\nwork\n> to github more frequently, but the latter isn't appropriate for\neverything;\n> twice recently I've lost a fair bit of work as a result.\n> \n> Q: Is there a mode in which I can run git that would make it a bit more\nrobust\n> against crashes, at the cost of being a bit slower?\n> \n> \n> The primary symptom is that files modified shortly before a crash show up\n> existing but zero-length after the crash. For source files I mostly know\nwhat\n> to do in that situation, but `git fsck` shows a lot of files under\n'.git/objects'\n> that are empty, which seems to make things hard to recover:\n> \n> % git fsck\n> error: object file\n> .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty\n> error: unable to mmap\n> .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file\n> or directory\n> error: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or\n> missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533\n> [... a dozen similar entries ...]\n> error: object file\n> .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty\n> error: unable to mmap\n> .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or\n> directory\n> error: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or\n> missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5\n> Checking object directories: 100% (256/256), done.\n> Checking objects: 100% (1577/1577), done.\n> error: refs/stash: invalid sha1 pointer\n> 0000000000000000000000000000000000000000\n> error: bad ref for .git/logs/refs/stash\n> dangling commit 1c0ea4e6159952501957012d2b9db7d68b52d107\n> %\n> \n> Last time I checked out the previous state from github in a new directory\nand\n> was able to find and copy over most of my work before continuing. On this\n> occasion I did a `git stash save` shortly before the crash, and I'm not\nsure\n> how to get that back. I see René Scharfe's suggestion of:\n>   git fsck --unreachable |\n>   grep commit | cut -d\\  -f3 |\n>   xargs git log --merges --no-walk --grep=WIP from a recent message, but\n> that is only showing me an older stash item.\n\nI would suggest turning off write-through buffering on your disk. Let writes\ncomplete immediately instead of being deferred to sync. Also, this does feel\nlike a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.\n\nGood luck,\nRandall\n\n-- Brief whoami:\n NonStop developer since approximately 211288444200000000\n UNIX developer since approximately 421664400\n-- In my real life, I talk too much.\n\n\n\n"},{"id":"408308","messageId":"20201025030606.GF860779@camp.crustytoothpaste.net","threadId":"54498","inReplyTo":"016001d6aa52$b1cbc510$15634f30$@nexbridge.com","subject":"Re: safer git?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-10-25T03:06:06Z","receivedAt":"2020-10-25T03:13:52Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"[I somehow didn't get the original message, so replying inline below.]\n\nOn 2020-10-24 at 22:11:54, Randall S. Becker wrote:\n> On October 24, 2020 4:19 PM, hv@crypt.org wrote:\n> > Q: Is there a mode in which I can run git that would make it a bit\n> > more robust\n> > against crashes, at the cost of being a bit slower?\n> > \n> > \n> > The primary symptom is that files modified shortly before a crash\n> > show up existing but zero-length after the crash. For source files I\n> > mostly know what to do in that situation, but `git fsck` shows a lot\n> > of files under '.git/objects' that are empty, which seems to make\n> > things hard to recover:\n> > \n> > % git fsck\n> > error: object file\n> > .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty\n> > error: unable to mmap\n> > .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file\n> > or directory\n> > error: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or\n> > missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533\n> > [... a dozen similar entries ...]\n> > error: object file\n> > .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty\n> > error: unable to mmap\n> > .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or\n> > directory\n> > error: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or\n> > missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5\n> > Checking object directories: 100% (256/256), done.\n> > Checking objects: 100% (1577/1577), done.\n> > error: refs/stash: invalid sha1 pointer\n> > 0000000000000000000000000000000000000000\n> > error: bad ref for .git/logs/refs/stash\n> > dangling commit 1c0ea4e6159952501957012d2b9db7d68b52d107\n> > %\n\nYou can try setting core.fsyncObjectFiles to true.  That's the only knob\nthat Git has for that at the moment, although there was some discussion\nabout adding a new feature for other files[0].\n\nI suspect a lot of the zero-byte files and any files that end up as\nall-zeros are due to your file system.  The default file system on\nUbuntu is ext4, IIRC, and if that's what you're using, you can set\ndata=journal instead of data=ordered as a mount option.  For the root\nfile system, you'll need to pass rootflags=data=journal as a kernel\nboot option.\n\nThat may be significantly slower, but until you get your hardware\nproblem sorted out, it may very well be worth it for you.  I'd try this\noption before the one below because it'll have less of an impact on\nperformance and may solve most or all of your problems.\n\n> > Last time I checked out the previous state from github in a new directory\n> and\n> > was able to find and copy over most of my work before continuing. On this\n> > occasion I did a `git stash save` shortly before the crash, and I'm not\n> sure\n> > how to get that back. I see René Scharfe's suggestion of:\n> >   git fsck --unreachable |\n> >   grep commit | cut -d\\  -f3 |\n> >   xargs git log --merges --no-walk --grep=WIP from a recent message, but\n> > that is only showing me an older stash item.\n> \n> I would suggest turning off write-through buffering on your disk. Let writes\n> complete immediately instead of being deferred to sync. Also, this does feel\n> like a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.\n\nTurning off buffering and caching for your disk drive may make things\n_really_ slow, but it will definitely improve data integrity.\n\nI know hardware problems are always a hassle, so I hope you get things\nfigured out and fixed soon.\n\n[0] I admit I am not running the very latest version and the new feature\nmay have already landed; if so, I apologize for the out-of-date\ninformation and for not keeping up with the list.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"408346","messageId":"202010251245.09PCjIP26203@crypt.org","threadId":"54498","inReplyTo":"20201025030606.GF860779@camp.crustytoothpaste.net","subject":"Re: safer git?","fromName":"","fromEmail":"hv@crypt.org","sentAt":"2020-10-25T12:45:18Z","receivedAt":"2020-10-25T13:40:35Z","isPatch":false,"sender":{"key":"hv@crypt.org","avatar":null},"body":"\"brian m. carlson\" wrote:\n:You can try setting core.fsyncObjectFiles to true.\n\nSuper, that's exactly the sort of thing I was hoping to find.\n\n:I suspect a lot of the zero-byte files and any files that end up as\n:all-zeros are due to your file system.  The default file system on\n:Ubuntu is ext4, IIRC, and if that's what you're using, you can set\n:data=journal instead of data=ordered as a mount option.\n\nIt is indeed ext4. I'll consider this option; for now I've turned off\nwrite caching as suggested by Randall, which feels like a lighter-weight\napproach that should give almost all of the benefit.\n\n:\"Randall S. Becker\" wrote:\n:> I would suggest turning off write-through buffering on your disk. Let writes\n:> complete immediately instead of being deferred to sync. Also, this does feel\n:> like a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.\n\nfsck doesn't seem to be complaining, but I've set it to run every\n20 mounts. What I do see is a handful of \"orphaned inodes\" being\nreclaimed on boot after every crash.\n\n:Turning off buffering and caching for your disk drive may make things\n:_really_ slow, but it will definitely improve data integrity.\n\nI haven't noticed a big slowdown so far; I'm rarely doing a _lot_ of\nwrites.\n\nThanks greatly to both of you for the suggestions.\n\nHugo\n"},{"id":"408349","messageId":"017f01d6aae1$f870e4c0$e952ae40$@nexbridge.com","threadId":"54498","inReplyTo":"20201025030606.GF860779@camp.crustytoothpaste.net","subject":"RE: safer git?","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2020-10-25T15:17:30Z","receivedAt":"2020-10-25T15:17:55Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On October 24, 2020 11:06 PM, brian m. carlson wrote:\n> [I somehow didn't get the original message, so replying inline below.]\n> \n> On 2020-10-24 at 22:11:54, Randall S. Becker wrote:\n> > On October 24, 2020 4:19 PM, hv@crypt.org wrote:\n> > > Q: Is there a mode in which I can run git that would make it a bit\n> > > more robust against crashes, at the cost of being a bit slower?\n> > >\n> > >\n> > > The primary symptom is that files modified shortly before a crash\n> > > show up existing but zero-length after the crash. For source files I\n> > > mostly know what to do in that situation, but `git fsck` shows a lot\n> > > of files under '.git/objects' that are empty, which seems to make\n> > > things hard to recover:\n> > >\n> > > % git fsck\n> > > error: object file\n> > > .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty\n> > > error: unable to mmap\n> > > .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such\n> file\n> > > or directory\n> > > error: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or\n> > > missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533\n> > > [... a dozen similar entries ...]\n> > > error: object file\n> > > .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty\n> > > error: unable to mmap\n> > > .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such\n> file\n> > > or directory\n> > > error: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or\n> > > missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5\n> > > Checking object directories: 100% (256/256), done.\n> > > Checking objects: 100% (1577/1577), done.\n> > > error: refs/stash: invalid sha1 pointer\n> > > 0000000000000000000000000000000000000000\n> > > error: bad ref for .git/logs/refs/stash dangling commit\n> > > 1c0ea4e6159952501957012d2b9db7d68b52d107\n> > > %\n> \n> You can try setting core.fsyncObjectFiles to true.  That's the only knob that\n> Git has for that at the moment, although there was some discussion about\n> adding a new feature for other files[0].\n> \n> I suspect a lot of the zero-byte files and any files that end up as all-zeros are\n> due to your file system.  The default file system on Ubuntu is ext4, IIRC, and\n> if that's what you're using, you can set data=journal instead of data=ordered\n> as a mount option.  For the root file system, you'll need to pass\n> rootflags=data=journal as a kernel boot option.\n> \n> That may be significantly slower, but until you get your hardware problem\n> sorted out, it may very well be worth it for you.  I'd try this option before the\n> one below because it'll have less of an impact on performance and may solve\n> most or all of your problems.\n> \n> > > Last time I checked out the previous state from github in a new\n> > > directory\n> > and\n> > > was able to find and copy over most of my work before continuing. On\n> > > this occasion I did a `git stash save` shortly before the crash, and\n> > > I'm not\n> > sure\n> > > how to get that back. I see René Scharfe's suggestion of:\n> > >   git fsck --unreachable |\n> > >   grep commit | cut -d\\  -f3 |\n> > >   xargs git log --merges --no-walk --grep=WIP from a recent message,\n> > > but that is only showing me an older stash item.\n> >\n> > I would suggest turning off write-through buffering on your disk. Let\n> > writes complete immediately instead of being deferred to sync. Also,\n> > this does feel like a disk issue, so fsck or chkdsk /f (or whatever) on your\n> disk urgently.\n> \n> Turning off buffering and caching for your disk drive may make things\n> _really_ slow, but it will definitely improve data integrity.\n> \n> I know hardware problems are always a hassle, so I hope you get things\n> figured out and fixed soon.\n> \n> [0] I admit I am not running the very latest version and the new feature may\n> have already landed; if so, I apologize for the out-of-date information and\n> for not keeping up with the list.\n\nThanks for pointing out the git knob. I didn't know about that one and it seems helpful in general.\n\nRegards,\nRandall\n\n"}]}