Git bug report - disk errors on Windows after push

Thank you for filling out a Git bug report! Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue) Pushing to a directory directly.

What did you expect to happen? (Expected behavior) Good behavior

What happened instead? (Actual behavior) Disk errors. After pushing to a directory (my origin field is C:\Work for example) the git repository will be corrupted. Moreover causing disk errors not only in the target git bare repository. I've loosed some other files. Cannot be reproduced exactly. However I have decided to report this problem. I've experienced this problem for several years.

What's different between what you expected and what actually happened? Described.

Anything else you want to add:

Please review the rest of the bug report below. You can delete any lines you don't wish to share.

[System Info] git version: git version 2.31.1.windows.1 cpu: x86_64 built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb sizeof-long: 4 sizeof-size_t: 8 shell-path: /bin/sh feature: fsmonitor--daemon uname: Windows 10.0 19044 compiler info: gnuc: 10.2 libc info: no libc information available $SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe

[Enabled Hooks]

Hey,

On 2022-04-18 at 07:47:07, Attila Csosz wrote:

There's nothing that Git should be able to do as a normal operating system that causes disk errors. If you're seeing those kinds of errors, the likelihood is that your disk is bad and you need to replace it.

Do note that if you're storing your data in a directory managed by a cloud syncing service (e.g., Dropbox or OneDrive), then it is known that those file systems corrupt Git repositories and you shouldn't use them with Git. That's due to the way they sync data.

However, if this is a regular file system (e.g., NTFS), then Git shouldn't be able to cause disk problems, and if it could, that would be a serious defect in Windows, so it's almost certainly going to be a hardware problem.

I've experienced the problem at least 3 different external disk. I'm not using cloud syncing service (e.g., Dropbox or OneDrive) for git.

Attila

On 4/18/2022 19:37, brian m. carlson wrote:

[Please avoid top-posting on this list.]

On 2022-04-19 at 03:52:23, Attila Csosz wrote:

I don't doubt that's the case. However, it's not possible for Git to corrupt a disk just by writing to it. Git, as a normal unprivileged program, can only open files and perform normal read and write operations on them. Management of the file system, including integrity, is the responsibility of the operating system. If running Git on Windows causes files outside of the repository to be corrupted, that's either a hardware problem or a bug in Windows.

If you're seeing this problem, it could be that you have a series of bad disks (say, if you bought a set of cheap flash drives), that the drive isn't being removed properly, that you have some sort of broken driver or malware, that there's some other hardware problem (e.g., a bad dock, USB device, CPU, or memory), or just that there's a previously unknown bug in Windows.

None of this is Git's fault, and it's really up to you to figure out which of these it is. You might try isolating it by using a different computer to write to the repository first, to see if something about your current machine is the problem, and then try isolating other causes.

On 4/18/2022 8:52 PM, Attila Csosz wrote:

Hi Attila, Could you please share a bit more information about the disks that are being corrupted and the specific error messages you're seeing?

What is the underlying storage type (e.g. USB key, external hard drive, etc)? Also, which filesystem is the drive formatted as?

Could you please save one instance of a volume in the corrupted state so that we can consider further investigations?

I believe the output of "fsutil fsinfo volumeinfo <drive letter>" and "fsutil fsinfo drivetype <drive letter>" should give us some useful info about the volume that's being corrupted.

Thanks, Neeraj Singh, Windows FileSystem Dev

Please check this link also https://github.community/t/what-could-be-the-reason-my-git-repositories-keep-getting-corrupted-broken/184711

You can found many link on web on 'git corrupted repository'. In my opinion when a git repository is corrupted on Windows there is a good chance to HDD errors.

I suggest also run intensive test on Windows (run a bat file commiting and pushing to a bare directory type repository). Check your HDD healthy status after 10-20 run of 'git push'.

My git software is come form 'git-scm.com' and I'm using TortoiseGit. My git software is up to date. Last time my laptop HDD is crashed loosing sensitive datas. I will never push to a local repo I'm using online repo-s like github.

/// fsutil fsinfo volumeinfo e: Volume Name : Seagate Backup Plus Drive Volume Serial Number : <removed by hand> Max Component Length : 255 File System Name : NTFS Is ReadWrite Not Thinly-Provisioned Supports Case-sensitive filenames Preserves Case of filenames Supports Unicode in filenames Preserves & Enforces ACL's Supports file-based Compression Supports Disk Quotas Supports Sparse files Supports Reparse Points Returns Handle Close Result Information Supports POSIX-style Unlink and Rename Supports Object Identifiers Supports Encrypted File System Supports Named Streams Supports Transactions Supports Hard Links Supports Extended Attributes Supports Open By FileID Supports USN Journal ///

Maybe this email not reaching the main mailing list I'm somehow blacklisted.

Attila

On Thu, Apr 21, 2022 at 12:54 AM Neeraj Singh <nksingh85@gmail.com> wrote:

On 4/20/2022 7:29 PM, Attila Csosz wrote:

Do you currently have a drive in the bad state? Is write-caching enabled for the drive (https://docs.microsoft.com/en-us/troubleshoot/windows-server/backup-and-storage/turn-disk-write-caching-on-off)?

Git hammers the filesystem in a way that's not typically done for a USB drive (especially issuing some "FlushFileBuffers" calls). So it possibly exposes some inherent flakiness in your setup. For instance, your drives might be good, but something could be wrong with your USB cable or the power supply.

If you have a drive in the bad state, a "chkdsk /scan /X <drive letter>" might give some hints about what specifically is wrong with that filesystem.

I'd recommend preferring a cloud-based backup system for your Git repos (e.g. GitHub or GitLab). Also, for local backups, something solid-state like an SSD might be more reliable in the long run.

Thanks, Neeraj

On Mon, Apr 18 2022, brian m. carlson wrote:

I think you're most likely right that this is some local HW issue, but the report doesn't have enough information to say either way.

But generally as we've been discussing at length in various threads related to git's fsync() behavior we can and will corrupt repositories due only to our own (mis)use of FS APIs, i.e. not doing fsyncing and the like correctly, it's been improved a bit recently.

But we're still in a state of not doing it correctly, and/or relying on some implicit assumptions.

For something like an external disk that may have its power cable yanked I'd give it about even odds that it's HW v.s. git's own FS syncing logic being at fault.

We get away with a lot of shenanigans because we're usually on a journaled FS, which e.g. on a FAT32 external volume we won't benefit from.

Attila: I have no idea if this even exists on Windows, but aside from
other suggestions in this thread I'd suggest enabling whatever Window's
equivalent of "sync" mounting is (if it even exists). I.e. to have every
single FS operation by synced to the metal. 

It will make your repository *much much slower*, but it will be safe as far as any logic errors in git vis-a-vis syncing go.

I'm using git on Windows for 6-7 years sometimes facing this problem (different computers, different HDDs). Last time the problem occured on my internal laptop HDD. So I have decided to report the problem.

If unstable to use local bare repos on Windows you should mention in somewhere or completly disable it. Disk errors are unacceptable even the git repo is corrupted.

Please check this link also https://github.community/t/what-could-be-the-reason-my-git-repositories-keep-getting-corrupted-broken/184711

Attila

On Fri, Apr 22, 2022 at 11:27 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

Given that the original report had this:

    Disk errors. After pushing to a directory (my origin field is C:\Work
    for example) the git repository will be corrupted.
    Moreover causing disk errors not only in the target git bare
    repository. I've loosed some other files.

that clearly stated that the corruption spreads outside repositories managed (and written and fsynced) by Git, it does not seem likely that Git's relying on "closing is enough as sync daemon will eventually flush them" is the primary source of corruption.

The recent "are we fsyncing correctly?" topic is about how likely we would survive in the presense of actions like unplugging without unmounting. It does not help a bit against flaky hardware, and even when it helps, it is merely making the damage caused by such actions less severe.