# Git status extremely slow if any file is a multiple of 8GBi 3 messages from 2022-05-04 to 2022-05-04. Participants: Jason Hatton, Junio C Hamano. Thread: https://gitlist.dev/t/57840 ## Jason Hatton, 2022-05-04 00:15 Subject: Git status extremely slow if any file is a multiple of 8GBi Message-ID: URL: https://gitlist.dev/e/CY4PR16MB1655A1F7DD8B2FABCC30C9DAAFC39%40CY4PR16MB1655.namprd16.prod.outlook.com ``` Git is extremely slow if any file is a multiple of 8GBi. Below is a working example. > git init > dd if=/dev/zero of=8gb bs=1024 count=$((1024*1024*8)) > git lfs track 8gb > git add . > git commit > time git status > time git status > time git status This seems related to git using uint32_t for storing the file size in the index. A file of zero size has a special meaning. I have an open issue on git-for-windows where @dscho has a lot of good technical information. https://github.com/git-for-windows/git/issues/3833#issuecomment-1116544918 I have a proposed idea that may or may not help. Would it be possible for any file that is a multiple of 2^32 to be adjusted to a file size of 1 instead of zero? Git is already functioning with mangling file sizes over 4GB in the index, so maybe bumping up the size of 2^32 multiple files would mitigate the issue. Using different versions of git would just cause 2^32 multiple files to be re-checked again, which is the current behavior anyway. 0 would retain its special meaning. If the file size is changed from 2^32 to 2^32 + 1, git may not notice that the file changed because both files will look like they are size 1. The ctime and mtime may still catch this. // Some pseudo code near: https://github.com/git-for-windows/git/blob/dc88e3cd72a2f0bbe2fe513acfc72bd66b577851/read-cache.c#L176sd- sd->sd_size = st->st_size; if (sd->sd_size == 0 && st->st_size != 0) { sd->sd_size = 1; } Thanks -- Jason D. Hatton ``` ## Junio C Hamano, 2022-05-04 13:55 Subject: Re: Git status extremely slow if any file is a multiple of 8GBi Message-ID: URL: https://gitlist.dev/e/xmqqczgtpfl5.fsf%40gitster.g In-Reply-To: ``` Jason Hatton writes: > I have a proposed idea that may or may not help. Would it be possible for any > file that is a multiple of 2^32 to be adjusted to a file size of 1 instead of > zero? Git is already functioning with mangling file sizes over 4GB in the > index, so maybe bumping up the size of 2^32 multiple files would mitigate the > issue. Clever. The condition sd_size==0 is used as a signal for "no, we really need to compare the contents", and causes the contents to be hashed, and if the contents match the object name recorded in the index, the on-disk size is stored in sd_size and the entry is marked as CE_UPTODATE. Alas, if the truncated st_size is 0, the resulting entry would have sd_size==0 again, so a workaround like what you outlined is needed. You'd need to make sure that you tweaked if (sd->sd_size != (unsigned int) st->st_size) changed |= DATA_CHANGED; that appears at the end of read-cache.c::match_stat_data() in a way that is consistent with how you munge the sd_size member in fill_stat_data(), if you are going to go that route. Thanks. ``` ## Junio C Hamano, 2022-05-04 16:08 Subject: Re: Git status extremely slow if any file is a multiple of 8GBi Message-ID: URL: https://gitlist.dev/e/xmqq4k25nuvj.fsf%40gitster.g In-Reply-To: ``` Junio C Hamano writes: > The condition sd_size==0 is used as a signal for "no, we really need > to compare the contents", and causes the contents to be hashed, and > if the contents match the object name recorded in the index, the > on-disk size is stored in sd_size and the entry is marked as > CE_UPTODATE. Alas, if the truncated st_size is 0, the resulting > entry would have sd_size==0 again, so a workaround like what you > outlined is needed. This is of secondary importance, but the fact that Jason observed 8GBi files gets hashed over and over unnecessarily means that we would do the same for an empty file, opening, reading 0-bytes, hashing, and closing, without taking advantage of the fact that CE_UPTODATE bit says the file contents should be up-to-date with respect to the cached object name, doesn't it? Or do we have "if st_size == 0 and sd_size == 0 then we know what it hashes to (i.e. EMPTY_BLOB_SHA*) and there is no need to do the usual open-read-hash-close dance" logic (I didn't check)? ```