{"thread":{"id":"57840","subject":"Git status extremely slow if any file is a multiple of 8GBi","startedAt":"2022-05-04T00:23:12Z","lastAt":"2022-05-04T16:08:23Z","messageCount":3,"participants":["Jason Hatton","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"454805","messageId":"CY4PR16MB1655A1F7DD8B2FABCC30C9DAAFC39@CY4PR16MB1655.namprd16.prod.outlook.com","threadId":"57840","inReplyTo":null,"subject":"Git status extremely slow if any file is a multiple of 8GBi","fromName":"Jason Hatton","fromEmail":"jhatton@globalfinishing.com","sentAt":"2022-05-04T00:15:34Z","receivedAt":"2022-05-04T00:23:12Z","isPatch":false,"sender":{"key":"jhatton@globalfinishing.com","avatar":"https://avatars.githubusercontent.com/u/36747462?v=4"},"body":"Git is extremely slow if any file is a multiple of 8GBi. Below is\na working example.\n\n> git init\n> dd if=/dev/zero of=8gb bs=1024 count=$((1024*1024*8))\n> git lfs track 8gb\n> git add .\n> git commit\n> time git status\n> time git status\n> time git status\n\nThis seems related to git using uint32_t for storing the file size in the index.\nA file of zero size has a special meaning.\n\nI have an open issue on git-for-windows where @dscho has a lot of good technical\ninformation.\n\nhttps://github.com/git-for-windows/git/issues/3833#issuecomment-1116544918\n\nI have a proposed idea that may or may not help. Would it be possible for any\nfile that is a multiple of 2^32 to be adjusted to a file size of 1 instead of\nzero? Git is already functioning with mangling file sizes over 4GB in the\nindex, so maybe bumping up the size of 2^32 multiple files would mitigate the\nissue. Using different versions of git would just cause 2^32 multiple files to\nbe re-checked again, which is the current behavior anyway. 0 would retain its\nspecial meaning. If the file size is changed from 2^32 to 2^32 + 1, git may not\nnotice that the file changed because both files will look like they are size 1.\nThe ctime and mtime may still catch this.\n\n// Some pseudo code near: \nhttps://github.com/git-for-windows/git/blob/dc88e3cd72a2f0bbe2fe513acfc72bd66b577851/read-cache.c#L176sd-\n\nsd->sd_size = st->st_size;\nif (sd->sd_size == 0 && st->st_size != 0) {\n    sd->sd_size = 1;\n}\n\nThanks\n--\nJason D. Hatton\n\n"},{"id":"454811","messageId":"xmqqczgtpfl5.fsf@gitster.g","threadId":"57840","inReplyTo":"CY4PR16MB1655A1F7DD8B2FABCC30C9DAAFC39@CY4PR16MB1655.namprd16.prod.outlook.com","subject":"Re: Git status extremely slow if any file is a multiple of 8GBi","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-04T13:55:34Z","receivedAt":"2022-05-04T13:55:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jason Hatton <jhatton@globalfinishing.com> writes:\n\n> I have a proposed idea that may or may not help. Would it be possible for any\n> file that is a multiple of 2^32 to be adjusted to a file size of 1 instead of\n> zero? Git is already functioning with mangling file sizes over 4GB in the\n> index, so maybe bumping up the size of 2^32 multiple files would mitigate the\n> issue.\n\nClever.  \n\nThe condition sd_size==0 is used as a signal for \"no, we really need\nto compare the contents\", and causes the contents to be hashed, and\nif the contents match the object name recorded in the index, the\non-disk size is stored in sd_size and the entry is marked as\nCE_UPTODATE.  Alas, if the truncated st_size is 0, the resulting\nentry would have sd_size==0 again, so a workaround like what you\noutlined is needed.\n\nYou'd need to make sure that you tweaked\n\n\tif (sd->sd_size != (unsigned int) st->st_size)\n\t\tchanged |= DATA_CHANGED;\n\nthat appears at the end of read-cache.c::match_stat_data() in a way\nthat is consistent with how you munge the sd_size member in\nfill_stat_data(), if you are going to go that route.\n\nThanks.\n"},{"id":"454828","messageId":"xmqq4k25nuvj.fsf@gitster.g","threadId":"57840","inReplyTo":"xmqqczgtpfl5.fsf@gitster.g","subject":"Re: Git status extremely slow if any file is a multiple of 8GBi","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-04T16:08:16Z","receivedAt":"2022-05-04T16:08:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> The condition sd_size==0 is used as a signal for \"no, we really need\n> to compare the contents\", and causes the contents to be hashed, and\n> if the contents match the object name recorded in the index, the\n> on-disk size is stored in sd_size and the entry is marked as\n> CE_UPTODATE.  Alas, if the truncated st_size is 0, the resulting\n> entry would have sd_size==0 again, so a workaround like what you\n> outlined is needed.\n\nThis is of secondary importance, but the fact that Jason observed\n8GBi files gets hashed over and over unnecessarily means that we\nwould do the same for an empty file, opening, reading 0-bytes,\nhashing, and closing, without taking advantage of the fact that\nCE_UPTODATE bit says the file contents should be up-to-date with\nrespect to the cached object name, doesn't it?\n\nOr do we have \"if st_size == 0 and sd_size == 0 then we know what it\nhashes to (i.e. EMPTY_BLOB_SHA*) and there is no need to do the\nusual open-read-hash-close dance\" logic (I didn't check)?\n"}]}