{"thread":{"id":"57866","subject":"Weird behaviour of git diff-index in container","startedAt":"2022-05-09T22:42:25Z","lastAt":"2022-05-10T16:47:48Z","messageCount":4,"participants":["Timo Funke","Junio C Hamano","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"455061","messageId":"VI1PR0402MB28779C7A41783472B2EF6823BFC69@VI1PR0402MB2877.eurprd04.prod.outlook.com","threadId":"57866","inReplyTo":null,"subject":"Weird behaviour of git diff-index in container","fromName":"Timo Funke","fromEmail":"timoses@msn.com","sentAt":"2022-05-09T22:42:14Z","receivedAt":"2022-05-09T22:42:25Z","isPatch":false,"sender":{"key":"timoses@msn.com","avatar":null},"body":"What did you do before the bug happened? (Steps to reproduce your issue)\n\nmkdir test\ncd test\ngit init\ntouch test\ngit add test\ngit commit -m 'init'\npodman run --rm -it -v `pwd`:/git:z --entrypoint sh docker.io/alpine\n> container# apk add git\n> container# cd /git\n> container# git diff-index --quiet HEAD -- ; echo $?\n1\n> container# git diff-index --quiet HEAD -- ; echo $?\n1\n> container# git status\nOn branch master\nnothing to commit, working tree clean\n> container# git diff-index --quiet HEAD -- ; echo $?\n0\n\n\nWhat did you expect to happen? (Expected behavior)\n`git diff-index --quiet HEAD -- ; echo $?` should return `0`\neven without executing `git status`.\n\nWhat happened instead? (Actual behavior)\nWithout executing `git status` `git diff-index --quiet HEAD -- ; echo $?`\nwill repeatedly print `1`.\n\nWhat's different between what you expected and what actually happened?\nIt is odd that `git diff-index --quiet HEAD -- ; echo $?` prints\ndifferent results depending on whether `git status` was executed.\n\nAnything else you want to add: Perhaps this has to do with running git in a container?\n\n\n[System Info]\ngit version:\ngit version 2.34.2\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Linux 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Thu Mar 10 20:59:28 UTC 2022 x86_64\ncompiler info: gnuc: 10.3\nlibc info: no libc information available\n$SHELL (typically, interactive shell): <unset>\n\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show"},{"id":"455063","messageId":"xmqqy1za9tx3.fsf@gitster.g","threadId":"57866","inReplyTo":"VI1PR0402MB28779C7A41783472B2EF6823BFC69@VI1PR0402MB2877.eurprd04.prod.outlook.com","subject":"Re: Weird behaviour of git diff-index in container","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-09T23:18:32Z","receivedAt":"2022-05-09T23:18:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Timo Funke <timoses@msn.com> writes:\n\n>> container# git diff-index --quiet HEAD -- ; echo $?\n> 1\n>> container# git status\n> On branch master\n> nothing to commit, working tree clean\n>> container# git diff-index --quiet HEAD -- ; echo $?\n> 0\n\nThis is unfortunately very much expected and doubly unfortunately\nnot very well documented.  Patches to update documentation is very\nmuch welcomed, but such a patch cannot be written in void, so let's\nexplain what is going on.\n\nTo detect paths that have not been modified quickly, Git uses the\nmechanism called \"cached stat data\" in the index.  Among the cached\nstat data is the timestamp of the last modification of each file.\nBy noting that the fact that the last time it checked, the contents\nin the file on the filesystem hasn't been modified, together with\nthe file timestamp observed at the time of such a check, the next\ntime somebody asks \"please compute 'git diff'\", Git can notice that\nthe timestamp of the working tree file hasn't changed and say \"no,\nthere is no change\" without looking at the contents.\n\nNow, when the file on the filesystem is \"touched\" in a way that its\ntimestamp gets updated without changing the contents (hence, if\nthere weren't the above optimization, diff would have said \"no\nchange\"), Git will think there is a change in the file.\n\nThere are two levels of Git subcommands.  Porcelain commands, like\n\"git diff\", are end-user facing and are optimized more for usability\nthan performance.  \"git diff --quiet HEAD --\" in the above scenario\nWILL notice that there is no change in the contents after all and\nexit with 0 (unless diff.autoRefreshIndex configuration is set to\nfalse).  The way they do so is by refreshing the \"cached stat data\"\nautomatically before using, and that operation is called \"refreshing\nthe index\" (hence the configuration variable name to disable it).\n\nOn the other hand, plumbing commands, like \"git diff-files\" and \"git\ndiff-index\", are designed to be used in scripts, number of times,\nand do not want to pay the cost of refreshing the index always\nbefore working.  The correct way to use them in a repository whose\ncurrent state you do not know about is to first \"refresh the index\"\nby running the command to do so,  e.g. \"git update-index --refresh\"\nbefore doing anything else.\n\nIf you were to run \"git diff-files\" and \"git diff-index HEAD\" in a\nrow in order to compute what \"git status\" would give you, for\nexample, you do not need to and want to pay the cost of refreshing\nthe index twice.  You run \"git update-index --refresh\" once, and\nthen run \"git diff-files\".  Doing so would not change the contents\nof the working tree files, so you do not have to refresh the index\nagain after that, before running \"git diff-index HEAD\".  That is why\nthese plumbing commands do not refresh the index themselves.  They\nexpect you to be refreshing the index before you call them.\n\n\"git status\" is one of the commands (as a Porcelain) that refreshes\nthe index automatically, so it is very much understandable that the\nsame \"diff-index --quiet\" behaves differently after running it once\nand until you touch/smudge the working tree files.\n\n"},{"id":"455071","messageId":"YnnRIq3kudurSq4c@google.com","threadId":"57866","inReplyTo":"VI1PR0402MB28779C7A41783472B2EF6823BFC69@VI1PR0402MB2877.eurprd04.prod.outlook.com","subject":"Re: Weird behaviour of git diff-index in container","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2022-05-10T02:42:42Z","receivedAt":"2022-05-10T02:42:50Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi!\n\nTimo Funke wrote:\n\n> podman run --rm -it -v `pwd`:/git:z --entrypoint sh docker.io/alpine\n> > container# apk add git\n> > container# cd /git\n> > container# git diff-index --quiet HEAD -- ; echo $?\n> 1\n> > container# git diff-index --quiet HEAD -- ; echo $?\n> 1\n> > container# git status\n> On branch master\n> nothing to commit, working tree clean\n> > container# git diff-index --quiet HEAD -- ; echo $?\n> 0\n>\n>\n> What did you expect to happen? (Expected behavior)\n> `git diff-index --quiet HEAD -- ; echo $?` should return `0`\n> even without executing `git status`.\n>\n> What happened instead? (Actual behavior)\n> Without executing `git status` `git diff-index --quiet HEAD -- ; echo $?`\n> will repeatedly print `1`.\n>\n> What's different between what you expected and what actually happened?\n> It is odd that `git diff-index --quiet HEAD -- ; echo $?` prints\n> different results depending on whether `git status` was executed.\n\nI love this example.  Thanks for writing.\n\nI checked \"git help diff-index\" to see whether it describes this\npitfall, and I didn't see an explanation.  So at the very least you\nhave uncovered a documentation bug.\n\nThe difference between diff-index and status here is a difference\nbetween \"porcelain\" (user-facing) commands and \"plumbing\"\n(script-facing) commands.  In Git's index file there is stat(2)\ninformation for each file; if that stat(2) information matches the\ncorresponding file in the working directory then we know it hasn't\nbeen modified relative to what is in the index.  If the stat(2)\ninformation differs from the working copy, on the other hand, the\nbehavior depends on whether the command being run is porcelain or\nplumbing:\n\n - plumbing commands assume that the script author has run \"git\n   update-index --refresh -q\" first to update the stat(2) information\n   if the file hasn't changed.  This allows efficient scripts to\n   refresh the index once and then run multiple commands that rely on\n   the result of that:\n\n\tgit update-index --refresh -q || :\n\tfor rev in \"${revs[@]}\"\n\tdo\n\t\tif git diff-index --quiet \"$rev\" --\n\t\tthen\n\t\t\t... do something ...\n\t\tfi\n\tdone\n\n - porcelain commands such as \"git status\" implicitly refresh the\n   index before doing anything else.  This allows them to produce the\n   expected result even if the repository is a copy made using \"cp -a\"\n   or has been transferred across machines on a USB stick.\n\nSome places I expected to find an explanation of this:\n\n- documentation for the \"git diff-index\" command (\"git help\n  diff-index\").  It does not mention this behavior.\n\n- documentation for the \"git diff\" command (\"git help diff\").  It also\n  doesn't mention this.  That's particularly surprising because it\n  would be a great place to document the diff.autoRefreshIndex setting\n  that affects this behavior of the \"git diff\" command (described in\n  Documentation/config/diff.txt).\n\n- the Git user manual (Documentation/user-manual.txt).  It describes\n  \"git update-index --refresh\" but very briefly.  It doesn't describe\n  the above scripting pattern.\n\n- Git's command-line conventions (\"git help cli\").  No mention.\n\n- overview of plumbing and porcelain commands (\"man git\").  No\n  mention.\n\n- the Git scripting manual (\"git help core-tutorial\").  It describes\n  \"git update-index --refresh\" after a \"cp -a\" but not its use in\n  scripts.\n\n- the history of Git's contrib/examples/.  This contains many examples\n  of the above scripting pattern but is not very discoverable.\n\nSo there are many opportunities for someone to document this better.\nIf you'd be interested in pursuing that, I'd be happy to provide some\npointers.\n\nThanks,\nJonathan\n"},{"id":"455086","messageId":"xmqq8rr98hci.fsf@gitster.g","threadId":"57866","inReplyTo":"YnnRIq3kudurSq4c@google.com","subject":"Re: Weird behaviour of git diff-index in container","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-10T16:47:41Z","receivedAt":"2022-05-10T16:47:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> I love this example.  Thanks for writing.\n\nI guess our mails crossed ;-)\n\n> Some places I expected to find an explanation of this:\n>\n> - documentation for the \"git diff-index\" command (\"git help\n>   diff-index\").  It does not mention this behavior.\n\nYes, diff-index and diff-files should at least have a pointer to\n\"update-index --refresh\".  Ideally they should share a write-up\nbased on what both of us covered in these responses.\n\n> - documentation for the \"git diff\" command (\"git help diff\").  It also\n>   doesn't mention this.  That's particularly surprising because it\n>   would be a great place to document the diff.autoRefreshIndex setting\n>   that affects this behavior of the \"git diff\" command (described in\n>   Documentation/config/diff.txt).\n\nAnd the autorefreshindex documentation is a tad stale (it is on by\ndefault these days) and does not say why you would want it.  I do\nnot mind config/diff.txt having it, but that should eventually refer\nto the same page that is designed to help the readers of the\ndiff-index and diff-files documentation.\n\nI do not think anywhere else the missing info belongs to, but\nstepping back a bit, it may help to have a write up on scripting\nusing the plumbing commands in general, not limited to \"diff-*\"\nfamily of commands.  I actually am torn a bit, as we have long\nneglected to give matching improvement to plumbing commands when we\nadd shiny new toys to commands at the Porcelain level, so Git may\nhave grown much more hostile to scripters over the years X-<.\n\n"}]}