{"thread":{"id":"52660","subject":"Stat cache in .git/index hinders syncing of repositories","startedAt":"2020-01-18T00:04:03Z","lastAt":"2020-01-24T09:16:34Z","messageCount":9,"participants":["Christoph Groth","Junio C Hamano","brian m. carlson","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"390034","messageId":"87v9p9skjz.fsf@drac","threadId":"52660","inReplyTo":null,"subject":"Stat cache in .git/index hinders syncing of repositories","fromName":"Christoph Groth","fromEmail":"christoph@grothesque.org","sentAt":"2020-01-17T23:57:36Z","receivedAt":"2020-01-18T00:04:03Z","isPatch":false,"sender":{"key":"christoph@grothesque.org","avatar":null},"body":"Hello,\n\nI am using unison to sync home directories across multiple machines.\nThis includes a fair number of git repositories and works very well.\nUnison recently acquired a new feature that allows to treat selected\nsubdirectories (like .git) atomically.  This makes the syncing perfectly\nsafe.\n\nSome people say that one should use git itself to sync git working\ndirectories, but IMHO these people oversee the difference between\ncollaboration (using git) and being able to continue one’s own\nunfinished work on a different machine, including uncommitted files,\nstashes, and - if it has to be - in the middle of a merge.  Moreover, it\nis simpler not to have to treat git repositories specially when syncing.\nSyncing git repositories is thus clearly useful.\n\nHowever, there is one problem with syncing git repositories, that has\nbeen noticed by multiple people [1]: The file .git/index contains not\nonly the “git index”, but also a cache of stat-data of the files in the\nworking directory.  Some file synchronizers are able to sync mtimes, but\nsyncing ctimes would be bizarre (if it is even possible).\n\nSo, say that machines A and B are synced.  A new git repository appears\non machine A.  The synchronizer is run which results in copying all the\nfiles of the new repo verbatim to machine B.  Note that now on machine\nB the cache inside the file .git/index contains invalid stat\ninformation.  So when \"git status\" is run on B .git/index gets\nrewritten, and the next sync operation copies it back to A, where again\nit is rewritten even by something as harmless as \"git status\".  And so\non, and so forth...\n\nIn my opinion the root of this ping-pong problem is that .git/index\nmixes information about the status of the repository (=what has been\nstaged) that should be synced with a cache of machine-specific\nfilesystem metadata.\n\nI am not an expert of git-internals, but perhaps it would be a good idea\nto move the cache into a separate file that could be put on a \"ignore\"\nlist for synchronizers?  It seems to me that this has been already\nproposed in a different context [2], and I would not be surprised if\nfactoring out the cache had other beneficial effects.\n\nIf it is not feasible to separate the cache, perhaps another possibility\nwould be to add a new possible value for core.checkStat that would\ndisable stat structure checking except for file sizes?\n\nAs a workaround for now, I exclude .git/index from syncing.  This seems\nto work quite well, but I would be scared to sync unfinished merges like\nthis.\n\nThanks\nChristoph\n\n[1] https://stackoverflow.com/questions/12126247/why-does-git-index-change-when-i-havent-done-anything-to-my-repository\n[2] https://www.mail-archive.com/git@vger.kernel.org/msg48065.html\n"},{"id":"390056","messageId":"xmqq7e1od41x.fsf@gitster-ct.c.googlers.com","threadId":"52660","inReplyTo":"87v9p9skjz.fsf@drac","subject":"Re: Stat cache in .git/index hinders syncing of repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-01-18T18:15:22Z","receivedAt":"2020-01-18T18:15:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christoph Groth <christoph@grothesque.org> writes:\n\n> However, there is one problem with syncing git repositories, that has\n> been noticed by multiple people [1]: The file .git/index contains not\n> only the “git index”, but also a cache of stat-data of the files in the\n> working directory.  Some file synchronizers are able to sync mtimes, but\n> syncing ctimes would be bizarre (if it is even possible).\n\nThe stat-data in the index file is meant to be a mere optimization,\nand after copying .git/index and the working tree files to a new\nbox, running \"git update-index --refresh\" would make them in sync,\nno?\n"},{"id":"390060","messageId":"87pnfgshxu.fsf@drac","threadId":"52660","inReplyTo":"xmqq7e1od41x.fsf@gitster-ct.c.googlers.com","subject":"Re: Stat cache in .git/index hinders syncing of repositories","fromName":"Christoph Groth","fromEmail":"christoph@grothesque.org","sentAt":"2020-01-18T19:06:21Z","receivedAt":"2020-01-18T19:14:23Z","isPatch":false,"sender":{"key":"christoph@grothesque.org","avatar":null},"body":"Junio C Hamano wrote:\n\n> Christoph Groth <christoph@grothesque.org> writes:\n>\n> > However, there is one problem with syncing git repositories, that\n> > has been noticed by multiple people [1]: The file .git/index\n> > contains not only the “git index”, but also a cache of stat-data of\n> > the files in the working directory.  Some file synchronizers are\n> > able to sync mtimes, but syncing ctimes would be bizarre (if it is\n> > even possible).\n>\n> The stat-data in the index file is meant to be a mere optimization,\n> and after copying .git/index and the working tree files to a new box,\n> running \"git update-index --refresh\" would make them in sync, no?\n\nLet’s assume that one somehow manages to teach the synchronizer to run\n\"git update-index --refresh\" immediately after copying the index file\nand to consider the resulting modified .git/index file as identical to\nthe version on the other side.  (That would be already really difficult,\nbecause it’s against the design of synchronizers to consider differing\nfiles to be identical.)\n\nEven then, .git/index will change again whenever the mtime of any\ntracked file changes.  That can easily happen through \"touch\" or\nmodifying a file and then reverting the change.  There’s no way for the\nsynchronizer to tell the difference between relevant and irrelevant\nchanges to .git/index.  (Short of writing a specialized tool for\ncomparing git index files and embedding it.)\n\nWould it be feasible to move this optimization data to a separate file\nwithout breaking backwards and forwards compatibility?  I guess it\nwould: the data format of .git/index could remain identical, but newer\nversions of git could ignore the stat data in the index file and use\na separate file for that.  If an older version of git is used with the\nrepository, it would simply notice that the stat data in .git/index is\nnot up-to-date.\n\nBut if the above is not feasible for some reason, would it be possible\nto provide a switch for disabling stat caching optimization?\n\nI believe that synchronizing files between machines is something that\nwill become ever more important with new tools like syncthing.  So it\nwould be really cool if git were to support such usage.  It *almost*\ndoes!\n"},{"id":"390061","messageId":"20200118194204.GC6570@camp.crustytoothpaste.net","threadId":"52660","inReplyTo":"87pnfgshxu.fsf@drac","subject":"Re: Stat cache in .git/index hinders syncing of repositories","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-01-18T19:42:04Z","receivedAt":"2020-01-18T19:42:12Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-01-18 at 19:06:21, Christoph Groth wrote:\n> But if the above is not feasible for some reason, would it be possible\n> to provide a switch for disabling stat caching optimization?\n\nGit is going to perform really terribly on repositories of any size if\nyou disable stat caching, so we're not very likely to implement such a\nfeature.  Even if we did implement it, you probably wouldn't want to use\nit.\n\nHowever, there are the core.checkStat and core.trustctime options which\ncan control which information is used in the stat caching.  You can\nrestrict it to the whole second part of mtime and the file size if you\nwant.  See git-config(1) for more details.\n\nNote that this assumes that (a) your sync tool can honor mtimes and (b)\nthat your sync tool syncs to another system of the same type.  You may\nstill run into problems if you share files between Linux and Windows\nbecause symbolic links are different sizes there.  (This also bites\nWSL.)  Since rsync can do the former, I think it's a reasonable\nexpectation that other tools can as well.\n\nOne final word of caution: you probably want to activate your sync tool\nonly manually and only when the repository is idle.  Tools like Dropbox\nthat automatically sync files one by one have been known to corrupt\nrepositories because the way they sync data leaves the repository in an\ninconsistent state and doesn't honor standard POSIX file system\nsemantics which Git relies on for integrity.\n\nHopefully with that information you can find a configuration and tool\nthat work for you.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"390064","messageId":"87d0bgs9o4.fsf@drac","threadId":"52660","inReplyTo":"20200118194204.GC6570@camp.crustytoothpaste.net","subject":"Re: Stat cache in .git/index hinders syncing of repositories","fromName":"Christoph Groth","fromEmail":"christoph@grothesque.org","sentAt":"2020-01-18T22:04:59Z","receivedAt":"2020-01-18T22:05:11Z","isPatch":false,"sender":{"key":"christoph@grothesque.org","avatar":null},"body":"brian m. carlson wrote:\n> On 2020-01-18 at 19:06:21, Christoph Groth wrote:\n> > But if the above is not feasible for some reason, would it be\n> > possible to provide a switch for disabling stat caching\n> > optimization?\n>\n> Git is going to perform really terribly on repositories of any size if\n> you disable stat caching, so we're not very likely to implement such\n> a feature.  Even if we did implement it, you probably wouldn't want to\n> use it.\n\nOK, I see.  But please consider (one day) to split up the index file to\nseparate the local stat cache from the globally valid data.\n\n(By the way, even after 12 years of using Git intensely I am confused\nabout what actually is the index.  I believed that it is the \"staging\narea\", like in \"git-add - Add file contents to the index\".  But then the\n.git/index file reflects all the tracked files, and not just staged\nones.  This usage is also reflected by the command \"git update-index\".)\n\n> However, there are the core.checkStat and core.trustctime options\n> which can control which information is used in the stat caching.  You\n> can restrict it to the whole second part of mtime and the file size if\n> you want.  See git-config(1) for more details.\n\nThanks a lot, that did the trick!  I’ve been already syncing mtimes.\nSetting both core.checkStat and core.trustctime to the \"weak\" values\nmade the spurious modifications go away.\n\nStill, this is a workaround, and the price is reduced robustness of file\nmodification detection.  Technically, that wouldn’t be necessary...\nI hope that in practice it won’t matter.\n\n> One final word of caution: you probably want to activate your sync\n> tool only manually and only when the repository is idle.  Tools like\n> Dropbox that automatically sync files one by one have been known to\n> corrupt repositories because the way they sync data leaves the\n> repository in an inconsistent state and doesn't honor standard POSIX\n> file system semantics which Git relies on for integrity.\n\nYes, that’s why I still prefer Unison to more automatic real-time\ntools.\n"},{"id":"390072","messageId":"nycvar.QRO.7.76.6.2001201248480.46@tvgsbejvaqbjf.bet","threadId":"52660","inReplyTo":"87d0bgs9o4.fsf@drac","subject":"Re: Stat cache in .git/index hinders syncing of repositories","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-01-20T12:01:54Z","receivedAt":"2020-01-20T12:02:11Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Christoph,\n\nOn Sat, 18 Jan 2020, Christoph Groth wrote:\n\n> brian m. carlson wrote:\n> > On 2020-01-18 at 19:06:21, Christoph Groth wrote:\n> > > But if the above is not feasible for some reason, would it be\n> > > possible to provide a switch for disabling stat caching\n> > > optimization?\n> >\n> > Git is going to perform really terribly on repositories of any size if\n> > you disable stat caching, so we're not very likely to implement such\n> > a feature.  Even if we did implement it, you probably wouldn't want to\n> > use it.\n>\n> OK, I see.  But please consider (one day) to split up the index file to\n> separate the local stat cache from the globally valid data.\n\nI am sure that this has been considered even before Git was publicly\nannounced, and I would wager a guess that it was determined that it would\nbe better to keep all of Git's private data in one place.\n\nNow, you are totally free to disagree, and even to work on a patch series\nto separate the stat cache and offer a compelling argument why this change\nshould be made. If I were you, I would not expect any other person to be\ninterested in working on this.\n\n> (By the way, even after 12 years of using Git intensely I am confused\n> about what actually is the index.  I believed that it is the \"staging\n> area\", like in \"git-add - Add file contents to the index\".  But then the\n> .git/index file reflects all the tracked files, and not just staged\n> ones.  This usage is also reflected by the command \"git update-index\".)\n\nThe concept of the Git index is slightly different from what is actually\nstored inside `.git/index`. You should consider the latter to be an\nimplementation detail that is of concern only if you want to work on\ninternals. Otherwise the description of the index as a staging area is a\npretty good image.\n\nThe staging area contains of course more than just the stages you changed.\nIt contains the entire tree that is staged in order to become the next\ncommit.\n\nIf you asked a worker at a theater to make a minor change to the stage,\nyou would not expect the staging area to be empty, either.\n\n> > However, there are the core.checkStat and core.trustctime options\n> > which can control which information is used in the stat caching.  You\n> > can restrict it to the whole second part of mtime and the file size if\n> > you want.  See git-config(1) for more details.\n>\n> Thanks a lot, that did the trick!  I’ve been already syncing mtimes.\n> Setting both core.checkStat and core.trustctime to the \"weak\" values\n> made the spurious modifications go away.\n\nAnd of course now you have a less performant setup because files have a\nmuch better chance of being \"racily clean\", i.e. their mtime could be\nidentical to the `.git/index` file, in which case Git has to assume that\nthe file might have changed, and the index has to be refreshed.\n\nJust saying that what you think of as a silver bullet comes at a price.\n\n> Still, this is a workaround, and the price is reduced robustness of file\n> modification detection.\n\nYou misunderstand how Git detects whether a file is modified or not.\n\nA file is re-hashed if its mtime is newer than, _or equal to_, the mtime\nof `.git/index`.\n\nSo no, it is not the robustness that is the problem. It is no less robust.\nThe problem is that you force re-hashing where it would not be necessary\notherwise.\n\nIn general, I am not sure that you are using the right tool for\nsynchronizing. If you cannot guarantee that a snapshot of the directory is\ncopied, you will always run the risk of inconsistent data, which is worse\nthan not having a backup at all: at least without a backup you do not have\na false sense of security.\n\nCiao,\nJohannes\n"},{"id":"390141","messageId":"87ftg9n0r1.fsf@drac","threadId":"52660","inReplyTo":"nycvar.QRO.7.76.6.2001201248480.46@tvgsbejvaqbjf.bet","subject":"Re: Stat cache in .git/index hinders syncing of repositories","fromName":"Christoph Groth","fromEmail":"christoph@grothesque.org","sentAt":"2020-01-20T23:53:22Z","receivedAt":"2020-01-20T23:53:31Z","isPatch":false,"sender":{"key":"christoph@grothesque.org","avatar":null},"body":"Johannes Schindelin wrote:\n>\n> On Sat, 18 Jan 2020, Christoph Groth wrote:\n>\n> > OK, I see.  But please consider (one day) to split up the index file\n> > to separate the local stat cache from the globally valid data.\n>\n> I am sure that this has been considered even before Git was publicly\n> announced,\n\nI would be very interested to hear the rationale for keeping the\ninformation about what is staged and the stat cache together in the same\nfile.  I, or someone else, might actually work on a patch one day, but\nbefore starting, it would be good to understand the reasoning behind the\ncurrent design.\n\n> and I would wager a guess that it was determined that it would be\n> better to keep all of Git's private data in one place.\n\nMy point is that it’s not just private data: When I excluded .git/index\nfrom synchronization, staging files for a commit was no longer\nsynchronized.\n\n> > (By the way, even after 12 years of using Git intensely I am\n> > confused about what actually is the index.  I believed that it is\n> > the \"staging area\", like in \"git-add - Add file contents to the\n> > index\".  But then the .git/index file reflects all the tracked\n> > files, and not just staged ones.  This usage is also reflected by\n> > the command \"git update-index\".)\n>\n> The concept of the Git index is slightly different from what is\n> actually stored inside `.git/index`. You should consider the latter to\n> be an implementation detail that is of concern only if you want to\n> work on internals. Otherwise the description of the index as a staging\n> area is a pretty good image.\n\nTo me, it does not seem to be a mere implementation detail.  For example\nthe command ’git update-index --refresh’ is part of the \"public API\" and\nits action is to update the stat cache.  It does not modify what is\nstaged or not.\n\n> > Still, this is a workaround, and the price is reduced robustness of\n> > file modification detection.\n>\n> You misunderstand how Git detects whether a file is modified or not.\n>\n> A file is re-hashed if its mtime is newer than, _or equal to_, the\n> mtime of `.git/index`.\n\nYou must mean \"the mtime in ’.git/index’\", but OK, I see.  Makes sense\nof course.  So setting core.trustctime to false and core.checkstat to\nminimal only means that some avoidable rehashings may be made.  But this\nwould require two modifications of a file in the same second, without\na change to the file size.\n\n> In general, I am not sure that you are using the right tool for\n> synchronizing. If you cannot guarantee that a snapshot of the\n> directory is copied, you will always run the risk of inconsistent\n> data, which is worse than not having a backup at all: at least without\n> a backup you do not have a false sense of security.\n\nI do not understand what makes you think so.\n\nUnison is very robust software, I never had any problems with it and\nnever heard of anyone having any.  Moreover, as I noted in the opening\nmessage of this thread, it recently gained an option to treat chosen\ndirectories as atomic.  I’m using this for \".git\" subdirectories.\n\nChristoph\n"},{"id":"390145","messageId":"20200121025311.GA4113372@camp.crustytoothpaste.net","threadId":"52660","inReplyTo":"87ftg9n0r1.fsf@drac","subject":"Re: Stat cache in .git/index hinders syncing of repositories","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-01-21T02:53:11Z","receivedAt":"2020-01-21T02:53:20Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-01-20 at 23:53:22, Christoph Groth wrote:\n> Johannes Schindelin wrote:\n> >\n> > On Sat, 18 Jan 2020, Christoph Groth wrote:\n> >\n> > > OK, I see.  But please consider (one day) to split up the index file\n> > > to separate the local stat cache from the globally valid data.\n> >\n> > I am sure that this has been considered even before Git was publicly\n> > announced,\n> \n> I would be very interested to hear the rationale for keeping the\n> information about what is staged and the stat cache together in the same\n> file.  I, or someone else, might actually work on a patch one day, but\n> before starting, it would be good to understand the reasoning behind the\n> current design.\n> \n> > and I would wager a guess that it was determined that it would be\n> > better to keep all of Git's private data in one place.\n> \n> My point is that it’s not just private data: When I excluded .git/index\n> from synchronization, staging files for a commit was no longer\n> synchronized.\n\nTo try to answer this question, Git stores all of its state about the\nworking tree in the index.  Bare repositories don't typically have an\nindex because they don't have a working tree.  Whether that state is\nstaged contents or stat information, all of it is in one file.\n\nStoring all of this data in one file means that only one file need be\nmapped into memory and rewritten.  Git writes to the index by atomically\ncreating a lock file along side of it and writing the new contents into\nit, and then doing an atomic replace.  This approach wouldn't be\npossible with multiple files, and any update to it wouldn't be atomic.\n\nThere is support for a split index mode which means that the main index\nneed not be rewritten as often, which is helpful when making small\nupdates to large trees, where the cost of rewriting the index is\nsignificant.  I don't know how locking is handled there[0], but I assume\nthat it is, because the people who implemented and reviewed it are\ncapable and thoughtful.\n\nHowever, having said that, nobody has provided a compelling case for\nusing multiple files for storing different types of working tree state.\nThe existing options are available for cases like yours and others', and\nthey work.  Since there are clear benefits to the current model,\nincluding simplicity and robustness, and few downsides, nobody has\ndecided to change it.\n\nI should add that even if, for some reason, we did add support for\nsplitting this data out, I'm not sure if we'd support syncing only part\nof the repository state and blowing away other state.  We don't really\nsupport that now (other than through tools like fetch and clone) and I\ndon't think we'd want to encourage that behavior in the future.\n\n[0] And I have not had the interest to look at this present moment.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"390345","messageId":"8736c5tdst.fsf@drac","threadId":"52660","inReplyTo":"20200121025311.GA4113372@camp.crustytoothpaste.net","subject":"Re: Stat cache in .git/index hinders syncing of repositories","fromName":"Christoph Groth","fromEmail":"christoph@grothesque.org","sentAt":"2020-01-24T09:16:18Z","receivedAt":"2020-01-24T09:16:34Z","isPatch":false,"sender":{"key":"christoph@grothesque.org","avatar":null},"body":"brian m. carlson wrote:\n> On 2020-01-20 at 23:53:22, Christoph Groth wrote:\n\n> > My point is that it’s not just private data: When I excluded\n> > .git/index from synchronization, staging files for a commit was no\n> > longer synchronized.\n>\n> (...)\n>\n> Storing all of this data in one file means that only one file need be\n> mapped into memory and rewritten.  Git writes to the index by\n> atomically creating a lock file along side of it and writing the new\n> contents into it, and then doing an atomic replace.  This approach\n> wouldn't be possible with multiple files, and any update to it\n> wouldn't be atomic.\n\nThanks a lot for the explanation.  To me, it still seems less\nsatisfying, from a design point of view, to mix state (=what changes\nhave been staged) with an ephemeral cache that is specific to\na particular file system.  Without having thought deeply about it,\nI have the impression that it wouldn’t matter if the stat cache and the\n“staging state” of the repository would be atomic each on their own.\n\nBut I understand now that all of this hardly matters in practice (see\nbelow), so I’m not motivated to work on this, and probably no one else\nis. :-)\n\n> However, having said that, nobody has provided a compelling case for\n> using multiple files for storing different types of working tree\n> state.  The existing options are available for cases like yours and\n> others', and they work.  Since there are clear benefits to the current\n> model, including simplicity and robustness, and few downsides, nobody\n> has decided to change it.\n\nIndeed, I do see hardly any disadvantages of globally setting\n\n\ttrustctime = false\n\tcheckstat = minimal\n\nas I do now.  In fact, I wonder what is the purpose of caching the\nsubsecond part of mtime and the ctime in the first place.  Perhaps it\nmatters for scripted use of git where several operations can occur in\nthe same second, but even then only changes that keep file sizes\nconstant would be affected.\n\n> I should add that even if, for some reason, we did add support for\n> splitting this data out, I'm not sure if we'd support syncing only\n> part of the repository state and blowing away other state.  We don't\n> really support that now (other than through tools like fetch and\n> clone) and I don't think we'd want to encourage that behavior in the\n> future.\n\nThe stat cache file would not be really part of the state of the\nrepository, since deleting it would not change anything, but only slow\ndown the next operation.  (That’s at least my understanding currently,\nperhaps I’m still overseeing something.)\n\nBrian, Johannes, Junio, thanks a lot for taking the time to clarify this\nissue.\n\nChristoph\n"}]}