{"thread":{"id":"47741","subject":"Git on Windows maps creation time onto changed time","startedAt":"2018-02-01T23:10:52Z","lastAt":"2018-02-02T23:01:24Z","messageCount":7,"participants":["Keith Goldfarb","Ævar Arnfjörð Bjarmason","Johannes Sixt","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"338054","messageId":"CCB76F06-DBA9-4342-8830-402F8AF9854A@blackthorn-media.com","threadId":"47741","inReplyTo":null,"subject":"Git on Windows maps creation time onto changed time","fromName":"Keith Goldfarb","fromEmail":"keith@blackthorn-media.com","sentAt":"2018-02-01T23:10:45Z","receivedAt":"2018-02-01T23:10:52Z","isPatch":false,"sender":{"key":"keith@blackthorn-media.com","avatar":null},"body":"Dear git,\n\nWhile tracking down a problem with a filesystem shared by Windows and Ubuntu, I came across the following code in compat/mingw.c (ming_fstat(), also in do_lstat()):\n\n\tif (GetFileInformationByHandle(fh, &fdata)) {\n\t\tbuf->st_ino = 0;\n\t\tbuf->st_gid = 0;\n\t\tbuf->st_uid = 0;\n\t\tbuf->st_nlink = 1;\n\t\tbuf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);\n\t\tbuf->st_size = fdata.nFileSizeLow |\n\t\t\t(((off_t)fdata.nFileSizeHigh)<<32);\n\t\tbuf->st_dev = buf->st_rdev = 0; /* not used by Git */\n\t\tbuf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));\n\t\tbuf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));\n\t\tbuf->st_ctime = filetime_to_time_t(&(fdata.ftCreationTime));\n\t\treturn 0;\n\t}\n\nThe assignment of buf->st_ctime doesn’t seem right to me. I understand there’s no good choice here, but I think a better choice would be to duplicate the definition used for st_mtime.\n\nBackground: When I do a git status on Windows and then later on Ubuntu (or the other order), it is extremely slow, as the entire tree is being traversed. I tracked it down to this difference in definition of c_time. Yes, I know about the core.trustctime variable, but my problem aside this seems like an unwise choice.\n\nThanks for listening,\n\nK.\n\n"},{"id":"338055","messageId":"CACBZZX7=NKKYMTmQNe8JqjPqZF6kYsw7k+UpFkgZGhFModpr1g@mail.gmail.com","threadId":"47741","inReplyTo":"CCB76F06-DBA9-4342-8830-402F8AF9854A@blackthorn-media.com","subject":"Re: Git on Windows maps creation time onto changed time","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-02-01T23:22:07Z","receivedAt":"2018-02-01T23:22:34Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Feb 2, 2018 at 12:10 AM, Keith Goldfarb\n wrote:\n> Dear git,\n>\n> While tracking down a problem with a filesystem shared by Windows and Ubuntu, I came across the following code in compat/mingw.c (ming_fstat(), also in do_lstat()):\n>\n> if (GetFileInformationByHandle(fh, &fdata)) {\n> buf->st_ino = 0;\n> buf->st_gid = 0;\n> buf->st_uid = 0;\n> buf->st_nlink = 1;\n> buf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);\n> buf->st_size = fdata.nFileSizeLow |\n> (((off_t)fdata.nFileSizeHigh)<<32);\n> buf->st_dev = buf->st_rdev = 0; /* not used by Git */\n> buf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));\n> buf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));\n> buf->st_ctime = filetime_to_time_t(&(fdata.ftCreationTime));\n> return 0;\n> }\n>\n> The assignment of buf->st_ctime doesn’t seem right to me. I understand there’s no good choice here, but I think a better choice would be to duplicate the definition used for st_mtime.\n>\n> Background: When I do a git status on Windows and then later on Ubuntu (or the other order), it is extremely slow, as the entire tree is being traversed. I tracked it down to this difference in definition of c_time. Yes, I know about the core.trustctime variable, but my problem aside this seems like an unwise choice.\n>\n> Thanks for listening,\n\nLet's CC Marius Storm-Olsen who added this behavior (CC'd).\n"},{"id":"338101","messageId":"c1e419ae-feae-5a31-ead6-20b25fa92f85@kdbg.org","threadId":"47741","inReplyTo":"CCB76F06-DBA9-4342-8830-402F8AF9854A@blackthorn-media.com","subject":"Re: Git on Windows maps creation time onto changed time","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2018-02-02T17:38:31Z","receivedAt":"2018-02-02T17:40:13Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 02.02.2018 um 00:10 schrieb Keith Goldfarb:\n> Dear git,\n> \n> While tracking down a problem with a filesystem shared by Windows and Ubuntu, I came across the following code in compat/mingw.c (ming_fstat(), also in do_lstat()):\n> \n> \tif (GetFileInformationByHandle(fh, &fdata)) {\n> \t\tbuf->st_ino = 0;\n> \t\tbuf->st_gid = 0;\n> \t\tbuf->st_uid = 0;\n> \t\tbuf->st_nlink = 1;\n> \t\tbuf->st_mode = file_attr_to_st_mode(fdata.dwFileAttributes);\n> \t\tbuf->st_size = fdata.nFileSizeLow |\n> \t\t\t(((off_t)fdata.nFileSizeHigh)<<32);\n> \t\tbuf->st_dev = buf->st_rdev = 0; /* not used by Git */\n> \t\tbuf->st_atime = filetime_to_time_t(&(fdata.ftLastAccessTime));\n> \t\tbuf->st_mtime = filetime_to_time_t(&(fdata.ftLastWriteTime));\n> \t\tbuf->st_ctime = filetime_to_time_t(&(fdata.ftCreationTime));\n> \t\treturn 0;\n> \t}\n> \n> The assignment of buf->st_ctime doesn’t seem right to me. I\n> understand there’s no good choice here, but I think a better choice\n> would be to duplicate the definition used for st_mtime.\n\nThe purpose of these values is to allow to notice a change on the file \nsystem without going through the actual file data. Duplicating st_mtime \nwould be pointless.\n\n> Background: When I do a git status on Windows and then later on\n> Ubuntu (or the other order), it is extremely slow, as the entire tree\n> is being traversed. I tracked it down to this difference in\n> definition of c_time. Yes, I know about the core.trustctime variable,\n> but my problem aside this seems like an unwise choice.\n\nDon't do that then. Use core.trustctime.\n\n-- Hannes\n"},{"id":"338107","messageId":"040E3C3E-BC8E-4658-A882-5149D10AD408@blackthorn-media.com","threadId":"47741","inReplyTo":"c1e419ae-feae-5a31-ead6-20b25fa92f85@kdbg.org","subject":"Re: Git on Windows maps creation time onto changed time","fromName":"Keith Goldfarb","fromEmail":"keith@blackthorn-media.com","sentAt":"2018-02-02T18:45:59Z","receivedAt":"2018-02-02T18:46:10Z","isPatch":false,"sender":{"key":"keith@blackthorn-media.com","avatar":null},"body":"> The purpose of these values is to allow to notice a change on the file system without going through the actual file data. Duplicating st_mtime would be pointless.\n\nWell, I agree with the first statement. I disagree with the 2nd. It’s not pointless to fix a problem, and in theory the creation date of a file can never change.\n\n> Don't do that then. Use core.trustctime.\n\nI am. Unfortunately, my problem isn’t solved by that alone. Perhaps this deserves its own thread, but on Windows the st_ino field is set to zero. This can also trigger a false positive, causing the whole cache to be rebuilt unnecessarily.\n\nK."},{"id":"338119","messageId":"nycvar.QRO.7.76.6.1802022216370.35@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"47741","inReplyTo":"040E3C3E-BC8E-4658-A882-5149D10AD408@blackthorn-media.com","subject":"Re: Git on Windows maps creation time onto changed time","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-02-02T21:18:14Z","receivedAt":"2018-02-02T21:18:23Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Keith,\n\nOn Fri, 2 Feb 2018, Keith Goldfarb wrote:\n\n> > The purpose of these values is to allow to notice a change on the file\n> > system without going through the actual file data. Duplicating\n> > st_mtime would be pointless.\n> \n> Well, I agree with the first statement. I disagree with the 2nd. It’s\n> not pointless to fix a problem, and in theory the creation date of a\n> file can never change.\n> \n> > Don't do that then. Use core.trustctime.\n> \n> I am. Unfortunately, my problem isn’t solved by that alone. Perhaps this\n> deserves its own thread, but on Windows the st_ino field is set to zero.\n> This can also trigger a false positive, causing the whole cache to be\n> rebuilt unnecessarily.\n\nYou cannot assume that the inode numbers are identical between file\nsystems/operating systems. That's simply not going to work.\n\nI know you really want *so hard* for the same working directory to be\naccessible from both Windows and Linux. I have a lot of sympathy for that\nsentiment. Though I do not see much chance for success on that front.\n\nCiao,\nJohannes"},{"id":"338126","messageId":"0B8B61C6-0EB6-4134-BDA9-FA7750B79BB2@blackthorn-media.com","threadId":"47741","inReplyTo":"nycvar.QRO.7.76.6.1802022216370.35@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: Git on Windows maps creation time onto changed time","fromName":"Keith Goldfarb","fromEmail":"keith@blackthorn-media.com","sentAt":"2018-02-02T22:44:44Z","receivedAt":"2018-02-02T22:44:51Z","isPatch":false,"sender":{"key":"keith@blackthorn-media.com","avatar":null},"body":"> On Feb 2, 2018, at 1:18 PM, Johannes Schindelin wrote:\n> \n> You cannot assume that the inode numbers are identical between file\n> systems/operating systems. That's simply not going to work.\n\nYes, I agree with you, I cannot assume this, so I checked. In my case, the inode numbers are indeed identical.\n\n> I know you really want *so hard* for the same working directory to be\n> accessible from both Windows and Linux. I have a lot of sympathy for that\n> sentiment. Though I do not see much chance for success on that front.\n\nI’m certainly willing to accept that there are going to be limitations by using a filesystem from two different operating systems. But regardless of the problems caused by that pattern, would you agree that the Windows code should be using the actual inode number (well, 32-bits of it) instead of zero?\n\nK."},{"id":"338129","messageId":"nycvar.QRO.7.76.6.1802022353220.35@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"47741","inReplyTo":"0B8B61C6-0EB6-4134-BDA9-FA7750B79BB2@blackthorn-media.com","subject":"Re: Git on Windows maps creation time onto changed time","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-02-02T23:01:13Z","receivedAt":"2018-02-02T23:01:24Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Keith,\n\nOn Fri, 2 Feb 2018, Keith Goldfarb wrote:\n\n> > On Feb 2, 2018, at 1:18 PM, Johannes Schindelin wrote:\n> > \n> > You cannot assume that the inode numbers are identical between file\n> > systems/operating systems. That's simply not going to work.\n> \n> Yes, I agree with you, I cannot assume this, so I checked. In my case, the inode numbers are indeed identical.\n> \n> > I know you really want *so hard* for the same working directory to be\n> > accessible from both Windows and Linux. I have a lot of sympathy for that\n> > sentiment. Though I do not see much chance for success on that front.\n> \n> I’m certainly willing to accept that there are going to be limitations by using a filesystem from two different operating systems. But regardless of the problems caused by that pattern, would you agree that the Windows code should be using the actual inode number (well, 32-bits of it) instead of zero?\n\nIn all likelihood, you have enabled the FS Cache feature. And for a good\nreason: it makes many Git operations much, much faster.\n\nStill not as fast as on Linux because Git is quite tuned to assume all the\nLinux semantics. So instead of using tell-tales for file modifications\nthat are portable, Git uses tell-tales that are readily available on\nLinux. And instead of using access patterns that are portable, Git assumes\nPOSIX semantics where you simply use opendir()/readdir()/closedir() and in\nthat inner loop, you call lstat() to get all the stat data.\n\nThis is obviously not portable beyond POSIX. On Windows, for example,\nthere are really fast FindFirstFile()/FindNextFile() APIs that even allow\nyou to specify patterns to match (which Linux has to do manually, by\ninspecting the file name obtained via readdir()). You even get some stat\ndata back at the same time, really fast, without additional API calls,\nunlike on Linux.\n\nBut Git's access patterns really are tuned to the *dir() way, and we have\nto emulate it in Git for Windows. This is very, very slow.\n\nThe FS Cache feature tries to gain back some speed by using\nFindFirstFile()/FindNextFile() pre-emptively and caching the data, looking\nit up upon the next time stat data is queried.\n\nThe stat data obtained that way does not, however, include Change Time nor\ninode number.\n\nAnd that, dear children, was the story for the day: this is why Git for\nWindows uses the CreationTime instead of the Change Time and why it marks\ninode as 0; Because to do it accurately would be a lot slower, and we\ncannot afford even slower because we are already super slow because we\nhave to work around Git's assumptions as to what APIs are available.\n\nCiao,\nJohannes"}]}