{"thread":{"id":"47877","subject":"Git should preserve modification times at least on request","startedAt":"2018-02-19T21:45:12Z","lastAt":"2018-02-26T11:28:13Z","messageCount":28,"participants":["Peter Backes","Johannes Schindelin","Randall S. Becker","Hilco Wijbenga","Theodore Ts'o","Jeff King","Junio C Hamano","Jacob Keller","Derek Fawcus","Phillip Wood","Ævar Arnfjörð Bjarmason","'Peter Backes'","Konstantin Khomoutov","Andreas Krey"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"339643","messageId":"20180219212235.GA9891@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":null,"subject":"Git should preserve modification times at least on request","fromName":"Peter Backes","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-19T21:22:36Z","receivedAt":"2018-02-19T21:45:12Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"Hello,\n\nplease ensure to CC me if you reply as I am not subscribed to the list.\n\nhttps://git.wiki.kernel.org/index.php/Git_FAQ#Why_isn.27t_Git_preserving_modification_time_on_files.3F \nargues that git isn't preserving modification times because it needs to \nensure that build tools work properly.\n\nI agree that modification times should not be restored by default, \nbecause of the principle of least astonishment. But should it be \nimpossible? The principle of least astonishment does not mandate this; \nit is not a paternalistic principle.\n\nThus, I do not get at all\n- why git doesn't *store* modification times, perhaps by default, but \nat least on request\n- why git doesn't restore modification times *on request*\n\nIt is pretty annoying that git cannot, even if I know what I am doing, \nand explicitly want it to, preserve the modification time.\n\nOne use case: I have lots of file lying around in my build directory \nand for some of them, the modification time in important information to \nme. Those files are not at all used with the build tool. In contrast to \ngit pull, git pull --rebase needs those to be stashed. But after the \npull and unstash, the mtime is gone. Boo.\n\nPlease provide options to store and restore modification times. It \nshouldn't be hard to do, given that other metadata such as the mode is \nalready stored. It would make live so much easier. And the fact that \nthis has made into the FAQ clearly suggests that there are many others \nwho think so.\n\nBest wishes\nPeter\n\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339644","messageId":"nycvar.QRO.7.76.6.1802192257100.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"47877","inReplyTo":"20180219212235.GA9891@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-02-19T21:58:12Z","receivedAt":"2018-02-19T21:58:22Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peter,\n\nOn Mon, 19 Feb 2018, Peter Backes wrote:\n\n> please ensure to CC me if you reply as I am not subscribed to the list.\n> \n> https://git.wiki.kernel.org/index.php/Git_FAQ#Why_isn.27t_Git_preserving_modification_time_on_files.3F \n> argues that git isn't preserving modification times because it needs to \n> ensure that build tools work properly.\n> \n> I agree that modification times should not be restored by default, \n> because of the principle of least astonishment. But should it be \n> impossible? The principle of least astonishment does not mandate this; \n> it is not a paternalistic principle.\n> \n> Thus, I do not get at all\n> - why git doesn't *store* modification times, perhaps by default, but \n> at least on request\n> - why git doesn't restore modification times *on request*\n> \n> It is pretty annoying that git cannot, even if I know what I am doing, \n> and explicitly want it to, preserve the modification time.\n> \n> One use case: I have lots of file lying around in my build directory \n> and for some of them, the modification time in important information to \n> me. Those files are not at all used with the build tool. In contrast to \n> git pull, git pull --rebase needs those to be stashed. But after the \n> pull and unstash, the mtime is gone. Boo.\n> \n> Please provide options to store and restore modification times. It \n> shouldn't be hard to do, given that other metadata such as the mode is \n> already stored. It would make live so much easier. And the fact that \n> this has made into the FAQ clearly suggests that there are many others \n> who think so.\n\nSince you already assessed that it shouldn't be hard to do, you probably\nwant to put your money where your mouth is and come up with a patch, and\nthen offer it up for discussion on this here mailing list.\n\nCiao,\nJohannes\n"},{"id":"339645","messageId":"20180219220819.GA10466@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"nycvar.QRO.7.76.6.1802192257100.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: Git should preserve modification times at least on request","fromName":"Peter Backes","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-19T22:08:19Z","receivedAt":"2018-02-19T22:08:41Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"Hi Johannes,\n\nOn Mon, Feb 19, 2018 at 10:58:12PM +0100, Johannes Schindelin wrote:\n> Since you already assessed that it shouldn't be hard to do, you probably\n> want to put your money where your mouth is and come up with a patch, and\n> then offer it up for discussion on this here mailing list.\n\nWell, it would be good to discuss this a bit beforehand, since my time \nis wasted if there's no chance to get it accepted. Perhaps there is \nsome counterargument I don't know about.\n\nIs there some existing code that could be used? I think I read \nsomewhere that git once did preserve mtimes, but that this code was \nremoved because of the build tool issues. Perhaps that code could \nsimply be put back in, and surrounded by conditions.\n\nBest wishes\nPeter\n\nPS: Given the opportunity, I want to thank you very much for \nmaintaining the git repository for my cvsclone tool.\n\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339647","messageId":"007901d3a9d2$2edd7cd0$8c987670$@nexbridge.com","threadId":"47877","inReplyTo":"nycvar.QRO.7.76.6.1802192257100.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"RE: Git should preserve modification times at least on request","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-02-19T22:37:05Z","receivedAt":"2018-02-19T22:37:23Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On February 19, 2018 4:58 PM Johannes wrote:\n> On Mon, 19 Feb 2018, Peter Backes wrote:\n> \n> > please ensure to CC me if you reply as I am not subscribed to the list.\n> >\n> > https://git.wiki.kernel.org/index.php/Git_FAQ#Why_isn.27t_Git_preservi\n> > ng_modification_time_on_files.3F argues that git isn't preserving\n> > modification times because it needs to ensure that build tools work\n> > properly.\n> >\n> > I agree that modification times should not be restored by default,\n> > because of the principle of least astonishment. But should it be\n> > impossible? The principle of least astonishment does not mandate this;\n> > it is not a paternalistic principle.\n> >\n> > Thus, I do not get at all\n> > - why git doesn't *store* modification times, perhaps by default, but\n> > at least on request\n> > - why git doesn't restore modification times *on request*\n> >\n> > It is pretty annoying that git cannot, even if I know what I am doing,\n> > and explicitly want it to, preserve the modification time.\n> >\n> > One use case: I have lots of file lying around in my build directory\n> > and for some of them, the modification time in important information\n> > to me. Those files are not at all used with the build tool. In\n> > contrast to git pull, git pull --rebase needs those to be stashed. But\n> > after the pull and unstash, the mtime is gone. Boo.\n> >\n> > Please provide options to store and restore modification times. It\n> > shouldn't be hard to do, given that other metadata such as the mode is\n> > already stored. It would make live so much easier. And the fact that\n> > this has made into the FAQ clearly suggests that there are many others\n> > who think so.\n> \n> Since you already assessed that it shouldn't be hard to do, you probably\n> want to put your money where your mouth is and come up with a patch, and\n> then offer it up for discussion on this here mailing list.\n\nPutting my large-production-user hat on, there are (at least) three\nconditions that exist in this space:\n\n1. Build systems - this typically need the file modification time to be set\nto the time at which git touches a file (e.g., checkout). This permits build\nsystems to detect that files are modified (even if an older version is\nchecked out, make, for example, still needs to see the change to initiate a\nbuild. My understanding is that current git behaviour is modeled on this use\ncase.\n\n2. Commit linkage - in some environments, files that are checked out are set\nto the timestamp of the commit rather than the original file time or the\ncheckout time. This permits a faster production resolution of when changes\nwere run through the system as a group. I have implemented this strategy\n(somewhat grudgingly) in a few places. It is a possible desire for some\nusers. I particularly dislike this approach because merge/cherry-pick/rebase\ncan mess with the preceptive \"when\" of a change and if you are going to do\nthis, make sure that your metadata is suitably managed.\n\n3. Original file times - as Peter asked, storing the original file time has\nsome legacy advantages. This emulates the behaviour of some legacy SCM\nsystems and makes people feel better about things. From an audit point of\nview, this has value for systems other than git. In git, you use the\nhash-object to figure out what the file really is, so there is no real audit\nneed anymore for timestamps, which can be spoofed at whim anyway. The\nhash-object comment applies to 2 also. Same comment here for dealing with\nnon-touching but modifying. For example: what is the timestamp on a\nmerge-squash? I would contend that it is the time of the merge-squash, not\nthe original time. It could also be an interim term, when a conflict was\nresolved.\n\nJust remember that #2 and #3 break #1, unless you essentially rebuild from\nscratch in every build (ant/maven models). With that said, I seen many repo\nadmins who want all of the above, so making them all available would make\ntheir lives easier.\n\nMy $0.02. Cheers,\nRandall\n\n-- Brief whoami:\n  NonStop developer since approximately NonStop(211288444200000000)\n  UNIX developer since approximately 421664400\n-- In my real life, I talk too much.\n\n\n\n"},{"id":"339687","messageId":"CAE1pOi00dRYGgLbvep=pC1azAqc+qX=K1+iM-SZycygZyMBg6w@mail.gmail.com","threadId":"47877","inReplyTo":"007901d3a9d2$2edd7cd0$8c987670$@nexbridge.com","subject":"Re: Git should preserve modification times at least on request","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2018-02-19T23:22:40Z","receivedAt":"2018-02-19T23:23:07Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On Mon, Feb 19, 2018 at 2:37 PM, Randall S. Becker\n<rsbecker@nexbridge.com> wrote:\n> On February 19, 2018 4:58 PM Johannes wrote:\n>> On Mon, 19 Feb 2018, Peter Backes wrote:\n>>\n>> > please ensure to CC me if you reply as I am not subscribed to the list.\n>> >\n>> > https://git.wiki.kernel.org/index.php/Git_FAQ#Why_isn.27t_Git_preservi\n>> > ng_modification_time_on_files.3F argues that git isn't preserving\n>> > modification times because it needs to ensure that build tools work\n>> > properly.\n>> >\n>> > I agree that modification times should not be restored by default,\n>> > because of the principle of least astonishment. But should it be\n>> > impossible? The principle of least astonishment does not mandate this;\n>> > it is not a paternalistic principle.\n>> >\n>> > Thus, I do not get at all\n>> > - why git doesn't *store* modification times, perhaps by default, but\n>> > at least on request\n>> > - why git doesn't restore modification times *on request*\n>> >\n>> > It is pretty annoying that git cannot, even if I know what I am doing,\n>> > and explicitly want it to, preserve the modification time.\n>> >\n>> > One use case: I have lots of file lying around in my build directory\n>> > and for some of them, the modification time in important information\n>> > to me. Those files are not at all used with the build tool. In\n>> > contrast to git pull, git pull --rebase needs those to be stashed. But\n>> > after the pull and unstash, the mtime is gone. Boo.\n>> >\n>> > Please provide options to store and restore modification times. It\n>> > shouldn't be hard to do, given that other metadata such as the mode is\n>> > already stored. It would make live so much easier. And the fact that\n>> > this has made into the FAQ clearly suggests that there are many others\n>> > who think so.\n>>\n>> Since you already assessed that it shouldn't be hard to do, you probably\n>> want to put your money where your mouth is and come up with a patch, and\n>> then offer it up for discussion on this here mailing list.\n>\n> Putting my large-production-user hat on, there are (at least) three\n> conditions that exist in this space:\n>\n> 1. Build systems - this typically need the file modification time to be set\n> to the time at which git touches a file (e.g., checkout). This permits build\n> systems to detect that files are modified (even if an older version is\n> checked out, make, for example, still needs to see the change to initiate a\n> build. My understanding is that current git behaviour is modeled on this use\n> case.\n>\n> 2. Commit linkage - in some environments, files that are checked out are set\n> to the timestamp of the commit rather than the original file time or the\n> checkout time. This permits a faster production resolution of when changes\n> were run through the system as a group. I have implemented this strategy\n> (somewhat grudgingly) in a few places. It is a possible desire for some\n> users. I particularly dislike this approach because merge/cherry-pick/rebase\n> can mess with the preceptive \"when\" of a change and if you are going to do\n> this, make sure that your metadata is suitably managed.\n>\n> 3. Original file times - as Peter asked, storing the original file time has\n> some legacy advantages. This emulates the behaviour of some legacy SCM\n> systems and makes people feel better about things. From an audit point of\n> view, this has value for systems other than git. In git, you use the\n> hash-object to figure out what the file really is, so there is no real audit\n> need anymore for timestamps, which can be spoofed at whim anyway. The\n> hash-object comment applies to 2 also. Same comment here for dealing with\n> non-touching but modifying. For example: what is the timestamp on a\n> merge-squash? I would contend that it is the time of the merge-squash, not\n> the original time. It could also be an interim term, when a conflict was\n> resolved.\n>\n> Just remember that #2 and #3 break #1, unless you essentially rebuild from\n> scratch in every build (ant/maven models). With that said, I seen many repo\n> admins who want all of the above, so making them all available would make\n> their lives easier.\n\nAside from exactly which modification times should be used (which I\nwould love to have a bit more control over as well), something else\nI'd like to see is that, when switching between branches, files that\nare the same on both branches should not have their modification time\nchanged.\n"},{"id":"339689","messageId":"20180220012219.GA9791@thunk.org","threadId":"47877","inReplyTo":"20180219220819.GA10466@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2018-02-20T01:22:19Z","receivedAt":"2018-02-20T01:22:31Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, Feb 19, 2018 at 11:08:19PM +0100, Peter Backes wrote:\n> Is thetre some existing code that could be used? I think I read \n> somewhere that git once did preserve mtimes, but that this code was \n> removed because of the build tool issues. Perhaps that code could \n> simply be put back in, and surrounded by conditions.\n\nI don't believe that was ever true, because the mod times is simply\nnot *stored* anywhere.\n\nYou might want to consider trying to implement it as hook scripts\nfirst, and see how well/poorly it works for you.  I do have a use\ncase, which is to maintain the timestamps for guilt (a quilt-like\npatch management system which uses git).  At the moment I just use a\nmanual script, save-timestamps, which looks like this:\n\n#!/bin/sh\nstat -c \"touch -d @%Y %n\" * | sort -k 3 | grep -v \"~$\" | sort -k3 > timestamps\n\nand then I just include the timestamps file in thhe commit.  When I\nunpack the file elsewhere, I just run the command \". timestamps\", or\nif I am manually editing a single file, I might do:\n\n\tgrep file-name-of-patch timestamps | sht\n\nThis works because the timestamps file has lines which look like\nthis:\n\ntouch -d @1519007593 jbd2-clarify-recovery-checksum-error-msg\n\nI've been too lazy to automate this using a \"pre-commit\" and\n\"post-checkout\" hook, but it *really* wouldn't be that hard.  Right\nnow it also only works for files in the top-level of the repo, which\nis all I have in my guilt patch repo.  Making this work in a\nmultiple-directory environment is also left as an exercise to the\nreader.  :-)\n\nCheers,\n\n\t\t\t\t\t\t- Ted\n\nP.S.  Also left to the reader is making it work on legacy OS's like\nWindows.  :-)\n"},{"id":"339697","messageId":"nycvar.QRO.7.76.6.1802201127140.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"47877","inReplyTo":"20180219220819.GA10466@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-02-20T10:46:38Z","receivedAt":"2018-02-20T10:46:47Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peter,\n\nOn Mon, 19 Feb 2018, Peter Backes wrote:\n\n> On Mon, Feb 19, 2018 at 10:58:12PM +0100, Johannes Schindelin wrote:\n> > Since you already assessed that it shouldn't be hard to do, you\n> > probably want to put your money where your mouth is and come up with a\n> > patch, and then offer it up for discussion on this here mailing list.\n> \n> Well, it would be good to discuss this a bit beforehand, since my time \n> is wasted if there's no chance to get it accepted. Perhaps there is \n> some counterargument I don't know about.\n\nOh, sorry. I understood your mail as if you had told the core Git\ndevelopers that they should implement the feature you desire. I did not\nunderstand that you hinted at a discussion first, and that you would then\ngo and implement the feature you asked for.\n\n> Is there some existing code that could be used? I think I read \n> somewhere that git once did preserve mtimes, but that this code was \n> removed because of the build tool issues. Perhaps that code could \n> simply be put back in, and surrounded by conditions.\n\nI don't think that code was ever there. Maybe you heard about some file\nmode being preserved overzealously (we stored the octal file mode\nverbatim, but then decided to store only 644 or 755).\n\n(This is to add to Theodore's reply, giving a bit more depth.)\n\nAs you can see from the code decoding a tree entry:\n\nhttps://github.com/git-for-windows/git/blob/e1848984d/tree-walk.c#L25-L52\n\nthere is no mtime at all in the on-disk format of tree objects. There is\nthe hash, the mode, and the file name.\n\nAs your main use case would be stashing and unstashing (which uses tree\nobjects as storage format), this means you would have to find a different\nway to store the information you desire.\n\nIf I were you, and if I had the time to implement this feature, I would go\nabout it by adding a note (using `git notes` from a script first, but only\nfor proof of concept, because I saw too many things go wrong with Unix\nshell scripts in production) for the tree object, say, in\nrefs/notes/mtimes. I would probably invent a file format\n(`<mtime><TAB><path><LF>`) to store the information, and for starters I\nwould only store the mtimes of the files that were stashed, then extend\nthe script into a full Git builtin with a subcommand that can generates\nthese notes, a subcommand to replay them, and a subcommand to inspect\nthem.\n\nThen I would extend `git-stash.sh` to take an option (and later, to heed a\nnew config setting to do this automatically) to generate those mtime notes\nfor the newest stash's top-level tree object (storing only the times of\nthe files that were modified by the `stash` command), and to replay them\nif such an mtime note is found for the stash that is being applied.\n\nYou will not be able to convince the core Git developers to make this the\ndefault, I don't think. But if you make it an opt-in as I outlined above,\nI believe your chances would be good to get that feature if you put in the\neffort to implement it.\n\nOh, and if you implement the feature using notes, the same feature can be\nused not only for stashing and unstashing. These notes are maintained in\nregular Git refs, i.e. they can be shared. And since those notes would be\nfor tree objects, you could even apply the mtimes on a fresh clone, if\nyou have a use case for that.\n\nCiao,\nJohannes\n"},{"id":"339703","messageId":"20180220115350.GA18760@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"nycvar.QRO.7.76.6.1802201127140.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: Git should preserve modification times at least on request","fromName":"Peter Backes","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-20T11:53:50Z","receivedAt":"2018-02-20T11:54:24Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"Hello Johannes,\n\nOn Tue, Feb 20, 2018 at 11:46:38AM +0100, Johannes Schindelin wrote:\n> Oh, sorry. I understood your mail as if you had told the core Git\n> developers that they should implement the feature you desire. I did not\n> understand that you hinted at a discussion first, and that you would then\n> go and implement the feature you asked for.\n\nWell, sorry for being misunderstandable. It was my impression from the \nFAQ that the reason for why this feature doesn't exist was a strong \nopinion that it would cause technical problems. The FAQ doesn't mention \nanything like a lack of manpower. As I stated it was my \nimpression that this feature would not be too hard to implement.\n\nBecause of this my email presupposed it was not manpower that prevented \nthis feature.\n\nMy statement \"Please provide options\" was thus targeted at reviewing \nand discussing the perceived technical reasons for not implementing \nthis feature at least as an option. It wasn't supposed to demand free \nlunch from anyone.\n\nOf course I can offer to do some work to the best of my abilitites if \nthat's the issue. That should go without saying for Free Software \nprojects. Perhaps even my employer would be happy to pay me for \nimplementing the feature during workign hours. This shouldn't be the \nissue. The issue is the seemingly dogmatic reply in the FAQ which makes \nme reluctant to put work into this in fear that a patch submission \nwould be met with strong rejection.\n\n> You will not be able to convince the core Git developers to make this the\n> default, I don't think.\n\nI have stressed very clearly in my mail that I am not asking the \ndefaults about mtime restoring to be changed. I agree that those \ndefaults are reasonable and in line with the principle of least \nastonishment.\n\nWhat bugs me is my impression from the FAQ that even as an option, the \nfeature might be unwelcome.\n\nBest wishes\nPeter\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339714","messageId":"CAE1pOi1FMbdrgpN44BN3MxXoMLA=osPSFiLLHN-FEYABb=NPzA@mail.gmail.com","threadId":"47877","inReplyTo":"CAE1pOi00dRYGgLbvep=pC1azAqc+qX=K1+iM-SZycygZyMBg6w@mail.gmail.com","subject":"Re: Git should preserve modification times at least on request","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2018-02-20T16:42:39Z","receivedAt":"2018-02-20T16:43:06Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On Mon, Feb 19, 2018 at 3:22 PM, Hilco Wijbenga\n<hilco.wijbenga@gmail.com> wrote:\n> Aside from exactly which modification times should be used (which I\n> would love to have a bit more control over as well), something else\n> I'd like to see is that, when switching between branches, files that\n> are the same on both branches should not have their modification time\n> changed.\n\nAs Junio pointed out to me, Git actually already does what I want when\nswitching branches. To verify, I switched between 5 branches after\nsetting a specific timestamp on a particular file, and it did not\nchange throughout the process. Now I'm left wondering when this\nchanged or whether my memory is faulty. I could have sworn this did\nnot work previously. :-)\n"},{"id":"339743","messageId":"20180220210554.GA24474@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"nycvar.QRO.7.76.6.1802201127140.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: Git should preserve modification times at least on request","fromName":"Peter Backes","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-20T21:05:54Z","receivedAt":"2018-02-20T21:06:15Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"Hi Johannes,\n\nOn Tue, Feb 20, 2018 at 11:46:38AM +0100, Johannes Schindelin wrote:\n> If I were you [...]\n\nIt seems all pretty straight forward, except for\n\n> I would probably invent a file format (`<mtime><TAB><path><LF>`)\n\nI'm stuck there because of <path> being munged.\n\nTo obtain or set the mtime of the file, I need the unmunged path.\n\nHow to get it?\n\n----\n\nWhat follows is irrelevant for progress.\n\n> I don't think that code was ever there. Maybe you heard about some file\n> mode being preserved overzealously (we stored the octal file mode\n> verbatim, but then decided to store only 644 or 755).\n\nI'm not sure. I'm not able to find that source anymore, though.\n\n> As you can see from the code decoding a tree entry:\n> \n> https://github.com/git-for-windows/git/blob/e1848984d/tree-walk.c#L25-L52\n> \n> there is no mtime at all in the on-disk format of tree objects. There is\n> the hash, the mode, and the file name.\n\nI didn't comletely get the code in tree-walk.c since the parsing \narchitecture seems to pass around pointers via global variables. \nIt seems that in addition to hash, mode and file name, the on-disk \nformat has at least the object type, see git cat-file -p master^{tree} \nPerhaps I got it wrong.\n\nBest wishes\nPeter\n\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339745","messageId":"20180220211634.GA15232@sigill.intra.peff.net","threadId":"47877","inReplyTo":"20180219212235.GA9891@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-20T21:16:34Z","receivedAt":"2018-02-20T21:16:41Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 19, 2018 at 10:22:36PM +0100, Peter Backes wrote:\n\n> please ensure to CC me if you reply as I am not subscribed to the list.\n> \n> https://git.wiki.kernel.org/index.php/Git_FAQ#Why_isn.27t_Git_preserving_modification_time_on_files.3F \n> argues that git isn't preserving modification times because it needs to \n> ensure that build tools work properly.\n\nI think there are some references buried somewhere in that wiki, but did\nyou look at any of the third-party tools that store file metadata\nalongside the files in the repository? E.g.:\n\n  https://etckeeper.branchable.com/\n\nor\n\n  https://github.com/przemoc/metastore\n\nI didn't see either of those mentioned in this thread (though I also do\nnot have personal experience with them, either).\n\nModification times are a subset of the total metadata you might care\nabout, so they are solving a much more general problem. Which may also\npartially answer your question about why this isn't built into git. The\ngeneral problem gets much bigger when you start wanting to carry things\nlike modes (which git doesn't actually track; we really only care about\nthe executable bit) or extended attributes (acls, etc).\n\n-Peff\n"},{"id":"339750","messageId":"20180220220525.GA25134@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"20180220211634.GA15232@sigill.intra.peff.net","subject":"Re: Git should preserve modification times at least on request","fromName":"Peter Backes","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-20T22:05:25Z","receivedAt":"2018-02-20T22:05:50Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"Hi Jeff,\n\nOn Tue, Feb 20, 2018 at 04:16:34PM -0500, Jeff King wrote:\n> I think there are some references buried somewhere in that wiki, but did\n> you look at any of the third-party tools that store file metadata\n> alongside the files in the repository? E.g.:\n> \n>   https://etckeeper.branchable.com/\n> \n> or\n> \n>   https://github.com/przemoc/metastore\n> \n> I didn't see either of those mentioned in this thread (though I also do\n> not have personal experience with them, either).\n> \n> Modification times are a subset of the total metadata you might care\n> about, so they are solving a much more general problem. Which may also\n> partially answer your question about why this isn't built into git. The\n> general problem gets much bigger when you start wanting to carry things\n> like modes (which git doesn't actually track; we really only care about\n> the executable bit) or extended attributes (acls, etc).\n\nI know about those, but that's not what I am looking for. Those tools \nserve entirely different purposes, ie., tracking file system changes. \nI, however, am specifically interested in version control.\n\nIn version control, the user checks out his own copy of the tree for \nworking. For this purpose, it is thus pointless to track ownership, \npermissions (except for the x bit), xattrs, or any other metadata. In \nfact, it can be considered the wrong thing to do.\n\nThe modification time, however, is special. It clearly has its place in \nversion control. It tells us when the last modification was actually \ndone to the file. I am often working on some feature, and one part is \nfinished and is lying around, but I am still working on other parts in \nother files. Then, maybe after some weeks, the other parts are \nfinished. Now, when committing, the information about modification time \nis lost. Maybe some weeks later I want to figure out when I last \nmodified those files that were committed. But that information is now \ngone, at least in the git repository. Sure, I could do lots of WIP \ncommits, but this would clutter up the history unneccessarly and I \nwould have lots of versions that might not even compile, let alone run.\n\nAs far as I remember, bitkeeper had this distinction between checkins \nand commits. You could check in a file at any time, and any number of \ntimes, and then group all those checkins together with a commit. Git \nseems to have avoided this principle, or have kept it only \nrudimentarily via git add (but git add cannot add more than one version \nof the same file). Perhaps for simplificiation of use, perhaps for \nsimplification of implementation, I don't know.\n\nI assume, if it were not for the build tool issues, git would have \ntracked mtime from the very start.\n\nBest wishes\nPeter\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339756","messageId":"nycvar.QRO.7.76.6.1802202329540.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"47877","inReplyTo":"20180220210554.GA24474@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-02-20T22:32:23Z","receivedAt":"2018-02-20T22:32:32Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peter,\n\nOn Tue, 20 Feb 2018, Peter Backes wrote:\n\n> On Tue, Feb 20, 2018 at 11:46:38AM +0100, Johannes Schindelin wrote:\n> \n> > I would probably invent a file format (`<mtime><TAB><path><LF>`)\n> \n> I'm stuck there because of <path> being munged.\n\nFrom which command do you want to get it? If you are looking at `git\ndiff`, you may want to use the `-z --name-only` options to avoid munging\nthe paths.\n\nOr are you looking at a different source for the paths?\n\nCiao,\nJohannes\n"},{"id":"339759","messageId":"20180220224808.GA25678@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"nycvar.QRO.7.76.6.1802202329540.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: Git should preserve modification times at least on request","fromName":"Peter Backes","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-20T22:48:09Z","receivedAt":"2018-02-20T22:48:30Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"On Tue, Feb 20, 2018 at 11:32:23PM +0100, Johannes Schindelin wrote:\n> Hi Peter,\n> \n> On Tue, 20 Feb 2018, Peter Backes wrote:\n> \n> > On Tue, Feb 20, 2018 at 11:46:38AM +0100, Johannes Schindelin wrote:\n> > \n> > > I would probably invent a file format (`<mtime><TAB><path><LF>`)\n> > \n> > I'm stuck there because of <path> being munged.\n> \n> From which command do you want to get it? If you are looking at `git\n> diff`, you may want to use the `-z --name-only` options to avoid munging\n> the paths.\n\nI plan to use \"git diff-tree --name-only $w_tree HEAD\" and subtract\nall lines from \"git diff-index --name-only HEAD\" to get the files for \nwhich the timestamp should be stored..\n\nIf I use \"-z\" I get the non-munged path, but I cannot safely store such \npaths in the proposed file format; they might contain newlines (sigh). \nSo at one point I have to munge. Then the same question arises when I \nhave to get the actual path from the munged path when restoring the \ntimestamps.\n\nIf there's no ready-made functionality to munge and unmunge paths, I \nhave to write some awk for this. At first I thought this might add one \nmore dependency to git, but it seems that awk is already used in \ngit-mergetool.sh, so I suppose it's okay to use in git-stash.sh etc, \ntoo.\n\nBest wishes\nPeter\n\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339764","messageId":"xmqqeflf1b82.fsf@gitster-ct.c.googlers.com","threadId":"47877","inReplyTo":"20180220211634.GA15232@sigill.intra.peff.net","subject":"Re: Git should preserve modification times at least on request","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-20T23:40:29Z","receivedAt":"2018-02-20T23:40:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Modification times are a subset of the total metadata you might care\n> about, so they are solving a much more general problem. Which may also\n> partially answer your question about why this isn't built into git. The\n> general problem gets much bigger when you start wanting to carry things\n> like modes (which git doesn't actually track; we really only care about\n> the executable bit) or extended attributes (acls, etc).\n\n\"modes\" are interesting, especially when you think about group\npermissions, as it would make you design how you store group and\nowner, which in turn forces you to think how \"peter\" on one system\nrelates to \"peter\" on another system (answer: there generally isn't\nany relationship) ;-)\n\n"},{"id":"339799","messageId":"CA+P7+xr=HXGi9ufeC5=Sm7bR3csxZPMExjD=QDrc2T9Pp46Yjg@mail.gmail.com","threadId":"47877","inReplyTo":"20180220220525.GA25134@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2018-02-21T09:48:25Z","receivedAt":"2018-02-21T09:48:55Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Tue, Feb 20, 2018 at 2:05 PM, Peter Backes <rtc@helen.plasma.xg8.de> wrote:\n> Hi Jeff,\n>\n> On Tue, Feb 20, 2018 at 04:16:34PM -0500, Jeff King wrote:\n>> I think there are some references buried somewhere in that wiki, but did\n>> you look at any of the third-party tools that store file metadata\n>> alongside the files in the repository? E.g.:\n>>\n>>   https://etckeeper.branchable.com/\n>>\n>> or\n>>\n>>   https://github.com/przemoc/metastore\n>>\n>> I didn't see either of those mentioned in this thread (though I also do\n>> not have personal experience with them, either).\n>>\n>> Modification times are a subset of the total metadata you might care\n>> about, so they are solving a much more general problem. Which may also\n>> partially answer your question about why this isn't built into git. The\n>> general problem gets much bigger when you start wanting to carry things\n>> like modes (which git doesn't actually track; we really only care about\n>> the executable bit) or extended attributes (acls, etc).\n>\n> I know about those, but that's not what I am looking for. Those tools\n> serve entirely different purposes, ie., tracking file system changes.\n> I, however, am specifically interested in version control.\n>\n> In version control, the user checks out his own copy of the tree for\n> working. For this purpose, it is thus pointless to track ownership,\n> permissions (except for the x bit), xattrs, or any other metadata. In\n> fact, it can be considered the wrong thing to do.\n>\n> The modification time, however, is special. It clearly has its place in\n> version control. It tells us when the last modification was actually\n> done to the file. I am often working on some feature, and one part is\n> finished and is lying around, but I am still working on other parts in\n> other files. Then, maybe after some weeks, the other parts are\n> finished. Now, when committing, the information about modification time\n> is lost. Maybe some weeks later I want to figure out when I last\n> modified those files that were committed. But that information is now\n> gone, at least in the git repository. Sure, I could do lots of WIP\n> commits, but this would clutter up the history unneccessarly and I\n> would have lots of versions that might not even compile, let alone run.\n\nYou could have git figure this out by the commit time of the last\ncommit which modified a file. This gets a bit weird for cherry-picks\nor other things like rebase, but that should get what you want.\n\nIf you only ever need this information sometimes, you can look it up\nby doing something like:\n\ngit log -1 --pretty=\"%cd\" -- <path to file>\n\nThat should show the commit time of the latest commit which touches\nthat file, which is \"essentially\" the modify time of the file in terms\nof  the version control history.\n\nObviously, this wouldn't work if you continually amend a change of\nmultiple files, since git wouldn't track the files separately, and\nthis only really shows you the time of the last commit.\n\nHowever, in \"version control\" sense, this *is* the last time a file\nwas modified, since it doesn't really care about the stuff that\nhappens outside of version control.\n\nI'm not really sure if this is enough for you, or if you really want\nto store the actual mtime for some reason? (I think you can likely\nsolve your problem in some other way though).\n\n>\n> As far as I remember, bitkeeper had this distinction between checkins\n> and commits. You could check in a file at any time, and any number of\n> times, and then group all those checkins together with a commit. Git\n> seems to have avoided this principle, or have kept it only\n> rudimentarily via git add (but git add cannot add more than one version\n> of the same file). Perhaps for simplificiation of use, perhaps for\n> simplification of implementation, I don't know.\n>\n\nYou can do lots of commits on a branch and then one merge commit to\nmerge it into the main line. This is a common strategy used by many\npeople.\n\nThanks,\nJake\n\n> I assume, if it were not for the build tool issues, git would have\n> tracked mtime from the very start.\n>\n\nMaybe. Personally, I would hate having my mtime not be \"the time I\nchecked the file out\", since this is intuitive to me at this point.\nI'm sure if I lived in a different world I'd be used to that way also,\nthough.\n\nThe build issue *is* important though, because many build systems rely\non the mtime to figure out what to rebuild, and a complete rebuild\nisn't a good idea for very large projects.\n\nThanks,\nJake\n\n> Best wishes\n> Peter\n> --\n> Peter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339836","messageId":"20180221210339.GA43094@accordion.employees.org","threadId":"47877","inReplyTo":"20180219212235.GA9891@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Derek Fawcus","fromEmail":"dfawcus+lists-git@employees.org","sentAt":"2018-02-21T21:03:39Z","receivedAt":"2018-02-21T21:10:12Z","isPatch":false,"sender":{"key":"dfawcus+lists-git@employees.org","avatar":null},"body":"On Mon, Feb 19, 2018 at 10:22:36PM +0100, Peter Backes wrote:\n> \n> It is pretty annoying that git cannot, even if I know what I am doing, \n> and explicitly want it to, preserve the modification time.\n\nThe use case I've come across where it would be of value is for code\narcheology, either importing a bunch of tar files, or importing a\nrepo from some other VCS.\n\nThere preserving the mod times can be useful when one is subsequently\nfiguring out what changed, and the scope of the 'commits' is too big\n(i.e. the granularity of the tar files themselves).\n\ne.g. initial commits are done on tar boundaries, but one may try to\nfigure out individual changes from a ChangeLog file.  I've done this\na couple of times, but to date it has required keeping the untarred\ntrees around (or a timestamp list file from each tree), in addition\nto the git repro in to which one is then synthesizing smaller commits.\n\nDF\n"},{"id":"339837","messageId":"0a9f7d75-b362-bb9d-48ef-c4a6a921ff96@talktalk.net","threadId":"47877","inReplyTo":"20180220224808.GA25678@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Phillip Wood","fromEmail":"phillip.wood@talktalk.net","sentAt":"2018-02-21T21:30:50Z","receivedAt":"2018-02-21T21:30:57Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 20/02/18 22:48, Peter Backes wrote:\n> \n> On Tue, Feb 20, 2018 at 11:32:23PM +0100, Johannes Schindelin wrote:\n>> Hi Peter,\n>>\n>> On Tue, 20 Feb 2018, Peter Backes wrote:\n>>\n>>> On Tue, Feb 20, 2018 at 11:46:38AM +0100, Johannes Schindelin wrote:\n>>>\n>>>> I would probably invent a file format (`<mtime><TAB><path><LF>`)\n>>>\n>>> I'm stuck there because of <path> being munged.\n>>\n>>  From which command do you want to get it? If you are looking at `git\n>> diff`, you may want to use the `-z --name-only` options to avoid munging\n>> the paths.\n> \n> I plan to use \"git diff-tree --name-only $w_tree HEAD\" and subtract\n> all lines from \"git diff-index --name-only HEAD\" to get the files for\n> which the timestamp should be stored..\n> \n> If I use \"-z\" I get the non-munged path, but I cannot safely store such\n> paths in the proposed file format; they might contain newlines (sigh).\n> So at one point I have to munge. Then the same question arises when I\n> have to get the actual path from the munged path when restoring the\n> timestamps.\n> \n> If there's no ready-made functionality to munge and unmunge paths, I\n> have to write some awk for this. At first I thought this might add one\n> more dependency to git, but it seems that awk is already used in\n> git-mergetool.sh, so I suppose it's okay to use in git-stash.sh etc,\n> too.\n\nIn recent versions of git there's unquote_path() in Git.pm, you could \npossibly use that with perl -e from your script\n\nBest Wishes\n\nPhillip\n> Best wishes\n> Peter\n> \n\n"},{"id":"339838","messageId":"87bmgif2pa.fsf@evledraar.gmail.com","threadId":"47877","inReplyTo":"20180221210339.GA43094@accordion.employees.org","subject":"Re: Git should preserve modification times at least on request","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-02-21T21:33:05Z","receivedAt":"2018-02-21T21:33:13Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 21 2018, Derek Fawcus jotted:\n\n> On Mon, Feb 19, 2018 at 10:22:36PM +0100, Peter Backes wrote:\n>>\n>> It is pretty annoying that git cannot, even if I know what I am doing,\n>> and explicitly want it to, preserve the modification time.\n>\n> The use case I've come across where it would be of value is for code\n> archeology, either importing a bunch of tar files, or importing a\n> repo from some other VCS.\n>\n> There preserving the mod times can be useful when one is subsequently\n> figuring out what changed, and the scope of the 'commits' is too big\n> (i.e. the granularity of the tar files themselves).\n>\n> e.g. initial commits are done on tar boundaries, but one may try to\n> figure out individual changes from a ChangeLog file.  I've done this\n> a couple of times, but to date it has required keeping the untarred\n> trees around (or a timestamp list file from each tree), in addition\n> to the git repro in to which one is then synthesizing smaller commits.\n\nThis sounds like a sensible job for a git import tool, i.e. import a\ntarget directory into git, and instead of 'git add'-ing the whole thing\nit would look at the mtimes, sort files by mtime, then add them in order\nand only commit those files that had the same mtime in the same commit\n(or within some boundary).\n\nThe advantage of doing this via such a tool is that you could tweak it\nto commit by any criteria you wanted, e.g. not mtime but ctime or even\natime.\n\nYou'd get the same thing as you'd get if git's tree format would change\nto include mtimes (which isn't going to happen), but with a lot more\nflexibility.\n"},{"id":"339844","messageId":"20180221221420.GA7743@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"87bmgif2pa.fsf@evledraar.gmail.com","subject":"Re: Git should preserve modification times at least on request","fromName":"Peter Backes","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-21T22:14:20Z","receivedAt":"2018-02-21T22:14:51Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"On Wed, Feb 21, 2018 at 10:33:05PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> This sounds like a sensible job for a git import tool, i.e. import a\n> target directory into git, and instead of 'git add'-ing the whole thing\n> it would look at the mtimes, sort files by mtime, then add them in order\n> and only commit those files that had the same mtime in the same commit\n> (or within some boundary).\n\nI think that this would be The Wrong Thing to do.\n\nThe commit time is just that: The time the commit was done. The commit \nis an atomic group of changes to a number of files that hopefully bring \nthe tree from one usable state into the next.\n\nThe mtime, in contrast, tells us when a file was most recently modified.\n\nIt may well be that main.c was most recently modified yesterday, and \nfeature.c was modified this morning, and that only both changes taken \ntogether make sense as a commit, despite the long time in between.\n\nEven worse, it may be that feature A took a long time to implement, so \nwe have huge gaps in between the mtimes, but feature B was quickly done \nafter A was finished. Such an algorithm would probably split feature A \nincorrectly into several commits, and group the more recently changed \nfiles of feature A with those of feature B.\n\nAnd if Feature A and Feature B were developed in parallel, things get \ncompletely messy.\n\n> The advantage of doing this via such a tool is that you could tweak it\n> to commit by any criteria you wanted, e.g. not mtime but ctime or even\n> atime.\n\nMaybe, but it would be rather useless to commit by ctime or atime. You \ndo one grep -r and the atime is different. You do one chmod or chown \nand the ctime is different. Those timestamps are really only useful for \nvery limited purposes.\n\nThat ctime exists seems reasonable, since it's only ever updated when \nthe inode is written anyway.\n\natime, in contrast, was clearly one of the rather nonsensical \ninnovations of UNIX: Do one write to the disk for each read from the \ndisk. C'mon, really? It would have been a lot more reasonable to simply \nprovide a generic way for tracing read() system calls instead; then \nuserspace could decide what to do with that information and which of it \nis useful and should be kept and perhaps stored on disk. Now we have \nthis ugly hack called relatime to deal with the problem.\n\n> You'd get the same thing as you'd get if git's tree format would change\n> to include mtimes (which isn't going to happen), but with a lot more\n> flexibility.\n\nWell, from basic logic, I don't see how a decision not to implement a \nfeature could possibly increase flexility. The opposite seems to be the \ncase.\n\nBest wishes\nPeter\n\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339849","messageId":"87a7w2ezeq.fsf@evledraar.gmail.com","threadId":"47877","inReplyTo":"20180221221420.GA7743@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-02-21T22:44:13Z","receivedAt":"2018-02-21T22:44:23Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 21 2018, Peter Backes jotted:\n\n> On Wed, Feb 21, 2018 at 10:33:05PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> This sounds like a sensible job for a git import tool, i.e. import a\n>> target directory into git, and instead of 'git add'-ing the whole thing\n>> it would look at the mtimes, sort files by mtime, then add them in order\n>> and only commit those files that had the same mtime in the same commit\n>> (or within some boundary).\n>\n> I think that this would be The Wrong Thing to do.\n\nI'm merely pointing out that if you have the use-case Derek Fawcus\ndescribes you can get per-file mtimes via something similar to the the\nhook method Theodore Ts'o described today with a simple import tool with\nno changes to git or its object format required.\n\nTo the extent that there's a convention for this in git that's the\nconvention, e.g. if you use github or gitlab they'll render the\nmodification time of a file in the tree view, and that time is the time\nof the commit that last touched it: https://github.com/git/git\n\n> The commit time is just that: The time the commit was done. The commit\n> is an atomic group of changes to a number of files that hopefully bring\n> the tree from one usable state into the next.\n>\n> The mtime, in contrast, tells us when a file was most recently modified.\n>\n> It may well be that main.c was most recently modified yesterday, and\n> feature.c was modified this morning, and that only both changes taken\n> together make sense as a commit, despite the long time in between.\n>\n> Even worse, it may be that feature A took a long time to implement, so\n> we have huge gaps in between the mtimes, but feature B was quickly done\n> after A was finished.\n\n...\n\n> Such an algorithm would probably split feature A\n> incorrectly into several commits, and group the more recently changed\n> files of feature A with those of feature B.\n\nRight, but that's a trade-off you can pick at import time in this\nhypothetical tar-to-commits tool, you could decide to do no merging and\nsuffer to signal loss.\n\n> And if Feature A and Feature B were developed in parallel, things get\n> completely messy.\n>\n>> The advantage of doing this via such a tool is that you could tweak it\n>> to commit by any criteria you wanted, e.g. not mtime but ctime or even\n>> atime.\n>\n> Maybe, but it would be rather useless to commit by ctime or atime. You\n> do one grep -r and the atime is different. You do one chmod or chown\n> and the ctime is different. Those timestamps are really only useful for\n> very limited purposes.\n>\n> That ctime exists seems reasonable, since it's only ever updated when\n> the inode is written anyway.\n>\n> atime, in contrast, was clearly one of the rather nonsensical\n> innovations of UNIX: Do one write to the disk for each read from the\n> disk. C'mon, really? It would have been a lot more reasonable to simply\n> provide a generic way for tracing read() system calls instead; then\n> userspace could decide what to do with that information and which of it\n> is useful and should be kept and perhaps stored on disk. Now we have\n> this ugly hack called relatime to deal with the problem.\n\nYes, that [ac]time example was a stretch. A better example would be\ncommitting the file mode, or extended attributes, or \"this is on a\ndifferent FS\", or whatever other per-file/dir attribute we're not\ncurrently capturing.\n\n>> You'd get the same thing as you'd get if git's tree format would change\n>> to include mtimes (which isn't going to happen), but with a lot more\n>> flexibility.\n>\n> Well, from basic logic, I don't see how a decision not to implement a\n> feature could possibly increase flexility. The opposite seems to be the\n> case.\n\nI'm not trying to argue the usefulness of this mtime-per-file thing in\ntheory, just providing Derek Fawcus with a suggestion for a viable\nworkaround.\n\nWhat I meant by this offhand comment, and which you may or may not know\n(and I see no references to it from skimming the thread) is that there's\nsimply no space in the tree objects to add *anything* without breaking\nthe object format and requiring a major upgrade, although the plan to\nswitch to a new hash function is relevant to this.\n\nEven if we suppose that git was being implemented today I don't think\nthis would make any sense as a first-level feature.\n\nEmpirical evidence suggests that people use git on a massive scale\nlargely without caring about this, and the users who do have a\nworkaround.\n\nIf it were added as a first-level feature to git it would present a lot\nof UX confusion. E.g. you run \"git add\" and it'll be showing the mtime\nsomehow, or you get a formatted patch over E-Mail and it doesn't only\ninclude the commit time but also times for individual files.\n\nThe VC systems that had this feature in the past were centralized, so\nthey could (in theory anyway) ensure that timestamps were monotonically\nincreasing. This won't be the case with git, we have plenty of timestamp\ndrift in e.g. linux.git and other git repos.\n\nSo if these mtimes were used by default they'd interact badly with stuff\nlike \"make\" in those cases, because you might check out a modified\nversion with a timestamp in the past.\n\nOr maybe I've misunderstood how that worked in CVS/SVN/Bitkeeper, but in\nany case, I just wanted to point out a workaround (but then digressed\ninto critiquing the idea above...).\n"},{"id":"339857","messageId":"20180221231234.GA8509@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"87a7w2ezeq.fsf@evledraar.gmail.com","subject":"Re: Git should preserve modification times at least on request","fromName":"Peter Backes","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-21T23:12:34Z","receivedAt":"2018-02-21T23:13:08Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"On Wed, Feb 21, 2018 at 11:44:13PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> If it were added as a first-level feature to git it would present a lot\n> of UX confusion. E.g. you run \"git add\" and it'll be showing the mtime\n> somehow, or you get a formatted patch over E-Mail and it doesn't only\n> include the commit time but also times for individual files.\n\nBut that's pretty standard. patch format has timestamp fields for \nprecisely this purpose:\n\n% echo a > x  \n% echo b > y\n% diff -u x y\n--- x\t2018-02-21 23:56:29.574029523 +0100\n+++ y\t2018-02-21 23:56:31.430003389 +0100\n@@ -1 +1 @@\n-a\n+b\n\nAt present, git simply leaves those fields blank...\n\n> The VC systems that had this feature in the past were centralized, so\n> they could (in theory anyway) ensure that timestamps were monotonically\n> increasing. This won't be the case with git, we have plenty of timestamp\n> drift in e.g. linux.git and other git repos.\n\nI don't see where monotonicity would be an issue any more than it is \nfor centralized version control systems.\n\nEven in the centralized setting, monotonicity is not guaranteed, since \nyou might have local timestamps deviating from the repository; you \nmight have added a line, compiled, and removed it again later on, \nwithout running make again. Now if you checkout changes from the \nrepository, and it sets the timestamp, that timestamp might be older \nthan before the compile, and the file would not be rebuilt if you run \nmake. So you cannot avoid those issues in centralized setttings either.\n\n> So if these mtimes were used by default they'd interact badly with stuff\n> like \"make\" in those cases, because you might check out a modified\n> version with a timestamp in the past.\n\nThat's very clearly the case, and I have stressed in my initial email \nthat I fully agree with the reasoning of the FAQ in this regard. It is, \nhowever, merely an argument against *restoring* the timestamps *by \ndefault*, to comply with the principle of least astonishment. It is, by \nitself, not an argument against *storing* the timestamps, let alone \nagainst restoring them *on request*.\n\nFor the initial checkout, it should not even be harmful to restore the \ntimestamps by default.\n\n> any case, I just wanted to point out a workaround (but then digressed\n> into critiquing the idea above...).\n\nWell, Johannes's proposed solution seems pretty reasonable and \nrealistic to me.  Thanks to Phillip's hint about unquote_path() in \nGit.pm it seems I now have all the needed ingredients to implement this \nfeature.\n\nBest wishes\nPeter\n\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339871","messageId":"007d01d3ab6f$e5439f10$afcadd30$@nexbridge.com","threadId":"47877","inReplyTo":"20180221231234.GA8509@helen.PLASMA.Xg8.DE","subject":"RE: Git should preserve modification times at least on request","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-02-21T23:58:34Z","receivedAt":"2018-02-21T23:58:55Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On February 21, 2018 6:13 PM, Peter Backes wrote:\n> On Wed, Feb 21, 2018 at 11:44:13PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > If it were added as a first-level feature to git it would present a\n> > lot of UX confusion. E.g. you run \"git add\" and it'll be showing the\n> > mtime somehow, or you get a formatted patch over E-Mail and it doesn't\n> > only include the commit time but also times for individual files.\n> \n> But that's pretty standard. patch format has timestamp fields for\nprecisely\n> this purpose:\n> \n> % echo a > x\n> % echo b > y\n> % diff -u x y\n> --- x\t2018-02-21 23:56:29.574029523 +0100\n> +++ y\t2018-02-21 23:56:31.430003389 +0100\n\nMay I suggest storing the date/time in UTC+0 in all cases. I can see\npotential issues a couple of times a year where holes exist. I cannot even\nfathom what would happen on a merge or edit of history.\n\nCheers,\nRandall\n\n"},{"id":"339897","messageId":"20180222020535.GA11063@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"007d01d3ab6f$e5439f10$afcadd30$@nexbridge.com","subject":"Re: Git should preserve modification times at least on request","fromName":"'Peter Backes'","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-22T02:05:35Z","receivedAt":"2018-02-22T02:06:09Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"On Wed, Feb 21, 2018 at 06:58:34PM -0500, Randall S. Becker wrote:\n> May I suggest storing the date/time in UTC+0 in all cases. I can see\n> potential issues a couple of times a year where holes exist. I cannot even\n> fathom what would happen on a merge or edit of history.\n\nI consider storing the timestamp simply in the traditional \nseconds-since-epoch UNIX timestamp format. But I'm not entirely sure \nyet (see below).\n\nIf a timestamp includes the offset, there shouldn't be any issue with \nholes. UTC+0 is nice, too, of course, though some might want to \npreserve the timezone in which the timestamp was actually created.\n\nThe bigger issue is usually to copy with those pesky leap seconds. It \nmakes a difference whether one uses solar seconds (\"posix\" style; those \nare more commonly seen) or atomic seconds (\"right\" style) for the UNIX \ntimestamp. Those differences accumulate over time, so you can have \nalmost half a minute delta if you are not careful with timestamp \nconversion. If I remember correctly, rcs uses some rather awkward \ninterative convergence algorithm to portably convert from \nhuman-readable date and time to UNIX timestamps.\n\nThus I'm still not sure whether it will be a UNIX-format timestamp or \nwhether a human-readable date/time might be preferrable.\n\nBest wishes\nPeter\n\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"339989","messageId":"20180222232411.GA54558@accordion.employees.org","threadId":"47877","inReplyTo":"87a7w2ezeq.fsf@evledraar.gmail.com","subject":"Re: Git should preserve modification times at least on request","fromName":"Derek Fawcus","fromEmail":"dfawcus+lists-git@employees.org","sentAt":"2018-02-22T23:24:11Z","receivedAt":"2018-02-22T23:24:17Z","isPatch":false,"sender":{"key":"dfawcus+lists-git@employees.org","avatar":null},"body":"On Wed, Feb 21, 2018 at 11:44:13PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, Feb 21 2018, Peter Backes jotted:\n> > On Wed, Feb 21, 2018 at 10:33:05PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> >> This sounds like a sensible job for a git import tool, i.e. import a\n> >> target directory into git, and instead of 'git add'-ing the whole thing\n> >> it would look at the mtimes, sort files by mtime, then add them in order\n> >> and only commit those files that had the same mtime in the same commit\n> >> (or within some boundary).\n> >\n> > I think that this would be The Wrong Thing to do.\n\nAgreed, but probably for a different reason.\n\n> I'm merely pointing out that if you have the use-case Derek Fawcus\n> describes you can get per-file mtimes via something similar to the the\n> hook method Theodore Ts'o described today with a simple import tool with\n> no changes to git or its object format required.\n\nActually, I was not proposing any change to the git objects.\nI was simply suggesting a case where I'd have found a optional mechanism\nfor mtime restoration useful.\n\nWhat would be useful is a better version of the hook based scheme which\nTed mentioned.  The import could be via a wrapper script, but checkouts\nwould have to be via a hook such that the original timestamps could then\nbe applied; and those stamps would have to be part of the tar-file commit.\n\nThe idea of automatically generating a bunch of commits in time order\nwould be the wrong thing here. That is because one file could well\ncontain changes from more than one logical commit (as guided by the\nChangelog), and that one logical commit can be spread across a few\nfiles with diffrent mode time, one has to manually tease those apart.\n\nSo here the purpose behind restoring the timestamps is as an aid in\nguiding the examination of files to find the changes referenced in\nthe Changelog.\n\nGit is quite useful for this sort of effort, as once a sensible commit\nhas been synthsized, rebase of the next tar-file commit then helps\nreveal the next set of changes.\n\nSo what I'm thinking of is for stuff like this: https://github.com/DoctorWkt/unix-jun72\n(and the other repros there), where one wishes to figure out and\nregenerate a history of changes.  Since git is quite useful for\nrepresenting the end result, it is just that other scripting\nmay make it easier to use for such cases.\n\nDF\n"},{"id":"340033","messageId":"20180223122826.zuqncwduvf2n2all@tigra","threadId":"47877","inReplyTo":"20180221221420.GA7743@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Konstantin Khomoutov","fromEmail":"kostix@bswap.ru","sentAt":"2018-02-23T12:28:26Z","receivedAt":"2018-02-23T13:25:48Z","isPatch":false,"sender":{"key":"kostix@bswap.ru","avatar":null},"body":"On Wed, Feb 21, 2018 at 11:14:20PM +0100, Peter Backes wrote:\n\n[...]\n> atime, in contrast, was clearly one of the rather nonsensical \n> innovations of UNIX: Do one write to the disk for each read from the \n> disk. C'mon, really? It would have been a lot more reasonable to simply \n> provide a generic way for tracing read() system calls instead; then \n> userspace could decide what to do with that information and which of it \n> is useful and should be kept and perhaps stored on disk. Now we have \n> this ugly hack called relatime to deal with the problem.\n[...]\n\nIIUC, the purpose of atime can be more apparent if you consider it in\nthe context of the time it appeared: the systems were multi-user but the\ndisks were small, so a question \"what files are lying there but appear\nto be unused\" was rather sensical to ask as such files could be found,\nreported and then considered for deletion of moving off rotating media\nto tapes etc.\n\n"},{"id":"340291","messageId":"20180226110435.GA23573@helen.PLASMA.Xg8.DE","threadId":"47877","inReplyTo":"20180226105642.GA6549@inner.h.apk.li","subject":"Re: Git should preserve modification times at least on request","fromName":"'Peter Backes'","fromEmail":"rtc@helen.plasma.xg8.de","sentAt":"2018-02-26T11:04:35Z","receivedAt":"2018-02-26T11:05:15Z","isPatch":false,"sender":{"key":"rtc@helen.plasma.xg8.de","avatar":null},"body":"On Mon, Feb 26, 2018 at 11:56:42AM +0100, Andreas Krey wrote:\n> > The bigger issue is usually to copy with those pesky leap seconds. It \n> > makes a difference whether one uses solar seconds (\"posix\" style; those \n> > are more commonly seen) or atomic seconds (\"right\" style) for the UNIX \n> > timestamp.\n> \n> Is there any system, unix or otherwise, that uses 'right'-style seconds,\n> i.e. TAI, as its base?\n\nMost certainly there is. This depends on the individual configuration \nof the system. On my Fedora system, the commonly used tzdata package \noff the shelf contains support for 'right' style versions of all \ntimezones in /usr/share/zoneinfo/right If the user links one of those \ntimezones to /etc/localtime or manually specifies them (like \nTZ=right/Europe/Berlin ls -l) they will be used.\n\nYou don't find a lot of those systems today, but those who used to use \nthe 'right' timestamps might for legacy reasons explicitly configure \ntheir system to use those timezone variants. I personally did this for \na number of years, but then converted the filesystems timestamps to \n'posix' and I am now exclusively using 'posix' ones.\n\nBest wishes\nPeter\n-- \nPeter Backes, rtc@helen.PLASMA.Xg8.DE\n"},{"id":"340295","messageId":"20180226105642.GA6549@inner.h.apk.li","threadId":"47877","inReplyTo":"20180222020535.GA11063@helen.PLASMA.Xg8.DE","subject":"Re: Git should preserve modification times at least on request","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2018-02-26T10:56:42Z","receivedAt":"2018-02-26T11:28:13Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Thu, 22 Feb 2018 03:05:35 +0000, 'Peter Backes' wrote:\n...\n> The bigger issue is usually to copy with those pesky leap seconds. It \n> makes a difference whether one uses solar seconds (\"posix\" style; those \n> are more commonly seen) or atomic seconds (\"right\" style) for the UNIX \n> timestamp.\n\nIs there any system, unix or otherwise, that uses 'right'-style seconds,\ni.e. TAI, as its base?\n\n(I.e. one where (time(0)%60) does not indicate the current position\nof the second hand of an accurate clock?)\n\n- Andreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds <torvalds@*.org>\nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"}]}