{"thread":{"id":"35508","subject":"Setting file timestamps to commit time (git-checkout)","startedAt":"2013-12-09T11:25:28Z","lastAt":"2013-12-11T07:37:11Z","messageCount":11,"participants":["Dominik Vogt","Junio C Hamano","Jonathan Nieder","Duy Nguyen","Andreas Schwab","Constantine A. Murenin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"231786","messageId":"20131209112528.GA5309@linux.vnet.ibm.com","threadId":"35508","inReplyTo":null,"subject":"Setting file timestamps to commit time (git-checkout)","fromName":"Dominik Vogt","fromEmail":"vogt@linux.vnet.ibm.com","sentAt":"2013-12-09T11:25:28Z","receivedAt":"2013-12-09T11:25:28Z","isPatch":false,"sender":{"key":"vogt@linux.vnet.ibm.com","avatar":null},"body":"Me and some colleagues work on gcc in lots of different branches.\nFor each branch there is a separate build directory for each\nbranch, e.g. build-a, build-b and build-c. Let's assume that all\nbranches are identical at the moment. If a file in branch a is\nchanged that triggers a complete rebuild of gcc (e.g.\n.opt), rebuilding in build-a takes about an hour. Now,\n when I switch to one of the other branches, said file is not\nidentical anymore and stamped with the _current_ time during\ncheckout. Although branch b and c have not changed at all, they\nwill now be rebuilt completely because the timestamp on that files\nhas changed. I.e. a chance on one branch forces a rebuild on n\nother branches, which can take many hours.\n\nI think this situation could be improved with an option to\ngit-checkout with the following logic:\n\n$ git checkout \n FOR EACH in working directory of \n IF is identical to the version in the \n THEN leave the file untouched\n ELSE IF of the HEAD of the \n is in the future\n THEN checkout the new version of and stamp it with\n the current time\n ELSE (commit timestamp is current or in the past)\n THEN checkout the new version of and stamp it with\n the commit timestamp of the current HEAD of \n\nAny comments? Is there already a way to do this?\n\n(Please do not cc me on replies, I'm subscribed to the list.)\n\nCiao\n\nDominik ^_^ ^_^\n\n-- \n\nDominik Vogt\nIBM Germany\n"},{"id":"231808","messageId":"xmqqsiu1yd7p.fsf@gitster.dls.corp.google.com","threadId":"35508","inReplyTo":"20131209112528.GA5309@linux.vnet.ibm.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-12-09T20:35:38Z","receivedAt":"2013-12-09T20:35:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dominik Vogt writes:\n\n> Me and some colleagues work on gcc in lots of different branches.\n> For each branch there is a separate build directory for each\n> branch, e.g. build-a, build-b and build-c. Let's assume that all\n> branches are identical at the moment. If a file in branch a is\n> changed that triggers a complete rebuild of gcc (e.g.\n> .opt), rebuilding in build-a takes about an hour. Now,\n> when I switch to one of the other branches, said file is not\n> identical anymore and stamped with the _current_ time during\n> checkout. Although branch b and c have not changed at all, they\n> will now be rebuilt completely because the timestamp on that files\n> has changed.\n\nI am not quite sure I follow your set-up. Do you have three working\ntrees connected to a repository (via contrib/workdir/git-new-workdir\nperhaps), each having a checkout of its own branch? And in one\nworking directory that has build-a checked out, a new commit touches\none file, .opt, to make a new commit:\n\nBefore:\n\n ---o---o---X\n ^ refs/heads/build-a\n refs/heads/build-b\n refs/heads/build-c\n\nAfter:\n v refs/heads/build-a\n ---o---o---X---Y\n ^ refs/heads/build-b\n refs/heads/build-c\n\nBecause you said that branch b and c hasn't changed at all, I do not\nsee how your build-b and/or build-c directories become dirty.\n"},{"id":"231812","messageId":"20131209204815.GV29959@google.com","threadId":"35508","inReplyTo":"20131209112528.GA5309@linux.vnet.ibm.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-12-09T20:48:16Z","receivedAt":"2013-12-09T20:48:16Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nDominik Vogt wrote:\n\n> Now,\n> when I switch to one of the other branches, said file is not\n> identical anymore and stamped with the _current_ time during\n> checkout. Although branch b and c have not changed at all, they\n> will now be rebuilt completely because the timestamp on that files\n> has changed. I.e. a chance on one branch forces a rebuild on n\n> other branches, which can take many hours.\n>\n> I think this situation could be improved with an option to\n> git-checkout with the following logic:\n>\n> $ git checkout \n> FOR EACH in working directory of \n> IF is identical to the version in the \n> THEN leave the file untouched\n> ELSE IF of the HEAD of the \n> is in the future\n> THEN checkout the new version of and stamp it with\n> the current time\n> ELSE (commit timestamp is current or in the past)\n> THEN checkout the new version of and stamp it with\n> the commit timestamp of the current HEAD of \n\nWouldn't that break \"make\"? When you switch to an old branch, changed\nfiles would then a timestamp *before* the corresponding build targets,\ncausing the stale (wrong function signatures, etc) build results from\nthe newer branch to be reused and breaking the build.\n\nI suspect the simplest way to accomplish what you're looking for would\nbe to keep separate worktrees for each branch you regularly build.\nIt's possible to do that using entirely independent clones, clones\nsharing some objects (using \"git clone --shared\" from some master\ncopy), or even multiple worktrees for the same clone (using the\ngit-new-workdir script from contrib/workdir/).\n\nSee [1] and [2] for more hints.\n\n[...]\n> (Please do not cc me on replies, I'm subscribed to the list.)\n\nThe convention on this list is to always reply-to-all, but I'm happy\nto make an exception. :)\n\nHope that helps,\nJonathan\n\n[1] https://git.wiki.kernel.org/index.php/Git_FAQ#Why_isn.27t_Git_preserving_modification_time_on_files.3F\n[2] https://git.wiki.kernel.org/index.php/ExampleScripts#Setting_the_timestamps_of_the_files_to_the_commit_timestamp_of_the_commit_which_last_touched_them\n"},{"id":"231847","messageId":"20131210083531.GB4087@linux.vnet.ibm.com","threadId":"35508","inReplyTo":"xmqqsiu1yd7p.fsf@gitster.dls.corp.google.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Dominik Vogt","fromEmail":"vogt@linux.vnet.ibm.com","sentAt":"2013-12-10T08:35:31Z","receivedAt":"2013-12-10T08:35:31Z","isPatch":false,"sender":{"key":"vogt@linux.vnet.ibm.com","avatar":null},"body":"On Mon, Dec 09, 2013 at 12:35:38PM -0800, Junio C Hamano wrote:\n> Dominik Vogt writes:\n> \n> > Me and some colleagues work on gcc in lots of different branches.\n> > For each branch there is a separate build directory for each\n> > branch, e.g. build-a, build-b and build-c. Let's assume that all\n> > branches are identical at the moment. If a file in branch a is\n> > changed that triggers a complete rebuild of gcc (e.g.\n> > .opt), rebuilding in build-a takes about an hour. Now,\n> > when I switch to one of the other branches, said file is not\n> > identical anymore and stamped with the _current_ time during\n> > checkout. Although branch b and c have not changed at all, they\n> > will now be rebuilt completely because the timestamp on that files\n> > has changed.\n> \n> I am not quite sure I follow your set-up. Do you have three working\n> trees connected to a repository (via contrib/workdir/git-new-workdir\n> perhaps), each having a checkout of its own branch?\n\nNo, just one working tree, but three separate build directories\nfor various branches. Actually, the build directories could be\nlocated at some random place on disk, but it's convenient to keep\nthem inside the working tree. Personally I do not use multiple\nworking trees because in the past I had the impression that this\nkind of setup creates more problems than it solves. Just to give\nyou an idea how my current workspace looks like:\n\n ~/rpm/BUILD/gcc-4.1.2-20080825\n build-4.1/\n install-4.1/\n ...\n (branch \"master\")\n\n ~/rpm/BUILD/gcc-4.4.7-20120601\n build-4.4/\n install-4.1/\n (branch \"master\")\n\n ~/src/git/gcc-unpatched\n build/\n install/\n ...\n (branch \"master\")\n\n ~/src/git/gcc-patched\n build-4.8/\n build-4.9/\n build-somefeature/\n install-4.8/\n install-4.9/\n install-somefeature/\n ...\n (various feature branches)\n\n> [snip]\n\nHm, the case I described was too simple. Another try:\n\n* With the setup described above I have, say, eleven branches, namely\n a and b, b2, ..., b9:\n\n ---o---X <== a\n |\n `---Y <== b\n |\n |---o <== b2\n ...\n `---o <== b9\n\n* The two commits X and Y both touch a file that triggers a\n complete rebuild, say gcc/common.opt.\n\n* Each branch has a matching build directory build-, and\n all of them are built for the latest version of the\n corresponding branch.\n\n* Switch to branch a and do some work or just look at it.\n\n* When I switch back to any of the b-branches, gcc/common.opt gets\n stamped with the current time, i.e. \"make\" considers the whole\n build directory to be outdated and builds everything from\n scratch. Then I switch to another b-branch and the whole thing\n starts over etc. With gcc-bootstrapping enabled, such a build\n takes me almost an hour. In other words, just looking at branch\n a entails a full day just rebuilding branches that have not\n changed at all.\n\nI've discussed that with some of my co-workers, but we still\ncould not come up with a nice solution. The \"right\" way to \"fix\"\nthis might be to stash all file modification dates on a branch\nswitch and restore them when switching back to the original. But\nthat sounds awfully expensive, and really out of the scope of an\nRCS. The second best approach I could think of is to stamp files\nwith the timestamp of the last commit that touched that, but I\nguess that is not a cheap operation either.\n\nCiao\n\nDominik ^_^ ^_^\n\n-- \n\nDominik Vogt\nIBM Germany\n"},{"id":"231848","messageId":"20131210084622.GC4087@linux.vnet.ibm.com","threadId":"35508","inReplyTo":"20131209204815.GV29959@google.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Dominik Vogt","fromEmail":"vogt@linux.vnet.ibm.com","sentAt":"2013-12-10T08:46:22Z","receivedAt":"2013-12-10T08:46:22Z","isPatch":false,"sender":{"key":"vogt@linux.vnet.ibm.com","avatar":null},"body":"On Mon, Dec 09, 2013 at 12:48:16PM -0800, Jonathan Nieder wrote:\n> Dominik Vogt wrote:\n> > when I switch to one of the other branches, said file is not\n> > identical anymore and stamped with the _current_ time during\n> > checkout. Although branch b and c have not changed at all, they\n> > will now be rebuilt completely because the timestamp on that files\n> > has changed. I.e. a chance on one branch forces a rebuild on n\n> > other branches, which can take many hours.\n> >\n> > I think this situation could be improved with an option to\n> > git-checkout with the following logic:\n> >\n> > $ git checkout \n> > FOR EACH in working directory of \n> > IF is identical to the version in the \n> > THEN leave the file untouched\n> > ELSE IF of the HEAD of the \n> > is in the future\n> > THEN checkout the new version of and stamp it with\n> > the current time\n> > ELSE (commit timestamp is current or in the past)\n> > THEN checkout the new version of and stamp it with\n> > the commit timestamp of the current HEAD of \n> \n> Wouldn't that break \"make\"? When you switch to an old branch, changed\n> files would then a timestamp *before* the corresponding build targets,\n> causing the stale (wrong function signatures, etc) build results from\n> the newer branch to be reused and breaking the build.\n\nYes, if you share a common build directory, this logic would\nutterly break the build system. The point with gcc is, that you\ndo not build it in the source tree but in a separate build\ndirectory, and it's easy to have separate build directories for\nyour branches.\n\n> I suspect the simplest way to accomplish what you're looking for would\n> be to keep separate worktrees for each branch you regularly build.\n> It's possible to do that using entirely independent clones, clones\n> sharing some objects (using \"git clone --shared\" from some master\n> copy), or even multiple worktrees for the same clone (using the\n> git-new-workdir script from contrib/workdir/).\n\nI've tried the first two ways for separate workdirs in the past\nbut did not like them. How does git-new-workdir cope with\nrebasing (e.g. you have the same branch checked out in two working\ntrees and \"rebase -i\" it in one of them)? Is it really a working\noption?\n\n> > (Please do not cc me on replies, I'm subscribed to the list.)\n> \n> The convention on this list is to always reply-to-all, but I'm happy\n> to make an exception. :)\n\nIt's just a hint; anyway, I guess I should remove the Reply-To\nheader if I don't want direct replies. ;-)\n\nCiao\n\nDominik ^_^ ^_^\n\n-- \n\nDominik Vogt\nIBM Germany\n"},{"id":"231851","messageId":"CACsJy8Bsgfh=0mTHY4kFAXE6+y7ODx5AwVHcLzVgz01Biiy=7A@mail.gmail.com","threadId":"35508","inReplyTo":"20131210084622.GC4087@linux.vnet.ibm.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-10T10:34:17Z","receivedAt":"2013-12-10T10:34:17Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Dec 10, 2013 at 3:46 PM, Dominik Vogt wrote:\n>\n> > I suspect the simplest way to accomplish what you're looking for would\n> > be to keep separate worktrees for each branch you regularly build.\n> > It's possible to do that using entirely independent clones, clones\n> > sharing some objects (using \"git clone --shared\" from some master\n> > copy), or even multiple worktrees for the same clone (using the\n> > git-new-workdir script from contrib/workdir/).\n>\n> I've tried the first two ways for separate workdirs in the past\n> but did not like them. How does git-new-workdir cope with\n> rebasing (e.g. you have the same branch checked out in two working\n> trees and \"rebase -i\" it in one of them)? Is it really a working\n> option?\n\nI wonder if we could promote multiple worktree from a hack to a\nsupported feature. What I have in mind is when you \"clone\n--separate-worktree\" it would create a .git file that describes\nseparate worktree:\n\ngitbasedir: /path/to/the/original/.git\nname: foo\n\nHEAD, index and logs/HEAD would be stored in\n/path/to/the/original/.git/worktrees/foo/. GIT_DIR would be set to\n.../foo/, GIT_OBJECT_DIRECTORY, the new GIT_REF_DIRECTORY (which\ncovers root for all refs/, logs/ and packed-refs) and maybe\nGIT_HOOKS_DIRECTORY are pointed to directories in\n.../original/.git/... though.\n\nThis allows all worktrees to be aware of the others and locking could\nbe implemented so that no two worktrees check out the same branch (or\nthey can, but the other becomes detached if the ref is updated in this\nworktree)..\n-- \nDuy\n"},{"id":"231867","messageId":"87bo0olebe.fsf@igel.home","threadId":"35508","inReplyTo":"20131210083531.GB4087@linux.vnet.ibm.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2013-12-10T19:02:29Z","receivedAt":"2013-12-10T19:02:29Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Dominik Vogt writes:\n\n> The second best approach I could think of is to stamp files with the\n> timestamp of the last commit that touched that, but I guess that is\n> not a cheap operation either.\n\nI'm using this script for this:\n\n#!/bin/sh\ngit log --name-only --format=format:%n%ct -- \"$@\" |\nperl -e 'my $do_date = 0; chomp(my $cdup = `git rev-parse --show-cdup`);\n while (<>) {\n\tchomp;\n\tif ($do_date) {\n\t next if ($_ eq \"\");\n\t die \"Unexpected $_\\n\" unless /^[0-9]+$/;\n\t $d = $_;\n\t $do_date = 0;\n\t} elsif ($_ eq \"\") {\n\t $do_date = 1;\n\t} elsif (!defined($seen{$_})) {\n\t $seen{$_} = 1;\n \t utime $d, $d, \"$cdup$_\";\n \t}\n }'\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"231878","messageId":"20131211010145.GG2311@google.com","threadId":"35508","inReplyTo":"20131210084622.GC4087@linux.vnet.ibm.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-12-11T01:01:45Z","receivedAt":"2013-12-11T01:01:45Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Dominik Vogt wrote:\n\n> How does git-new-workdir cope with\n> rebasing (e.g. you have the same branch checked out in two working\n> trees and \"rebase -i\" it in one of them)?\n\nGenerally you don't have the same branch checked out in two working\ntrees. I tend to use \"git checkout --detach\" to not have *any*\nbranch checked out in most working trees, though that comes with its\nown set of problems since the HEAD reflog is not shared.\n\n> Is it really a working\n> option?\n\nYes, modulo the two warnings above. ;-)\n\nIf someone has time to work on it, the threads\n\n http://thread.gmane.org/gmane.comp.version-control.git/150559\n http://thread.gmane.org/gmane.comp.version-control.git/182821\n\ndescribe one way to make those caveats go away.\n\nThanks,\nJonathan\n"},{"id":"231879","messageId":"20131211010823.GH2311@google.com","threadId":"35508","inReplyTo":"CACsJy8Bsgfh=0mTHY4kFAXE6+y7ODx5AwVHcLzVgz01Biiy=7A@mail.gmail.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-12-11T01:08:23Z","receivedAt":"2013-12-11T01:08:23Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Duy Nguyen wrote:\n\n> I wonder if we could promote multiple worktree from a hack to a\n> supported feature. What I have in mind is when you \"clone\n> --separate-worktree\" it would create a .git file that describes\n> separate worktree:\n>\n> gitbasedir: /path/to/the/original/.git\n> name: foo\n>\n> HEAD, index and logs/HEAD would be stored in\n> /path/to/the/original/.git/worktrees/foo/.\n\nI like this idea a lot.\n\nJonathan\n"},{"id":"231881","messageId":"CAPKkNb6pUkJnJ=wW=BqgjFOpGixy1S=Hccv+OxWjoBipoYoq=A@mail.gmail.com","threadId":"35508","inReplyTo":"20131210083531.GB4087@linux.vnet.ibm.com","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Constantine A. Murenin","fromEmail":"mureninc@gmail.com","sentAt":"2013-12-11T01:39:05Z","receivedAt":"2013-12-11T01:39:05Z","isPatch":false,"sender":{"key":"mureninc@gmail.com","avatar":null},"body":"On 10 December 2013 00:35, Dominik Vogt wrote:\n> that sounds awfully expensive, and really out of the scope of an\n> RCS. The second best approach I could think of is to stamp files\n> with the timestamp of the last commit that touched that, but I\n> guess that is not a cheap operation either.\n\nYou can already do this with a very small third-party script:\n\n https://github.com/cnst/git-tools/blob/master/git-restore-mtime-core\n\nC.\n"},{"id":"231885","messageId":"20131211073711.GA4848@linux.vnet.ibm.com","threadId":"35508","inReplyTo":"87bo0olebe.fsf@igel.home","subject":"Re: Setting file timestamps to commit time (git-checkout)","fromName":"Dominik Vogt","fromEmail":"vogt@linux.vnet.ibm.com","sentAt":"2013-12-11T07:37:11Z","receivedAt":"2013-12-11T07:37:11Z","isPatch":false,"sender":{"key":"vogt@linux.vnet.ibm.com","avatar":null},"body":"On Tue, Dec 10, 2013 at 08:02:29PM +0100, Andreas Schwab wrote:\n> Dominik Vogt writes:\n> \n> > The second best approach I could think of is to stamp files with the\n> > timestamp of the last commit that touched that, but I guess that is\n> > not a cheap operation either.\n> \n> I'm using this script for this:\n[snip]\n\nHm, that runs 18 s on the local Gcc repository. That's not as\nexpensive as I would have thought, but definitely not suitable to\nrun automatically on each checkout. I wonder if performance could\nbe improved by integrating the script logic into the git-checkout\ncode (activated by a command line option).\n\nOn Tue, Dec 10, 2013 at 05:39:05PM -0800, Constantine A. Murenin wrote:\n> You can already do this with a very small third-party script:\n>\n> https://github.com/cnst/git-tools/blob/master/git-restore-mtime-core\n\nThat script just produces error messages for me.\n\nCiao\n\nDominik ^_^ ^_^\n\n-- \n\nDominik Vogt\nIBM Germany\n"}]}