{"thread":{"id":"44817","subject":"Refreshing index timestamps without reading content","startedAt":"2017-01-05T11:19:14Z","lastAt":"2017-01-10T14:13:58Z","messageCount":7,"participants":["Quentin Casasnovas","Duy Nguyen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"308817","messageId":"20170105112359.GN8116@chrystal.oracle.com","threadId":"44817","inReplyTo":null,"subject":"Refreshing index timestamps without reading content","fromName":"Quentin Casasnovas","fromEmail":"quentin.casasnovas@oracle.com","sentAt":"2017-01-05T11:23:59Z","receivedAt":"2017-01-05T11:19:14Z","isPatch":false,"sender":{"key":"quentin.casasnovas@oracle.com","avatar":null},"body":"Hi guys,\n\nApologies if this is documented somewhere, I have fairly bad search vudu\nskills.\n\nI'm looking for a way to cause a full refresh of the index without causing\nany read of the files, basically telling git \"trust me, all worktree files\nare matching the index, but their stat information have changed\".  I have\nread about the update-index --assume-unchanged and --skip-worktree flags in\nthe documentation, but these do not cause any index refresh - rather, they\nfake that the respective worktree files are matching the index until you\nremove those assume-unchanged/skip-worktree bits.\n\nThis might sound like a really weird thing to do, but I do have a use case\nfor it - we have some build farm setup where the resulting objects of a\ncompilation are stored on a shared server.  The source files are not stored\non the shared server, but locally on each of the build server (as to\ndecrease network load and make good use of local storage as caches).\n\nWe then use an onion filesystem to mount the compiled objects on top of the\nlocal sources - and change the modification time of the source to be older\nthan the object files, so that on subsequent builds, make does not rebuild\nthe whole world.\n\nThis works fine except for one thing, after changing the mtime of the\nsource files, the first subsequent git command needing to compare the tree\nwith the index will take a LONG time since it will read all of the object\ncontent:\n\n  cd linux-2.6\n\n  # Less than a second  when the index is up to date\n  time git status > /dev/null\n  git status 0.06s user 0.09s system 172% cpu 0.087 total\n                                              ~~~~~~~~~~~\n\n  # Change the mtime..\n  git ls-tree -r --name-only HEAD | xargs -n 1024 touch\n\n  # Now 30s..\n  time git status > /dev/null\n  git status  2.73s user 1.79s system 13% cpu 32.453 total\n                                              ~~~~~~~~~~~~\n\nThe timing information above was captured on my laptop SSD and the penalty\nis obviously much higher on spinning disks - especially when this operation\nis done on *hundreds* of different work tree in parallel, all hosted on the\nsame filesystem (it can take tens of minutes!).\n\nIs there any way to tell git, after the git ls-tree command above, to\nrefresh its stat cache information and trust us that the file content has\nnot changed, as to avoid any useless file read (though it will obviously\nwill have to stat all of them, but that's not something we can really\navoid)\n\nIf not, I am willing to implement a --assume-content-unchanged to the git\nupdate-index if you guys don't see something fundamentally wrong with this\napproach.\n\nThanks for any hints you can give! :)\n\nQ\n"},{"id":"309026","messageId":"CACsJy8BRfJG6L49VyC+qsrQ9Arz0gCGpMATpK9uLq61Lx6_Jtg@mail.gmail.com","threadId":"44817","inReplyTo":"20170105112359.GN8116@chrystal.oracle.com","subject":"Re: Refreshing index timestamps without reading content","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2017-01-09T12:02:45Z","receivedAt":"2017-01-09T12:03:37Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 5, 2017 at 6:23 PM, Quentin Casasnovas\n<quentin.casasnovas@oracle.com> wrote:\n> Hi guys,\n>\n> Apologies if this is documented somewhere, I have fairly bad search vudu\n> skills.\n>\n> I'm looking for a way to cause a full refresh of the index without causing\n> any read of the files, basically telling git \"trust me, all worktree files\n> are matching the index, but their stat information have changed\".  I have\n> read about the update-index --assume-unchanged and --skip-worktree flags in\n> the documentation, but these do not cause any index refresh - rather, they\n> fake that the respective worktree files are matching the index until you\n> remove those assume-unchanged/skip-worktree bits.\n>\n> This might sound like a really weird thing to do, but I do have a use case\n> for it - we have some build farm setup where the resulting objects of a\n> compilation are stored on a shared server.  The source files are not stored\n> on the shared server, but locally on each of the build server (as to\n> decrease network load and make good use of local storage as caches).\n>\n> We then use an onion filesystem to mount the compiled objects on top of the\n> local sources - and change the modification time of the source to be older\n> than the object files, so that on subsequent builds, make does not rebuild\n> the whole world.\n>\n> This works fine except for one thing, after changing the mtime of the\n> source files, the first subsequent git command needing to compare the tree\n> with the index will take a LONG time since it will read all of the object\n> content:\n>\n>   cd linux-2.6\n>\n>   # Less than a second  when the index is up to date\n>   time git status > /dev/null\n>   git status 0.06s user 0.09s system 172% cpu 0.087 total\n>                                               ~~~~~~~~~~~\n>\n>   # Change the mtime..\n>   git ls-tree -r --name-only HEAD | xargs -n 1024 touch\n>\n>   # Now 30s..\n>   time git status > /dev/null\n>   git status  2.73s user 1.79s system 13% cpu 32.453 total\n>                                               ~~~~~~~~~~~~\n>\n> The timing information above was captured on my laptop SSD and the penalty\n> is obviously much higher on spinning disks - especially when this operation\n> is done on *hundreds* of different work tree in parallel, all hosted on the\n> same filesystem (it can take tens of minutes!).\n>\n> Is there any way to tell git, after the git ls-tree command above, to\n> refresh its stat cache information and trust us that the file content has\n> not changed, as to avoid any useless file read (though it will obviously\n> will have to stat all of them, but that's not something we can really\n> avoid)\n\nI don't think there's any way to do that, unfortunately.\n\n> If not, I am willing to implement a --assume-content-unchanged to the git\n> update-index if you guys don't see something fundamentally wrong with this\n> approach.\n\nIf you do that, I think you should go with either of the following options\n\n- Extend git-update-index --index-info to take stat info as well (or\nmaybe make a new option instead). Then you can feed stat info directly\nto git without a use-case-specific \"assume-content-unchanged\".\n\n- Add \"git update-index --touch\" that does what \"touch\" does. In this\ncase, it blindly updates stat info to latest. But like touch, we can\nalso specify  mtime from command line if we need to. It's a bit less\ngeneric than the above option, but easier to use.\n\nCaveat: The options I'm proposing can be rejected. So maybe wait a bit\nto see how people feel and perhaps send an RFC patch, again to gauge\nthe reception.\n\n-- \nDuy\n"},{"id":"309027","messageId":"20170109121724.GC7000@chrystal.oracle.com","threadId":"44817","inReplyTo":"CACsJy8BRfJG6L49VyC+qsrQ9Arz0gCGpMATpK9uLq61Lx6_Jtg@mail.gmail.com","subject":"Re: Refreshing index timestamps without reading content","fromName":"Quentin Casasnovas","fromEmail":"quentin.casasnovas@oracle.com","sentAt":"2017-01-09T12:17:24Z","receivedAt":"2017-01-09T12:13:54Z","isPatch":false,"sender":{"key":"quentin.casasnovas@oracle.com","avatar":null},"body":"On Mon, Jan 09, 2017 at 07:02:45PM +0700, Duy Nguyen wrote:\n> On Thu, Jan 5, 2017 at 6:23 PM, Quentin Casasnovas\n> <quentin.casasnovas@oracle.com> wrote:\n> \n> > If not, I am willing to implement a --assume-content-unchanged to the git\n> > update-index if you guys don't see something fundamentally wrong with this\n> > approach.\n> \n> If you do that, I think you should go with either of the following options\n> \n> - Extend git-update-index --index-info to take stat info as well (or\n> maybe make a new option instead). Then you can feed stat info directly\n> to git without a use-case-specific \"assume-content-unchanged\".\n> \n> - Add \"git update-index --touch\" that does what \"touch\" does. In this\n> case, it blindly updates stat info to latest. But like touch, we can\n> also specify  mtime from command line if we need to. It's a bit less\n> generic than the above option, but easier to use.\n> \n> Caveat: The options I'm proposing can be rejected. So maybe wait a bit\n> to see how people feel and perhaps send an RFC patch, again to gauge\n> the reception.\n\n\nHey Duy,\n\nThanks for your answer.\n\nI've played with this a bit last week and added an extra command, which I\ncalled --refresh-stat, which works like your suggested --index.  It works\nvery well with my use case and improves the performances very significantly\non some of our use cases.\n\nIt is attached to this e-mail to gather comments, as you suggest, and is\nreally not meant to be reviewed for inclusion yet as it lacks test cases,\ndocumentation changes, etc.  It is just a convenient way to show what I\nneed and receive comments.\n\nThe logic is simple enugh and will just skip calling ie_modified() when\nrefreshing the index.\n\nCheers,\nQ\n\n\nFrom 461b4f75056fc476db6bbdf8bc3ff6e91a8ad08d Mon Sep 17 00:00:00 2001\nFrom: Quentin Casasnovas <quentin.casasnovas@oracle.com>\nDate: Sat, 7 Jan 2017 09:26:29 +0100\nSubject: [PATCH] git-update-index: add --refresh-stat option.\n\ngit-update-index --refresh, contrary to what the documentation says, not\nonly will refresh the stat information but will also update the sha1 of the\nobjects in the working tree if the stat information is found to be\nout-of-date.\n\nWe add a --refresh-stat option, which like --refresh will update the stat\ninformation, but will assume that any file in the working tree that is\npresent in the index matches the index - hence prevents unecessarily\nreading the files in the working tree.  It is different from\n--assume-unchanged or --skip-worktree in that the new stat information is\nrecorded in the index, hence subsequent git update-index --refresh will not\nread the files if their mtime hasn't changed.\n\nThis sounds like a very dangerous option to give to the user, since it\ncould potentially cause changed files to be missed, but can be needed when\nwe are sure the working tree files have not changed and are matching the\nindex.\n\nSigned-off-by: Quentin Casasnovas <quentin.casasnovas@oracle.com>\n---\n builtin/update-index.c | 10 ++++++++++\n cache.h                |  3 +++\n read-cache.c           |  7 +++++--\n 3 files changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex f3f07e7f1cb2..1215b6a09687 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -777,6 +777,12 @@ static int really_refresh_callback(const struct option *opt,\n \treturn refresh(opt->value, REFRESH_REALLY);\n }\n \n+static int refresh_stat_callback(const struct option *opt,\n+\t\t\t\tconst char *arg, int unset)\n+{\n+\treturn refresh(opt->value, REFRESH_STAT_ONLY);\n+}\n+\n static int chmod_callback(const struct option *opt,\n \t\t\t\tconst char *arg, int unset)\n {\n@@ -934,6 +940,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"refresh stat information\"),\n \t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG,\n \t\t\trefresh_callback},\n+\t\t{OPTION_CALLBACK, 0, \"refresh-stat\", &refresh_args, NULL,\n+\t\t\tN_(\"refresh only stat information (assume content has not changed)\"),\n+\t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG,\n+\t\t\trefresh_stat_callback},\n \t\t{OPTION_CALLBACK, 0, \"really-refresh\", &refresh_args, NULL,\n \t\t\tN_(\"like --refresh, but ignore assume-unchanged setting\"),\n \t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG,\ndiff --git a/cache.h b/cache.h\nindex a50a61a19787..57d0f9fffbe5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -611,6 +611,8 @@ extern void *read_blob_data_from_index(struct index_state *, const char *, unsig\n #define CE_MATCH_IGNORE_MISSING\t\t0x08\n /* enable stat refresh */\n #define CE_MATCH_REFRESH\t\t0x10\n+/* only stat refresh, assume content hasn't changed */\n+#define CE_MATCH_REFRESH_STAT_ONLY\t0x20\n extern int ie_match_stat(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n extern int ie_modified(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n \n@@ -643,6 +645,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_MISSING\t0x0008\t/* ignore non-existent */\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n+#define REFRESH_STAT_ONLY\t0x0040\t/* do not check content but update stat */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n extern struct cache_entry *refresh_cache_entry(struct cache_entry *, unsigned int);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex db5d91064266..e9334094c6f0 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1101,6 +1101,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n \tint ignore_missing = options & CE_MATCH_IGNORE_MISSING;\n+\tint assume_content_unchanged = options & CE_MATCH_REFRESH_STAT_ONLY;\n \n \tif (!refresh || ce_uptodate(ce))\n \t\treturn ce;\n@@ -1161,7 +1162,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\t}\n \t}\n \n-\tif (ie_modified(istate, ce, &st, options)) {\n+\tif (!assume_content_unchanged && ie_modified(istate, ce, &st, options)) {\n \t\tif (err)\n \t\t\t*err = EINVAL;\n \t\treturn NULL;\n@@ -1206,11 +1207,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tint quiet = (flags & REFRESH_QUIET) != 0;\n \tint not_new = (flags & REFRESH_IGNORE_MISSING) != 0;\n \tint ignore_submodules = (flags & REFRESH_IGNORE_SUBMODULES) != 0;\n+\tint refresh_stat_only = (flags & REFRESH_STAT_ONLY) != 0;\n \tint first = 1;\n \tint in_porcelain = (flags & REFRESH_IN_PORCELAIN);\n \tunsigned int options = (CE_MATCH_REFRESH |\n \t\t\t\t(really ? CE_MATCH_IGNORE_VALID : 0) |\n-\t\t\t\t(not_new ? CE_MATCH_IGNORE_MISSING : 0));\n+\t\t\t\t(not_new ? CE_MATCH_IGNORE_MISSING : 0) |\n+\t\t\t\t(refresh_stat_only ? CE_MATCH_REFRESH_STAT_ONLY : 0));\n \tconst char *modified_fmt;\n \tconst char *deleted_fmt;\n \tconst char *typechange_fmt;\n-- \n2.11.0\n\n"},{"id":"309028","messageId":"20170109122251.GD7000@chrystal.oracle.com","threadId":"44817","inReplyTo":"20170109121724.GC7000@chrystal.oracle.com","subject":"Re: Refreshing index timestamps without reading content","fromName":"Quentin Casasnovas","fromEmail":"quentin.casasnovas@oracle.com","sentAt":"2017-01-09T12:22:51Z","receivedAt":"2017-01-09T12:20:33Z","isPatch":false,"sender":{"key":"quentin.casasnovas@oracle.com","avatar":null},"body":"On Mon, Jan 09, 2017 at 01:17:24PM +0100, Quentin Casasnovas wrote:\n> On Mon, Jan 09, 2017 at 07:02:45PM +0700, Duy Nguyen wrote:\n> > On Thu, Jan 5, 2017 at 6:23 PM, Quentin Casasnovas\n> > <quentin.casasnovas@oracle.com> wrote:\n> > \n> > > If not, I am willing to implement a --assume-content-unchanged to the git\n> > > update-index if you guys don't see something fundamentally wrong with this\n> > > approach.\n> > \n> > If you do that, I think you should go with either of the following options\n> > \n> > - Extend git-update-index --index-info to take stat info as well (or\n> > maybe make a new option instead). Then you can feed stat info directly\n> > to git without a use-case-specific \"assume-content-unchanged\".\n> > \n> > - Add \"git update-index --touch\" that does what \"touch\" does. In this\n> > case, it blindly updates stat info to latest. But like touch, we can\n> > also specify  mtime from command line if we need to. It's a bit less\n> > generic than the above option, but easier to use.\n> > \n> > Caveat: The options I'm proposing can be rejected. So maybe wait a bit\n> > to see how people feel and perhaps send an RFC patch, again to gauge\n> > the reception.\n> \n> \n> Hey Duy,\n> \n> Thanks for your answer.\n> \n> I've played with this a bit last week and added an extra command, which I\n> called --refresh-stat, which works like your suggested --index.  It works\n                                                         ^^^^^^^\n\nWhoops, sorry!\n\n[...] *a bit* like your suggested *--touch*. [...]\n\nI don't really need it to support specific mtime from the CLI for my\ndifferent use cases, but if someone has some ideas in how that would be\nuseful, I can try implementing your --index-info change to support stat\ninformation to be passed around instead of my current simple\nimplementation.\n\nQ\n"},{"id":"309037","messageId":"xmqqr34cuvjj.fsf@gitster.mtv.corp.google.com","threadId":"44817","inReplyTo":"CACsJy8BRfJG6L49VyC+qsrQ9Arz0gCGpMATpK9uLq61Lx6_Jtg@mail.gmail.com","subject":"Re: Refreshing index timestamps without reading content","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-01-09T15:01:36Z","receivedAt":"2017-01-09T15:02:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> On Thu, Jan 5, 2017 at 6:23 PM, Quentin Casasnovas\n> <quentin.casasnovas@oracle.com> wrote:\n>> Is there any way to tell git, after the git ls-tree command above, to\n>> refresh its stat cache information and trust us that the file content has\n>> not changed, as to avoid any useless file read (though it will obviously\n>> will have to stat all of them, but that's not something we can really\n>> avoid)\n>\n> I don't think there's any way to do that, unfortunately.\n\nLose \"unfortunately\".\n\n>> If not, I am willing to implement a --assume-content-unchanged to the git\n>> update-index if you guys don't see something fundamentally wrong with this\n>> approach.\n>\n> If you do that, I think you should go with either of the following options\n>\n> - Extend git-update-index --index-info to take stat info as well (or\n> maybe make a new option instead). Then you can feed stat info directly\n> to git without a use-case-specific \"assume-content-unchanged\".\n>\n> - Add \"git update-index --touch\" that does what \"touch\" does. In this\n> case, it blindly updates stat info to latest. But like touch, we can\n> also specify  mtime from command line if we need to. It's a bit less\n> generic than the above option, but easier to use.\n\nEven if we assume that it is a good idea to let people muck with the\nindex like this, either of the above would be a usable addition,\nbecause the cached stat information does not consist solely of\nmtime.\n\n\"git update-index --index-info\" was invented for the case where a\nuser or a script _knows_ the object ID of the blob that _would_\nresult if a contents of a file on the filesystem were run through\nhash-object.  So from the interface's point of view, it may make\nsense to teach it to take an extra/optional argument that is the\npath to the file and take the stat info out of the named file when\nthe extra/optional argument was given.\n\nBut that assumes that it is a good idea to do this in the first\nplace.  It was deliberate design decision that setting the cached\nstat info for the entry was protected behind actual content\ncomparison, and removing that protection will open the index to\nabuse.\n\nThe userbase of Git has grown wide enough that it is harder to say\n\"If you lie that a file whose contents does not match the index is\nup to date using this mechanism, you will lose data and all bad\nthings happen---you can keep both halves\".  Once we release a\nversion of Git with such a \"feature\", the first bug report will be\n\"I did not want to run 'update-index --refresh' because it takes\ntime, and some index entries apparently did not match what is on the\nfilesystem, and I got a corrupt working file after a merge.  Git\nshould make sure that the contents match when using the new 'path to\nthe file' argument when updating the cached stat info!\".  I do not\nhave a good answer to such a bug report.\n\nSo...\n"},{"id":"309039","messageId":"20170109155537.GG7000@chrystal.oracle.com","threadId":"44817","inReplyTo":"xmqqr34cuvjj.fsf@gitster.mtv.corp.google.com","subject":"Re: Refreshing index timestamps without reading content","fromName":"Quentin Casasnovas","fromEmail":"quentin.casasnovas@oracle.com","sentAt":"2017-01-09T15:55:37Z","receivedAt":"2017-01-09T15:51:15Z","isPatch":false,"sender":{"key":"quentin.casasnovas@oracle.com","avatar":null},"body":"On Mon, Jan 09, 2017 at 07:01:36AM -0800, Junio C Hamano wrote:\n> Duy Nguyen <pclouds@gmail.com> writes:\n>\n> > On Thu, Jan 5, 2017 at 6:23 PM, Quentin Casasnovas\n> > <quentin.casasnovas@oracle.com> wrote:\n> >> Is there any way to tell git, after the git ls-tree command above, to\n> >> refresh its stat cache information and trust us that the file content has\n> >> not changed, as to avoid any useless file read (though it will obviously\n> >> will have to stat all of them, but that's not something we can really\n> >> avoid)\n> >\n> > I don't think there's any way to do that, unfortunately.\n>\n> Lose \"unfortunately\".\n>\n> >> If not, I am willing to implement a --assume-content-unchanged to the git\n> >> update-index if you guys don't see something fundamentally wrong with this\n> >> approach.\n> >\n> > If you do that, I think you should go with either of the following options\n> >\n> > - Extend git-update-index --index-info to take stat info as well (or\n> > maybe make a new option instead). Then you can feed stat info directly\n> > to git without a use-case-specific \"assume-content-unchanged\".\n> >\n> > - Add \"git update-index --touch\" that does what \"touch\" does. In this\n> > case, it blindly updates stat info to latest. But like touch, we can\n> > also specify  mtime from command line if we need to. It's a bit less\n> > generic than the above option, but easier to use.\n>\n> Even if we assume that it is a good idea to let people muck with the\n> index like this, either of the above would be a usable addition,\n> because the cached stat information does not consist solely of\n> mtime.\n>\n> \"git update-index --index-info\" was invented for the case where a\n> user or a script _knows_ the object ID of the blob that _would_\n> result if a contents of a file on the filesystem were run through\n> hash-object.  So from the interface's point of view, it may make\n> sense to teach it to take an extra/optional argument that is the\n> path to the file and take the stat info out of the named file when\n> the extra/optional argument was given.\n>\n> But that assumes that it is a good idea to do this in the first\n> place.  It was deliberate design decision that setting the cached\n> stat info for the entry was protected behind actual content\n> comparison, and removing that protection will open the index to\n> abuse.\n>\n\nHi Junio,\n\nThanks for your feedback, appreciated :)\n\nI do understand how it would be possible for someone to shoot themselves in\nthe feet with such option, but it solves real life use cases and improved\nbuild times very signficantly here.\n\nAnother use case we have is setting up very lightweight linux work trees,\nby reflinking from a base work-tree.  This allows for a completely\ndifferent work-tree taking up almost no size at first, whereas using a\nshared clone or the recent worktree subcommand would \"waste\" ~500MB*:\n\n # linux-2.6 is a shared clone of a bare clone residing locally\n ~ $ cp --reflink -a linux-2.6 linux-2.6-reflinked\n\n # At this point, the mtime inside linux-2.6-reflinked are matching the\n # mtime of the source linux-2.6 (since we used the '-a' option of 'cp)\n ~ $ diff -u <(stat linux-2.6/README) <(stat linux-2.6-reflinked/README)\n --- /proc/self/fd/11  2017-01-09 16:34:04.523438942 +0100\n +++ /proc/self/fd/12  2017-01-09 16:34:04.523438942 +0100\n @@ -1,8 +1,8 @@\n -  File: 'linux-2.6/README'\n +  File: 'linux-2.6-reflinked/README'\n    Size: 18372\t\tBlocks: 40         IO Block: 4096   regular file\n -Device: fd00h/64768d\tInode: 268467090   Links: 1\n +Device: fd00h/64768d\tInode: 805970606   Links: 1\n  Access: (0644/-rw-r--r--)  Uid: ( 1000/ quentin)   Gid: ( 1000/ quentin)\n  Access: 2017-01-09 12:04:15.317758718 +0100\n  Modify: 2017-01-09 12:04:12.566758772 +0100\n -Change: 2017-01-09 12:04:12.566758772 +0100\n +Change: 2017-01-09 16:29:48.305444003 +0100\n   Birth:\n\n  # Now let's check how long it takes to refresh the index from the source\n  # and destination..\n  ~/linux-2.6 $ time git update-index --refresh\n  git update-index --refresh  0.04s user 0.08s system 204% cpu 0.058 total\n                                                               ~~~~~~~~~~~\n  ~/linux-2.6-reflinked $ time git update-index --refresh\n  git update-index --refresh  2.40s user 1.43s system 38% cpu 10.003 total\n                                                              ~~~~~~~~~~~~\n\nThis is quite a high penalty when a power user knows that his lightweight\ncopy worktree matches the index!  That's on a single worktree on a fairly\ndecent SSD but our build farm would do this with hundreds of work trees in\nparallel, all residing on a spinning disks - the penalty would be much\nworse than 10 seconds wasted.\n\nI might have a look at adding reflinking awareness to the worktree\nsubcommand (if it hasn't been implemented already) to avoid this index\nforce refreshing but that still would not fix our other use case where we\nknowingly change the mtime of the files on our worktrees.\n\n* That's a rather low estimate:\n\n  git ls-tree -zr --name-only v4.9 | du -csh --files0-from=- | tail -n1\n  747M  total\n\n> The userbase of Git has grown wide enough that it is harder to say \"If\n> you lie that a file whose contents does not match the index is up to date\n> using this mechanism, you will lose data and all bad things happen---you\n> can keep both halves\".  Once we release a version of Git with such a\n> \"feature\", the first bug report will be \"I did not want to run\n> 'update-index --refresh' because it takes time, and some index entries\n> apparently did not match what is on the filesystem, and I got a corrupt\n> working file after a merge.  Git should make sure that the contents match\n> when using the new 'path to the file' argument when updating the cached\n> stat info!\".  I do not have a good answer to such a bug report.\n>\n\nThe good answer would probably be that this is intended behaviour and would\nexplicitly be mentionned in the manual for that specific option?  :)\n\nHow about renaming the option (which is more of a sub-command) to\n--i-know-this-can-eat-my-data-but-refresh-stat-only?  Then we can print a\nbig warning saying this is generally a VERY bad idea?\n\nQ\n"},{"id":"309143","messageId":"20170110141731.GH7000@chrystal.oracle.com","threadId":"44817","inReplyTo":"20170109155537.GG7000@chrystal.oracle.com","subject":"Re: Refreshing index timestamps without reading content","fromName":"Quentin Casasnovas","fromEmail":"quentin.casasnovas@oracle.com","sentAt":"2017-01-10T14:17:31Z","receivedAt":"2017-01-10T14:13:58Z","isPatch":false,"sender":{"key":"quentin.casasnovas@oracle.com","avatar":null},"body":"On Mon, Jan 09, 2017 at 04:55:37PM +0100, Quentin Casasnovas wrote:\n> On Mon, Jan 09, 2017 at 07:01:36AM -0800, Junio C Hamano wrote:\n> > Duy Nguyen <pclouds@gmail.com> writes:\n> >\n> > > On Thu, Jan 5, 2017 at 6:23 PM, Quentin Casasnovas\n> > > <quentin.casasnovas@oracle.com> wrote:\n> > >> Is there any way to tell git, after the git ls-tree command above, to\n> > >> refresh its stat cache information and trust us that the file content has\n> > >> not changed, as to avoid any useless file read (though it will obviously\n> > >> will have to stat all of them, but that's not something we can really\n> > >> avoid)\n> > >\n> > > I don't think there's any way to do that, unfortunately.\n> >\n> > Lose \"unfortunately\".\n> >\n> > >> If not, I am willing to implement a --assume-content-unchanged to the git\n> > >> update-index if you guys don't see something fundamentally wrong with this\n> > >> approach.\n> > >\n> > > If you do that, I think you should go with either of the following options\n> > >\n> > > - Extend git-update-index --index-info to take stat info as well (or\n> > > maybe make a new option instead). Then you can feed stat info directly\n> > > to git without a use-case-specific \"assume-content-unchanged\".\n> > >\n> > > - Add \"git update-index --touch\" that does what \"touch\" does. In this\n> > > case, it blindly updates stat info to latest. But like touch, we can\n> > > also specify  mtime from command line if we need to. It's a bit less\n> > > generic than the above option, but easier to use.\n> >\n> > Even if we assume that it is a good idea to let people muck with the\n> > index like this, either of the above would be a usable addition,\n> > because the cached stat information does not consist solely of\n> > mtime.\n> >\n> > \"git update-index --index-info\" was invented for the case where a\n> > user or a script _knows_ the object ID of the blob that _would_\n> > result if a contents of a file on the filesystem were run through\n> > hash-object.  So from the interface's point of view, it may make\n> > sense to teach it to take an extra/optional argument that is the\n> > path to the file and take the stat info out of the named file when\n> > the extra/optional argument was given.\n> >\n> > But that assumes that it is a good idea to do this in the first\n> > place.  It was deliberate design decision that setting the cached\n> > stat info for the entry was protected behind actual content\n> > comparison, and removing that protection will open the index to\n> > abuse.\n> >\n> \n> Hi Junio,\n> \n> Thanks for your feedback, appreciated :)\n> \n> I do understand how it would be possible for someone to shoot themselves in\n> the feet with such option, but it solves real life use cases and improved\n> build times very signficantly here.\n> \n> Another use case we have is setting up very lightweight linux work trees,\n> by reflinking from a base work-tree.  This allows for a completely\n> different work-tree taking up almost no size at first, whereas using a\n> shared clone or the recent worktree subcommand would \"waste\" ~500MB*:\n> \n>  # linux-2.6 is a shared clone of a bare clone residing locally\n>  ~ $ cp --reflink -a linux-2.6 linux-2.6-reflinked\n> \n>  # At this point, the mtime inside linux-2.6-reflinked are matching the\n>  # mtime of the source linux-2.6 (since we used the '-a' option of 'cp)\n>  ~ $ diff -u <(stat linux-2.6/README) <(stat linux-2.6-reflinked/README)\n>  --- /proc/self/fd/11  2017-01-09 16:34:04.523438942 +0100\n>  +++ /proc/self/fd/12  2017-01-09 16:34:04.523438942 +0100\n>  @@ -1,8 +1,8 @@\n>  -  File: 'linux-2.6/README'\n>  +  File: 'linux-2.6-reflinked/README'\n>     Size: 18372\t\tBlocks: 40         IO Block: 4096   regular file\n>  -Device: fd00h/64768d\tInode: 268467090   Links: 1\n>  +Device: fd00h/64768d\tInode: 805970606   Links: 1\n>   Access: (0644/-rw-r--r--)  Uid: ( 1000/ quentin)   Gid: ( 1000/ quentin)\n>   Access: 2017-01-09 12:04:15.317758718 +0100\n>   Modify: 2017-01-09 12:04:12.566758772 +0100\n>  -Change: 2017-01-09 12:04:12.566758772 +0100\n>  +Change: 2017-01-09 16:29:48.305444003 +0100\n>    Birth:\n> \n>   # Now let's check how long it takes to refresh the index from the source\n>   # and destination..\n>   ~/linux-2.6 $ time git update-index --refresh\n>   git update-index --refresh  0.04s user 0.08s system 204% cpu 0.058 total\n>                                                                ~~~~~~~~~~~\n>   ~/linux-2.6-reflinked $ time git update-index --refresh\n>   git update-index --refresh  2.40s user 1.43s system 38% cpu 10.003 total\n>                                                               ~~~~~~~~~~~~\n> \n\nAfter discussing this with my friend Vegard, he found the core.checkStat\nconfig which, if set to 'minimal', ignores the inode number which is enough\nfor the above use case to work just fine - so please excuse my ignorance!\n\nFor the initial problem I had when changing the mtime of all the files in\nthe tree, I should be able to change the mtime of the object files instead,\nhence I don't really need the patch I sent earlier.\n\nSorry for the wasted time! :)\n\nQ\n"}]}