{"thread":{"id":"50525","subject":"[PATCH 0/1] Introduce \"precious\" file attribute","startedAt":"2019-02-16T12:00:43Z","lastAt":"2019-02-22T18:07:48Z","messageCount":19,"participants":["Nguyễn Thái Ngọc Duy","Ævar Arnfjörð Bjarmason","Duy Nguyen","Junio C Hamano","Clemens Buchacher","Steffen Jost"],"isPatch":true,"patchVersion":1,"patchTotal":1},"messages":[{"id":"369465","messageId":"20190216114938.18843-1-pclouds@gmail.com","threadId":"50525","inReplyTo":null,"subject":"[PATCH 0/1] Introduce \"precious\" file attribute","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-16T11:49:37Z","receivedAt":"2019-02-16T12:00:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Compared to the last round [1], the \"precious\" attribute is now only\nused by \"git clean\". \"git merge\" and \"git checkout\" will not abort when\nthey are about to overwrite precious files.\n\n[1] https://public-inbox.org/git/20181126193804.30741-1-pclouds@gmail.com/\n\nNguyễn Thái Ngọc Duy (1):\n  Introduce \"precious\" file concept\n\n Documentation/git-clean.txt     |  3 ++-\n Documentation/gitattributes.txt | 11 +++++++++++\n Documentation/gitignore.txt     |  4 ++++\n attr.c                          | 12 ++++++++++++\n attr.h                          |  2 ++\n builtin/clean.c                 | 20 +++++++++++++++++---\n t/t7300-clean.sh                | 29 +++++++++++++++++++++++++++++\n 7 files changed, 77 insertions(+), 4 deletions(-)\n\n-- \n2.21.0.rc0.328.g0e39304f8d\n\n"},{"id":"369466","messageId":"20190216114938.18843-2-pclouds@gmail.com","threadId":"50525","inReplyTo":"20190216114938.18843-1-pclouds@gmail.com","subject":"[PATCH 1/1] Introduce \"precious\" file concept","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-16T11:49:38Z","receivedAt":"2019-02-16T12:00:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"A new attribute \"precious\" is added to indicate that certain files\nhave valuable content and should not be easily discarded even if they\nare ignored or untracked.\n\nSo far there are one part of Git that are made aware of precious\nfiles: \"git clean\" will leave precious files alone.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/git-clean.txt     |  3 ++-\n Documentation/gitattributes.txt | 11 +++++++++++\n Documentation/gitignore.txt     |  4 ++++\n attr.c                          | 12 ++++++++++++\n attr.h                          |  2 ++\n builtin/clean.c                 | 20 +++++++++++++++++---\n t/t7300-clean.sh                | 29 +++++++++++++++++++++++++++++\n 7 files changed, 77 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt\nindex 03056dad0d..a9beadfb12 100644\n--- a/Documentation/git-clean.txt\n+++ b/Documentation/git-clean.txt\n@@ -21,7 +21,8 @@ option is specified, ignored files are also removed. This can, for\n example, be useful to remove all build products.\n \n If any optional `<path>...` arguments are given, only those paths\n-are affected.\n+are affected. Ignored or untracked files with `precious` attributes\n+are not removed.\n \n OPTIONS\n -------\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 9b41f81c06..1f2e830241 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -1192,6 +1192,17 @@ If this attribute is not set or has an invalid value, the value of the\n (See linkgit:git-config[1]).\n \n \n+Precious files\n+~~~~~~~~~~~~~~\n+\n+`precious`\n+^^^^^^^^^^\n+\n+This attribute is set on files to indicate that their content is\n+valuable. Some commands will behave slightly different on precious\n+files. linkgit:git-clean[1] will leave precious files alone.\n+\n+\n USING MACRO ATTRIBUTES\n ----------------------\n \ndiff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt\nindex 1c94f08ff4..0e9614289e 100644\n--- a/Documentation/gitignore.txt\n+++ b/Documentation/gitignore.txt\n@@ -141,6 +141,10 @@ not tracked by Git remain untracked.\n To stop tracking a file that is currently tracked, use\n 'git rm --cached'.\n \n+Ignored files are generally considered discardable. See `precious`\n+attribute in linkgit:gitattributes[5] to change the behavior regarding\n+ignored files.\n+\n EXAMPLES\n --------\n \ndiff --git a/attr.c b/attr.c\nindex fdd110bec5..6349410e14 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -1157,3 +1157,15 @@ void attr_start(void)\n \tpthread_mutex_init(&g_attr_hashmap.mutex, NULL);\n \tpthread_mutex_init(&check_vector.mutex, NULL);\n }\n+\n+int is_precious_file(struct index_state *istate, const char *path)\n+{\n+\tstatic struct attr_check *check;\n+\tif (!check)\n+\t\tcheck = attr_check_initl(\"precious\", NULL);\n+\tif (!check)\n+\t\treturn 0;\n+\n+\tgit_check_attr(istate, path, check);\n+\treturn ATTR_TRUE(check->items[0].value);\n+}\ndiff --git a/attr.h b/attr.h\nindex b0378bfe5f..b9a9751a66 100644\n--- a/attr.h\n+++ b/attr.h\n@@ -82,4 +82,6 @@ void git_attr_set_direction(enum git_attr_direction new_direction);\n \n void attr_start(void);\n \n+int is_precious_file(struct index_state *istate, const char *path);\n+\n #endif /* ATTR_H */\ndiff --git a/builtin/clean.c b/builtin/clean.c\nindex aaba4af3c2..47ec5e58cf 100644\n--- a/builtin/clean.c\n+++ b/builtin/clean.c\n@@ -18,6 +18,7 @@\n #include \"color.h\"\n #include \"pathspec.h\"\n #include \"help.h\"\n+#include \"attr.h\"\n \n static int force = -1; /* unset */\n static int interactive;\n@@ -31,6 +32,8 @@ static const char *const builtin_clean_usage[] = {\n \n static const char *msg_remove = N_(\"Removing %s\\n\");\n static const char *msg_would_remove = N_(\"Would remove %s\\n\");\n+static const char *msg_skip_precious = N_(\"Skipping precious file %s\\n\");\n+static const char *msg_would_skip_precious = N_(\"Would skip precious file %s\\n\");\n static const char *msg_skip_git_dir = N_(\"Skipping repository %s\\n\");\n static const char *msg_would_skip_git_dir = N_(\"Would skip repository %s\\n\");\n static const char *msg_warn_remove_failed = N_(\"failed to remove %s\");\n@@ -154,6 +157,7 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag,\n \tstruct dirent *e;\n \tint res = 0, ret = 0, gone = 1, original_len = path->len, len;\n \tstruct string_list dels = STRING_LIST_INIT_DUP;\n+\tconst char *rel_path;\n \n \t*dir_gone = 1;\n \n@@ -193,9 +197,16 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag,\n \n \t\tstrbuf_setlen(path, len);\n \t\tstrbuf_addstr(path, e->d_name);\n-\t\tif (lstat(path->buf, &st))\n+\t\tif (lstat(path->buf, &st)) {\n \t\t\t; /* fall thru */\n-\t\telse if (S_ISDIR(st.st_mode)) {\n+\t\t} else if ((!prefix && is_precious_file(&the_index, path->buf)) ||\n+\t\t\t   (prefix && skip_prefix(path->buf, prefix, &rel_path) &&\n+\t\t\t    is_precious_file(&the_index, rel_path))) {\n+\t\t\tquote_path_relative(path->buf, prefix, &quoted);\n+\t\t\tprintf(dry_run ? _(msg_would_skip_precious) : _(msg_skip_precious), quoted.buf);\n+\t\t\t*dir_gone = 0;\n+\t\t\tcontinue;\n+\t\t} else if (S_ISDIR(st.st_mode)) {\n \t\t\tif (remove_dirs(path, prefix, force_flag, dry_run, quiet, &gone))\n \t\t\t\tret = 1;\n \t\t\tif (gone) {\n@@ -1019,7 +1030,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)\n \t\tif (lstat(abs_path.buf, &st))\n \t\t\tcontinue;\n \n-\t\tif (S_ISDIR(st.st_mode)) {\n+\t\tif (is_precious_file(&the_index, item->string)) {\n+\t\t\tqname = quote_path_relative(item->string, NULL, &buf);\n+\t\t\tprintf(dry_run ? _(msg_would_skip_precious) : _(msg_skip_precious), qname);\n+\t\t} else if (S_ISDIR(st.st_mode)) {\n \t\t\tif (remove_dirs(&abs_path, prefix, rm_flags, dry_run, quiet, &gone))\n \t\t\t\terrors++;\n \t\t\tif (gone && !quiet) {\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 7b36954d63..a478a08a27 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -669,4 +669,33 @@ test_expect_success 'git clean -d skips untracked dirs containing ignored files'\n \ttest_path_is_missing foo/b/bb\n '\n \n+test_expect_success 'git clean -xd leaves precious files alone' '\n+\tgit init precious &&\n+\t(\n+\t\tcd precious &&\n+\t\ttest_commit one &&\n+\t\tcat >.gitignore <<-\\EOF &&\n+\t\t*.o\n+\t\t*.mak\n+\t\tEOF\n+\t\tcat >.gitattributes <<-\\EOF &&\n+\t\t*.mak precious\n+\t\t.gitattributes precious\n+\t\t*.precious precious\n+\t\tEOF\n+\t\tmkdir sub &&\n+\t\ttouch one.o sub/two.o one.mak sub/two.mak &&\n+\t\ttouch one.untracked two.precious sub/also.precious &&\n+\t\tgit clean -fdx &&\n+\t\ttest_path_is_missing one.o &&\n+\t\ttest_path_is_missing sub/two.o &&\n+\t\ttest_path_is_missing one.untracked &&\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_path_is_file one.mak &&\n+\t\ttest_path_is_file sub/two.mak &&\n+\t\ttest_path_is_file two.precious &&\n+\t\ttest_path_is_file sub/also.precious\n+\t)\n+'\n+\n test_done\n-- \n2.21.0.rc0.328.g0e39304f8d\n\n"},{"id":"369474","messageId":"87wolzo7a1.fsf@evledraar.gmail.com","threadId":"50525","inReplyTo":"20190216114938.18843-2-pclouds@gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-16T19:36:06Z","receivedAt":"2019-02-16T19:37:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Feb 16 2019, Nguyễn Thái Ngọc Duy wrote:\n\n[Re-CC some people involved the last time around]\n\n> A new attribute \"precious\" is added to indicate that certain files\n> have valuable content and should not be easily discarded even if they\n> are ignored or untracked.\n>\n> So far there are one part of Git that are made aware of precious\n> files: \"git clean\" will leave precious files alone.\n\nThanks for bringing this up again. There were also some patches recently\nto save away clobbered files, do you/anyone else have any end goal in\nmind here that combines this & that, or some other thing I may not have\nkept up with?\n\nMy commentary on this whole thing is basically a repeat of what I said\nin https://public-inbox.org/git/87wop0yvxv.fsf@evledraar.gmail.com/\n\nI.e. we have a definite problem here somewhere, and there is some\nsolution, but this patch feels a bit like navigating that maze in the\ndark without a map.\n\nWe had users report that the likes of \"pull\" were eating their data, but\nnow with this iteration of \"precious\" only impacting \"clean\" the only\nproblem anyone with the current semantics is still left unaddressed. My\nmemory (I may be wrong) is that \"clean\" was just brought up (by you?) as\na \"what about this other related case?\" in that whole discussion.\n\nSo as noted in the E-Mail linked above I think the first step should be\nto enumerate/document/test the cases where we're now eating data\nimplicitly, and discuss how that relates to the semantics we desired\nwhen the data-eating behavior was first introduced (as noted in E-Mails\nlinked from the above, my own preliminary digging seems to reveal there\nisn't much of a relationship between the two).\n\nOnly when we have that list of XYZ cases we're supporting now, and can\nsee that XYZ is so important to maintain backwards compatibility for\nthat we can't change it should way say \"we eat your data by default\nbecause XYZ is so useful/backcompat, set 'precious' ...\".\n\nBut right now we don't even have the list of XYZ or tests for them (as\nmy RFC \"garbage\" attribute patch revealed). So this whole thing still\nfeels like jumping three steps ahead to me in terms of addressing *that*\nissue, but perhaps you have some orthogonal use-case in mind for this?\n"},{"id":"369478","messageId":"CACsJy8CR7VGp7htC_wKC9BUCaQsmkp5Zd4+M7bddPL-jKyfDMQ@mail.gmail.com","threadId":"50525","inReplyTo":"87wolzo7a1.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-17T09:31:56Z","receivedAt":"2019-02-17T09:32:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Feb 17, 2019 at 2:36 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n>\n> On Sat, Feb 16 2019, Nguyễn Thái Ngọc Duy wrote:\n>\n> [Re-CC some people involved the last time around]\n>\n> > A new attribute \"precious\" is added to indicate that certain files\n> > have valuable content and should not be easily discarded even if they\n> > are ignored or untracked.\n> >\n> > So far there are one part of Git that are made aware of precious\n> > files: \"git clean\" will leave precious files alone.\n>\n> Thanks for bringing this up again. There were also some patches recently\n> to save away clobbered files, do you/anyone else have any end goal in\n> mind here that combines this & that, or some other thing I may not have\n> kept up with?\n\nI assume you mean the clobbering untracked files by merge/checkout.\nThose files will be backed up [1] if backup-log is implemented. Even\nfiles deleted by \"git clean\" could be saved but that might go a little\ntoo far.\n\n[1] https://public-inbox.org/git/20181209104419.12639-20-pclouds@gmail.com/\n\n> My commentary on this whole thing is basically a repeat of what I said\n> in https://public-inbox.org/git/87wop0yvxv.fsf@evledraar.gmail.com/\n>\n> I.e. we have a definite problem here somewhere, and there is some\n> solution, but this patch feels a bit like navigating that maze in the\n> dark without a map.\n>\n> We had users report that the likes of \"pull\" were eating their data, but\n> now with this iteration of \"precious\" only impacting \"clean\" the only\n> problem anyone with the current semantics is still left unaddressed. My\n> memory (I may be wrong) is that \"clean\" was just brought up (by you?) as\n> a \"what about this other related case?\" in that whole discussion.\n>\n> So as noted in the E-Mail linked above I think the first step should be\n> to enumerate/document/test the cases where we're now eating data\n> implicitly, and discuss how that relates to the semantics we desired\n> when the data-eating behavior was first introduced (as noted in E-Mails\n> linked from the above, my own preliminary digging seems to reveal there\n> isn't much of a relationship between the two).\n>\n> Only when we have that list of XYZ cases we're supporting now, and can\n> see that XYZ is so important to maintain backwards compatibility for\n> that we can't change it should way say \"we eat your data by default\n> because XYZ is so useful/backcompat, set 'precious' ...\".\n>\n> But right now we don't even have the list of XYZ or tests for them (as\n> my RFC \"garbage\" attribute patch revealed). So this whole thing still\n> feels like jumping three steps ahead to me in terms of addressing *that*\n> issue, but perhaps you have some orthogonal use-case in mind for this?\n\nI'm not addressing the accidentally losing data in this patch. My\nanswer for that would still be backup-log, if it ever gets merged. But\nthis patch is about _known_ files that I want to keep when doing \"git\nclean\", no more.\n-- \nDuy\n"},{"id":"369523","messageId":"87va1ho222.fsf@evledraar.gmail.com","threadId":"50525","inReplyTo":"CACsJy8CR7VGp7htC_wKC9BUCaQsmkp5Zd4+M7bddPL-jKyfDMQ@mail.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-18T09:53:25Z","receivedAt":"2019-02-18T09:53:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Feb 17 2019, Duy Nguyen wrote:\n\n> On Sun, Feb 17, 2019 at 2:36 AM Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n>>\n>>\n>> On Sat, Feb 16 2019, Nguyễn Thái Ngọc Duy wrote:\n>>\n>> [Re-CC some people involved the last time around]\n>>\n>> > A new attribute \"precious\" is added to indicate that certain files\n>> > have valuable content and should not be easily discarded even if they\n>> > are ignored or untracked.\n>> >\n>> > So far there are one part of Git that are made aware of precious\n>> > files: \"git clean\" will leave precious files alone.\n>>\n>> Thanks for bringing this up again. There were also some patches recently\n>> to save away clobbered files, do you/anyone else have any end goal in\n>> mind here that combines this & that, or some other thing I may not have\n>> kept up with?\n>\n> I assume you mean the clobbering untracked files by merge/checkout.\n> Those files will be backed up [1] if backup-log is implemented. Even\n> files deleted by \"git clean\" could be saved but that might go a little\n> too far.\n\nAnd I suppose if we have some mechanism for \"don't backup but error out\nif it was detectes as needed\" we'll have the equivalent of inverting the\nmerge/checkout --force behavior now (& my \"garbage\" patch), i.e. we'd\nstall on potential clobbering and need to carry on with --force.\n\n> [1] https://public-inbox.org/git/20181209104419.12639-20-pclouds@gmail.com/\n>\n>> My commentary on this whole thing is basically a repeat of what I said\n>> in https://public-inbox.org/git/87wop0yvxv.fsf@evledraar.gmail.com/\n>>\n>> I.e. we have a definite problem here somewhere, and there is some\n>> solution, but this patch feels a bit like navigating that maze in the\n>> dark without a map.\n>>\n>> We had users report that the likes of \"pull\" were eating their data, but\n>> now with this iteration of \"precious\" only impacting \"clean\" the only\n>> problem anyone with the current semantics is still left unaddressed. My\n>> memory (I may be wrong) is that \"clean\" was just brought up (by you?) as\n>> a \"what about this other related case?\" in that whole discussion.\n>>\n>> So as noted in the E-Mail linked above I think the first step should be\n>> to enumerate/document/test the cases where we're now eating data\n>> implicitly, and discuss how that relates to the semantics we desired\n>> when the data-eating behavior was first introduced (as noted in E-Mails\n>> linked from the above, my own preliminary digging seems to reveal there\n>> isn't much of a relationship between the two).\n>>\n>> Only when we have that list of XYZ cases we're supporting now, and can\n>> see that XYZ is so important to maintain backwards compatibility for\n>> that we can't change it should way say \"we eat your data by default\n>> because XYZ is so useful/backcompat, set 'precious' ...\".\n>>\n>> But right now we don't even have the list of XYZ or tests for them (as\n>> my RFC \"garbage\" attribute patch revealed). So this whole thing still\n>> feels like jumping three steps ahead to me in terms of addressing *that*\n>> issue, but perhaps you have some orthogonal use-case in mind for this?\n>\n> I'm not addressing the accidentally losing data in this patch. My\n> answer for that would still be backup-log, if it ever gets merged. But\n> this patch is about _known_ files that I want to keep when doing \"git\n> clean\", no more.\n\nIndeed. My concern is that we're making incremental steps without a\nclear idea of the end state, and once we get there we might find that\nsome steps along the way box us in or weren't what we wanted to solve\nthe overall UX issue.\n\nMore so in the CL / commit message not describing where we are overall,\nwhere this patch fits in (backup log, etc.) than this whole thing\n(backup log is already 24 patches) needing to be sent as one giant\nseries...\n"},{"id":"369525","messageId":"CACsJy8CpRWNzHBk4qGf-5BVBUDuE4HpuKvftvh3E1DC_pNcBKA@mail.gmail.com","threadId":"50525","inReplyTo":"87va1ho222.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-18T10:14:46Z","receivedAt":"2019-02-18T10:15:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Feb 18, 2019 at 4:53 PM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n>\n> On Sun, Feb 17 2019, Duy Nguyen wrote:\n>\n> > On Sun, Feb 17, 2019 at 2:36 AM Ævar Arnfjörð Bjarmason\n> > <avarab@gmail.com> wrote:\n> >>\n> >>\n> >> On Sat, Feb 16 2019, Nguyễn Thái Ngọc Duy wrote:\n> >>\n> >> [Re-CC some people involved the last time around]\n> >>\n> >> > A new attribute \"precious\" is added to indicate that certain files\n> >> > have valuable content and should not be easily discarded even if they\n> >> > are ignored or untracked.\n> >> >\n> >> > So far there are one part of Git that are made aware of precious\n> >> > files: \"git clean\" will leave precious files alone.\n> >>\n> >> Thanks for bringing this up again. There were also some patches recently\n> >> to save away clobbered files, do you/anyone else have any end goal in\n> >> mind here that combines this & that, or some other thing I may not have\n> >> kept up with?\n> >\n> > I assume you mean the clobbering untracked files by merge/checkout.\n> > Those files will be backed up [1] if backup-log is implemented. Even\n> > files deleted by \"git clean\" could be saved but that might go a little\n> > too far.\n>\n> And I suppose if we have some mechanism for \"don't backup but error out\n> if it was detectes as needed\" we'll have the equivalent of inverting the\n> merge/checkout --force behavior now (& my \"garbage\" patch), i.e. we'd\n> stall on potential clobbering and need to carry on with --force.\n\nYes that's another way to go.\n\n> > [1] https://public-inbox.org/git/20181209104419.12639-20-pclouds@gmail.com/\n> >\n> >> My commentary on this whole thing is basically a repeat of what I said\n> >> in https://public-inbox.org/git/87wop0yvxv.fsf@evledraar.gmail.com/\n> >>\n> >> I.e. we have a definite problem here somewhere, and there is some\n> >> solution, but this patch feels a bit like navigating that maze in the\n> >> dark without a map.\n> >>\n> >> We had users report that the likes of \"pull\" were eating their data, but\n> >> now with this iteration of \"precious\" only impacting \"clean\" the only\n> >> problem anyone with the current semantics is still left unaddressed. My\n> >> memory (I may be wrong) is that \"clean\" was just brought up (by you?) as\n> >> a \"what about this other related case?\" in that whole discussion.\n> >>\n> >> So as noted in the E-Mail linked above I think the first step should be\n> >> to enumerate/document/test the cases where we're now eating data\n> >> implicitly, and discuss how that relates to the semantics we desired\n> >> when the data-eating behavior was first introduced (as noted in E-Mails\n> >> linked from the above, my own preliminary digging seems to reveal there\n> >> isn't much of a relationship between the two).\n> >>\n> >> Only when we have that list of XYZ cases we're supporting now, and can\n> >> see that XYZ is so important to maintain backwards compatibility for\n> >> that we can't change it should way say \"we eat your data by default\n> >> because XYZ is so useful/backcompat, set 'precious' ...\".\n> >>\n> >> But right now we don't even have the list of XYZ or tests for them (as\n> >> my RFC \"garbage\" attribute patch revealed). So this whole thing still\n> >> feels like jumping three steps ahead to me in terms of addressing *that*\n> >> issue, but perhaps you have some orthogonal use-case in mind for this?\n> >\n> > I'm not addressing the accidentally losing data in this patch. My\n> > answer for that would still be backup-log, if it ever gets merged. But\n> > this patch is about _known_ files that I want to keep when doing \"git\n> > clean\", no more.\n>\n> Indeed. My concern is that we're making incremental steps without a\n> clear idea of the end state, and once we get there we might find that\n> some steps along the way box us in or weren't what we wanted to solve\n> the overall UX issue.\n>\n> More so in the CL / commit message not describing where we are overall,\n> where this patch fits in (backup log, etc.) than this whole thing\n> (backup log is already 24 patches) needing to be sent as one giant\n> series...\n\nNot seeing the overall picture is probably because I don't have any.\nBoth backup-log and this precious attribute are more at plumbing level\nthat could be combined to make something, but no I don't have the\nwhole final UX worked out. The end game for backup log may be \"git\nundo\" (or just --undo option across commands). The \"precious\"\nattribute does not exactly have any role in \"git undo\" picture. It's\njust refining the file classification (untracked, tracked, ignored)\nthat we have although it could be used as hints for \"git undo\". That's\nall I got.\n-- \nDuy\n"},{"id":"369656","messageId":"xmqq8syb3b3j.fsf@gitster-ct.c.googlers.com","threadId":"50525","inReplyTo":"CACsJy8CR7VGp7htC_wKC9BUCaQsmkp5Zd4+M7bddPL-jKyfDMQ@mail.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-02-19T18:08:16Z","receivedAt":"2019-02-19T18:08:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> On Sun, Feb 17, 2019 at 2:36 AM Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n>>\n>>\n>> On Sat, Feb 16 2019, Nguyễn Thái Ngọc Duy wrote:\n>>\n>> [Re-CC some people involved the last time around]\n>>\n>> > A new attribute \"precious\" is added to indicate that certain files\n>> > have valuable content and should not be easily discarded even if they\n>> > are ignored or untracked.\n>> >\n>> > So far there are one part of Git that are made aware of precious\n>> > files: \"git clean\" will leave precious files alone.\n>>\n>> Thanks for bringing this up again. There were also some patches recently\n>> to save away clobbered files, do you/anyone else have any end goal in\n>> mind here that combines this & that, or some other thing I may not have\n>> kept up with?\n>\n> I assume you mean the clobbering untracked files by merge/checkout.\n> Those files will be backed up [1] if backup-log is implemented. Even\n> files deleted by \"git clean\" could be saved but that might go a little\n> too far.\n\nI agree with Ævar that it is a very good idea to ask what the\nendgame should look like.  I would have expected that, with an\nintroduction of new \"ignored but unexpendable\" class of file\n(i.e. \"precious\" here), operations such as merge and checkout will\nbe updated to keep them in situations where we would remove \"ignored\nand expendable\" files (i.e. \"ignored\").  And it is perfectly OK if\nthe very first introduction of the \"precious\" support begins only\nwith a single operation, such as \"clean\", as long as the end-goal is\nclear.\n\nI personally do not believe in \"backup log\"; if we can screw up and\ncan fail to stop an operation that must avoid losing info, then we\ncan screw up the same way and fail to design and implement \"backup\"\nto save info before an operation loses it.  If we do a good job in\nsupporting \"precious\" in various operations, we can rely less on\n\"backup log\" and still be safe ;-)\n\n"},{"id":"369688","messageId":"CACsJy8Dq9_uFofs40XwjLkmiBNWXCpic96W1MK_tjLQyaF0+BA@mail.gmail.com","threadId":"50525","inReplyTo":"xmqq8syb3b3j.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-20T01:35:41Z","receivedAt":"2019-02-20T01:36:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 20, 2019 at 1:08 AM Junio C Hamano <gitster@pobox.com> wrote:\n>\n> Duy Nguyen <pclouds@gmail.com> writes:\n>\n> > On Sun, Feb 17, 2019 at 2:36 AM Ævar Arnfjörð Bjarmason\n> > <avarab@gmail.com> wrote:\n> >>\n> >>\n> >> On Sat, Feb 16 2019, Nguyễn Thái Ngọc Duy wrote:\n> >>\n> >> [Re-CC some people involved the last time around]\n> >>\n> >> > A new attribute \"precious\" is added to indicate that certain files\n> >> > have valuable content and should not be easily discarded even if they\n> >> > are ignored or untracked.\n> >> >\n> >> > So far there are one part of Git that are made aware of precious\n> >> > files: \"git clean\" will leave precious files alone.\n> >>\n> >> Thanks for bringing this up again. There were also some patches recently\n> >> to save away clobbered files, do you/anyone else have any end goal in\n> >> mind here that combines this & that, or some other thing I may not have\n> >> kept up with?\n> >\n> > I assume you mean the clobbering untracked files by merge/checkout.\n> > Those files will be backed up [1] if backup-log is implemented. Even\n> > files deleted by \"git clean\" could be saved but that might go a little\n> > too far.\n>\n> I agree with Ævar that it is a very good idea to ask what the\n> endgame should look like.  I would have expected that, with an\n> introduction of new \"ignored but unexpendable\" class of file\n> (i.e. \"precious\" here), operations such as merge and checkout will\n> be updated to keep them in situations where we would remove \"ignored\n> and expendable\" files (i.e. \"ignored\").  And it is perfectly OK if\n> the very first introduction of the \"precious\" support begins only\n> with a single operation, such as \"clean\", as long as the end-goal is\n> clear.\n\nI think the sticking point is how to deal with the surprise factor and\n\"precious\" will not help at all in this aspect. In my mind there are\nthree classes\n\n - total expectation, i know i want git to not touch some files, i\ntell git so (e.g. with \"precious\")\n\n - surprises sometimes, but in known classes. This is the main use\ncase of backup log, where I may accidentally do \"git commit\n-amsomething\" after carefully preparing the index. Saving overwritten\nfiles by merge/checkout could be done here as an alternative to\n\"garbage\" attribute.\n\n> I personally do not believe in \"backup log\"; if we can screw up and\n> can fail to stop an operation that must avoid losing info, then we\n> can screw up the same way and fail to design and implement \"backup\"\n> to save info before an operation loses it.  If we do a good job in\n> supporting \"precious\" in various operations, we can rely less on\n> \"backup log\" and still be safe ;-)\n\nand this is the third class, something completely unexpected. Yes\nbackup-log can't help here, but I don't think \"precious\" can either.\nAnd I have no good proposal for this case.\n-- \nDuy\n"},{"id":"369696","messageId":"49F0F61F-9874-4027-8430-E313AA46C83D@gmx.net","threadId":"50525","inReplyTo":"CACsJy8Dq9_uFofs40XwjLkmiBNWXCpic96W1MK_tjLQyaF0+BA@mail.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Clemens Buchacher","fromEmail":"drizzd@gmx.net","sentAt":"2019-02-20T08:31:36Z","receivedAt":"2019-02-20T08:32:07Z","isPatch":true,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"\n\nOn February 20, 2019 2:35:41 AM GMT+01:00, Duy Nguyen <pclouds@gmail.com> wrote:\n>On Wed, Feb 20, 2019 at 1:08 AM Junio C Hamano <gitster@pobox.com>\n>wrote:\n>>\n>> Duy Nguyen <pclouds@gmail.com> writes:\n>>\n>> > On Sun, Feb 17, 2019 at 2:36 AM Ævar Arnfjörð Bjarmason\n>> > <avarab@gmail.com> wrote:\n>> >>\n>> >>\n>> >> On Sat, Feb 16 2019, Nguyễn Thái Ngọc Duy wrote:\n>> >>\n>> >> [Re-CC some people involved the last time around]\n>> >>\n>> >> > A new attribute \"precious\" is added to indicate that certain\n>files\n>> >> > have valuable content and should not be easily discarded even if\n>they\n>> >> > are ignored or untracked.\n>> >> >\n>> >> > So far there are one part of Git that are made aware of precious\n>> >> > files: \"git clean\" will leave precious files alone.\n>> >>\n>> >> Thanks for bringing this up again. There were also some patches\n>recently\n>> >> to save away clobbered files, do you/anyone else have any end goal\n>in\n>> >> mind here that combines this & that, or some other thing I may not\n>have\n>> >> kept up with?\n>> >\n>> > I assume you mean the clobbering untracked files by merge/checkout.\n>> > Those files will be backed up [1] if backup-log is implemented.\n>Even\n>> > files deleted by \"git clean\" could be saved but that might go a\n>little\n>> > too far.\n>>\n>> I agree with Ævar that it is a very good idea to ask what the\n>> endgame should look like.  I would have expected that, with an\n>> introduction of new \"ignored but unexpendable\" class of file\n>> (i.e. \"precious\" here), operations such as merge and checkout will\n>> be updated to keep them in situations where we would remove \"ignored\n>> and expendable\" files (i.e. \"ignored\").  And it is perfectly OK if\n>> the very first introduction of the \"precious\" support begins only\n>> with a single operation, such as \"clean\", as long as the end-goal is\n>> clear.\n>\n>I think the sticking point is how to deal with the surprise factor and\n>\"precious\" will not help at all in this aspect. In my mind there are\n>three classes\n>\n> - total expectation, i know i want git to not touch some files, i\n>tell git so (e.g. with \"precious\")\n>\n> - surprises sometimes, but in known classes. This is the main use\n>case of backup log, where I may accidentally do \"git commit\n>-amsomething\" after carefully preparing the index. Saving overwritten\n>files by merge/checkout could be done here as an alternative to\n>\"garbage\" attribute.\n>\n>> I personally do not believe in \"backup log\"; if we can screw up and\n>> can fail to stop an operation that must avoid losing info, then we\n>> can screw up the same way and fail to design and implement \"backup\"\n>> to save info before an operation loses it.  If we do a good job in\n>> supporting \"precious\" in various operations, we can rely less on\n>> \"backup log\" and still be safe ;-)\n>\n>and this is the third class, something completely unexpected. Yes\n>backup-log can't help here, but I don't think \"precious\" can either.\n>And I have no good proposal for this case.\n\nSorry for going off on a tangent here, but I have had this on my mind for a long time. For cases where merge can lead to loss of a non-ignored untracked file (t7607-merge-overwrite.sh), I have the following proposal:\n\n1. Merge the ORIG_HEAD and MERGE_HEAD commits without touching the index or the work tree. This is where we do rename detection, recursive merge, and content (line-by-line) merge. The result is CHECKOUT_HEAD, a tree with possible merge conflicts. For the switch branch operation CHECKOUT_HEAD is the tree to switch to. The remaining steps are the same for merge and switch branch operations.\n2. Merge CHECKOUT_HEAD and the index with ORIG_HEAD as the merge base. The result is the CHECKOUT_INDEX. Do this in order to keep staged changes which are not affected by the merge. Do not do rename detection or content merge. In case of conflict, rollback and error out.\n3. Merge CHECKOUT_INDEX with the work tree with the original index as merge base. Do this to simulate the work tree update. Dp not do remame detection or content merge. A conflict means that the checkout operation would touch untracked files or files with unstaged changes. In case of such a conflict, rollback and error out.\n\nI believe this algorithm would behave much like the current implementation. But it separates the rename/history/content aspects of the merge algorithm from the checkout operation. It greatly simplifies the implementation of the checkout operation and there are no special cases where we lose files. Implementing step 1 is the tricky part. But it may still be worthwhile because the merge algorithm does not have to worry about staged changes or unstaged changes. The merge algorithm could work on the hierarchical tree structure instead of the flattened index. This makes it trivial to detect directory/file conflicts (no need to do a lookahead when iterating index files). This is also a better fit for detecting directory renames. Maybe this will allow us to focus more on rename detection, such as directory renames or moved functions [*1*]. \n\n[*1*] Also: moved files where the original file is replaced with a wrapper for the moved file always fools rename detection because we don't detect renames for files which were not removed.\n"},{"id":"369697","messageId":"87h8cy6cme.fsf@evledraar.booking.com","threadId":"50525","inReplyTo":"xmqq8syb3b3j.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-20T09:19:21Z","receivedAt":"2019-02-20T09:19:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 19 2019, Junio C Hamano wrote:\n\n> Duy Nguyen <pclouds@gmail.com> writes:\n>\n>> On Sun, Feb 17, 2019 at 2:36 AM Ævar Arnfjörð Bjarmason\n>> <avarab@gmail.com> wrote:\n>>>\n>>>\n>>> On Sat, Feb 16 2019, Nguyễn Thái Ngọc Duy wrote:\n>>>\n>>> [Re-CC some people involved the last time around]\n>>>\n>>> > A new attribute \"precious\" is added to indicate that certain files\n>>> > have valuable content and should not be easily discarded even if they\n>>> > are ignored or untracked.\n>>> >\n>>> > So far there are one part of Git that are made aware of precious\n>>> > files: \"git clean\" will leave precious files alone.\n>>>\n>>> Thanks for bringing this up again. There were also some patches recently\n>>> to save away clobbered files, do you/anyone else have any end goal in\n>>> mind here that combines this & that, or some other thing I may not have\n>>> kept up with?\n>>\n>> I assume you mean the clobbering untracked files by merge/checkout.\n>> Those files will be backed up [1] if backup-log is implemented. Even\n>> files deleted by \"git clean\" could be saved but that might go a little\n>> too far.\n>\n> I agree with Ævar that it is a very good idea to ask what the\n> endgame should look like.  I would have expected that, with an\n> introduction of new \"ignored but unexpendable\" class of file\n> (i.e. \"precious\" here), operations such as merge and checkout will\n> be updated to keep them in situations where we would remove \"ignored\n> and expendable\" files (i.e. \"ignored\").  And it is perfectly OK if\n> the very first introduction of the \"precious\" support begins only\n> with a single operation, such as \"clean\", as long as the end-goal is\n> clear.\n\nFWIW I'm in full agreement with that.\n\n> I personally do not believe in \"backup log\"; if we can screw up and\n> can fail to stop an operation that must avoid losing info, then we\n> can screw up the same way and fail to design and implement \"backup\"\n> to save info before an operation loses it.\n\nYes, there could be some unforseen interaction between git commands\nwhere we should have such a backup log, but did not think to implement\nit. I'd hope such cases would be reported, and we could fix them.\n\nBut those sorts of cases aren't why we started discussing this, rather\nwe *know* what the data shredding command interaction is, but there\nwasn't a consensus for just not shredding data by default by making\nusers use \"checkout -f\" or \"merge -f\" to proceed. I.e. taking some\nvariant of my \"trashable\" patch[1].\n\n> If we do a good job in\n> supporting \"precious\" in various operations, we can rely less on\n> \"backup log\" and still be safe ;-)\n\nIs noted in previous discussions[2] I think that's entirely\nimplausible. I think at best the \"precious\" facility will be used to\nmark e.g *.o files as \"don't check in, but don't clean (Makefile handles\nit)\".\n\nMost git users are at the level of only knowing very basic\nadd/commit/pull/push command interaction. I feel strongly that we need\nto make our tools safe to use by default, and not require some\nrelatively advanced \"precious\"/attribute facility to be carefully\nconfigured in advance so we don't throw away uncommitted work on the\nlikes of merge/checkout.\n\n1. https://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\n2. https://public-inbox.org/git/871s7r4wuv.fsf@evledraar.gmail.com/\n"},{"id":"369698","messageId":"CACsJy8B15hORnaOdYW8TNE3Gniv9NBJopyLYmHR5iF0U3beq6g@mail.gmail.com","threadId":"50525","inReplyTo":"87h8cy6cme.fsf@evledraar.booking.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-20T09:41:51Z","receivedAt":"2019-02-20T09:42:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 20, 2019 at 4:19 PM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n> > I personally do not believe in \"backup log\"; if we can screw up and\n> > can fail to stop an operation that must avoid losing info, then we\n> > can screw up the same way and fail to design and implement \"backup\"\n> > to save info before an operation loses it.\n>\n> Yes, there could be some unforseen interaction between git commands\n> where we should have such a backup log, but did not think to implement\n> it. I'd hope such cases would be reported, and we could fix them.\n>\n> But those sorts of cases aren't why we started discussing this, rather\n> we *know* what the data shredding command interaction is, but there\n> wasn't a consensus for just not shredding data by default by making\n> users use \"checkout -f\" or \"merge -f\" to proceed. I.e. taking some\n> variant of my \"trashable\" patch[1].\n>\n> > If we do a good job in\n> > supporting \"precious\" in various operations, we can rely less on\n> > \"backup log\" and still be safe ;-)\n>\n> Is noted in previous discussions[2] I think that's entirely\n> implausible. I think at best the \"precious\" facility will be used to\n> mark e.g *.o files as \"don't check in, but don't clean (Makefile handles\n> it)\".\n>\n> Most git users are at the level of only knowing very basic\n> add/commit/pull/push command interaction. I feel strongly that we need\n> to make our tools safe to use by default, and not require some\n> relatively advanced \"precious\"/attribute facility to be carefully\n> configured in advance so we don't throw away uncommitted work on the\n> likes of merge/checkout.\n\nThere is a trade off somewhere. \"new user first\" should not come at\nthe cost for more experienced users.\n\nMaking \"git checkout/merge\" abort while it's working before breaks\nscripts. And requiring to mark trashable files manually duplicates a\nlot of ignore patterns. Have a look at any .gitignore file, the\nmajority of them is for discardable files because \"ignored\" class was\ncreated with those in mind (*.o and friends). So now you would need to\nadd more or less the same set of ignore rules in .gitattributes to\nmark them trashable, and gitignore/gitattributes rules are not exactly\ncompatible, you can't just blindly copy them over. Every time you add\none more .gitignore rule, there's a good chance you need to add a\nsimilar rule for trashable attribute.\n\nMaybe we just add a new \"newbie\" config knob and turn on the safety\nnets on. Leave the knob on by default. And I will turn it off in my\n~/.gitconfig as soon as it's real.\n-- \nDuy\n"},{"id":"369699","messageId":"6c40dcc2-5ef1-7cfb-858d-029b72fa3709@tcs.ifi.lmu.de","threadId":"50525","inReplyTo":"87h8cy6cme.fsf@evledraar.booking.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Steffen Jost","fromEmail":"jost@tcs.ifi.lmu.de","sentAt":"2019-02-20T09:36:51Z","receivedAt":"2019-02-20T09:42:42Z","isPatch":true,"sender":{"key":"jost@tcs.ifi.lmu.de","avatar":null},"body":"On 20.02.19 10:19, Ævar Arnfjörð Bjarmason wrote:\n> Most git users are at the level of only knowing very basic\n> add/commit/pull/push command interaction. I feel strongly that we need\n> to make our tools safe to use by default, and not require some\n> relatively advanced \"precious\"/attribute facility to be carefully\n> configured in advance so we don't throw away uncommitted work on the\n> likes of merge/checkout.\n> \n> 1. https://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\n> 2. https://public-inbox.org/git/871s7r4wuv.fsf@evledraar.gmail.com/\n> \n\n+1\nPlease consider that silently deleting files is a no-go.\n\nI teach computer science, and our switch from subversion to git for our second year programming projects caused a lot of grief, so much that my colleagues consider switching back to subversion as the point of first contact with revisioning.\n\nSilently deleting partially revisioned files is a major source: students regularly destroy IDE or OS specific config files that they cannot restore themselves. (Project participants use all kinds of different IDEs on different OSs and thus have all kinds of weird hidden files that always manage to get checked into the repository, wreaking havoc on another's machine. So they get deleted and thus disturb the student that needed those files.) We do provide a huge .gitignore that ought to prevent this, but despite numerous warnings they only add it later, which then causes previously checked-in files to be lost upon switching between branches.\n\nPlease, by default, issue at least a warning before files are irrevocably los - or maybe keep a local snapshot of everything for the last few checkout in order to undo them?\n\n\nThanks,\n  Steffen.\n\n-- \n+49-89-2180-9139\nhttp://www.tcs.ifi.lmu.de/~jost/\n\nLehr- und Forschungseinheit für Theoretische Informatik\nLudwig-Maximilians-Universität München\nOettingenstr. 67 (E111)\n80538 München\nBAVARIA\n"},{"id":"369700","messageId":"87ftsi68ke.fsf@evledraar.gmail.com","threadId":"50525","inReplyTo":"CACsJy8B15hORnaOdYW8TNE3Gniv9NBJopyLYmHR5iF0U3beq6g@mail.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-02-20T10:46:57Z","receivedAt":"2019-02-20T10:47:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 20 2019, Duy Nguyen wrote:\n\n> On Wed, Feb 20, 2019 at 4:19 PM Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n>> > I personally do not believe in \"backup log\"; if we can screw up and\n>> > can fail to stop an operation that must avoid losing info, then we\n>> > can screw up the same way and fail to design and implement \"backup\"\n>> > to save info before an operation loses it.\n>>\n>> Yes, there could be some unforseen interaction between git commands\n>> where we should have such a backup log, but did not think to implement\n>> it. I'd hope such cases would be reported, and we could fix them.\n>>\n>> But those sorts of cases aren't why we started discussing this, rather\n>> we *know* what the data shredding command interaction is, but there\n>> wasn't a consensus for just not shredding data by default by making\n>> users use \"checkout -f\" or \"merge -f\" to proceed. I.e. taking some\n>> variant of my \"trashable\" patch[1].\n>>\n>> > If we do a good job in\n>> > supporting \"precious\" in various operations, we can rely less on\n>> > \"backup log\" and still be safe ;-)\n>>\n>> Is noted in previous discussions[2] I think that's entirely\n>> implausible. I think at best the \"precious\" facility will be used to\n>> mark e.g *.o files as \"don't check in, but don't clean (Makefile handles\n>> it)\".\n>>\n>> Most git users are at the level of only knowing very basic\n>> add/commit/pull/push command interaction. I feel strongly that we need\n>> to make our tools safe to use by default, and not require some\n>> relatively advanced \"precious\"/attribute facility to be carefully\n>> configured in advance so we don't throw away uncommitted work on the\n>> likes of merge/checkout.\n>\n> There is a trade off somewhere. \"new user first\" should not come at\n> the cost for more experienced users.\n>\n> Making \"git checkout/merge\" abort while it's working before breaks\n> scripts. And requiring to mark trashable files manually duplicates a\n> lot of ignore patterns. Have a look at any .gitignore file, the\n> majority of them is for discardable files because \"ignored\" class was\n> created with those in mind (*.o and friends). So now you would need to\n> add more or less the same set of ignore rules in .gitattributes to\n> mark them trashable, and gitignore/gitattributes rules are not exactly\n> compatible, you can't just blindly copy them over. Every time you add\n> one more .gitignore rule, there's a good chance you need to add a\n> similar rule for trashable attribute.\n>\n> Maybe we just add a new \"newbie\" config knob and turn on the safety\n> nets on. Leave the knob on by default. And I will turn it off in my\n> ~/.gitconfig as soon as it's real.\n\nOh yes, as noted upthread (\"My commentary on this whole thing...\"[1] )\nmy position on what we should do at this point is not that we should\ndefinitely go one way or the other, but that more investigation is\nneeded.\n\nAs my \"trashable\"[2] patch makes clear we don't even have good tests or\ndocumentation for these cases, which would be a good first step.\n\nThe one thing that *is* clear from my digging a few months back is that\nthe behavior we have now in git is overzealous when we look at the\ninitial case reported by Shawn way back when it was added.\n\nSpecifically, the intention back in 2007 was to fix a case where \"git\ncheckout\" (\"read-tree -m\", but whatever) would barf on a branch switch\nwhere switching needed to replace a *tracked* \"smth\" with a *tracked*\n\"smth/file\", or the other way around[3].\n\nDoes that mean we can just back that behavior out? No, because people\nmight have come to rely on it, but we should start with seeing exactly\nwhat it *does* do, whether all those things are important or intended,\nand maybe we can weight the shredding/backcompat trade-off for some of\nthose differently than others.\n\nSo the obvious thing to try would be to see if we can narrowly keep the\nbehavior where we end up shredding a file on disk, *but* are switching\nbetween two trees A & B where that have/don't have that file.\n\nOr more generously, try to \"git hash-object\" arbitrary files we're about\nto shred, and check if it's already in the object database. That would\ncatch case where e.g. the user switching from A->B and would shred a\nfile, but it (or conflicting dir) is known to neither \"A\" nor \"B\", but\nexists as a checked-in file in unrelated commit \"C\", which the user\nrecently had checked out (and e.g. their editor auto saved it as-is or\nsomething...).\n\nBut it's entirely possible that after all that digging we'll come to the\nconclusion that we can't change this at all, and we're just going to\nlive with all the current caveats.\n\nThat doesn't mean that having what amounts to a power user feature to\nmitigate that damage if you know git well enough that it's going to be a\nproblem is going to help anything but a small minority of users. So\n\"dude, where's my data?\" problem will still exist.\n\nEven then there room to maneuver, e.g.:\n\n X. Perhaps after investigating it's not acceptable to change the\n    default for script use, but could we require --force if we detect\n    that we're connected to a terminal?\n\n Y. Or if even that is considered too much, we could have something like\n    how help.autoCorrect works, where if we detect we're about to eat\n    data we wait for 10 seconds, and invite the user to Ctrl+C now\n    because we're about to clobber file \"xyz\".\n\n Z. It's for whatever reason still unacceptable to do X or Y (or some\n    similar mitigation) for all cases of file shredding, but would be OK\n    for some specific sub-cases (e.g. the not known to git-hash-object\n    case above), and we have reason to suspect that such a narrow\n    mitigation strikes the right trade-off between backwards\n    compatibility and preventing the \"dude, where's my data?\" reports we\n    get about this periodically.\n\n1. https://public-inbox.org/git/87wolzo7a1.fsf@evledraar.gmail.com/\n2. https://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\n3. https://public-inbox.org/git/87wopj3661.fsf@evledraar.gmail.com/\n"},{"id":"369702","messageId":"B168DCB1-7A69-4729-89C7-B513464DD468@gmx.net","threadId":"50525","inReplyTo":"CACsJy8B15hORnaOdYW8TNE3Gniv9NBJopyLYmHR5iF0U3beq6g@mail.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Clemens Buchacher","fromEmail":"drizzd@gmx.net","sentAt":"2019-02-20T11:11:12Z","receivedAt":"2019-02-20T11:11:43Z","isPatch":true,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"\n\nOn February 20, 2019 10:41:51 AM GMT+01:00, Duy Nguyen <pclouds@gmail.com> wrote:\n>Making \"git checkout/merge\" abort while it's working before breaks\n>scripts.\n\nChange is always a trade-off. We should not reject change without considering the merits. Once we agree on the desired state, we can think about the migration strategy. \n\n>And requiring to mark trashable files manually duplicates a\n>lot of ignore patterns. Have a look at any .gitignore file, the\n>majority of them is for discardable files because \"ignored\" class was\n>created with those in mind (*.o and friends). So now you would need to\n>add more or less the same set of ignore rules in .gitattributes to\n>mark them trashable, and gitignore/gitattributes rules are not exactly\n>compatible, you can't just blindly copy them over. Every time you add\n>one more .gitignore rule, there's a good chance you need to add a\n>similar rule for trashable attribute.\n\nI agree that ignored precious files are typically a small subset of the ignore files. Maintaining separate rules for ignored files and for trashable files would result in a lot of duplication.\n\nOn the other hand, how frequently do we really have to trash ignored files? Trashing a file should only be necessary if a tracked file overwrites an ignored file. When does this happen? I don't think it will happen for *.o files. So in most cases, there is simply no need to specify which files are precious. The default could simply be that all files are precious.\n\nTo support more complex use cases, we could specify precious files in addition to ignored files. Only if we specify precious files (and/or enable the ignored-are-trashable config option on a repository level), all other files become trashable.\n\nFunctionally this is equivalent the newbie option which you suggest, but I think it is not an issue of newbie vs experienced users but an issue of common vs special use cases.\n\n>Maybe we just add a new \"newbie\" config knob and turn on the safety\n>nets on. Leave the knob on by default. And I will turn it off in my\n>~/.gitconfig as soon as it's real.\n"},{"id":"369743","messageId":"xmqqsgwium3o.fsf@gitster-ct.c.googlers.com","threadId":"50525","inReplyTo":"CACsJy8Dq9_uFofs40XwjLkmiBNWXCpic96W1MK_tjLQyaF0+BA@mail.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-02-20T22:32:59Z","receivedAt":"2019-02-20T22:33:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n>  - surprises sometimes, but in known classes. This is the main use\n> case of backup log, where I may accidentally do \"git commit\n> -amsomething\" after carefully preparing the index. Saving overwritten\n> files by merge/checkout could be done here as an alternative to\n> \"garbage\" attribute.\n\nThe problem with either of these is not the \"saving\" half, but\n\"restoring\".  For different people, and for different cases even to\nthe same person, granularity of what is perceived as a single action\nis different, so \"give me the state of my working tree files before\nthe last operation\" is a request that does not have a good\ndefinition.  After finishing \"git rebase\" that replays 3 changes,\none of which needs manual conflict resolution, you may realize that\nyou made an incorrect resolution for one path but not others.  How\nwould you let the user say \"no, I do not want to undo the whole\nrebase, I want to go back to the state where I replayed the first\nchange, saw the conflicts while replaying the second change, and\nresolved them in these files, but before touching that last one I\nscrewed up resolving, so that I can correct\"?\n\nPiling many \"backups\" (or \"snapshots\") on top of each other is the\neasier part; I'd expect that it would be a much harder design\nproblem to let users make use of them in meaningful ways, and that\nis the primary reason why I am skeptical.\n\n"},{"id":"369744","messageId":"xmqqo976ultb.fsf@gitster-ct.c.googlers.com","threadId":"50525","inReplyTo":"CACsJy8B15hORnaOdYW8TNE3Gniv9NBJopyLYmHR5iF0U3beq6g@mail.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-02-20T22:39:12Z","receivedAt":"2019-02-20T22:39:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> There is a trade off somewhere. \"new user first\" should not come at\n> the cost for more experienced users.\n\nProbably.  Nobody will stay being newbie forever.\n\n> Making \"git checkout/merge\" abort while it's working before breaks\n> scripts. And requiring to mark trashable files manually duplicates a\n> lot of ignore patterns. Have a look at any .gitignore file, the\n> majority of them is for discardable files because \"ignored\" class was\n> created with those in mind (*.o and friends).\n\nVery true.  That is why we were OK for so long with \"ignored\" that\nmeans \"ignored and expendable\".  We know in some situations we want\n\"ignored but precious\", and that is why we are discussing this topic.\n\n> So now you would need to\n> add more or less the same set of ignore rules in .gitattributes to\n> mark them trashable, and gitignore/gitattributes rules are not exactly\n> compatible, you can't just blindly copy them over. Every time you add\n> one more .gitignore rule, there's a good chance you need to add a\n> similar rule for trashable attribute.\n\nI am not sure why you would even need to _duplicate_.\n\nAre you saying for each and every rule that specify \"ignored and\nexpendable\" in .gitignore there always will be \"ignored but\nprecious\" exception that match the pattern?  Given that we have been\nOK for so long without even needing \"precious\", I find it somewhat\nunrealistic to assume so.\n"},{"id":"369896","messageId":"CACsJy8CB=P9T0XJMWQetExgwDyFN78nuJZq8FtmzG+V1fBY4ig@mail.gmail.com","threadId":"50525","inReplyTo":"xmqqo976ultb.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-22T09:35:44Z","receivedAt":"2019-02-22T09:36:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Feb 21, 2019 at 5:39 AM Junio C Hamano <gitster@pobox.com> wrote:\n> > So now you would need to\n> > add more or less the same set of ignore rules in .gitattributes to\n> > mark them trashable, and gitignore/gitattributes rules are not exactly\n> > compatible, you can't just blindly copy them over. Every time you add\n> > one more .gitignore rule, there's a good chance you need to add a\n> > similar rule for trashable attribute.\n>\n> I am not sure why you would even need to _duplicate_.\n>\n> Are you saying for each and every rule that specify \"ignored and\n> expendable\" in .gitignore there always will be \"ignored but\n> precious\" exception that match the pattern?  Given that we have been\n> OK for so long without even needing \"precious\", I find it somewhat\n> unrealistic to assume so.\n\nIf all ignored files are now redefined as precious and we mark them\nexpendable with trashable attribute, then we need to duplicate most of\nthe rules. The \"precious\" attribute of course does not have this\nproblem since precious-and-ignored files should be rare.\n-- \nDuy\n"},{"id":"369897","messageId":"CACsJy8C377NmLv9edNYjinKAQf-P1y5+Nwhdj3vRkz_E__x43Q@mail.gmail.com","threadId":"50525","inReplyTo":"B168DCB1-7A69-4729-89C7-B513464DD468@gmx.net","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-02-22T09:46:23Z","receivedAt":"2019-02-22T09:46:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Feb 20, 2019 at 6:11 PM Clemens Buchacher <drizzd@gmx.net> wrote:\n> >And requiring to mark trashable files manually duplicates a\n> >lot of ignore patterns. Have a look at any .gitignore file, the\n> >majority of them is for discardable files because \"ignored\" class was\n> >created with those in mind (*.o and friends). So now you would need to\n> >add more or less the same set of ignore rules in .gitattributes to\n> >mark them trashable, and gitignore/gitattributes rules are not exactly\n> >compatible, you can't just blindly copy them over. Every time you add\n> >one more .gitignore rule, there's a good chance you need to add a\n> >similar rule for trashable attribute.\n>\n> I agree that ignored precious files are typically a small subset of the ignore files. Maintaining separate rules for ignored files and for trashable files would result in a lot of duplication.\n>\n> On the other hand, how frequently do we really have to trash ignored files? Trashing a file should only be necessary if a tracked file overwrites an ignored file. When does this happen? I don't think it will happen for *.o files. So in most cases, there is simply no need to specify which files are precious. The default could simply be that all files are precious.\n>\n> To support more complex use cases, we could specify precious files in addition to ignored files. Only if we specify precious files (and/or enable the ignored-are-trashable config option on a repository level), all other files become trashable.\n>\n> Functionally this is equivalent the newbie option which you suggest, but I think it is not an issue of newbie vs experienced users but an issue of common vs special use cases.\n\nSo far the two conflicting cases are \"git checkout/merge\" and \"git\nclean\". Ignored files are valuable by default in the former, while\nit's expendable by default in the latter.\n\nSo if you add this ignored-are-trashable config key (defaults to\nfalse), git-clean -if will not do anything anymore. We _could_ advice\nthe user to turn the config on (with all the consequences). I don't\nknow if we have any other use cases that deserve the same advice.\n\nAnother option is simply leave the expendable/precious nature of\nignored files undefined like it is now and handle case by case:\n\n - git-clean learns to use a new attribute \"clean\". Undefined\nattribute is seen as +clean. To keep some files \"precious\" you update\n.gitattributes and add -clean rules.\n - git merge/checkout learns another attribute,\ncheckout-overwrite-ignore? Undefined attribute is seen as\n-checkout-overwrite-ignore (i.e. abort the operation).\n\nWe stay away from any generic attribute name in this direction to make\nclear it's only applicable to specific use cases.\n-- \nDuy\n"},{"id":"369962","messageId":"xmqqk1hrr91s.fsf@gitster-ct.c.googlers.com","threadId":"50525","inReplyTo":"CACsJy8CB=P9T0XJMWQetExgwDyFN78nuJZq8FtmzG+V1fBY4ig@mail.gmail.com","subject":"Re: [PATCH 1/1] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-02-22T18:07:43Z","receivedAt":"2019-02-22T18:07:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> On Thu, Feb 21, 2019 at 5:39 AM Junio C Hamano <gitster@pobox.com> wrote:\n>> > So now you would need to\n>> > add more or less the same set of ignore rules in .gitattributes to\n>> > mark them trashable, and gitignore/gitattributes rules are not exactly\n>> > compatible, you can't just blindly copy them over. Every time you add\n>> > one more .gitignore rule, there's a good chance you need to add a\n>> > similar rule for trashable attribute.\n>>\n>> I am not sure why you would even need to _duplicate_.\n>>\n>> Are you saying for each and every rule that specify \"ignored and\n>> expendable\" in .gitignore there always will be \"ignored but\n>> precious\" exception that match the pattern?  Given that we have been\n>> OK for so long without even needing \"precious\", I find it somewhat\n>> unrealistic to assume so.\n>\n> If all ignored files are now redefined as precious and we mark them\n> expendable with trashable attribute, then we need to duplicate most of\n> the rules. The \"precious\" attribute of course does not have this\n> problem since precious-and-ignored files should be rare.\n\nAh, so you are saying that ignored (the traditional one we always\nhad) plus precious is a better combination than ignored (repurposed\nto mean ignored-but-precious) plus trashable, because the latter\nwill cause people configure with a lot of duplication?\n\nIf that is the case, I'd agree ;-)\n"}]}