{"thread":{"id":"49800","subject":"[RFC PATCH] Introduce \"precious\" file concept","startedAt":"2018-11-11T09:53:01Z","lastAt":"2018-12-06T18:40:00Z","messageCount":40,"participants":["Nguyễn Thái Ngọc Duy","Bert Wesarg","Ævar Arnfjörð Bjarmason","Junio C Hamano","Duy Nguyen","Per Lundberg","Matthieu Moy","brian m. carlson","Eckhard Maaß","Jacob Keller"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"362935","messageId":"20181111095254.30473-1-pclouds@gmail.com","threadId":"49800","inReplyTo":null,"subject":"[RFC PATCH] Introduce \"precious\" file concept","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-11T09:52:54Z","receivedAt":"2018-11-11T09:53:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Since this topic has come up twice recently, I revisited this\n\"precious\" thingy that I started four years ago and tried to see if I\ncould finally finish it. There are a couple things to be sorted out...\n\nA new attribute \"precious\" is added to indicate that certain files\nhave valuable content and should not be easily discarded even if they\nare ignored or untracked (*).\n\nSo far there are two parts of Git that are made aware of precious\nfiles: \"git clean\" will leave precious files alone and unpack-trees.c\n(i.e. merges and branch switches) will not overwrite\nignored-but-precious files.\n\nIs there any other parts of Git that should be made aware of this\n\"precious\" attribute?\n\nAlso while \"precious\" is a fun name, but it does not sound serious.\nAny suggestions? Perhaps \"valuable\"?\n\nVery lightly tested. The patch is more to have something to discuss\nthan is bug free and ready to use.\n\n(*) Note that tracked files could be marked \"precious\" in the future\n too although the exact semantics is not very clear since tracked\n files are by default precious.\n\n But something like \"index log\" could use this to record all\n changes to precious files instead of just \"git add -p\" changes,\n for example. So these files are in a sense more precious than\n other tracked files.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-clean.txt | 3 ++-\n Documentation/gitattributes.txt | 13 +++++++++++++\n attr.c | 9 +++++++++\n attr.h | 2 ++\n builtin/clean.c | 19 ++++++++++++++++---\n unpack-trees.c | 3 ++-\n 6 files changed, 44 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt\nindex 03056dad0d..a9beadfb12 100644\n--- a/Documentation/git-clean.txt\n+++ b/Documentation/git-clean.txt\n@@ -21,7 +21,8 @@ option is specified, ignored files are also removed. This can, for\n example, be useful to remove all build products.\n \n If any optional `...` arguments are given, only those paths\n-are affected.\n+are affected. Ignored or untracked files with `precious` attributes\n+are not removed.\n \n OPTIONS\n -------\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex b8392fc330..c722479bdc 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -1188,6 +1188,19 @@ If this attribute is not set or has an invalid value, the value of the\n (See linkgit:git-config[1]).\n \n \n+Precious files\n+~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+`precious`\n+^^^^^^^^^^\n+\n+This attribute is set on files to indicate that their content is\n+valuable. Many commands will behave slightly different on precious\n+files. linkgit:git-clean[1] will leave precious files alone. Merging\n+and branch switching will not silently overwrite ignored files that\n+are marked \"precious\".\n+\n+\n USING MACRO ATTRIBUTES\n ----------------------\n \ndiff --git a/attr.c b/attr.c\nindex 60d284796d..d06ca0ae4b 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -1186,3 +1186,12 @@ void attr_start(void)\n \tpthread_mutex_init(&check_vector.mutex, NULL);\n #endif\n }\n+\n+int is_precious_file(struct index_state *istate, const char *path)\n+{\n+\tstatic struct attr_check *check;\n+\tif (!check)\n+\t\tcheck = attr_check_initl(\"precious\", NULL);\n+\tgit_check_attr(istate, path, check);\n+\treturn check && ATTR_TRUE(check->items[0].value);\n+}\ndiff --git a/attr.h b/attr.h\nindex b0378bfe5f..b9a9751a66 100644\n--- a/attr.h\n+++ b/attr.h\n@@ -82,4 +82,6 @@ void git_attr_set_direction(enum git_attr_direction new_direction);\n \n void attr_start(void);\n \n+int is_precious_file(struct index_state *istate, const char *path);\n+\n #endif /* ATTR_H */\ndiff --git a/builtin/clean.c b/builtin/clean.c\nindex 8d9a7dc206..9e554448a6 100644\n--- a/builtin/clean.c\n+++ b/builtin/clean.c\n@@ -17,6 +17,7 @@\n #include \"color.h\"\n #include \"pathspec.h\"\n #include \"help.h\"\n+#include \"attr.h\"\n \n static int force = -1; /* unset */\n static int interactive;\n@@ -30,6 +31,8 @@ static const char *const builtin_clean_usage[] = {\n \n static const char *msg_remove = N_(\"Removing %s\\n\");\n static const char *msg_would_remove = N_(\"Would remove %s\\n\");\n+static const char *msg_skip_precious = N_(\"Skipping precious file %s\\n\");\n+static const char *msg_would_skip_precious = N_(\"Would skip precious file %s\\n\");\n static const char *msg_skip_git_dir = N_(\"Skipping repository %s\\n\");\n static const char *msg_would_skip_git_dir = N_(\"Would skip repository %s\\n\");\n static const char *msg_warn_remove_failed = N_(\"failed to remove %s\");\n@@ -152,6 +155,7 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag,\n \tstruct dirent *e;\n \tint res = 0, ret = 0, gone = 1, original_len = path->len, len;\n \tstruct string_list dels = STRING_LIST_INIT_DUP;\n+\tconst char *rel_path;\n \n \t*dir_gone = 1;\n \n@@ -191,9 +195,15 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag,\n \n \t\tstrbuf_setlen(path, len);\n \t\tstrbuf_addstr(path, e->d_name);\n-\t\tif (lstat(path->buf, &st))\n+\t\tif (lstat(path->buf, &st)) {\n \t\t\t; /* fall thru */\n-\t\telse if (S_ISDIR(st.st_mode)) {\n+\t\t} else if ((!prefix || skip_prefix(path->buf, prefix, &rel_path)) &&\n+\t\t\t is_precious_file(&the_index, rel_path)) {\n+\t\t\tquote_path_relative(path->buf, prefix, "ed);\n+\t\t\tprintf(dry_run ? _(msg_would_skip_precious) : _(msg_skip_precious), quoted.buf);\n+\t\t\t*dir_gone = 0;\n+\t\t\tcontinue;\n+\t\t} else if (S_ISDIR(st.st_mode)) {\n \t\t\tif (remove_dirs(path, prefix, force_flag, dry_run, quiet, &gone))\n \t\t\t\tret = 1;\n \t\t\tif (gone) {\n@@ -1017,7 +1027,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)\n \t\tif (lstat(abs_path.buf, &st))\n \t\t\tcontinue;\n \n-\t\tif (S_ISDIR(st.st_mode)) {\n+\t\tif (is_precious_file(&the_index, item->string)) {\n+\t\t\tqname = quote_path_relative(item->string, NULL, &buf);\n+\t\t\tprintf(dry_run ? _(msg_would_skip_precious) : _(msg_skip_precious), qname);\n+\t\t} else if (S_ISDIR(st.st_mode)) {\n \t\t\tif (remove_dirs(&abs_path, prefix, rm_flags, dry_run, quiet, &gone))\n \t\t\t\terrors++;\n \t\t\tif (gone && !quiet) {\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 7570df481b..d49fe0f77e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1895,7 +1895,8 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\treturn 0;\n \n \tif (o->dir &&\n-\t is_excluded(o->dir, o->src_index, name, &dtype))\n+\t is_excluded(o->dir, o->src_index, name, &dtype) &&\n+\t !is_precious_file(o->src_index, name))\n \t\t/*\n \t\t * ce->name is explicitly excluded, so it is Ok to\n \t\t * overwrite it.\n-- \n2.19.1.1235.ga92291acdb\n\n"},{"id":"362939","messageId":"CAKPyHN3xB3o8eFssqr9014TGoX2qYUz8b9Zw0r-f=R4+sv95gw@mail.gmail.com","threadId":"49800","inReplyTo":"20181111095254.30473-1-pclouds@gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Bert Wesarg","fromEmail":"bert.wesarg@googlemail.com","sentAt":"2018-11-11T12:15:07Z","receivedAt":"2018-11-11T12:15:22Z","isPatch":true,"sender":{"key":"bert.wesarg@googlemail.com","avatar":"https://avatars.githubusercontent.com/u/111934?v=4"},"body":"On Sun, Nov 11, 2018 at 10:53 AM Nguyễn Thái Ngọc Duy wrote:\n>\n> Also while \"precious\" is a fun name, but it does not sound serious.\n> Any suggestions? Perhaps \"valuable\"?\n\n\"precious\" is also used by POSIX make:\n\nhttp://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html\n\nBert\n"},{"id":"362940","messageId":"871s7r4wuv.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"20181111095254.30473-1-pclouds@gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-11T12:33:44Z","receivedAt":"2018-11-11T12:33:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\n[CC-ing some of the people involved in recent threads about this]\n\nOn Sun, Nov 11 2018, Nguyễn Thái Ngọc Duy wrote:\n\n> Since this topic has come up twice recently, I revisited this\n> \"precious\" thingy that I started four years ago and tried to see if I\n> could finally finish it. There are a couple things to be sorted out...\n>\n> A new attribute \"precious\" is added to indicate that certain files\n> have valuable content and should not be easily discarded even if they\n> are ignored or untracked (*).\n>\n> So far there are two parts of Git that are made aware of precious\n> files: \"git clean\" will leave precious files alone and unpack-trees.c\n> (i.e. merges and branch switches) will not overwrite\n> ignored-but-precious files.\n>\n> Is there any other parts of Git that should be made aware of this\n> \"precious\" attribute?\n>\n> Also while \"precious\" is a fun name, but it does not sound serious.\n> Any suggestions? Perhaps \"valuable\"?\n>\n> Very lightly tested. The patch is more to have something to discuss\n> than is bug free and ready to use.\n>\n> (*) Note that tracked files could be marked \"precious\" in the future\n> too although the exact semantics is not very clear since tracked\n> files are by default precious.\n>\n> But something like \"index log\" could use this to record all\n> changes to precious files instead of just \"git add -p\" changes,\n> for example. So these files are in a sense more precious than\n> other tracked files.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Documentation/git-clean.txt | 3 ++-\n> Documentation/gitattributes.txt | 13 +++++++++++++\n> attr.c | 9 +++++++++\n> attr.h | 2 ++\n> builtin/clean.c | 19 ++++++++++++++++---\n> unpack-trees.c | 3 ++-\n> 6 files changed, 44 insertions(+), 5 deletions(-)\n>\n> diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt\n> index 03056dad0d..a9beadfb12 100644\n> --- a/Documentation/git-clean.txt\n> +++ b/Documentation/git-clean.txt\n> @@ -21,7 +21,8 @@ option is specified, ignored files are also removed. This can, for\n> example, be useful to remove all build products.\n>\n> If any optional `...` arguments are given, only those paths\n> -are affected.\n> +are affected. Ignored or untracked files with `precious` attributes\n> +are not removed.\n>\n> OPTIONS\n> -------\n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index b8392fc330..c722479bdc 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -1188,6 +1188,19 @@ If this attribute is not set or has an invalid value, the value of the\n> (See linkgit:git-config[1]).\n>\n>\n> +Precious files\n> +~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +`precious`\n> +^^^^^^^^^^\n> +\n> +This attribute is set on files to indicate that their content is\n> +valuable. Many commands will behave slightly different on precious\n> +files. linkgit:git-clean[1] will leave precious files alone. Merging\n> +and branch switching will not silently overwrite ignored files that\n> +are marked \"precious\".\n> +\n> +\n> USING MACRO ATTRIBUTES\n> ----------------------\n>\n> diff --git a/attr.c b/attr.c\n> index 60d284796d..d06ca0ae4b 100644\n> --- a/attr.c\n> +++ b/attr.c\n> @@ -1186,3 +1186,12 @@ void attr_start(void)\n> \tpthread_mutex_init(&check_vector.mutex, NULL);\n> #endif\n> }\n> +\n> +int is_precious_file(struct index_state *istate, const char *path)\n> +{\n> +\tstatic struct attr_check *check;\n> +\tif (!check)\n> +\t\tcheck = attr_check_initl(\"precious\", NULL);\n> +\tgit_check_attr(istate, path, check);\n> +\treturn check && ATTR_TRUE(check->items[0].value);\n> +}\n\nIf we merge two branches is this using the merged post-image of\n.gitattributes as a source?\n\n> \tif (o->dir &&\n> -\t is_excluded(o->dir, o->src_index, name, &dtype))\n> +\t is_excluded(o->dir, o->src_index, name, &dtype) &&\n> +\t !is_precious_file(o->src_index, name))\n> \t\t/*\n> \t\t * ce->name is explicitly excluded, so it is Ok to\n> \t\t * overwrite it.\n\nI wonder if instead we should just be reverting c81935348b (\"Fix\nswitching to a branch with D/F when current branch has file D.\",\n2007-03-15), which these days (haven't dug deeply) would just be this,\nright?:\n\n> diff --git a/unpack-trees.c b/unpack-trees.c\n index 7570df481b..b3efaddd4f 100644\n --- a/unpack-trees.c\n +++ b/unpack-trees.c\n @@ -1894,13 +1894,6 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \tif (ignore_case && icase_exists(o, name, len, st))\n \t\treturn 0;\n\n -\tif (o->dir &&\n -\t is_excluded(o->dir, o->src_index, name, &dtype))\n -\t\t/*\n -\t\t * ce->name is explicitly excluded, so it is Ok to\n -\t\t * overwrite it.\n -\t\t */\n -\t\treturn 0;\n \tif (S_ISDIR(st->st_mode)) {\n \t\t/*\n \t\t * We are checking out path \"foo\" and\n\nSomething like the approach you're taking will absolutely work from a\ntechnical standpoint, but I fear that it's going to be useless in\npractice.\n\nThe users who need protection against git deleting their files the most\nare exactly the sort of users who aren't expert-level enough to\nunderstand the nuances of how the semantics of .gitignore and \"precious\"\nare going to interact before git eats their data.\n\nThis is pretty apparent from the bug reports we're getting about\nthis. None of them are:\n\n \"Hey, I 100% understood .gitignore semantics including this one part\n of the docs where you say you'll do this, but just forgot one day\n and deleted my work. Can we get some more safety?\"\n\nBut rather (with some hyperbole for effect):\n\n \"ZOMG git deleted my file! Is this a bug??\"\n\nSo I think we should have the inverse of this \"precious\"\nattribute\". Just a change to the docs to say that .gitignore doesn't\nimply these eager deletion semantics on tree unpacking anymore, and if\nusers want it back they can define a \"garbage\" attribute\n(s/precious/garbage/).\n\nThat will lose no data, and in the very rare cases where a checkout of\ntracked files would overwrite an ignored pattern, we can just error out\n(as we do with the \"Ok to overwrite\" branch removed) and tell the user\nto delete the files to proceed.\n\nThree tests in our test suite fail with that patch applied, and they're\nexplicitly testing for exactly the sort of scenario where users are likely to lose data. I.e.:\n\n 1. Open a tracked file in an editor\n 2. Save it\n 3. Switch to a topic branch, that has different .gitignore semantics\n (e.g. let's say a build/ dir exists there)\n 4. Have their work deleted\n\nSo actually in writing this out I've become convinced that this\n\"precious\" approach can't work either, because *even if* you're an\nexpert who manages to perfectly define their .gitignore and \"precious\"\nrules in advance to avoid data deletion, those rules will *also* need to\ntake into account switching between branches (or even different\nhistories) where you have other sorts of rules.\n\nSo really, if there's ambiguity let's just not delete stuff by default\nand ask the user to resolve it.\n"},{"id":"362941","messageId":"87zhuf3gs0.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"871s7r4wuv.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-11T13:06:23Z","receivedAt":"2018-11-11T13:06:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Nov 11 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> [CC-ing some of the people involved in recent threads about this]\n>\n> On Sun, Nov 11 2018, Nguyễn Thái Ngọc Duy wrote:\n>\n>> Since this topic has come up twice recently, I revisited this\n>> \"precious\" thingy that I started four years ago and tried to see if I\n>> could finally finish it. There are a couple things to be sorted out...\n>>\n>> A new attribute \"precious\" is added to indicate that certain files\n>> have valuable content and should not be easily discarded even if they\n>> are ignored or untracked (*).\n>>\n>> So far there are two parts of Git that are made aware of precious\n>> files: \"git clean\" will leave precious files alone and unpack-trees.c\n>> (i.e. merges and branch switches) will not overwrite\n>> ignored-but-precious files.\n>>\n>> Is there any other parts of Git that should be made aware of this\n>> \"precious\" attribute?\n>>\n>> Also while \"precious\" is a fun name, but it does not sound serious.\n>> Any suggestions? Perhaps \"valuable\"?\n>>\n>> Very lightly tested. The patch is more to have something to discuss\n>> than is bug free and ready to use.\n>>\n>> (*) Note that tracked files could be marked \"precious\" in the future\n>> too although the exact semantics is not very clear since tracked\n>> files are by default precious.\n>>\n>> But something like \"index log\" could use this to record all\n>> changes to precious files instead of just \"git add -p\" changes,\n>> for example. So these files are in a sense more precious than\n>> other tracked files.\n>>\n>> Signed-off-by: Nguyễn Thái Ngọc Duy \n>> ---\n>> Documentation/git-clean.txt | 3 ++-\n>> Documentation/gitattributes.txt | 13 +++++++++++++\n>> attr.c | 9 +++++++++\n>> attr.h | 2 ++\n>> builtin/clean.c | 19 ++++++++++++++++---\n>> unpack-trees.c | 3 ++-\n>> 6 files changed, 44 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt\n>> index 03056dad0d..a9beadfb12 100644\n>> --- a/Documentation/git-clean.txt\n>> +++ b/Documentation/git-clean.txt\n>> @@ -21,7 +21,8 @@ option is specified, ignored files are also removed. This can, for\n>> example, be useful to remove all build products.\n>>\n>> If any optional `...` arguments are given, only those paths\n>> -are affected.\n>> +are affected. Ignored or untracked files with `precious` attributes\n>> +are not removed.\n>>\n>> OPTIONS\n>> -------\n>> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n>> index b8392fc330..c722479bdc 100644\n>> --- a/Documentation/gitattributes.txt\n>> +++ b/Documentation/gitattributes.txt\n>> @@ -1188,6 +1188,19 @@ If this attribute is not set or has an invalid value, the value of the\n>> (See linkgit:git-config[1]).\n>>\n>>\n>> +Precious files\n>> +~~~~~~~~~~~~~~~~~~~~~~~~\n>> +\n>> +`precious`\n>> +^^^^^^^^^^\n>> +\n>> +This attribute is set on files to indicate that their content is\n>> +valuable. Many commands will behave slightly different on precious\n>> +files. linkgit:git-clean[1] will leave precious files alone. Merging\n>> +and branch switching will not silently overwrite ignored files that\n>> +are marked \"precious\".\n>> +\n>> +\n>> USING MACRO ATTRIBUTES\n>> ----------------------\n>>\n>> diff --git a/attr.c b/attr.c\n>> index 60d284796d..d06ca0ae4b 100644\n>> --- a/attr.c\n>> +++ b/attr.c\n>> @@ -1186,3 +1186,12 @@ void attr_start(void)\n>> \tpthread_mutex_init(&check_vector.mutex, NULL);\n>> #endif\n>> }\n>> +\n>> +int is_precious_file(struct index_state *istate, const char *path)\n>> +{\n>> +\tstatic struct attr_check *check;\n>> +\tif (!check)\n>> +\t\tcheck = attr_check_initl(\"precious\", NULL);\n>> +\tgit_check_attr(istate, path, check);\n>> +\treturn check && ATTR_TRUE(check->items[0].value);\n>> +}\n>\n> If we merge two branches is this using the merged post-image of\n> .gitattributes as a source?\n>\n>> \tif (o->dir &&\n>> -\t is_excluded(o->dir, o->src_index, name, &dtype))\n>> +\t is_excluded(o->dir, o->src_index, name, &dtype) &&\n>> +\t !is_precious_file(o->src_index, name))\n>> \t\t/*\n>> \t\t * ce->name is explicitly excluded, so it is Ok to\n>> \t\t * overwrite it.\n>\n> I wonder if instead we should just be reverting c81935348b (\"Fix\n> switching to a branch with D/F when current branch has file D.\",\n> 2007-03-15), which these days (haven't dug deeply) would just be this,\n> right?:\n>\n>> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 7570df481b..b3efaddd4f 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1894,13 +1894,6 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n> \tif (ignore_case && icase_exists(o, name, len, st))\n> \t\treturn 0;\n>\n> -\tif (o->dir &&\n> -\t is_excluded(o->dir, o->src_index, name, &dtype))\n> -\t\t/*\n> -\t\t * ce->name is explicitly excluded, so it is Ok to\n> -\t\t * overwrite it.\n> -\t\t */\n> -\t\treturn 0;\n> \tif (S_ISDIR(st->st_mode)) {\n> \t\t/*\n> \t\t * We are checking out path \"foo\" and\n>\n> Something like the approach you're taking will absolutely work from a\n> technical standpoint, but I fear that it's going to be useless in\n> practice.\n>\n> The users who need protection against git deleting their files the most\n> are exactly the sort of users who aren't expert-level enough to\n> understand the nuances of how the semantics of .gitignore and \"precious\"\n> are going to interact before git eats their data.\n>\n> This is pretty apparent from the bug reports we're getting about\n> this. None of them are:\n>\n> \"Hey, I 100% understood .gitignore semantics including this one part\n> of the docs where you say you'll do this, but just forgot one day\n> and deleted my work. Can we get some more safety?\"\n>\n> But rather (with some hyperbole for effect):\n>\n> \"ZOMG git deleted my file! Is this a bug??\"\n>\n> So I think we should have the inverse of this \"precious\"\n> attribute\". Just a change to the docs to say that .gitignore doesn't\n> imply these eager deletion semantics on tree unpacking anymore, and if\n> users want it back they can define a \"garbage\" attribute\n> (s/precious/garbage/).\n>\n> That will lose no data, and in the very rare cases where a checkout of\n> tracked files would overwrite an ignored pattern, we can just error out\n> (as we do with the \"Ok to overwrite\" branch removed) and tell the user\n> to delete the files to proceed.\n>\n> Three tests in our test suite fail with that patch applied, and they're\n> explicitly testing for exactly the sort of scenario where users are likely to lose data. I.e.:\n>\n> 1. Open a tracked file in an editor\n> 2. Save it\n> 3. Switch to a topic branch, that has different .gitignore semantics\n> (e.g. let's say a build/ dir exists there)\n> 4. Have their work deleted\n>\n> So actually in writing this out I've become convinced that this\n> \"precious\" approach can't work either, because *even if* you're an\n> expert who manages to perfectly define their .gitignore and \"precious\"\n> rules in advance to avoid data deletion, those rules will *also* need to\n> take into account switching between branches (or even different\n> histories) where you have other sorts of rules.\n>\n> So really, if there's ambiguity let's just not delete stuff by default\n> and ask the user to resolve it.\n\nHere's a patch to implement that (which borrows from some of yours). It\npasses all of our tests:\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex b8392fc330..a6cad17899 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -1188,6 +1188,17 @@ If this attribute is not set or has an invalid value, the value of the\n (See linkgit:git-config[1]).\n\n\n+Trashable files\n+~~~~~~~~~~~~~~~\n+\n+`trashable`\n+^^^^^^^^^^\n+\n+Provides an escape hatch for re-enabling a potentially data destroying\n+feature which was enabled by default between Git versions 1.5.2 and\n+2.20. See the `NOTES` section of linkgit:gitignore[5] for details.\n+\n+\n USING MACRO ATTRIBUTES\n ----------------------\n\ndiff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt\nindex d107daaffd..39c6d5955a 100644\n--- a/Documentation/gitignore.txt\n+++ b/Documentation/gitignore.txt\n@@ -140,6 +140,13 @@ not tracked by Git remain untracked.\n To stop tracking a file that is currently tracked, use\n 'git rm --cached'.\n\n+Between Git versions 1.5.2 and 2.20 untracked files or directories\n+which were ignored and conflicted with a file about to be checked out\n+(e.g. during linkgit:git-checkout[1] or linkgit:git-merge[1]) would be\n+deleted. This could lead to loss of user data and is no longer the\n+default, See `trashable` in linkgit:gitattributes[5]. for how to\n+selectively enable this behavior.\n+\n EXAMPLES\n --------\n\ndiff --git a/attr.c b/attr.c\nindex 60d284796d..930af78650 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -1186,3 +1186,12 @@ void attr_start(void)\n \tpthread_mutex_init(&check_vector.mutex, NULL);\n #endif\n }\n+\n+int is_trashable_file(struct index_state *istate, const char *path)\n+{\n+\tstatic struct attr_check *check;\n+\tif (!check)\n+\t\tcheck = attr_check_initl(\"trashable\", NULL);\n+\tgit_check_attr(istate, path, check);\n+\treturn check && ATTR_TRUE(check->items[0].value);\n+}\ndiff --git a/attr.h b/attr.h\nindex b0378bfe5f..ccf4d4e6b5 100644\n--- a/attr.h\n+++ b/attr.h\n@@ -82,4 +82,6 @@ void git_attr_set_direction(enum git_attr_direction new_direction);\n\n void attr_start(void);\n\n+int is_trashable_file(struct index_state *istate, const char *path);\n+\n #endif /* ATTR_H */\ndiff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh\nindex 016391723c..d2ceee33d2 100755\n--- a/t/lib-submodule-update.sh\n+++ b/t/lib-submodule-update.sh\n@@ -844,6 +844,8 @@ test_submodule_switch_recursing_with_args () {\n \t\t\tgit branch -t add_sub1 origin/add_sub1 &&\n \t\t\t: >sub1 &&\n \t\t\techo sub1 >.git/info/exclude &&\n+\t\t\ttest_must_fail $command add_sub1 &&\n+\t\t\techo sub1 trashable >.gitattributes &&\n \t\t\t$command add_sub1 &&\n \t\t\ttest_superproject_content origin/add_sub1 &&\n \t\t\ttest_submodule_content sub1 origin/add_sub1\ndiff --git a/t/t1004-read-tree-m-u-wf.sh b/t/t1004-read-tree-m-u-wf.sh\nindex c13578a635..2243cd955e 100755\n--- a/t/t1004-read-tree-m-u-wf.sh\n+++ b/t/t1004-read-tree-m-u-wf.sh\n@@ -63,8 +63,10 @@ test_expect_success 'two-way with incorrect --exclude-per-directory (2)' '\n \tfi\n '\n\n-test_expect_success 'two-way clobbering a ignored file' '\n+test_expect_success 'two-way keeping a ignored file, trashing a trashable file' '\n\n+\tread_tree_u_must_fail -m -u --exclude-per-directory=.gitignore master side &&\n+\techo file2 trashable >.gitattributes &&\n \tread_tree_u_must_succeed -m -u --exclude-per-directory=.gitignore master side\n '\n\n@@ -106,7 +108,7 @@ test_expect_success 'three-way not clobbering a working tree file' '\n\n echo >.gitignore file3\n\n-test_expect_success 'three-way not complaining on an untracked file' '\n+test_expect_success 'three-way complaining on an untracked file, trashing a trashable file' '\n\n \tgit reset --hard &&\n \trm -f file2 subdir/file2 file3 subdir/file3 &&\n@@ -114,6 +116,8 @@ test_expect_success 'three-way not complaining on an untracked file' '\n \techo >file3 file three created in master, untracked &&\n \techo >subdir/file3 file three created in master, untracked &&\n\n+\tread_tree_u_must_fail -m -u --exclude-per-directory=.gitignore branch-point master side &&\n+\techo file3 trashable >.gitattributes &&\n \tread_tree_u_must_succeed -m -u --exclude-per-directory=.gitignore branch-point master side\n '\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 7570df481b..e9a7fb6583 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1895,9 +1895,10 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\treturn 0;\n\n \tif (o->dir &&\n-\t is_excluded(o->dir, o->src_index, name, &dtype))\n+\t is_excluded(o->dir, o->src_index, name, &dtype) &&\n+\t is_trashable_file(o->src_index, name))\n \t\t/*\n-\t\t * ce->name is explicitly excluded, so it is Ok to\n+\t\t * ce->name is explicitly trashable, so it is Ok to\n \t\t * overwrite it.\n \t\t */\n \t\treturn 0;\n"},{"id":"362942","messageId":"xmqqva534vnb.fsf@gitster-ct.c.googlers.com","threadId":"49800","inReplyTo":"20181111095254.30473-1-pclouds@gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-11T12:59:52Z","receivedAt":"2018-11-11T13:09:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Also while \"precious\" is a fun name, but it does not sound serious.\n> Any suggestions? Perhaps \"valuable\"?\n\nFWIW, I am reasonably sure that I was the first in Git circle who\nused the term \"precious\" in discussions wrt .gitignore, i.e. \"Git\nhas ignored but not precious category\". Since it was not my\ninvention but was a borrowed term from tla (aka GNU arch), I'd\nsuggest to keep using that term, unless there is a strong reason not\nto follow longstanding precedent.\n"},{"id":"362945","messageId":"CACsJy8CYpuc7-CZhk7kQQVQFxOfLFZu4TVpG=b0a7j8P1J394Q@mail.gmail.com","threadId":"49800","inReplyTo":"871s7r4wuv.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-11T15:41:19Z","receivedAt":"2018-11-11T15:41:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Nov 11, 2018 at 1:33 PM Ævar Arnfjörð Bjarmason\n wrote:\n> The users who need protection against git deleting their files the most\n> are exactly the sort of users who aren't expert-level enough to\n> understand the nuances of how the semantics of .gitignore and \"precious\"\n> are going to interact before git eats their data.\n>\n> This is pretty apparent from the bug reports we're getting about\n> this. None of them are:\n>\n> \"Hey, I 100% understood .gitignore semantics including this one part\n> of the docs where you say you'll do this, but just forgot one day\n> and deleted my work. Can we get some more safety?\"\n>\n> But rather (with some hyperbole for effect):\n>\n> \"ZOMG git deleted my file! Is this a bug??\"\n>\n> So I think we should have the inverse of this \"precious\"\n> attribute\". Just a change to the docs to say that .gitignore doesn't\n> imply these eager deletion semantics on tree unpacking anymore, and if\n> users want it back they can define a \"garbage\" attribute\n> (s/precious/garbage/).\n>\n> That will lose no data, and in the very rare cases where a checkout of\n> tracked files would overwrite an ignored pattern, we can just error out\n> (as we do with the \"Ok to overwrite\" branch removed) and tell the user\n> to delete the files to proceed.\n\nThere's also the other side of the coin. If this refuse to overwrite\ntriggers too often, it can become an annoyance. So far I've seen two\nreports of accident overwriting which make me think turning precious\nto trashable may be too extreme. Plus ignored files are trashable by\ndefault (or at least by design so far), adding trashable attribute\nchanges how we handle ignored files quite significantly.\n-- \nDuy\n"},{"id":"362946","messageId":"87wopj3661.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"CACsJy8CYpuc7-CZhk7kQQVQFxOfLFZu4TVpG=b0a7j8P1J394Q@mail.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-11T16:55:34Z","receivedAt":"2018-11-11T16:55:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Nov 11 2018, Duy Nguyen wrote:\n\n> On Sun, Nov 11, 2018 at 1:33 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>> The users who need protection against git deleting their files the most\n>> are exactly the sort of users who aren't expert-level enough to\n>> understand the nuances of how the semantics of .gitignore and \"precious\"\n>> are going to interact before git eats their data.\n>>\n>> This is pretty apparent from the bug reports we're getting about\n>> this. None of them are:\n>>\n>> \"Hey, I 100% understood .gitignore semantics including this one part\n>> of the docs where you say you'll do this, but just forgot one day\n>> and deleted my work. Can we get some more safety?\"\n>>\n>> But rather (with some hyperbole for effect):\n>>\n>> \"ZOMG git deleted my file! Is this a bug??\"\n>>\n>> So I think we should have the inverse of this \"precious\"\n>> attribute\". Just a change to the docs to say that .gitignore doesn't\n>> imply these eager deletion semantics on tree unpacking anymore, and if\n>> users want it back they can define a \"garbage\" attribute\n>> (s/precious/garbage/).\n>>\n>> That will lose no data, and in the very rare cases where a checkout of\n>> tracked files would overwrite an ignored pattern, we can just error out\n>> (as we do with the \"Ok to overwrite\" branch removed) and tell the user\n>> to delete the files to proceed.\n>\n> There's also the other side of the coin. If this refuse to overwrite\n> triggers too often, it can become an annoyance. So far I've seen two\n> reports of accident overwriting which make me think turning precious\n> to trashable may be too extreme. Plus ignored files are trashable by\n> default (or at least by design so far), adding trashable attribute\n> changes how we handle ignored files quite significantly.\n\nYeah I'm not trying to make the argument that we should just go with\nthese user bug reports, clearly that's just going to give us selection\nbias and we could continue to flip between the two behaviors with that\napproach. Just that an advanced opt-in feature to prevent dataloss will\nnot prevent it in practice.\n\nIs taking my patch the right thing? I don't know. I'm leaning in that\ndirection, but more making a devil's advocate argument to see if anyone\nfinds good cases that'll demonstrate how it's bad. I haven't read/seen\nthem so far, and the test suite didn't have any.\n\nI did go through the list archives as Junio suggested in\nhttps://public-inbox.org/git/7viq39avay.fsf@alter.siamese.dyndns.org/\nand found these two:\nhttps://public-inbox.org/git/?q=d%3A20070301..20070331+verify_absent\n\nIt seems to me that the reason we ended up with this behavior is a bug\nreport from Shawn that was describing a similar but not quite the same\nproblem:\n\n \"[...]a bug in read-tree -m that prevents him from switching\n branches when the type of a path changes between a directory and a\n file.[...]\"\n\nThat's not the same as when a now-tracked file clobbers a .gitignored\nfile. As far as I can tell (but may not have read carefully enough) that\nwasn't a problem anyone reported, but was changed while fixing another\nbug in c81935348b (\"Fix switching to a branch with D/F when current\nbranch has file D.\", 2007-03-15).\n"},{"id":"362968","messageId":"591ab1f7-ef39-13e5-83b8-76fe372ecc2c@hibox.tv","threadId":"49800","inReplyTo":"CACsJy8CYpuc7-CZhk7kQQVQFxOfLFZu4TVpG=b0a7j8P1J394Q@mail.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Per Lundberg","fromEmail":"per.lundberg@hibox.tv","sentAt":"2018-11-12T07:35:29Z","receivedAt":"2018-11-12T07:35:51Z","isPatch":true,"sender":{"key":"per.lundberg@hibox.tv","avatar":"https://gravatar.com/avatar/037abe105bde31ba7f4fe80a1436ef76a06c9ca8f2c0c153e6ca66a7527cd637?d=mp&s=160"},"body":"On 11/11/18 5:41 PM, Duy Nguyen wrote:\n> On Sun, Nov 11, 2018 at 1:33 PM Ævar Arnfjörð Bjarmason\n> wrote:\n >\n>> That will lose no data, and in the very rare cases where a checkout of\n>> tracked files would overwrite an ignored pattern, we can just error out\n>> (as we do with the \"Ok to overwrite\" branch removed) and tell the user\n>> to delete the files to proceed. \n> There's also the other side of the coin. If this refuse to overwrite\n> triggers too often, it can become an annoyance.\n\nSure, but doing \"git checkout -f some_ref\" instead of \"git checkout \nsome_ref\" isn't really _that_ annoying, is it? I think, people (because \nof not having read/studied the .gitignore semantics well enough) having \ntheir files being overwritten _without realizing it_ is a bigger danger. \nBut obviously there is a bit of treading a thin line here.\n\nIf we feel thrashable is stretching it too far (which I don't think it \nis), we could add a \"core.ignore_files_are_trashable\" setting that \nbrings back the old semantics, for those who have a strong feeling about it.\n\nIt's also quite possible to do it the other way around - i.e. set \n\"core.ignore_files_are_trashable\" to true by default, and let the \"new\" \nbehavior be opt-in. However, this might \"miss the mark\" in that those \npeople who would really benefit from the new semantics might miss this \nsetting, just like they could risk missing the \"precious\" setting.\n\n(I also think \"trashable\" sounds better and is more clear & precise than \n\"precious\", for whatever that is worth.)\n--\nPer Lundberg\n"},{"id":"362980","messageId":"87o9au39s7.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"1205132135.1189562.1542013731020.JavaMail.zimbra@matthieu-moy.fr","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-12T09:49:44Z","receivedAt":"2018-11-12T09:49:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 12 2018, Matthieu Moy wrote:\n\n> \"Per Lundberg\" wrote:\n>\n>> On 11/11/18 5:41 PM, Duy Nguyen wrote:\n>> > On Sun, Nov 11, 2018 at 1:33 PM Ævar Arnfjörð Bjarmason\n>> > wrote:\n>>\n>> >> That will lose no data, and in the very rare cases where a checkout of\n>> >> tracked files would overwrite an ignored pattern, we can just error out\n>> >> (as we do with the \"Ok to overwrite\" branch removed) and tell the user\n>> >> to delete the files to proceed.\n>> > There's also the other side of the coin. If this refuse to overwrite\n>> > triggers too often, it can become an annoyance.\n>\n> I may have missed some cases, but to me the cases when checkout may try\n> to overwrite an ignored file are essentially:\n>\n> * Someone \"git add\"ed a file meant to be ignored by mistake (e.g.\n> \"git add -f *.o\").\n>\n> * A file that was meant to be kept private (e.g. config.mak.dev) ends\n> up being tracked. This may happen when we find a way to make per-developer\n> settings the same for everyone.\n\nYes, the cases under discussion here are all cases where a tracked file\nclobbers a file matching a pattern in in .gitignore.\n\nWhat I'd add to your list is:\n\n* Some projects (I've seen this in the wild) add e.g. *.mp3 or whatever\n else usually doesn't belong in the repo as a \"soft ignore\". This is\n something we've never recommended, but have implicitly supported since\n the only caveats are a) you need a one-off \"git add -f\" and then\n they're tracked b) them being missing from \"status\" before being\n tracked c) the issue under discussion here.\n\n* Cases where e.g. filename changes to a directory or globs because of\n that make this more complex.\n\n> I both cases I'd want at least to be notified that something is going on,\n> and in the second I'd probably want to keep my local file around.\n>> If we feel thrashable is stretching it too far (which I don't think it\n>> is), we could add a \"core.ignore_files_are_trashable\" setting that\n>> brings back the old semantics, for those who have a strong feeling about it.\n>\n> May I remind an idea I sugested in an old thread: add an intermediate level\n> where ignored files to be overwritten are renamed (eg. foo -> foo~ like Emacs'\n> backup files):\n>\n> https://public-inbox.org/git/vpqd3t9656k.fsf@bauges.imag.fr/\n>\n> One advantage of the \"rename\" behavior is that it's safer that the current,\n> but still not very disturbing for people who like the current behavior. This\n> makes it a good candidate for a default behavior.\n>\n> This could come in complement with this thread's \"precious\" concept:\n>\n> * If you know what you're doing and know that such or such file is precious,\n> mark it as such and Git will never overwrite it.\n>\n> * If you don't know about precious files, just keep the default setting and\n> the worse that can happen is to get your file overwritten with a bakup\n> of the old version kept around.\n>\n> This would probably play better with a notion of \"precious\" files than with\n> a notion of \"trashable\" files.\n\nI used to think this foo -> foo~ approach made the most sense (and said\nas much in\nhttps://public-inbox.org/git/871s8qdzph.fsf@evledraar.gmail.com/) but I\nthink it's probably best not to do it and just error out, because:\n\n * We'd still need to handle the cases where \"tests\" the file collides\n with \"tests\" the directory. Then where do we move the colliding file?\n ~/.git/lost+found/* ? We could handle the subdir case with another\n special-case though...\n\n * I think such silent action will just leave users more confused, and\n in many cases (e.g. a merge) whatever message we print out will be\n missed in a deluge of other messaging, and they'll probably miss it.\n\n I'd like to avoid a case where a bunch of *~ files get committed\n because the user's workflow is (and some beginner git users do this):\n\n git pull && git add . && git commit && git push\n\n As the \"pull\" would now invoke a merge that would do this rename.\n\n * If I have the \"foo\" file open in my editor (a plausible way to run\n into this) I switch to another terminal, do the merge, miss the\n message, then re-save \"foo\". Now I have both \"foo\" and \"foo~\"\n on-disk. Another case where we should just refuse until the user\n resolves the situation to avoid the confusion.\n"},{"id":"362981","messageId":"1205132135.1189562.1542013731020.JavaMail.zimbra@matthieu-moy.fr","threadId":"49800","inReplyTo":"591ab1f7-ef39-13e5-83b8-76fe372ecc2c@hibox.tv","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Matthieu Moy","fromEmail":"git@matthieu-moy.fr","sentAt":"2018-11-12T09:08:51Z","receivedAt":"2018-11-12T09:55:05Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"\"Per Lundberg\" wrote:\n\n> On 11/11/18 5:41 PM, Duy Nguyen wrote:\n> > On Sun, Nov 11, 2018 at 1:33 PM Ævar Arnfjörð Bjarmason\n> > wrote:\n>\n> >> That will lose no data, and in the very rare cases where a checkout of\n> >> tracked files would overwrite an ignored pattern, we can just error out\n> >> (as we do with the \"Ok to overwrite\" branch removed) and tell the user\n> >> to delete the files to proceed.\n> > There's also the other side of the coin. If this refuse to overwrite\n> > triggers too often, it can become an annoyance.\n\nI may have missed some cases, but to me the cases when checkout may try\nto overwrite an ignored file are essentially:\n\n* Someone \"git add\"ed a file meant to be ignored by mistake (e.g.\n \"git add -f *.o\").\n\n* A file that was meant to be kept private (e.g. config.mak.dev) ends\n up being tracked. This may happen when we find a way to make per-developer\n settings the same for everyone.\n\nI both cases I'd want at least to be notified that something is going on,\nand in the second I'd probably want to keep my local file around.\n\n> If we feel thrashable is stretching it too far (which I don't think it\n> is), we could add a \"core.ignore_files_are_trashable\" setting that\n> brings back the old semantics, for those who have a strong feeling about it.\n\nMay I remind an idea I sugested in an old thread: add an intermediate level\nwhere ignored files to be overwritten are renamed (eg. foo -> foo~ like Emacs'\nbackup files):\n\nhttps://public-inbox.org/git/vpqd3t9656k.fsf@bauges.imag.fr/\n\nOne advantage of the \"rename\" behavior is that it's safer that the current,\nbut still not very disturbing for people who like the current behavior. This\nmakes it a good candidate for a default behavior.\n\nThis could come in complement with this thread's \"precious\" concept:\n\n* If you know what you're doing and know that such or such file is precious,\n mark it as such and Git will never overwrite it.\n\n* If you don't know about precious files, just keep the default setting and\n the worse that can happen is to get your file overwritten with a bakup\n of the old version kept around.\n\nThis would probably play better with a notion of \"precious\" files than with\na notion of \"trashable\" files.\n\n-- \nMatthieu Moy\nhttps://matthieu-moy.fr/\n"},{"id":"362985","messageId":"xmqqk1li1thy.fsf@gitster-ct.c.googlers.com","threadId":"49800","inReplyTo":"87o9au39s7.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-12T10:26:49Z","receivedAt":"2018-11-12T10:26:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> What I'd add to your list is:\n>\n> * Some projects (I've seen this in the wild) add e.g. *.mp3 or whatever\n> else usually doesn't belong in the repo as a \"soft ignore\". This is\n> something we've never recommended, but have implicitly supported since\n> the only caveats are a) you need a one-off \"git add -f\" and then\n> they're tracked b) them being missing from \"status\" before being\n> tracked c) the issue under discussion here.\n\nOr only selected \"*.o\" (vendor supplied binary blob) kept tracked\nwhile everything else is built from the source.\n\nI do not know who you are referring to \"we\" in your sentence, but as\nfar as I am concerned, it has been and still is a BCP recommendation\non this list to deal with a case like that.\n"},{"id":"363003","messageId":"87in1231o2.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"xmqqk1li1thy.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-12T12:45:01Z","receivedAt":"2018-11-12T12:45:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 12 2018, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> What I'd add to your list is:\n>>\n>> * Some projects (I've seen this in the wild) add e.g. *.mp3 or whatever\n>> else usually doesn't belong in the repo as a \"soft ignore\". This is\n>> something we've never recommended, but have implicitly supported since\n>> the only caveats are a) you need a one-off \"git add -f\" and then\n>> they're tracked b) them being missing from \"status\" before being\n>> tracked c) the issue under discussion here.\n>\n> Or only selected \"*.o\" (vendor supplied binary blob) kept tracked\n> while everything else is built from the source.\n>\n> I do not know who you are referring to \"we\" in your sentence, but as\n> far as I am concerned, it has been and still is a BCP recommendation\n> on this list to deal with a case like that.\n\nI mean that this use-case of having a \"soft\" ignore by carrying it\nacross the \"git add\" barrier with a one-off \"-f\" isn't something\nexplicitly documented, and apparently not something many\nexpect. I.e. you / Matthieu have mentioned .gitignore in the past for\nonly-generated *.o use-case.\n\nBut it also does get used for \"mostly we don't want this file, but\nsometimes we do\" use-case, so that's something we need to deal with in\npractice. Like many workflows in git it's not something that was forseen\nor intended, but does happen in the wild.\n"},{"id":"363008","messageId":"xmqqbm6u1maq.fsf@gitster-ct.c.googlers.com","threadId":"49800","inReplyTo":"87in1231o2.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-12T13:02:21Z","receivedAt":"2018-11-12T13:02:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>> Or only selected \"*.o\" (vendor supplied binary blob) kept tracked\n>> while everything else is built from the source.\n>> ...\n> But it also does get used for \"mostly we don't want this file, but\n> sometimes we do\" use-case, so that's something we need to deal with in\n> practice.\n\nExactly. \"Mostly we don't want *.o as we prefer to build from the\nsource, but we have only object files for some selected ones\" is an\noften cited use case where it is the BCP to have *.o in .gitignore\nand use \"add -f\" to add the \"selected\" ones initially.\n\n"},{"id":"363055","messageId":"CACsJy8C3rOFv0kQeJrWufQQzbnfU4mSxJtphEYBGMmrroFFN-A@mail.gmail.com","threadId":"49800","inReplyTo":"1205132135.1189562.1542013731020.JavaMail.zimbra@matthieu-moy.fr","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-12T16:07:56Z","receivedAt":"2018-11-12T16:08:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 12, 2018 at 10:09 AM Matthieu Moy wrote:\n> May I remind an idea I sugested in an old thread: add an intermediate level\n> where ignored files to be overwritten are renamed (eg. foo -> foo~ like Emacs'\n> backup files):\n>\n> https://public-inbox.org/git/vpqd3t9656k.fsf@bauges.imag.fr/\n>\n> One advantage of the \"rename\" behavior is that it's safer that the current,\n> but still not very disturbing for people who like the current behavior. This\n> makes it a good candidate for a default behavior.\n\nI have something else in the bag that does something like this. The\nidea is that we go ahead and do destructive things but we let the user\nundo.\n\nSome more background in [1] but basically we hash \"every\" change and\nstore in the object database (in this case we store \"foo\" content\nbefore overwriting it). We maintain a list of these hashes so that\nundo is possible, but of course we don't keep infinite change history,\neventually too old changes will be pruned. [1] talks about index\nchanges (e.g. \"git add -p\") but it could apply to worktree changes as\nwell (and I'm also eyeing $GIT_DIR/config changes).\n\nThe upside: a similar undo mechanism that works for more than just\nthis case and it allows undoing multiple times while foo~ only allow\nonce. The downside: hashing is definitely heavier than renaming foo to\nfoo~. So this will feature be opt-in in most cases. But for\n\"dangerous\" overwrite like this case, I think we value the file\ncontent more and make it opt-out.\n\n[1] https://public-inbox.org/git/CACsJy8A3QCYY6QeJQYkbCKYh=7Q7pj=rer_OQHLGoAMqTNomNA@mail.gmail.com/\n-- \nDuy\n"},{"id":"363057","messageId":"CACsJy8BDjNW-eGMDiwpm0WKKe69dmQLDf=sO6-bC4zuC+hPURQ@mail.gmail.com","threadId":"49800","inReplyTo":"87zhuf3gs0.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-12T16:14:40Z","receivedAt":"2018-11-12T16:15:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Nov 11, 2018 at 2:06 PM Ævar Arnfjörð Bjarmason\n wrote:\n> +Trashable files\n> +~~~~~~~~~~~~~~~\n> +\n> +`trashable`\n> +^^^^^^^^^^\n> +\n> +Provides an escape hatch for re-enabling a potentially data destroying\n> +feature which was enabled by default between Git versions 1.5.2 and\n> +2.20. See the `NOTES` section of linkgit:gitignore[5] for details.\n\nHow does this interact with \"git clean -x\"? Most ignored files will\nnot have trashable attribute, so we don't remove any of them? Making\n\"git clean\" completely ignore this attribute is also possible, I\nguess, if we rename it somehow to avoid confusion.\n-- \nDuy\n"},{"id":"363099","messageId":"20181112232209.GK890086@genre.crustytoothpaste.net","threadId":"49800","inReplyTo":"871s7r4wuv.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-12T23:22:09Z","receivedAt":"2018-11-12T23:22:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Nov 11, 2018 at 01:33:44PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> The users who need protection against git deleting their files the most\n> are exactly the sort of users who aren't expert-level enough to\n> understand the nuances of how the semantics of .gitignore and \"precious\"\n> are going to interact before git eats their data.\n> \n> This is pretty apparent from the bug reports we're getting about\n> this. None of them are:\n> \n> \"Hey, I 100% understood .gitignore semantics including this one part\n> of the docs where you say you'll do this, but just forgot one day\n> and deleted my work. Can we get some more safety?\"\n> \n> But rather (with some hyperbole for effect):\n> \n> \"ZOMG git deleted my file! Is this a bug??\"\n> \n> So I think we should have the inverse of this \"precious\"\n> attribute\". Just a change to the docs to say that .gitignore doesn't\n> imply these eager deletion semantics on tree unpacking anymore, and if\n> users want it back they can define a \"garbage\" attribute\n> (s/precious/garbage/).\n> \n> That will lose no data, and in the very rare cases where a checkout of\n> tracked files would overwrite an ignored pattern, we can just error out\n> (as we do with the \"Ok to overwrite\" branch removed) and tell the user\n> to delete the files to proceed.\n\nThis is going to totally hose automation. My last job had files which\nmight move from tracked to untracked (a file that had become generated),\nand long-running CI and build systems would need to be able to check out\none status and switch to the other. Your proposed change will prevent\nthose systems from working, whereas they previously did.\n\nI agree that your proposal would have been a better design originally,\nbut breaking the way automated systems currently work is probably going\nto be a dealbreaker.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"364068","messageId":"280aa9c3-0b67-c992-1a79-fc87bbc74906@hibox.tv","threadId":"49800","inReplyTo":"20181112232209.GK890086@genre.crustytoothpaste.net","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Per Lundberg","fromEmail":"per.lundberg@hibox.tv","sentAt":"2018-11-26T09:30:13Z","receivedAt":"2018-11-26T09:31:03Z","isPatch":true,"sender":{"key":"per.lundberg@hibox.tv","avatar":"https://gravatar.com/avatar/037abe105bde31ba7f4fe80a1436ef76a06c9ca8f2c0c153e6ca66a7527cd637?d=mp&s=160"},"body":"On 11/13/18 1:22 AM, brian m. carlson wrote:\n> This is going to totally hose automation. My last job had files which\n> might move from tracked to untracked (a file that had become generated),\n> and long-running CI and build systems would need to be able to check out\n> one status and switch to the other. Your proposed change will prevent\n> those systems from working, whereas they previously did.\n> \n> I agree that your proposal would have been a better design originally,\n> but breaking the way automated systems currently work is probably going\n> to be a dealbreaker.\n\nHow about something like this:\n\n1. Introduce a concept with \"garbage\" files, which git is \"permitted to \ndelete\" without prompting.\n\n2. Retain the current default, i.e. \"ignored files are garbage\" for now, \nmaking the new behavior _opt in_ to avoid breaking automated \nsystems/existing scripts for anyone. Put the setting for this behind a \nnew core.* config flag.\n\n3. In the plan for version 3.0 (a new major version where some breakage \ncan be tolerable, according to Semantic Versioning), change the default \nso that \"only explicit garbage is garbage\". Include very clear notices \nof this in the release notes. The config flag is retained, but its \ndefault changes from true->false or vice versa. People who dislike the \nnew behavior can easily change back to the 2.x semantics.\n\nWould this be a reasonable compromise for everybody?\n-- \nPer Lundberg\n\n"},{"id":"364069","messageId":"87wop0yvxv.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"280aa9c3-0b67-c992-1a79-fc87bbc74906@hibox.tv","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-26T10:28:28Z","receivedAt":"2018-11-26T10:28:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 26 2018, Per Lundberg wrote:\n\n> On 11/13/18 1:22 AM, brian m. carlson wrote:\n>> This is going to totally hose automation. My last job had files which\n>> might move from tracked to untracked (a file that had become generated),\n>> and long-running CI and build systems would need to be able to check out\n>> one status and switch to the other. Your proposed change will prevent\n>> those systems from working, whereas they previously did.\n>>\n>> I agree that your proposal would have been a better design originally,\n>> but breaking the way automated systems currently work is probably going\n>> to be a dealbreaker.\n>\n> How about something like this:\n>\n> 1. Introduce a concept with \"garbage\" files, which git is \"permitted to\n> delete\" without prompting.\n>\n> 2. Retain the current default, i.e. \"ignored files are garbage\" for now,\n> making the new behavior _opt in_ to avoid breaking automated\n> systems/existing scripts for anyone. Put the setting for this behind a\n> new core.* config flag.\n>\n> 3. In the plan for version 3.0 (a new major version where some breakage\n> can be tolerable, according to Semantic Versioning), change the default\n> so that \"only explicit garbage is garbage\". Include very clear notices\n> of this in the release notes. The config flag is retained, but its\n> default changes from true->false or vice versa. People who dislike the\n> new behavior can easily change back to the 2.x semantics.\n>\n> Would this be a reasonable compromise for everybody?\n\nPossibly, but I think there's an earlier step zero there for anyone\ninterested in pursuing this (and currently I can't make time for it),\nwhich is to submit a patch with tests and documentation showing exactly\nthe sort of scenarios where we clobber or don't clobber existing files.\n\nAs my https://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\nshows we have tests for this, but they're not explicit, and some want to\ntest some unrelated thing.\n\nI.e. to test the cases where we clobber foo.c because foo.c now\nexplicitly exists, or cases where dir/foo.c is clobbered because \"dir\"\nis now a tracked text file etc., are those the only two cases? I vaguely\nsuspect that there were other interesting cases, but at this point the\ninformation has been paged out of the working set of my wetware. The\nthread at\nhttps://public-inbox.org/git/87o9au39s7.fsf@evledraar.gmail.com/ has\nsome notes about this.\n\nThen as noted in\nhttps://public-inbox.org/git/87wopj3661.fsf@evledraar.gmail.com/ the\nreason we have this behavior seems to be something that grew organically\nfrom a semi-related bugfix.\n\nSo I don't think we're at a point where we're all dug into our trenches\nand some people want X and others want Y and in the name of backwards\ncompatibility we're going to stay with X. It may turn out that we just\nwant to retain 10% of X, and can get 99% of the safety of Y by doing\nthat.\n"},{"id":"364070","messageId":"xmqqzhtwuhpc.fsf@gitster-ct.c.googlers.com","threadId":"49800","inReplyTo":"280aa9c3-0b67-c992-1a79-fc87bbc74906@hibox.tv","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-26T12:49:35Z","receivedAt":"2018-11-26T12:49:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Per Lundberg writes:\n\n> How about something like this:\n> ...\n> Would this be a reasonable compromise for everybody?\n\nI do not think you'd need to introduce such a deliberately breaking\nchange at all. Just introduce a new \"precious\" class, perhaps mark\nthem with the atttribute mechanism, and that would be the endgame.\nEarly adopters would start marking ignored but not expendable paths\nwith the \"precious\" attribute and they won't be clobbered. As the\nmechanism becomes widely known and mature, more and more people use\nit. And even after that happens, early adopters do not have to change\nany attribute setting, and late adopters would have plenty of examples\nto imitate. Those who do not need any \"precious\" class do not have\nto do anything and you won't break any existing automation that way.\n\n\n\n"},{"id":"364073","messageId":"CACsJy8AzmgkCm=_pJpcXY4xwujnfx9vFKJgbJ_BB__4UybACTQ@mail.gmail.com","threadId":"49800","inReplyTo":"280aa9c3-0b67-c992-1a79-fc87bbc74906@hibox.tv","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-26T15:26:54Z","receivedAt":"2018-11-26T15:27:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 26, 2018 at 10:30 AM Per Lundberg wrote:\n>\n> On 11/13/18 1:22 AM, brian m. carlson wrote:\n> > This is going to totally hose automation. My last job had files which\n> > might move from tracked to untracked (a file that had become generated),\n> > and long-running CI and build systems would need to be able to check out\n> > one status and switch to the other. Your proposed change will prevent\n> > those systems from working, whereas they previously did.\n> >\n> > I agree that your proposal would have been a better design originally,\n> > but breaking the way automated systems currently work is probably going\n> > to be a dealbreaker.\n>\n> How about something like this:\n>\n> 1. Introduce a concept with \"garbage\" files, which git is \"permitted to\n> delete\" without prompting.\n>\n> 2. Retain the current default, i.e. \"ignored files are garbage\" for now,\n> making the new behavior _opt in_ to avoid breaking automated\n> systems/existing scripts for anyone. Put the setting for this behind a\n> new core.* config flag.\n>\n> 3. In the plan for version 3.0 (a new major version where some breakage\n> can be tolerable, according to Semantic Versioning), change the default\n> so that \"only explicit garbage is garbage\". Include very clear notices\n> of this in the release notes. The config flag is retained, but its\n> default changes from true->false or vice versa. People who dislike the\n> new behavior can easily change back to the 2.x semantics.\n\nHow does this garbage thing interact with \"git clean -x\"? My\ninterpretation of this flag/attribute is that at version 3.0 by\ndefault all ignored files are _not_ garbage, so \"git clean -x\" should\nnot remove any of them. Which is weird because most of ignored files\nare like *.o that should be removed.\n\nI also need to mark \"precious\" on untracked or even tracked files (*).\nNot sure how this \"garbage\" attribute interacts with that.\n\n(*) I was hoping I could get the idea [1] implemented in somewhat good\nshape before presenting here. But I'm a bit slow on that front. So\nyeah this \"precious\" on untracked/tracked thingy may be even\nirrelevant if the patch series will be rejected.\n\n[1] https://public-inbox.org/git/CACsJy8C3rOFv0kQeJrWufQQzbnfU4mSxJtphEYBGMmrroFFN-A@mail.gmail.com/\n\n> Would this be a reasonable compromise for everybody?\n-- \nDuy\n"},{"id":"364074","messageId":"87sgznzwcp.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"CACsJy8AzmgkCm=_pJpcXY4xwujnfx9vFKJgbJ_BB__4UybACTQ@mail.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-26T15:34:14Z","receivedAt":"2018-11-26T15:34:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 26 2018, Duy Nguyen wrote:\n\n> On Mon, Nov 26, 2018 at 10:30 AM Per Lundberg wrote:\n>>\n>> On 11/13/18 1:22 AM, brian m. carlson wrote:\n>> > This is going to totally hose automation. My last job had files which\n>> > might move from tracked to untracked (a file that had become generated),\n>> > and long-running CI and build systems would need to be able to check out\n>> > one status and switch to the other. Your proposed change will prevent\n>> > those systems from working, whereas they previously did.\n>> >\n>> > I agree that your proposal would have been a better design originally,\n>> > but breaking the way automated systems currently work is probably going\n>> > to be a dealbreaker.\n>>\n>> How about something like this:\n>>\n>> 1. Introduce a concept with \"garbage\" files, which git is \"permitted to\n>> delete\" without prompting.\n>>\n>> 2. Retain the current default, i.e. \"ignored files are garbage\" for now,\n>> making the new behavior _opt in_ to avoid breaking automated\n>> systems/existing scripts for anyone. Put the setting for this behind a\n>> new core.* config flag.\n>>\n>> 3. In the plan for version 3.0 (a new major version where some breakage\n>> can be tolerable, according to Semantic Versioning), change the default\n>> so that \"only explicit garbage is garbage\". Include very clear notices\n>> of this in the release notes. The config flag is retained, but its\n>> default changes from true->false or vice versa. People who dislike the\n>> new behavior can easily change back to the 2.x semantics.\n>\n> How does this garbage thing interact with \"git clean -x\"? My\n> interpretation of this flag/attribute is that at version 3.0 by\n> default all ignored files are _not_ garbage, so \"git clean -x\" should\n> not remove any of them. Which is weird because most of ignored files\n> are like *.o that should be removed.\n>\n> I also need to mark \"precious\" on untracked or even tracked files (*).\n> Not sure how this \"garbage\" attribute interacts with that.\n>\n> (*) I was hoping I could get the idea [1] implemented in somewhat good\n> shape before presenting here. But I'm a bit slow on that front. So\n> yeah this \"precious\" on untracked/tracked thingy may be even\n> irrelevant if the patch series will be rejected.\n\nI think a garbage (or trashable) flag, if implemented, wouldn't need any\nspecial case in git-clean, i.e. -x would remove all untracked files,\nwhether ignored or garbage/trashable. That's what my patch to implement\nit does:\nhttps://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\n\nI think that makes sense. Users running \"git clean\" have \"--dry-run\" and\nunlike \"checkout a branch\" or \"merge this commit\" where we'll now shred\ndata implicitly it's obvious that git-clean is going to shred your data.\n"},{"id":"364076","messageId":"CACsJy8C4deg=M+sjmTBM-qs_=zZ9KarND3MNaR6-MqxukBJoSA@mail.gmail.com","threadId":"49800","inReplyTo":"87sgznzwcp.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-26T15:40:48Z","receivedAt":"2018-11-26T15:41:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 26, 2018 at 4:34 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Mon, Nov 26 2018, Duy Nguyen wrote:\n>\n> > On Mon, Nov 26, 2018 at 10:30 AM Per Lundberg wrote:\n> >>\n> >> On 11/13/18 1:22 AM, brian m. carlson wrote:\n> >> > This is going to totally hose automation. My last job had files which\n> >> > might move from tracked to untracked (a file that had become generated),\n> >> > and long-running CI and build systems would need to be able to check out\n> >> > one status and switch to the other. Your proposed change will prevent\n> >> > those systems from working, whereas they previously did.\n> >> >\n> >> > I agree that your proposal would have been a better design originally,\n> >> > but breaking the way automated systems currently work is probably going\n> >> > to be a dealbreaker.\n> >>\n> >> How about something like this:\n> >>\n> >> 1. Introduce a concept with \"garbage\" files, which git is \"permitted to\n> >> delete\" without prompting.\n> >>\n> >> 2. Retain the current default, i.e. \"ignored files are garbage\" for now,\n> >> making the new behavior _opt in_ to avoid breaking automated\n> >> systems/existing scripts for anyone. Put the setting for this behind a\n> >> new core.* config flag.\n> >>\n> >> 3. In the plan for version 3.0 (a new major version where some breakage\n> >> can be tolerable, according to Semantic Versioning), change the default\n> >> so that \"only explicit garbage is garbage\". Include very clear notices\n> >> of this in the release notes. The config flag is retained, but its\n> >> default changes from true->false or vice versa. People who dislike the\n> >> new behavior can easily change back to the 2.x semantics.\n> >\n> > How does this garbage thing interact with \"git clean -x\"? My\n> > interpretation of this flag/attribute is that at version 3.0 by\n> > default all ignored files are _not_ garbage, so \"git clean -x\" should\n> > not remove any of them. Which is weird because most of ignored files\n> > are like *.o that should be removed.\n> >\n> > I also need to mark \"precious\" on untracked or even tracked files (*).\n> > Not sure how this \"garbage\" attribute interacts with that.\n> >\n> > (*) I was hoping I could get the idea [1] implemented in somewhat good\n> > shape before presenting here. But I'm a bit slow on that front. So\n> > yeah this \"precious\" on untracked/tracked thingy may be even\n> > irrelevant if the patch series will be rejected.\n>\n> I think a garbage (or trashable) flag, if implemented, wouldn't need any\n> special case in git-clean, i.e. -x would remove all untracked files,\n> whether ignored or garbage/trashable. That's what my patch to implement\n> it does:\n> https://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\n>\n> I think that makes sense. Users running \"git clean\" have \"--dry-run\" and\n> unlike \"checkout a branch\" or \"merge this commit\" where we'll now shred\n> data implicitly it's obvious that git-clean is going to shred your data.\n\nThen that't not what I want. If I'm going to mark to keep \"config.mak\"\naround, I'm not going to carefully move it away before doing \"git\nclean -fdx\" then move it back. No \"git clean --dry-run\" telling me to\nmake a backup of config.mak is no good.\n-- \nDuy\n"},{"id":"364079","messageId":"87pnurzvr6.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"CACsJy8C4deg=M+sjmTBM-qs_=zZ9KarND3MNaR6-MqxukBJoSA@mail.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-26T15:47:09Z","receivedAt":"2018-11-26T15:47:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 26 2018, Duy Nguyen wrote:\n\n> On Mon, Nov 26, 2018 at 4:34 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>>\n>> On Mon, Nov 26 2018, Duy Nguyen wrote:\n>>\n>> > On Mon, Nov 26, 2018 at 10:30 AM Per Lundberg wrote:\n>> >>\n>> >> On 11/13/18 1:22 AM, brian m. carlson wrote:\n>> >> > This is going to totally hose automation. My last job had files which\n>> >> > might move from tracked to untracked (a file that had become generated),\n>> >> > and long-running CI and build systems would need to be able to check out\n>> >> > one status and switch to the other. Your proposed change will prevent\n>> >> > those systems from working, whereas they previously did.\n>> >> >\n>> >> > I agree that your proposal would have been a better design originally,\n>> >> > but breaking the way automated systems currently work is probably going\n>> >> > to be a dealbreaker.\n>> >>\n>> >> How about something like this:\n>> >>\n>> >> 1. Introduce a concept with \"garbage\" files, which git is \"permitted to\n>> >> delete\" without prompting.\n>> >>\n>> >> 2. Retain the current default, i.e. \"ignored files are garbage\" for now,\n>> >> making the new behavior _opt in_ to avoid breaking automated\n>> >> systems/existing scripts for anyone. Put the setting for this behind a\n>> >> new core.* config flag.\n>> >>\n>> >> 3. In the plan for version 3.0 (a new major version where some breakage\n>> >> can be tolerable, according to Semantic Versioning), change the default\n>> >> so that \"only explicit garbage is garbage\". Include very clear notices\n>> >> of this in the release notes. The config flag is retained, but its\n>> >> default changes from true->false or vice versa. People who dislike the\n>> >> new behavior can easily change back to the 2.x semantics.\n>> >\n>> > How does this garbage thing interact with \"git clean -x\"? My\n>> > interpretation of this flag/attribute is that at version 3.0 by\n>> > default all ignored files are _not_ garbage, so \"git clean -x\" should\n>> > not remove any of them. Which is weird because most of ignored files\n>> > are like *.o that should be removed.\n>> >\n>> > I also need to mark \"precious\" on untracked or even tracked files (*).\n>> > Not sure how this \"garbage\" attribute interacts with that.\n>> >\n>> > (*) I was hoping I could get the idea [1] implemented in somewhat good\n>> > shape before presenting here. But I'm a bit slow on that front. So\n>> > yeah this \"precious\" on untracked/tracked thingy may be even\n>> > irrelevant if the patch series will be rejected.\n>>\n>> I think a garbage (or trashable) flag, if implemented, wouldn't need any\n>> special case in git-clean, i.e. -x would remove all untracked files,\n>> whether ignored or garbage/trashable. That's what my patch to implement\n>> it does:\n>> https://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\n>>\n>> I think that makes sense. Users running \"git clean\" have \"--dry-run\" and\n>> unlike \"checkout a branch\" or \"merge this commit\" where we'll now shred\n>> data implicitly it's obvious that git-clean is going to shred your data.\n>\n> Then that't not what I want. If I'm going to mark to keep \"config.mak\"\n> around, I'm not going to carefully move it away before doing \"git\n> clean -fdx\" then move it back. No \"git clean --dry-run\" telling me to\n> make a backup of config.mak is no good.\n\nUnderstood. I mean this in the context of solving the problem users have\nwith running otherwise non-data-destroying commands like \"checkout\" and\n\"merge\" and getting their data destroyed, which is overwhelmingly why\nthis topic gets resurrected.\n\nSome of the solutions overlap with this thing you want, but I think it's\nworth keeping the distinction between the two in mind. I.e. I can\nimagine us finding some acceptable solution to the data shredding\nproblem that doesn't implement this mode for \"git-clean\", or the other\nway around.\n"},{"id":"364080","messageId":"CACsJy8Ck7CZ7JWaN6ark=wrAngywJJh76y-FvJ87gE2ckVS8pg@mail.gmail.com","threadId":"49800","inReplyTo":"87pnurzvr6.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-26T15:55:13Z","receivedAt":"2018-11-26T15:55:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 26, 2018 at 4:47 PM Ævar Arnfjörð Bjarmason\n wrote:\n> >> >> How about something like this:\n> >> >>\n> >> >> 1. Introduce a concept with \"garbage\" files, which git is \"permitted to\n> >> >> delete\" without prompting.\n> >> >>\n> >> >> 2. Retain the current default, i.e. \"ignored files are garbage\" for now,\n> >> >> making the new behavior _opt in_ to avoid breaking automated\n> >> >> systems/existing scripts for anyone. Put the setting for this behind a\n> >> >> new core.* config flag.\n> >> >>\n> >> >> 3. In the plan for version 3.0 (a new major version where some breakage\n> >> >> can be tolerable, according to Semantic Versioning), change the default\n> >> >> so that \"only explicit garbage is garbage\". Include very clear notices\n> >> >> of this in the release notes. The config flag is retained, but its\n> >> >> default changes from true->false or vice versa. People who dislike the\n> >> >> new behavior can easily change back to the 2.x semantics.\n> >> >\n> >> > How does this garbage thing interact with \"git clean -x\"? My\n> >> > interpretation of this flag/attribute is that at version 3.0 by\n> >> > default all ignored files are _not_ garbage, so \"git clean -x\" should\n> >> > not remove any of them. Which is weird because most of ignored files\n> >> > are like *.o that should be removed.\n> >> >\n> >> > I also need to mark \"precious\" on untracked or even tracked files (*).\n> >> > Not sure how this \"garbage\" attribute interacts with that.\n> >> >\n> >> > (*) I was hoping I could get the idea [1] implemented in somewhat good\n> >> > shape before presenting here. But I'm a bit slow on that front. So\n> >> > yeah this \"precious\" on untracked/tracked thingy may be even\n> >> > irrelevant if the patch series will be rejected.\n> >>\n> >> I think a garbage (or trashable) flag, if implemented, wouldn't need any\n> >> special case in git-clean, i.e. -x would remove all untracked files,\n> >> whether ignored or garbage/trashable. That's what my patch to implement\n> >> it does:\n> >> https://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\n> >>\n> >> I think that makes sense. Users running \"git clean\" have \"--dry-run\" and\n> >> unlike \"checkout a branch\" or \"merge this commit\" where we'll now shred\n> >> data implicitly it's obvious that git-clean is going to shred your data.\n> >\n> > Then that't not what I want. If I'm going to mark to keep \"config.mak\"\n> > around, I'm not going to carefully move it away before doing \"git\n> > clean -fdx\" then move it back. No \"git clean --dry-run\" telling me to\n> > make a backup of config.mak is no good.\n>\n> Understood. I mean this in the context of solving the problem users have\n> with running otherwise non-data-destroying commands like \"checkout\" and\n> \"merge\" and getting their data destroyed, which is overwhelmingly why\n> this topic gets resurrected.\n>\n> Some of the solutions overlap with this thing you want, but I think it's\n> worth keeping the distinction between the two in mind.\n\nOn the other hand all use cases should be considered. It's going to be\na mess to have \"trashable\" attribute that applies to some commands\nwhile \"precious\" to some others (and even worse when they overlap,\nimagine having to define both in .gitattributes)\n\n> I.e. I can\n> imagine us finding some acceptable solution to the data shredding\n> problem that doesn't implement this mode for \"git-clean\", or the other\n> way around.\n-- \nDuy\n"},{"id":"364082","messageId":"20181126160226.GA20885@esm","threadId":"49800","inReplyTo":"20181112232209.GK890086@genre.crustytoothpaste.net","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Eckhard Maaß","fromEmail":"eckhard.s.maass@googlemail.com","sentAt":"2018-11-26T16:02:26Z","receivedAt":"2018-11-26T16:02:31Z","isPatch":true,"sender":{"key":"eckhard.s.maass@googlemail.com","avatar":"https://avatars.githubusercontent.com/u/21984134?v=4"},"body":"On Mon, Nov 12, 2018 at 11:22:09PM +0000, brian m. carlson wrote:\n> This is going to totally hose automation. My last job had files which\n> might move from tracked to untracked (a file that had become generated),\n> and long-running CI and build systems would need to be able to check out\n> one status and switch to the other. Your proposed change will prevent\n> those systems from working, whereas they previously did.\n\nWouldn't those systems not use -f right now? And shouldn't Git have the\nsame semantic for -f to clobber everything in the proposed use case?\nLike it does right now for untracked files which are not ignored. So to\nbe save going back and forth I would expect those systems to use -f\nanyway. Have I missed something here?\n\nRegards,\nEckhard\n"},{"id":"364090","messageId":"20181126193804.30741-1-pclouds@gmail.com","threadId":"49800","inReplyTo":"20181111095254.30473-1-pclouds@gmail.com","subject":"[PATCH v2 0/2] Precios files round two","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-26T19:38:02Z","receivedAt":"2018-11-26T19:38:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Let's see if I can make everybody almost happy with this.\n\nPatch 1 is not that much different from round one except now with\ntests. It is still about \"precious\" attributes that prevent content\nloss with \"git clean\", \"git checkout \" or \"git merge\".\n\nPatch 2 goes Ævar and Per are pursuing. It adds an opt-in config that\nturns all ignored files precious, but only for unpack-trees\noperations.\n\nNguyễn Thái Ngọc Duy (2):\n Introduce \"precious\" file concept\n unpack-trees: support core.allIgnoredFilesArePreciousWhenMerging\n\n Documentation/config/core.txt | 6 ++++++\n Documentation/git-clean.txt | 3 ++-\n Documentation/gitattributes.txt | 13 +++++++++++++\n Documentation/gitignore.txt | 5 +++++\n attr.c | 12 ++++++++++++\n attr.h | 2 ++\n builtin/clean.c | 20 +++++++++++++++++---\n t/t1004-read-tree-m-u-wf.sh | 6 ++++++\n t/t7300-clean.sh | 29 +++++++++++++++++++++++++++++\n unpack-trees.c | 20 ++++++++++++++++++++\n 10 files changed, 112 insertions(+), 4 deletions(-)\n\n-- \n2.19.1.1327.g328c130451.dirty\n\n"},{"id":"364091","messageId":"20181126193804.30741-2-pclouds@gmail.com","threadId":"49800","inReplyTo":"20181126193804.30741-1-pclouds@gmail.com","subject":"[PATCH v2 1/2] Introduce \"precious\" file concept","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-26T19:38:03Z","receivedAt":"2018-11-26T19:38:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"A new attribute \"precious\" is added to indicate that certain files\nhave valuable content and should not be easily discarded even if they\nare ignored or untracked.\n\nSo far there are two parts of Git that are made aware of precious\nfiles: \"git clean\" will leave precious files alone and unpack-trees.c\n(i.e. merges and branch switches) will not overwrite\nignored-but-precious files.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-clean.txt | 3 ++-\n Documentation/gitattributes.txt | 13 +++++++++++++\n Documentation/gitignore.txt | 4 ++++\n attr.c | 12 ++++++++++++\n attr.h | 2 ++\n builtin/clean.c | 20 +++++++++++++++++---\n t/t1004-read-tree-m-u-wf.sh | 6 ++++++\n t/t7300-clean.sh | 29 +++++++++++++++++++++++++++++\n unpack-trees.c | 1 +\n 9 files changed, 86 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt\nindex 03056dad0d..a9beadfb12 100644\n--- a/Documentation/git-clean.txt\n+++ b/Documentation/git-clean.txt\n@@ -21,7 +21,8 @@ option is specified, ignored files are also removed. This can, for\n example, be useful to remove all build products.\n \n If any optional `...` arguments are given, only those paths\n-are affected.\n+are affected. Ignored or untracked files with `precious` attributes\n+are not removed.\n \n OPTIONS\n -------\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex b8392fc330..b027abea4a 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -1188,6 +1188,19 @@ If this attribute is not set or has an invalid value, the value of the\n (See linkgit:git-config[1]).\n \n \n+Precious files\n+~~~~~~~~~~~~~~\n+\n+`precious`\n+^^^^^^^^^^\n+\n+This attribute is set on files to indicate that their content is\n+valuable. Many commands will behave slightly different on precious\n+files. linkgit:git-clean[1] will leave precious files alone. Merging\n+and branch switching will not silently overwrite ignored files that\n+are marked \"precious\".\n+\n+\n USING MACRO ATTRIBUTES\n ----------------------\n \ndiff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt\nindex 1c94f08ff4..0e9614289e 100644\n--- a/Documentation/gitignore.txt\n+++ b/Documentation/gitignore.txt\n@@ -141,6 +141,10 @@ not tracked by Git remain untracked.\n To stop tracking a file that is currently tracked, use\n 'git rm --cached'.\n \n+Ignored files are generally considered discardable. See `precious`\n+attribute in linkgit:gitattributes[5] to change the behavior regarding\n+ignored files.\n+\n EXAMPLES\n --------\n \ndiff --git a/attr.c b/attr.c\nindex eaece6658d..b07d8cd835 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -1172,3 +1172,15 @@ void attr_start(void)\n \tpthread_mutex_init(&g_attr_hashmap.mutex, NULL);\n \tpthread_mutex_init(&check_vector.mutex, NULL);\n }\n+\n+int is_precious_file(struct index_state *istate, const char *path)\n+{\n+\tstatic struct attr_check *check;\n+\tif (!check)\n+\t\tcheck = attr_check_initl(\"precious\", NULL);\n+\tif (!check)\n+\t\treturn 0;\n+\n+\tgit_check_attr(istate, path, check);\n+\treturn ATTR_TRUE(check->items[0].value);\n+}\ndiff --git a/attr.h b/attr.h\nindex b0378bfe5f..b9a9751a66 100644\n--- a/attr.h\n+++ b/attr.h\n@@ -82,4 +82,6 @@ void git_attr_set_direction(enum git_attr_direction new_direction);\n \n void attr_start(void);\n \n+int is_precious_file(struct index_state *istate, const char *path);\n+\n #endif /* ATTR_H */\ndiff --git a/builtin/clean.c b/builtin/clean.c\nindex bbcdeb2d9e..42cd040849 100644\n--- a/builtin/clean.c\n+++ b/builtin/clean.c\n@@ -17,6 +17,7 @@\n #include \"color.h\"\n #include \"pathspec.h\"\n #include \"help.h\"\n+#include \"attr.h\"\n \n static int force = -1; /* unset */\n static int interactive;\n@@ -30,6 +31,8 @@ static const char *const builtin_clean_usage[] = {\n \n static const char *msg_remove = N_(\"Removing %s\\n\");\n static const char *msg_would_remove = N_(\"Would remove %s\\n\");\n+static const char *msg_skip_precious = N_(\"Skipping precious file %s\\n\");\n+static const char *msg_would_skip_precious = N_(\"Would skip precious file %s\\n\");\n static const char *msg_skip_git_dir = N_(\"Skipping repository %s\\n\");\n static const char *msg_would_skip_git_dir = N_(\"Would skip repository %s\\n\");\n static const char *msg_warn_remove_failed = N_(\"failed to remove %s\");\n@@ -153,6 +156,7 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag,\n \tstruct dirent *e;\n \tint res = 0, ret = 0, gone = 1, original_len = path->len, len;\n \tstruct string_list dels = STRING_LIST_INIT_DUP;\n+\tconst char *rel_path;\n \n \t*dir_gone = 1;\n \n@@ -192,9 +196,16 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag,\n \n \t\tstrbuf_setlen(path, len);\n \t\tstrbuf_addstr(path, e->d_name);\n-\t\tif (lstat(path->buf, &st))\n+\t\tif (lstat(path->buf, &st)) {\n \t\t\t; /* fall thru */\n-\t\telse if (S_ISDIR(st.st_mode)) {\n+\t\t} else if ((!prefix && is_precious_file(&the_index, path->buf)) ||\n+\t\t\t (prefix && skip_prefix(path->buf, prefix, &rel_path) &&\n+\t\t\t is_precious_file(&the_index, rel_path))) {\n+\t\t\tquote_path_relative(path->buf, prefix, "ed);\n+\t\t\tprintf(dry_run ? _(msg_would_skip_precious) : _(msg_skip_precious), quoted.buf);\n+\t\t\t*dir_gone = 0;\n+\t\t\tcontinue;\n+\t\t} else if (S_ISDIR(st.st_mode)) {\n \t\t\tif (remove_dirs(path, prefix, force_flag, dry_run, quiet, &gone))\n \t\t\t\tret = 1;\n \t\t\tif (gone) {\n@@ -1018,7 +1029,10 @@ int cmd_clean(int argc, const char **argv, const char *prefix)\n \t\tif (lstat(abs_path.buf, &st))\n \t\t\tcontinue;\n \n-\t\tif (S_ISDIR(st.st_mode)) {\n+\t\tif (is_precious_file(&the_index, item->string)) {\n+\t\t\tqname = quote_path_relative(item->string, NULL, &buf);\n+\t\t\tprintf(dry_run ? _(msg_would_skip_precious) : _(msg_skip_precious), qname);\n+\t\t} else if (S_ISDIR(st.st_mode)) {\n \t\t\tif (remove_dirs(&abs_path, prefix, rm_flags, dry_run, quiet, &gone))\n \t\t\t\terrors++;\n \t\t\tif (gone && !quiet) {\ndiff --git a/t/t1004-read-tree-m-u-wf.sh b/t/t1004-read-tree-m-u-wf.sh\nindex c13578a635..17dc626f62 100755\n--- a/t/t1004-read-tree-m-u-wf.sh\n+++ b/t/t1004-read-tree-m-u-wf.sh\n@@ -63,6 +63,12 @@ test_expect_success 'two-way with incorrect --exclude-per-directory (2)' '\n \tfi\n '\n \n+test_expect_success 'two-way not clobbering a precious ignored file' '\n+\ttest_when_finished rm -f .git/info/attributes &&\n+\techo \"file2 precious\" >.git/info/attributes &&\n+\tread_tree_u_must_fail -m -u --exclude-per-directory=.gitignore master side\n+'\n+\n test_expect_success 'two-way clobbering a ignored file' '\n \n \tread_tree_u_must_succeed -m -u --exclude-per-directory=.gitignore master side\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 7b36954d63..a478a08a27 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -669,4 +669,33 @@ test_expect_success 'git clean -d skips untracked dirs containing ignored files'\n \ttest_path_is_missing foo/b/bb\n '\n \n+test_expect_success 'git clean -xd leaves precious files alone' '\n+\tgit init precious &&\n+\t(\n+\t\tcd precious &&\n+\t\ttest_commit one &&\n+\t\tcat >.gitignore <<-\\EOF &&\n+\t\t*.o\n+\t\t*.mak\n+\t\tEOF\n+\t\tcat >.gitattributes <<-\\EOF &&\n+\t\t*.mak precious\n+\t\t.gitattributes precious\n+\t\t*.precious precious\n+\t\tEOF\n+\t\tmkdir sub &&\n+\t\ttouch one.o sub/two.o one.mak sub/two.mak &&\n+\t\ttouch one.untracked two.precious sub/also.precious &&\n+\t\tgit clean -fdx &&\n+\t\ttest_path_is_missing one.o &&\n+\t\ttest_path_is_missing sub/two.o &&\n+\t\ttest_path_is_missing one.untracked &&\n+\t\ttest_path_is_file .gitattributes &&\n+\t\ttest_path_is_file one.mak &&\n+\t\ttest_path_is_file sub/two.mak &&\n+\t\ttest_path_is_file two.precious &&\n+\t\ttest_path_is_file sub/also.precious\n+\t)\n+'\n+\n test_done\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 7570df481b..9a5aadc084 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1895,6 +1895,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\treturn 0;\n \n \tif (o->dir &&\n+\t !is_precious_file(o->src_index, name) &&\n \t is_excluded(o->dir, o->src_index, name, &dtype))\n \t\t/*\n \t\t * ce->name is explicitly excluded, so it is Ok to\n-- \n2.19.1.1327.g328c130451.dirty\n\n"},{"id":"364092","messageId":"20181126193804.30741-3-pclouds@gmail.com","threadId":"49800","inReplyTo":"20181126193804.30741-1-pclouds@gmail.com","subject":"[PATCH v2 2/2] unpack-trees: support core.allIgnoredFilesArePreciousWhenMerging","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-26T19:38:04Z","receivedAt":"2018-11-26T19:38:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Ignored files can be marked precious to prevent being overwritten\nduring a merge (or even a branch switch). If you really want to make\nsure no ignored files are overwritten, this config variable is for\nyou.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config/core.txt | 6 ++++++\n Documentation/gitignore.txt | 5 +++--\n unpack-trees.c | 21 ++++++++++++++++++++-\n 3 files changed, 29 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex d0e6635fe0..bff5834c13 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -1,3 +1,9 @@\n+core.allIgnoredFilesArePreciousWhenMerging::\n+\tDuring a merge operation, if \"precious\" attribute is unset,\n+\tconsider it set. You can explicitly remove \"precious\"\n+\tattribute if needed. See linkgit:gitattributes[5] for more\n+\tinformation.\n+\n core.fileMode::\n \tTells Git if the executable bit of files in the working tree\n \tis to be honored.\ndiff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt\nindex 0e9614289e..2832df7178 100644\n--- a/Documentation/gitignore.txt\n+++ b/Documentation/gitignore.txt\n@@ -142,8 +142,9 @@ To stop tracking a file that is currently tracked, use\n 'git rm --cached'.\n \n Ignored files are generally considered discardable. See `precious`\n-attribute in linkgit:gitattributes[5] to change the behavior regarding\n-ignored files.\n+attribute in linkgit:gitattributes[5] and\n+`core.allIgnoredFilesArePreciousWhenMerging` in linkgit:git-config[1]\n+to change the behavior regarding ignored files.\n \n EXAMPLES\n --------\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 9a5aadc084..df3b163e2e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1877,6 +1877,25 @@ static int icase_exists(struct unpack_trees_options *o, const char *name, int le\n \treturn src && !ie_match_stat(o->src_index, src, st, CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE);\n }\n \n+static int is_precious_ignored_file(struct index_state *istate, const char *path)\n+{\n+\tstatic struct attr_check *check;\n+\tint all_precious;\n+\n+\tif (!check)\n+\t\tcheck = attr_check_initl(\"precious\", NULL);\n+\tif (!check)\n+\t\treturn 0;\n+\n+\tgit_check_attr(istate, path, check);\n+\tif (ATTR_UNSET(check->items[0].value) &&\n+\t !git_config_get_bool(\"core.allignoredfilesarepreciouswhenmerging\",\n+\t\t\t\t &all_precious) &&\n+\t all_precious)\n+\t\treturn 1;\n+\treturn ATTR_TRUE(check->items[0].value);\n+}\n+\n static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\t\t const struct cache_entry *ce, struct stat *st,\n \t\t\t enum unpack_trees_error_types error_type,\n@@ -1895,7 +1914,7 @@ static int check_ok_to_remove(const char *name, int len, int dtype,\n \t\treturn 0;\n \n \tif (o->dir &&\n-\t !is_precious_file(o->src_index, name) &&\n+\t !is_precious_ignored_file(o->src_index, name) &&\n \t is_excluded(o->dir, o->src_index, name, &dtype))\n \t\t/*\n \t\t * ce->name is explicitly excluded, so it is Ok to\n-- \n2.19.1.1327.g328c130451.dirty\n\n"},{"id":"364122","messageId":"f6aad4a2-8cb4-acc2-af9b-9c9c82059b89@hibox.tv","threadId":"49800","inReplyTo":"CACsJy8Ck7CZ7JWaN6ark=wrAngywJJh76y-FvJ87gE2ckVS8pg@mail.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Per Lundberg","fromEmail":"per.lundberg@hibox.tv","sentAt":"2018-11-27T09:43:17Z","receivedAt":"2018-11-27T09:43:25Z","isPatch":true,"sender":{"key":"per.lundberg@hibox.tv","avatar":"https://gravatar.com/avatar/037abe105bde31ba7f4fe80a1436ef76a06c9ca8f2c0c153e6ca66a7527cd637?d=mp&s=160"},"body":"On 11/26/18 5:55 PM, Duy Nguyen wrote:\n> On Mon, Nov 26, 2018 at 4:47 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>> Some of the solutions overlap with this thing you want, but I think it's\n>> worth keeping the distinction between the two in mind.\n> \n> On the other hand all use cases should be considered. It's going to be\n> a mess to have \"trashable\" attribute that applies to some commands\n> while \"precious\" to some others (and even worse when they overlap,\n> imagine having to define both in .gitattributes)\n\nAgree - I think it would be a very bad idea to have a \"mix\" of both \ntrashable and precious. IMO, we should try to find which one of these \nconcepts suits most general use cases best and causes less churn for \nexisting scripts/users' existing \"semantic expectations\", and pick that one.\n--\nPer Lundberg\n"},{"id":"364129","messageId":"CA+P7+xri1=peNpEiZCE802HwCXhojyp2BDvOR+6BBSoRtsZyzA@mail.gmail.com","threadId":"49800","inReplyTo":"f6aad4a2-8cb4-acc2-af9b-9c9c82059b89@hibox.tv","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2018-11-27T12:55:50Z","receivedAt":"2018-11-27T12:56:05Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Tue, Nov 27, 2018 at 1:45 AM Per Lundberg wrote:\n>\n> On 11/26/18 5:55 PM, Duy Nguyen wrote:\n> > On Mon, Nov 26, 2018 at 4:47 PM Ævar Arnfjörð Bjarmason\n> > wrote:\n> >> Some of the solutions overlap with this thing you want, but I think it's\n> >> worth keeping the distinction between the two in mind.\n> >\n> > On the other hand all use cases should be considered. It's going to be\n> > a mess to have \"trashable\" attribute that applies to some commands\n> > while \"precious\" to some others (and even worse when they overlap,\n> > imagine having to define both in .gitattributes)\n>\n> Agree - I think it would be a very bad idea to have a \"mix\" of both\n> trashable and precious. IMO, we should try to find which one of these\n> concepts suits most general use cases best and causes less churn for\n> existing scripts/users' existing \"semantic expectations\", and pick that one.\n> --\n> Per Lundberg\n\nPersonally, I would rather err on the side which requires the least\ninteraction from users to avoid silently clobbering an ignored file.\n\nEither Duy's solution with a sort of \"untracked\" reflog, or the\ngarbage/trashable notion.\n\nI don't like the idea of precious because it means people have to know\nand remember to opt in, and it's quite possible they will not do so\nuntil after they've lost real data.\n\nI'd only have trashable apply in the case where it was implicit. i.e.\ngit clean -fdx would still delete them, as this is an explicit\noperation that (hopefully?) users know will delete data.\n\nThanks,\nJake\n"},{"id":"364133","messageId":"56ffc52f-644c-2c1d-6a16-c1005b064385@hibox.tv","threadId":"49800","inReplyTo":"CA+P7+xri1=peNpEiZCE802HwCXhojyp2BDvOR+6BBSoRtsZyzA@mail.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Per Lundberg","fromEmail":"per.lundberg@hibox.tv","sentAt":"2018-11-27T14:50:34Z","receivedAt":"2018-11-27T14:50:45Z","isPatch":true,"sender":{"key":"per.lundberg@hibox.tv","avatar":"https://gravatar.com/avatar/037abe105bde31ba7f4fe80a1436ef76a06c9ca8f2c0c153e6ca66a7527cd637?d=mp&s=160"},"body":"On 11/27/18 2:55 PM, Jacob Keller wrote:\n\n> Personally, I would rather err on the side which requires the least\n> interaction from users to avoid silently clobbering an ignored file.\n> \n > [...]\n> \n> I don't like the idea of precious because it means people have to know\n> and remember to opt in, and it's quite possible they will not do so\n> until after they've lost real data.\n\nI agree strongly with this personally; if we must choose between \"might \nbreak automation\" and \"might delete non-garbage files\", I would say the \nformer is the lesser evil of the two.\n\nBut, if I had 10 000 000 servers set up using automated scripts that \nwould break because of this, I might think differently. Quite likely so, \nin fact.\n\nWhat are these automation scenarios _more specifically_? Junio or Brian, \nwould you care to elaborate? Is it for build servers where you want \"git \nclean -dfx\" to always reset the working copy to a pristine state or are \nwe talking about some other scenarios?\n\n> I'd only have trashable apply in the case where it was implicit. i.e.\n> git clean -fdx would still delete them, as this is an explicit\n> operation that (hopefully?) users know will delete data.\n\nThis is one of the tougher calls, unfortunately.\n\nIf I was a user (which I am), and I was typing \"git clean -dfx\", what \nwould I expect?\n\nThe help text (currently) states \"-x remove ignored files, too\".\n\nWould it be safe to assume that people would understand that \"ignored \n_does not_ mean trashable when doing \"git checkout some-ref\" BUT it \n_does_ mean trashable in the \"git clean -dfx\" context\"? I'm not so \ncertain. It would be one of those perceived inconsistencies that would \nmake people scream in anger because they _presumed_ that with the new \n\"trashable\" concept, \"git clean -dfx\" would no longer hit them in the leg.\n\nAnd the other way around: if we change \"git clean -dfx\" to _not_ treat \n\"ignored == trashable\", it is likely to \"hose automation\" as it has been \npreviously stated. People who might be using this syntax and _want_ it \nto remove ignored files would be upset, and rightfully so.\n\nSo in my POV, it's a tough decision between two, less-than-optimal \nalternatives.\n\nBut I would perhaps be able to live with the current semantics for \"git \nclean -dfx\" _as long as we update the help text_ so that \"-x\" indicates \nmore clearly that non-trashable files can be deleted. It doesn't make \nthings _worse_ than they currently are and if this is what it takes to \nget the trashable concept implemented and accepted by the community, \nit's a compromise I'd be willing to make.\n--\nPer Lundberg\n\n"},{"id":"364134","messageId":"87mupuzhfx.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"xmqqzhtwuhpc.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-27T15:08:34Z","receivedAt":"2018-11-27T15:08:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 26 2018, Junio C Hamano wrote:\n\n> Per Lundberg writes:\n>\n>> How about something like this:\n>> ...\n>> Would this be a reasonable compromise for everybody?\n>\n> I do not think you'd need to introduce such a deliberately breaking\n> change at all. Just introduce a new \"precious\" class, perhaps mark\n> them with the atttribute mechanism, and that would be the endgame.\n> Early adopters would start marking ignored but not expendable paths\n> with the \"precious\" attribute and they won't be clobbered. As the\n> mechanism becomes widely known and mature, more and more people use\n> it. And even after that happens, early adopters do not have to change\n> any attribute setting, and late adopters would have plenty of examples\n> to imitate. Those who do not need any \"precious\" class do not have\n> to do anything and you won't break any existing automation that way.\n\nThe patch I submitted in <87zhuf3gs0.fsf@evledraar.gmail.com>[1] changed\nthe behavior for read-tree & checkout & merge etc.\n\nIt was an RFC more in the spirit of showing what in our current tests\nhad to change to spur some discussion.\n\nBut I'm very sympathetic to this line of argument. I.e. in my patch I'm\nchanging the semantics of read-tree, which is plumbing.\n\nWhat do you think about some patch like that which retains the plumbing\nbehavior for things like read-tree, doesn't introduce \"precious\" or\n\"trashable\", and just makes you specify \"[checkout|merge|...] --force\"\nin cases where we'd have clobbering?\n\nThis would give scripts which relied on our stable plumbing consistent\nbehavior, while helping users who're using our main porcelain not to\nlose data. I could then add a --force option to the likes of read-tree\n(on by default), so you could get porcelain-like behavior with\n--no-force.\n\n1. https://public-inbox.org/git/87zhuf3gs0.fsf@evledraar.gmail.com/\n"},{"id":"364135","messageId":"CACsJy8AtM4uar1TB=bevC4dnctN9h+V3P4OsSnCk9fGseXDrog@mail.gmail.com","threadId":"49800","inReplyTo":"CA+P7+xri1=peNpEiZCE802HwCXhojyp2BDvOR+6BBSoRtsZyzA@mail.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-27T15:19:42Z","receivedAt":"2018-11-27T15:20:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Nov 27, 2018 at 1:56 PM Jacob Keller wrote:\n>\n> On Tue, Nov 27, 2018 at 1:45 AM Per Lundberg wrote:\n> >\n> > On 11/26/18 5:55 PM, Duy Nguyen wrote:\n> > > On Mon, Nov 26, 2018 at 4:47 PM Ævar Arnfjörð Bjarmason\n> > > wrote:\n> > >> Some of the solutions overlap with this thing you want, but I think it's\n> > >> worth keeping the distinction between the two in mind.\n> > >\n> > > On the other hand all use cases should be considered. It's going to be\n> > > a mess to have \"trashable\" attribute that applies to some commands\n> > > while \"precious\" to some others (and even worse when they overlap,\n> > > imagine having to define both in .gitattributes)\n> >\n> > Agree - I think it would be a very bad idea to have a \"mix\" of both\n> > trashable and precious. IMO, we should try to find which one of these\n> > concepts suits most general use cases best and causes less churn for\n> > existing scripts/users' existing \"semantic expectations\", and pick that one.\n> > --\n> > Per Lundberg\n>\n> Personally, I would rather err on the side which requires the least\n> interaction from users to avoid silently clobbering an ignored file.\n>\n> Either Duy's solution with a sort of \"untracked\" reflog, or the\n> garbage/trashable notion.\n>\n> I don't like the idea of precious because it means people have to know\n> and remember to opt in, and it's quite possible they will not do so\n> until after they've lost real data.\n>\n> I'd only have trashable apply in the case where it was implicit. i.e.\n> git clean -fdx would still delete them, as this is an explicit\n> operation that (hopefully?) users know will delete data.\n\nYes I know it will delete ignored files. But I don't want it to delete\nsome files. There is no way I can tell Git to do that.\n\nIt's the same with merge/checkout's overwriting problem. Once the\ninitial surprise is over, I want control over what files I want Git to\njust delete and not annoy me, what Git should not delete.\n-- \nDuy\n"},{"id":"364178","messageId":"20181128012142.GT890086@genre.crustytoothpaste.net","threadId":"49800","inReplyTo":"56ffc52f-644c-2c1d-6a16-c1005b064385@hibox.tv","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-28T01:21:42Z","receivedAt":"2018-11-28T01:21:52Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Tue, Nov 27, 2018 at 02:50:34PM +0000, Per Lundberg wrote:\n> I agree strongly with this personally; if we must choose between \"might\n> break automation\" and \"might delete non-garbage files\", I would say the\n> former is the lesser evil of the two.\n> \n> But, if I had 10 000 000 servers set up using automated scripts that\n> would break because of this, I might think differently. Quite likely so,\n> in fact.\n> \n> What are these automation scenarios _more specifically_? Junio or Brian,\n> would you care to elaborate? Is it for build servers where you want \"git\n> clean -dfx\" to always reset the working copy to a pristine state or are\n> we talking about some other scenarios?\n\nWe had long-running CI servers, since bootstrapping a new system took an\nhour. These would check out the branch to test and run some process\n(essentially, a \"make\" and \"make test\"). Then, another branch would be\ntested, and so on. The old branch would likely not be merged at this\npoint.\n\nThe scenario I'm thinking of is when a file (say a CSS file) became\nbuilt instead of stored in the repository. Then the file would be added\nto .gitignore in the new commit, and it would be generated as part of\nthe make step. It would be important to blow away that file away when\nchecking out a new commit, because not doing so would mean that the CI\nsystem would simply fail to work and require manual intervention.\n\nMoreover, a CI job might fail, due to either a flaky test or a\nlegitimate failures, so the job might need to be re-run multiple times.\nRequiring human intervention, especially when such jobs might be running\nat odd hours, would be undesirable.\n\nAnother thing we did was to use a specially named gitignore file in our\nbuild step. We created a new repository, copied the special gitignore\nfile in as .gitignore, copied in the source and build products, ran git\nadd and git commit, and then ran git clean -dfx to remove proprietary\nsource code, packaging the result. A change to the behavior of git\nclean -dfx would be devastating here.\n\nI point this out to underscore how fundamental this change is. People\noverwhelmingly do not read the release notes, so expecting people to\nrealize that a change has been made, especially when many people only\nupgrade Git because of a security issue, may result in unexpected\nconsequences. Just because we don't think of this use of Git as normal\nor desirable doesn't mean people don't do it and don't expect it to\nkeep working. People do read and rely on our documentation.\n\nI think any change we make here has to be opt-in, at least until Git\n3.0. A config knob would probably be the right way to go. I realize\nthat may not provide all the benefits we want out of the box, but it\nlets people turn the option on once and forget about it. It also lets\npeople who don't desire this new behavior explicitly turn it off.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"364182","messageId":"xmqqa7ltua2s.fsf@gitster-ct.c.googlers.com","threadId":"49800","inReplyTo":"87mupuzhfx.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-28T03:58:51Z","receivedAt":"2018-11-28T03:59:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> What do you think about some patch like that which retains the plumbing\n> behavior for things like read-tree, doesn't introduce \"precious\" or\n> \"trashable\", and just makes you specify \"[checkout|merge|...] --force\"\n> in cases where we'd have clobbering?\n\nWhether you like it or not, don't people's automation use tons of\ninvocations of \"git merge\", \"git checkout\", etc.? You'd be breaking\nthem by such a change. Other than that, if we never had Git before\nand do not have to worry about existing users, I'd think it would be\na lot closer to the ideal than today's system if \"checkout \nfoo.o\" rejected overwriting \"foo.o\" that is not tracked in the\ncurrent index but matches an ignore pattern, and required a\n\"--force\" option to overwrite it.\n\nA user, during a conflict resolution, may say \"I want this 'git\ncheckout foo/' to ignore conflicted paths in that directory, so I\nwould give \"--force\" option to it, but now \"--force\" also implies\nthat I am willing to clobber ignored paths, which means I cannot use\nit\".\n\nI would think that a proper automation needs per-path hint from the\nuser and/or the project, not just a single-size-fits-all --force\noption, and \"unlike all the *.o ignored files that are expendable,\nthis vendor-supplied-object.o is not\" is one way to give such a\nper-path hint.\n\n> This would give scripts which relied on our stable plumbing consistent\n> behavior, while helping users who're using our main porcelain not to\n> lose data. I could then add a --force option to the likes of read-tree\n> (on by default), so you could get porcelain-like behavior with\n> --no-force.\n\nAt that low level, I suspect that a single size fits all \"--force\"\nwould work even less well.\n\n"},{"id":"364195","messageId":"08c3997b-b6fd-b1df-cc1c-f8b054f69f67@hibox.tv","threadId":"49800","inReplyTo":"20181128012142.GT890086@genre.crustytoothpaste.net","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Per Lundberg","fromEmail":"per.lundberg@hibox.tv","sentAt":"2018-11-28T06:54:46Z","receivedAt":"2018-11-28T06:54:56Z","isPatch":true,"sender":{"key":"per.lundberg@hibox.tv","avatar":"https://gravatar.com/avatar/037abe105bde31ba7f4fe80a1436ef76a06c9ca8f2c0c153e6ca66a7527cd637?d=mp&s=160"},"body":"On 11/28/18 3:21 AM, brian m. carlson wrote:\n\nThanks for the elaboration, Brian - good to get things down to a \npractical, real-world level.\n\n > [...]\n >\n> I point this out to underscore how fundamental this change is. People\n> overwhelmingly do not read the release notes, so expecting people to\n> realize that a change has been made, especially when many people only\n> upgrade Git because of a security issue, may result in unexpected\n> consequences.\n\nThis is one of the more important things of software engineering. _Don't \nmix security fixes with breaking changes_. They are very different \nthings and like you say, we can't really expect people to real release \nnotes for every little incremental release we do.\n\nThat's an important part of the SemVer guarantee: a minor version \nbump/patch level increase means \"bug fix\" or \"added functionality in a \nbackwards-compatible way\". So: no changing of default behavior or \nsemantics, but adding a new behavior which is opt-in is perfectly fine.\n\n> Just because we don't think of this use of Git as normal or desirable > doesn't mean people don't do it and don't expect it to keep working.\n\nIn other words, we need to be \"bug-by-bug\" compatible with previous \nversions. :-) What some people would consider a bug, others would \nconsider a feature.\n\n> I think any change we make here has to be opt-in, at least until Git\n> 3.0. A config knob would probably be the right way to go. \n\nAgree. It's less than optimal but I think it's something that we all \ncould live with. Deciding to switching the default (or not) is then \nrightfully postponed to a later time, and we can revisit the pros and \ncons then. The important thing now is to get the functionality \nimplemented in a good way, tested and eventually merged.\n--\nPer Lundberg\n"},{"id":"364259","messageId":"875zwgzx4v.fsf@evledraar.gmail.com","threadId":"49800","inReplyTo":"xmqqa7ltua2s.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-28T21:54:08Z","receivedAt":"2018-11-28T21:54:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 28 2018, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> What do you think about some patch like that which retains the plumbing\n>> behavior for things like read-tree, doesn't introduce \"precious\" or\n>> \"trashable\", and just makes you specify \"[checkout|merge|...] --force\"\n>> in cases where we'd have clobbering?\n>\n> Whether you like it or not, don't people's automation use tons of\n> invocations of \"git merge\", \"git checkout\", etc.? You'd be breaking\n> them by such a change.\n\nI'm so sympathetic to this argument that I tried to convince you of\nsomething like this around a year and a half ago:\nhttps://public-inbox.org/git/CACBZZX59KXPOEjiUKtZLN6zjO_xpiWve7Xga6q-53J2LwvfZyw@mail.gmail.com/\n:)\n\nI was probing for what your current stance on this sort of thing is,\nbecause discussions like this tend to get bogged down in the irrelevant\ndistraction of whether something is plumbing or porcelain, which almost\nnone of our users care about, and we've effectively stopped caring about\nourselves.\n\nBut we must have some viable way to repair warts in the tools, and\nlosing user data is a *big* wart.\n\nI don't think something like the endgame you've described in\nhttps://public-inbox.org/git/xmqqzhtwuhpc.fsf@gitster-ct.c.googlers.com/\nis ever going to work. Novice git users (the vast majority) are not\ngoing to diligently update both .gitignore and some .gitattribute\nmechanism in lockstep. I'd bet most git users haven't read more than a\nfew paragraphs of our entire documentation at best.\n\nSo what's the way forward? I think ultimately we must move to something\nwhere we effectively version the entire CLI UI similar to stable API\nversions. I.e. for things like this that would break some things (or\nDuy's new \"split checkout\") introduce them as flags first, then bundle\nup all such flags and cut a major release \"Git 3, 4, ...\", and\neventually remove old functionality.\n\n> Other than that, if we never had Git before and do not have to worry\n> about existing users, I'd think it would be a lot closer to the ideal\n> than today's system if \"checkout foo.o\" rejected overwriting\n> \"foo.o\" that is not tracked in the current index but matches an ignore\n> pattern, and required a \"--force\" option to overwrite it.\n>\n> A user, during a conflict resolution, may say \"I want this 'git\n> checkout foo/' to ignore conflicted paths in that directory, so I\n> would give \"--force\" option to it, but now \"--force\" also implies\n> that I am willing to clobber ignored paths, which means I cannot use\n> it\".\n>\n> I would think that a proper automation needs per-path hint from the\n> user and/or the project, not just a single-size-fits-all --force\n> option, and \"unlike all the *.o ignored files that are expendable,\n> this vendor-supplied-object.o is not\" is one way to give such a\n> per-path hint.\n>\n>> This would give scripts which relied on our stable plumbing consistent\n>> behavior, while helping users who're using our main porcelain not to\n>> lose data. I could then add a --force option to the likes of read-tree\n>> (on by default), so you could get porcelain-like behavior with\n>> --no-force.\n>\n> At that low level, I suspect that a single size fits all \"--force\"\n> would work even less well.\n\nYeah I don't think the one-size-fits-all way out of this is a single\n--force flag.\n"},{"id":"364284","messageId":"xmqqzhtspj91.fsf@gitster-ct.c.googlers.com","threadId":"49800","inReplyTo":"875zwgzx4v.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-29T05:04:10Z","receivedAt":"2018-11-29T05:04:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> I don't think something like the endgame you've described in\n> https://public-inbox.org/git/xmqqzhtwuhpc.fsf@gitster-ct.c.googlers.com/\n> is ever going to work. Novice git users (the vast majority) are not\n> going to diligently update both .gitignore and some .gitattribute\n> mechanism in lockstep.\n\nThat goes both ways, no? Forcing people to add the same pattern,\ne.g. *.o, to .gitignore and .gitattribute to say they are\nexpendable, when most of the time they are, is not something you\nshould expect from the normal population.\n\n>> I would think that a proper automation needs per-path hint from the\n>> user and/or the project, not just a single-size-fits-all --force\n>> option, and \"unlike all the *.o ignored files that are expendable,\n>> this vendor-supplied-object.o is not\" is one way to give such a\n>> per-path hint.\n>>\n>>> This would give scripts which relied on our stable plumbing consistent\n>>> behavior, while helping users who're using our main porcelain not to\n>>> lose data. I could then add a --force option to the likes of read-tree\n>>> (on by default), so you could get porcelain-like behavior with\n>>> --no-force.\n>>\n>> At that low level, I suspect that a single size fits all \"--force\"\n>> would work even less well.\n>\n> Yeah I don't think the one-size-fits-all way out of this is a single\n> --force flag.\n\nYes, indeed. That's why I prefer the \"precious\" bit. The system\nwould behave the same way with or without it, but projects (not\nindividual endusers) can take advantage of the feature if they\nwanted to.\n\n"},{"id":"364413","messageId":"CACsJy8D7rypM-0gYx3Xtx0jDQ=xMO2GxEmf6Mvc84Ch3AnSkug@mail.gmail.com","threadId":"49800","inReplyTo":"875zwgzx4v.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-12-01T06:21:36Z","receivedAt":"2018-12-01T06:22:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Nov 28, 2018 at 10:54 PM Ævar Arnfjörð Bjarmason\n wrote:\n> But we must have some viable way to repair warts in the tools, and\n> losing user data is a *big* wart.\n>\n> I don't think something like the endgame you've described in\n> https://public-inbox.org/git/xmqqzhtwuhpc.fsf@gitster-ct.c.googlers.com/\n> is ever going to work. Novice git users (the vast majority) are not\n> going to diligently update both .gitignore and some .gitattribute\n> mechanism in lockstep. I'd bet most git users haven't read more than a\n> few paragraphs of our entire documentation at best.\n>\n> So what's the way forward? I think ultimately we must move to something\n> where we effectively version the entire CLI UI similar to stable API\n> versions. I.e. for things like this that would break some things (or\n> Duy's new \"split checkout\") introduce them as flags first, then bundle\n> up all such flags and cut a major release \"Git 3, 4, ...\", and\n> eventually remove old functionality.\n\nRelated to Duy's new \"split chekckout\", I just realized that I added\n--overwrite-ignore (enabled by default) [1] years ago to allow to out\nout of this behavior. We could turn --no-overwrite-ignore by default\non the new command \"git switch-branch\" to err on the safe side. Then\nthe user could switch to --overwrite-ignore once they learn more about\ngitignore and gitattributes (or the coming \"backup log\"). I'm not sure\nif I really like this, but at least it's one of the options.\n\n[1] https://public-inbox.org/git/1322388933-6284-2-git-send-email-pclouds@gmail.com/\n-- \nDuy\n"},{"id":"364699","messageId":"CACsJy8BJanw33m9ei3TFL+kt1OH3htUwemhSE=goUh=6py3+xg@mail.gmail.com","threadId":"49800","inReplyTo":"CA+P7+xri1=peNpEiZCE802HwCXhojyp2BDvOR+6BBSoRtsZyzA@mail.gmail.com","subject":"Re: [RFC PATCH] Introduce \"precious\" file concept","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-12-06T18:39:30Z","receivedAt":"2018-12-06T18:40:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Nov 27, 2018 at 1:56 PM Jacob Keller wrote:\n> Personally, I would rather err on the side which requires the least\n> interaction from users to avoid silently clobbering an ignored file.\n>\n> Either Duy's solution with a sort of \"untracked\" reflog, or the\n> garbage/trashable notion.\n\nThe \"untracked reflog\" is partially functional now [1] if you want to\nhave a look. I'm not going to post the series until post-2.20, but if\nyou do look, I suggest the first patch that lays out the design and a\nplumbing command to manage it. Basically you'll do\n\n git backup-log --id=worktree log \n\nthen pick up the version you like with \"git backup-log cat\" and do\nwhatever you want with it. High level UI is not there and will be a\ntopic of discussion.\n\nThe precious/trashable/garbage notion can be used to suppress making backups.\n\n[1] https://gitlab.com/pclouds/git/commits/backup-log\n-- \nDuy\n"}]}