{"thread":{"id":"60338","subject":"[RFC] Define \"precious\" attribute and support it in `git clean`","startedAt":"2023-10-10T12:47:21Z","lastAt":"2023-10-29T06:44:18Z","messageCount":28,"participants":["Sebastian Thiel","Kristoffer Haugsbakk","Josh Triplett","Junio C Hamano","Richard Kerry","Jeff King","Phillip Wood","Oswald Buddenhagen","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"482976","messageId":"79901E6C-9839-4AB2-9360-9EBCA1AAE549@icloud.com","threadId":"60338","inReplyTo":null,"subject":"[RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Sebastian Thiel","fromEmail":"sebastian.thiel@icloud.com","sentAt":"2023-10-10T12:37:36Z","receivedAt":"2023-10-10T12:47:21Z","isPatch":false,"sender":{"key":"sebastian.thiel@icloud.com","avatar":"https://avatars.githubusercontent.com/u/63622?v=4"},"body":"[Note: I'm collaborating with Josh Triplett (CCed) on the design.]\n\nI'd like to propose adding a new standard gitattribute \"precious\". I've\nincluded proposed documentation at the end of this mail, and I'm happy to write\nthe code. I wanted to get feedback on the concept first.\n\nWhat's a 'precious' file?\n\n\"Precious\" files are files that are specific to a user or local configuration\nand thus not tracked by Git. As such, a user spent time to create or generate\nthem, and to tune them to fit their needs. They are typically also ignored by\n`git` due to `.gitignore` configuration, preventing them to be tracked by\naccident.\n\nThis proposal suggests to make them known to Git using git-attributes so that\n`git clean` can be taught to treat them with care.\n\nExample: A Linux Kernel .config file\n\nUsers can mark the `.config` file as 'precious' using `.gitattributes`:\n\n /.config precious\n\nWhen checking which ignored files `git clean -nx` would remove, we would see\nthe following.\n\n Would remove precious .config\n Would remove scripts/basic/.fixdep.cmd\n Would remove scripts/basic/fixdep\n Would remove scripts/kconfig/.conf.cmd\n\n\nThis highlights precious files by calling them out, but doesn't change the\nbehaviour of existing flags. Instead, the new flag `-p` is added which lets\n`git clean` spare precious files.\n\nThus `git clean -np` would print:\n\n Would remove scripts/basic/.fixdep.cmd\n Would remove scripts/basic/fixdep\n Would remove scripts/kconfig/.conf.cmd\n\nThe precious file is not part of the set of files to be removed anymore.\n\n`git clean -[n|f] -xp` will fail with an error indicating that `-x` and `-p`\nare mutually exclusive. The hope is that people can replace some of their\nusage of `-x` with `-p` to preserve precious files, while continuing to use\n`-x` if they want a completely clean working directory.\n\nAdditional Benefits\n\n`git clean -fdp` can now be used to restore the user's directory to a pristine\npost-clone state while keeping all files and directories the project or user\nidentifies as precious. There is less fear of accidentally deleting files\nwhich are required for local development or otherwise represent a time\ninvestment.\n\nExample: A precious IDE configuration directory.\n\nTo keep IDE configuration, one can also mark entire directories - the following\ncould go into a user-specific gitattributes file denoted by the\n`core.attributesFile` configuration.\n\n /.idea/** precious\n\nWith this attributes file in place, `git clean -ndx` would produce the\nfollowing output...\n\n Would remove .DS_Store\n Would remove precious .idea/\n\n...while `git clean -ndp` would look like this:\n\n Would remove .DS_Store\n\nHere's a patch showing what the documentation could look like. Happy to write\nthe corresponding code.\n\n---\ndiff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt\nindex 5e1a3d5148..5b2eab6573 100644\n--- a/Documentation/git-clean.txt\n+++ b/Documentation/git-clean.txt\n@@ -60,6 +60,10 @@ OPTIONS\n \tUse the given exclude pattern in addition to the standard ignore rules\n \t(see linkgit:gitignore[5]).\n \n+-p::\n+\tRemove ignored files as well (like `-x`), but preserve \"precious\"\n+\tfiles (see linkgit:gitattributes[5]).\n+\n -x::\n \tDon't use the standard ignore rules (see linkgit:gitignore[5]), but\n \tstill use the ignore rules given with `-e` options from the command\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 6deb89a296..f68aadc3c2 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -1248,6 +1248,20 @@ If this attribute is not set or has an invalid value, the value of the\n (See linkgit:git-config[1]).\n \n \n+Preserving precious files\n+~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+`precious`\n+^^^^^^^^^^\n+\n+A file marked as `precious` will be preserved when running linkgit:git-clean[1]\n+with the `-p` option. Use this attribute for files such as a Linux kernel\n+`.config` file, which are not tracked by git because they contain user-specific\n+or build-specific configuration, but which contain valuable information that a\n+user spent time and effort to create.\n+\n+\n+\n USING MACRO ATTRIBUTES\n ----------------------\n \n\nWhat do you think?\n\nThanks for your feedback,\nSebastian"},{"id":"482982","messageId":"98387b86-1732-42bc-9ac5-d64a6617b2db@app.fastmail.com","threadId":"60338","inReplyTo":"79901E6C-9839-4AB2-9360-9EBCA1AAE549@icloud.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Kristoffer Haugsbakk","fromEmail":"code@khaugsbakk.name","sentAt":"2023-10-10T13:38:51Z","receivedAt":"2023-10-10T13:40:30Z","isPatch":false,"sender":{"key":"code@khaugsbakk.name","avatar":"https://avatars.githubusercontent.com/u/2229597?v=4"},"body":"Hi Sebastian\n\nOn Tue, Oct 10, 2023, at 14:37, Sebastian Thiel wrote:\n> This highlights precious files by calling them out, but doesn't change the\n> behaviour of existing flags. Instead, the new flag `-p` is added which lets\n> `git clean` spare precious files.\n\nWhy can't `clean` preserve precious files by default? And then delete them\nas well with something like `--no-keep-precious`? Is there some backwards\ncompatibility concern?\n"},{"id":"482984","messageId":"ZSVbUSRUQlNy0bj-@localhost","threadId":"60338","inReplyTo":"98387b86-1732-42bc-9ac5-d64a6617b2db@app.fastmail.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2023-10-10T14:10:25Z","receivedAt":"2023-10-10T14:10:46Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Tue, Oct 10, 2023 at 03:38:51PM +0200, Kristoffer Haugsbakk wrote:\n> Hi Sebastian\n> \n> On Tue, Oct 10, 2023, at 14:37, Sebastian Thiel wrote:\n> > This highlights precious files by calling them out, but doesn't change the\n> > behaviour of existing flags. Instead, the new flag `-p` is added which lets\n> > `git clean` spare precious files.\n> \n> Why can't `clean` preserve precious files by default? And then delete them\n> as well with something like `--no-keep-precious`? Is there some backwards\n> compatibility concern?\n\nWhile I'd love for it to default to that and require an extra option to\nclean away precious files, I'd expect that that would break people's\nworkflows and finger memory. If someone expects `git clean -x -d -f` to\nclean away everything, including `.config`, and then it leaves some\nfiles in place, that seems likely to cause problems. (Leaving aside that\nit might break scripted workflows.)\n\nIt seems safer to keep the existing behavior for existing options, and\nadd a new option for \"remove everything except precious files\".\n"},{"id":"482990","messageId":"xmqqttqytnqb.fsf@gitster.g","threadId":"60338","inReplyTo":"79901E6C-9839-4AB2-9360-9EBCA1AAE549@icloud.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-10T17:02:20Z","receivedAt":"2023-10-10T17:02:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sebastian Thiel writes:\n\n> I'd like to propose adding a new standard gitattribute \"precious\".\n\n;-).\n\nOver the years, I've seen many times scenarios that would have been\nhelped if we had not just \"tracked? ignored? unignored?\" but also\nthe fourth kind [*]. The word \"ignored\" (or \"excluded\") has always\nmeant \"not tracked, not to be tracked, and expendable\" to Git, and\n\"ignored but unexpendable\" class was missing. I even used the term\n\"precious\" myself in those discussions. At the concept level, I\nsupport the effort 100%, but as always, the devil will be in the\ndetails.\n\nScenarios that people wished for \"precious\" traditionally have been\n\n * You are working on 'master'. You have in your .gitignore or\n .git/info/exclude a line to ignore path A, and have random\n scribbles in a throw-away file there. There is another branch\n 'seen', where they added some tracked contents at path A/B. You\n do \"git checkout seen\" and your file A that is an expendable file,\n because it is listed as ignored in .git/info/exclude, is removed\n to make room for creating A/B.\n\n * Similar situation, but this time, 'seen' branch added a tracked\n contents at path A. Again, \"git checkout seen\" will discard the\n expendable file A and replace it with tracked contents.\n\n * Instead of \"git checkout\", you decide to merge the branch 'seen'\n to the checkout of 'master', where you have an ignored path A.\n Because merging 'seen' would need to bring the tracked contents\n of either A/B (in the first scenario above) or A (in the second\n scenario), your \"expendable\" A will be removed to make room.\n\nIn previous discussions, nobody was disturbed that \"git clean\" was\nunaware of the \"precious\" class, but if we were to have the\n\"precious\" class in addition to \"ignored\" aka \"expendable\", I would\nnot oppose to teach \"git clean\" about it, too.\n\nThere was an early and rough design draft there in\n\nhttps://lore.kernel.org/git/7vipsnar23.fsf@alter.siamese.dyndns.org/\n\nwhich probably is worth a read, too.\n\nEven though I referred to the precious _attribute_ in some of these\ndiscussions, between the attribute mechanism and the ignore\nmechanism, I am actually leaning toward suggesting to extend the\nexclude/ignore mechanism to introduce the \"precious\" class. That\nway, we can avoid possible snafu arising from marking a path in\n.gitignore as ignored, and in .gitattrbutes as precious, and have to\nfigure out how these two settings are to work together.\n\nIn any case, the \"precious\" paths are expected to be small minority\nof what people never want to \"git add\" or \"git commit\", so coming up\nwith a special syntax to be used in .gitignore, even if that special\nsyntax is ugly and cumbersome to type, would be perfectly OK.\n\n\n[Reference]\n\n * https://lore.kernel.org/git/7viptp9jos.fsf@alter.siamese.dyndns.org/\n * https://lore.kernel.org/git/xmqqva534vnb.fsf@gitster-ct.c.googlers.com/\n"},{"id":"482991","messageId":"xmqqo7h6tnib.fsf@gitster.g","threadId":"60338","inReplyTo":"ZSVbUSRUQlNy0bj-@localhost","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-10T17:07:08Z","receivedAt":"2023-10-10T17:07:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Triplett writes:\n\n> While I'd love for it to default to that and require an extra option to\n> clean away precious files, I'd expect that that would break people's\n> workflows and finger memory. If someone expects `git clean -x -d -f` to\n> clean away everything, including `.config`, and then it leaves some\n> files in place, that seems likely to cause problems. (Leaving aside that\n> it might break scripted workflows.)\n\nI thought the point of introducing the new \"precious\" class of\npaths, in addition to the current \"tracked\", \"ignored, untracked,\nand expendable\", \"not ignored and untracked\", is so that people can\ndo \"git clean -x -d -f\" and expect the \".config\" that is marked as\n\"precious\" to stay. Before their Git learned the precious class, if\nthey marked \".config\" as \"ignored, untracked, and expendable\", then\nsuch an invocation of \"clean\" would have removed it, but if they add\nit to the new \"precious\" class, their expectation ought to be that\nprecious ones are not removed, no? Otherwise I am not quite sure\nwhat the point of adding such a new protection is.\n"},{"id":"482997","messageId":"25b25127-aa10-4179-bc02-065fe12d01ef@app.fastmail.com","threadId":"60338","inReplyTo":"ZSVbUSRUQlNy0bj-@localhost","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Kristoffer Haugsbakk","fromEmail":"code@khaugsbakk.name","sentAt":"2023-10-10T19:10:34Z","receivedAt":"2023-10-10T19:11:15Z","isPatch":false,"sender":{"key":"code@khaugsbakk.name","avatar":"https://avatars.githubusercontent.com/u/2229597?v=4"},"body":" Hi Josh\n\nOn Tue, Oct 10, 2023, at 16:10, Josh Triplett wrote:\n> > [snip]\n>\n> While I'd love for it to default to that and require an extra option to\n> clean away precious files, I'd expect that that would break people's\n> workflows and finger memory. If someone expects `git clean -x -d -f` to\n> clean away everything, including `.config`, and then it leaves some\n> files in place, that seems likely to cause problems. (Leaving aside that\n> it might break scripted workflows.)\n>\n> It seems safer to keep the existing behavior for existing options, and\n> add a new option for \"remove everything except precious files\".\n\nWhat's a scenario where it breaks? I'm guessing:\n\n1. Someone clones a project\n2. That project has precious files marked via `.gitattributes`\n3. They later do a `clean`\n4. The precious files are left alone even though they expected them to be\n deleted; they don't check what `clean` did (it deletes everything\n untracked (they expect) so nothing to check)\n5. This hurts them somehow\n\nIt seems that the only files that should be deleted with expediency are\nsecrets. But then why or how would:\n\n1. The project mark such files as precious\n2. The user introduces these files (they are precious hence they were not\n part of the clone)\n3. They are never deleted\n\nThis sounds unlikely to me. And if it was some kind of malignant vector\nthen all would be vulnerable to it (not just legacy scripts/legacy hands).\n\nWhat am I missing?\n\n-- \nKristoffer\n"},{"id":"483041","messageId":"AS8PR02MB73027943EE0A30DD8DAAD4639CCCA@AS8PR02MB7302.eurprd02.prod.outlook.com","threadId":"60338","inReplyTo":"xmqqttqytnqb.fsf@gitster.g","subject":"RE: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Richard Kerry","fromEmail":"richard.kerry@eviden.com","sentAt":"2023-10-11T10:06:25Z","receivedAt":"2023-10-11T10:11:47Z","isPatch":false,"sender":{"key":"richard.kerry@eviden.com","avatar":null},"body":"\n> > I'd like to propose adding a new standard gitattribute \"precious\".\n> \n> ;-).\n\nThe version of CVS that I used to use, CVSNT, was a lot more careful about the user's files than Git is inclined to be.\nIf CVSNT, while doing an Update, came across a non-tracked file that was in the way of something that it wanted to write, then the Update would be aborted showing a list of any files that were \"in the way\". The user could then rename/delete them or redo the Update with a \"force\" parameter to indicate that such items could be overwritten.\nGit has tended to take an approach of \"if it's important it'll be tracked by Git - anything else can be trashed with impunity.\". Over the years people have been caught out by this and lost work. It may well be that in a Linux development world anything other than tracked source files can be summarily deleted, but in a wider world, like Windows, or environments that are not software development, or that need special files lying around, this is not always an entirely reasonable approach.\n\n> Over the years, I've seen many times scenarios that would have been helped\n> if we had not just \"tracked? ignored? unignored?\" but also the fourth kind\n> [*]. The word \"ignored\" (or \"excluded\") has always meant \"not tracked, not\n> to be tracked, and expendable\" to Git, and \"ignored but unexpendable\" class\n> was missing. I even used the term \"precious\" myself in those discussions. At\n> the concept level, I support the effort 100%, but as always, the devil will be in\n> the details.\n> \n> Scenarios that people wished for \"precious\" traditionally have been\n> \n> * You are working on 'master'. You have in your .gitignore or\n> .git/info/exclude a line to ignore path A, and have random\n> scribbles in a throw-away file there. There is another branch\n> 'seen', where they added some tracked contents at path A/B. You\n> do \"git checkout seen\" and your file A that is an expendable file,\n> because it is listed as ignored in .git/info/exclude, is removed\n> to make room for creating A/B.\n\nSo checkout aborts, saying \"A is in the way\".\n\n> * Similar situation, but this time, 'seen' branch added a tracked\n> contents at path A. Again, \"git checkout seen\" will discard the\n> expendable file A and replace it with tracked contents.\n\nSo checkout aborts, saying \"A is in the way\".\n\n> * Instead of \"git checkout\", you decide to merge the branch 'seen'\n> to the checkout of 'master', where you have an ignored path A.\n> Because merging 'seen' would need to bring the tracked contents\n> of either A/B (in the first scenario above) or A (in the second\n> scenario), your \"expendable\" A will be removed to make room.\n\nSo merge aborts, saying \"A is in the way\". It is entirely conventional to have merge conflicts that the user needs to resolve. This is just another kind of conflict.\n\n> In previous discussions, nobody was disturbed that \"git clean\" was unaware\n> of the \"precious\" class, but if we were to have the \"precious\" class in addition\n> to \"ignored\" aka \"expendable\", I would not oppose to teach \"git clean\" about\n> it, too.\n\nIndeed, if something is explicitly precious then nothing should summarily delete it.\n\nI know this goes against some stated design decisions of early Git, but in the CVSNT world *all* files were considered precious and would always cause an update to be aborted if there were any inclination to replace them.\n\nAn option might be to state, in config, whether a project, or everything, should be managed on the basis of \"all untracked files are precious\" or \"files may be explicitly marked precious\", or, as now, \"nothing is precious\".\n\nRegards,\nRichard.\n\nPS. I think I've caught all places where my fingers typed \"previous\" when my brain meant \"precious\" - apologies if I've missed any.\n\n\n"},{"id":"483092","messageId":"b27782ce-8174-4d72-acc2-2d8f0e6e6ef1@app.fastmail.com","threadId":"60338","inReplyTo":"79901E6C-9839-4AB2-9360-9EBCA1AAE549@icloud.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Kristoffer Haugsbakk","fromEmail":"code@khaugsbakk.name","sentAt":"2023-10-11T21:41:26Z","receivedAt":"2023-10-11T21:42:51Z","isPatch":false,"sender":{"key":"code@khaugsbakk.name","avatar":"https://avatars.githubusercontent.com/u/2229597?v=4"},"body":"On Tue, Oct 10, 2023, at 14:37, Sebastian Thiel wrote:\n> [Note: I'm collaborating with Josh Triplett (CCed) on the design.]\n>\n> I'd like to propose adding a new standard gitattribute \"precious\". I've\n> included proposed documentation at the end of this mail, and I'm happy to write\n> the code. I wanted to get feedback on the concept first.\n>\n> What's a 'precious' file?\n>\n> \"Precious\" files are files that are specific to a user or local configuration\n> and thus not tracked by Git. As such, a user spent time to create or generate\n> them, and to tune them to fit their needs. They are typically also ignored by\n> `git` due to `.gitignore` configuration, preventing them to be tracked by\n> accident.\n\nHow do people deal with these precious files today?\n\nYou could track these precious files somewhere else. Maybe a Git directory\nwhich is a sibling of `.git`.\n\n .git-local\n\nFiles that are useless to the project but important to the individual.\n\nIt would get its own “excludes” file.\n\n .gitignore-local\n\nNow the normal repository (`.git`) needs to ignore these things.\n\n # Cast a wide net: could want other siblings\n # Use another pattern if you have something like `.git-blame-ignore-revs`\n printf '.git-*\\n' >> .git/info/exclude\n printf '.gitignore-*\\n' >> .git/info/exclude\n\nYou need to pass in two arguments to `git` every time you want to use\n`.git-local`.\n\n alias gitl='git --git-dir=.git-local -c core.excludesFile=.gitignore-local'\n\nGit Local should ignore everything by default. You should check with\n`.gitignore` to make sure that it ignores the files that Git Local does\n_not_ ignore.\n\n printf '*\\n' >> .gitignore-local\n printf '!.gitignore-local\\n' >> .gitignore-local\n printf '!.idea/**\\n' >> .gitignore-local\n\nNow you can backup your local files.\n\n gitl add .gitignore-local\n gitl add .idea\n gitl commit -m'Update local'\n\nBut you can also version control them by providing real (intentional)\nmessages.\n\n(Maybe `.idea/` is just an XML soup and thus hard to make a VCS narrative\naround; I don't know yet.)\n\n`git clean` won't help you. But an alias can.\n\nOr if writing a shell-oneliner alias is too hard for you, I mean me.\n\n #!/bin/sh\n # git-klean\n git --git-dir=.git-local -c core.excludesFile=.gitignore-local add --all\n git --git-dir=.git-local -c core.excludesFile=.gitignore-local commit -mUpdate\n git clean -e .gitignore-local -e .git-local \"$@\"\n\nThe two `-e` switches protect the Git Local things from being wiped by\n`-xd` (tested with `--dry-run`).\n\n§ Sibling repositories\n\nAt first I thought that `git clean` could use a `pre-clean` hook. But\nthat's not very satisfying.\n\nMaybe Git could be told about its siblings via a multi-valued\nconfiguration variable.\n\n sibling=local\n\nThen it expects there to exist `.git-` repository next to\nit (`.git/../.git-`).\n\nThen the rule becomes:\n\nOnly do destructive operations if all of the working trees[1] of the\nsibling repositories are clean (cannot override with `--force`).\n\nAdditionally one could say that directories that are Git repositories\nshould be ignored by `git clean`, always.[2] Or siblings which\nmatch the glob:\n\n .git-*\n\n... Or if you want something longer:\n\n .git-sibling-*\n\n(And also ignore `.gitignore-*` and maybe more things)\n\nThen the regular Git repository might still blow away your precious\nfiles. But they will be backed up by the siblings.\n\nOr you put these other Git repositories outside of `.git/..`. Sidestepping\nthe issue at the cost of some path confusion (for yourself). Maybe at:\n\n /home/user/git-siblings/repository1/local\n\n† 1: Worktrees not considered.\n† 2: And what would that break? People who make Git repositories in their\n working trees and then delete them? (Well they can still use `rm -r`.)\n\n⧫ ⧫\n\n§ Worktrees\n\nBut seriously: worktrees probably makes this not work.\n\n-- \nKristoffer\n"},{"id":"483105","messageId":"20231011224002.GD518221@coredump.intra.peff.net","threadId":"60338","inReplyTo":"AS8PR02MB73027943EE0A30DD8DAAD4639CCCA@AS8PR02MB7302.eurprd02.prod.outlook.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-10-11T22:40:02Z","receivedAt":"2023-10-11T22:40:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 11, 2023 at 10:06:25AM +0000, Richard Kerry wrote:\n\n> The version of CVS that I used to use, CVSNT, was a lot more careful\n> about the user's files than Git is inclined to be.\n> If CVSNT, while doing an Update, came across a non-tracked file that\n> was in the way of something that it wanted to write, then the Update\n> would be aborted showing a list of any files that were \"in the way\".\n> The user could then rename/delete them or redo the Update with a\n> \"force\" parameter to indicate that such items could be overwritten.\n> Git has tended to take an approach of \"if it's important it'll be\n> tracked by Git - anything else can be trashed with impunity.\". Over\n> the years people have been caught out by this and lost work. It may\n> well be that in a Linux development world anything other than tracked\n> source files can be summarily deleted, but in a wider world, like\n> Windows, or environments that are not software development, or that\n> need special files lying around, this is not always an entirely\n> reasonable approach.\n\nI'm not sure if you are just skipping the details of \".gitignore\" here,\nbut to be clear, blowing away untracked files is _not_ Git's default\nbehavior.\n\nFor example:\n\n [sample repo with established history]\n $ git init\n $ echo content >base\n $ git add base\n $ git commit -m base\n\n [one branch touches some-file]\n $ git checkout -b side-branch\n $ echo whatever >some-file\n $ git add some-file\n $ git commit -m 'add some-file'\n\n [but back on master/main, it is untracked]\n $ git checkout main\n $ echo precious >some-file\n\n [an operation that tries to overwrite the untracked file will fail]\n $ git checkout side-branch\n $ git checkout side-branch\n error: The following untracked working tree files would be overwritten by checkout:\n\tsome-file\n Please move or remove them before you switch branches.\n Aborting\n\n [providing --force will obliterate it]\n $ git checkout --force side-branch\n Switched to branch 'side-branch'\n\nThe issue that people sometimes find with Git is when the user has\nexplicitly listed a file in \".gitignore\", Git takes that to mean it\nshould never be tracked _and_ it is not precious. But people sometimes\nwant a way to say \"this should never be tracked, but keep treating it as\nprecious in the usual way\".\n\nFrom the description above it might sound like Git's current behavior is\nconflating two orthogonal things, but if you switched the default\nbehavior of .gitignore'd files to treat them as precious, you will find\nlots of cases that are annoying. E.g., if a file is generated by some\nparts of history and tracked in others, you'd have to use --force to\nmove between them to overwrite the generated version.\n\n-Peff\n"},{"id":"483120","messageId":"xmqqmswoivg9.fsf@gitster.g","threadId":"60338","inReplyTo":"AS8PR02MB73027943EE0A30DD8DAAD4639CCCA@AS8PR02MB7302.eurprd02.prod.outlook.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-11T23:35:34Z","receivedAt":"2023-10-11T23:35:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Richard Kerry writes:\n\n> An option might be to state, in config, whether a project, or\n> everything, should be managed on the basis of \"all untracked files\n> are precious\" or \"files may be explicitly marked precious\", or, as\n> now, \"nothing is precious\".\n\nI do not think there is any need to have a separate \"all or none\"\noption. We do not have to make things more complicated than\nnecessary.\n\nIf all untracked files are precious, a user should be able to say so\nwith an entry that matches all paths \"*\" to mark them precious, and\nnothing more needs to be done. By default nothing is ignored and\nnothing is precious, until you start marking paths with .gitignore\nentries.\n"},{"id":"483140","messageId":"ZSeynS5Vz_IQomp4@localhost","threadId":"60338","inReplyTo":"xmqqo7h6tnib.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2023-10-12T08:47:25Z","receivedAt":"2023-10-12T08:47:46Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Tue, Oct 10, 2023 at 10:07:08AM -0700, Junio C Hamano wrote:\n> Josh Triplett writes:\n> \n> > While I'd love for it to default to that and require an extra option to\n> > clean away precious files, I'd expect that that would break people's\n> > workflows and finger memory. If someone expects `git clean -x -d -f` to\n> > clean away everything, including `.config`, and then it leaves some\n> > files in place, that seems likely to cause problems. (Leaving aside that\n> > it might break scripted workflows.)\n> \n> I thought the point of introducing the new \"precious\" class of\n> paths, in addition to the current \"tracked\", \"ignored, untracked,\n> and expendable\", \"not ignored and untracked\", is so that people can\n> do \"git clean -x -d -f\" and expect the \".config\" that is marked as\n> \"precious\" to stay. Before their Git learned the precious class, if\n> they marked \".config\" as \"ignored, untracked, and expendable\", then\n> such an invocation of \"clean\" would have removed it, but if they add\n> it to the new \"precious\" class, their expectation ought to be that\n> precious ones are not removed, no? Otherwise I am not quite sure\n> what the point of adding such a new protection is.\n\nI'd expect a lot of projects to move things *from* the current \"ignored\"\nstate to \"precious\", once \"precious\" exists. Linux `.config`, for\ninstance.\n\nThat said, I do agree that the ideal behavior is for clean to preserve\nprecious files by default, and require an extra option to remove\nprecious files. If you think that doesn't have backwards-compatibility\nconsiderations, then it certainly seems much easier to jump directly to\nthat behavior.\n\n- Josh Triplett\n"},{"id":"483141","messageId":"ZSe2rvIWcNrrp-9R@localhost","threadId":"60338","inReplyTo":"25b25127-aa10-4179-bc02-065fe12d01ef@app.fastmail.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2023-10-12T09:04:46Z","receivedAt":"2023-10-12T09:05:00Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Tue, Oct 10, 2023 at 09:10:34PM +0200, Kristoffer Haugsbakk wrote:\n> Hi Josh\n> \n> On Tue, Oct 10, 2023, at 16:10, Josh Triplett wrote:\n> > > [snip]\n> >\n> > While I'd love for it to default to that and require an extra option to\n> > clean away precious files, I'd expect that that would break people's\n> > workflows and finger memory. If someone expects `git clean -x -d -f` to\n> > clean away everything, including `.config`, and then it leaves some\n> > files in place, that seems likely to cause problems. (Leaving aside that\n> > it might break scripted workflows.)\n> >\n> > It seems safer to keep the existing behavior for existing options, and\n> > add a new option for \"remove everything except precious files\".\n> \n> What's a scenario where it breaks? I'm guessing:\n> \n> 1. Someone clones a project\n> 2. That project has precious files marked via `.gitattributes`\n> 3. They later do a `clean`\n> 4. The precious files are left alone even though they expected them to be\n> deleted; they don't check what `clean` did (it deletes everything\n> untracked (they expect) so nothing to check)\n> 5. This hurts them somehow\n\nThe scenario I had in mind was:\n\n- Project has ignored files; git doesn't have a concept of \"precious\"\n- Users expect that `git clean -x -d -f` deletes everything that isn't\n part of the latest commit.\n- Git introduces the concept of \"precious\"\n- Project adopts \"precious\" and marks some of its ignored files as\n \"precious\" instead\n- Users' finger-macros around `git clean` stop cleaning up files they\n expected to be cleaned.\n\nThat said, given Junio's response I'm no longer concerned about this\nscenario.\n"},{"id":"483143","messageId":"0E44CB2C-57F2-4075-95BE-60FBFDD3CEE2@icloud.com","threadId":"60338","inReplyTo":"xmqqttqytnqb.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Sebastian Thiel","fromEmail":"sebastian.thiel@icloud.com","sentAt":"2023-10-12T10:55:19Z","receivedAt":"2023-10-12T10:55:39Z","isPatch":false,"sender":{"key":"sebastian.thiel@icloud.com","avatar":"https://avatars.githubusercontent.com/u/63622?v=4"},"body":"I liked the idea too see precious files as sub-class of ignored files, and\ninvestigated possibilities on how to achieve that while keeping the overall\neffort low and remove any potential for backwards-incompatibility as well.\n\nCurrently, `.gitignore` files only contain one pattern per line, which\noptionally may be prefixed with `!` to negate it. This can be escaped with `\\!`\n- and that's it.\n\nParsing patterns that way makes for simple parsing without a need for\nquoting.\n\n### What about a `$` syntax in `.gitignore` files?\n\nI looked into adding a new prefix, `$` to indicate the following path is\nprecious or… valuable. It can be escaped with `\\$` just like `\\!`. \n\nDoing so has the advantage that older `git` versions simply take the\ndeclaration as literal and would now exclude `$.config`, for example, whereas\nnewer `git` versions will consider them precious.\n\nThere is some potential for accidentally excluding files that previously\nwere untracked with older versions of git, but I'd think chances are low.\n\n#### Example: Linux kernel\n\n`.config` is ignored via `.gitignore: \n\n .*\n\n*Unfortunately*, users can't just add a local `.git/info/exclude` file with\n`$.config` in it and expect `.config` to be considered precious as the pattern\nsearch order will search this last as it's part of the exclude-globals. The\nsame is true for per-user git-ignore files. This means that any git would\nhave the `.*` pattern match before the `$.config` pattern, and stop right there\nconcluding that it's expendable, instead of precious. This is how users can\nexpect `.gitignore` files to work, and this is how `!negations` work as well -\nthe negation has to come after the actual exclusion to be effective.\n\nThus, to make this work, projects that ship the `.gitignore` files would *have\nto add patterns* that make certain files precious.\n\nAlternatively, users have to specify gitignore-overrides on the command-line,\nbut not all commands (if any except for `git check-ignore`) support that.\n\nIn the case of `git clean` one can already pass `--exclude=pattern`, but if\nthat's needed one doesn't need syntax for precious files in the first place.\n\n**This makes the $precious syntax useful only for projects who chose to opt in,\nbut makes overrides for users near impossible**.\n\nSuch opted-in projects would produce `.gitignore` files like these:\n\n .*\n $.config\n\nNote that due to the way ignore patterns are searched, the following would\nconsider `.config` trackable, not precious:\n\n .*\n $.config\n !.config\n\nIt's up the maintainer of the repository to configure their .gitignore files\ncorrectly, so nothing new either.\n\n#### Benefits\n\n* simple implementation, fast\n* backwards compatible\n\n#### Disadvantages\n\n* cannot easily be overridden by the user as part of their local settings.\n* needs repository-buy-in to be useful\n* $file could clash with the file '$file' and cause older git to ignore it\n\n### What about a `precious` attribute?\n\nThe search of `.gitattributes` works differently which makes it possible for\nusers to set attributes on any file or folder easily using their local files.\nUsing attributes has the added benefit of being extensible as one can start out\nwith:\n\n```gitattributes\n.config precious\n```\n\nand optionally continue with…\n\n```gitattributes\n.config precious=input\nkernel precious=output\n```\n\n…to further classify kinds of precious files, probably for their personal use.\nPlease note that currently pathspecs can't be used to filter by attribute\nfor files that are igonred and untracked or I couldn't figure out how.\nThat even makes sense as it wasn't considered a use-case yet.\n\n\n#### Benefits\n\n* backwards compatible\n* easily extendable with 'tags' or sub-classes of precious files using the\n assignment syntax.\n* overridable with user's local files\n\n#### Disadvantages\n\n* any 'exclude' query now also needs a .gitattribute query to support precious\n files (and it's not easy to optimize unless there is a flag to turn precious\n file support on or off)\n* `precious` might be in use by some repos which now gains a possibly different\n meaning in `git` as well.\n\n### Conclusion\n\nWeighing advantages and disadvantages of both approaches makes me prefer the\n`.gitignore` extension. The `.gitattributes` version of it *could* also be\nimplemented on top of it at a later date. However, it should be gated behind a\nconfiguration flag so users who need it as they want local overrides\ncan opt-in. Then they also pay for the feature which for most repositories \nwon't be an issue in the first place.\n\nAll this seems a bit too good to be true, and I hope you can show where\nit wouldn't work or which dangers or possible issues haven't been\nmentioned yet.\n\n"},{"id":"483157","messageId":"xmqqttqvg4lw.fsf@gitster.g","threadId":"60338","inReplyTo":"0E44CB2C-57F2-4075-95BE-60FBFDD3CEE2@icloud.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-12T16:58:19Z","receivedAt":"2023-10-12T16:58:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sebastian Thiel writes:\n\n> ### What about a `$` syntax in `.gitignore` files?\n>\n> I looked into adding a new prefix, `$` to indicate the following path is\n> precious or… valuable. It can be escaped with `\\$` just like `\\!`. \n\nI have been regretting that I did not make the quoting syntax not\nobviously extensible in f87f9497 (git-ls-files: --exclude mechanism\nupdates., 2005-07-24), which technically was a breaking change (as a\nrelative pathname that began with '!' were not special, but after\nthe change, it became necessary to '\\'-quote it). A relative\npathname that begins with '$' would be now broken the same way, but\nhopefully the fallout would be minor. I presume you picked '$'\nexactly because of this reason?\n\nI do not think it will be the end of the world if we don't do so,\nbut it would be really really nice if we at least explored a way (or\ntwo) to make a big enough hole in the syntax to not just add\n\"precious\", but leave room to later add other traits, without having\nto worry about breaking the backward compatibility again. A\nsimplest and suboptimal way may be to declare that a path that\nbegins with '$' now needs '\\'-quoting (just like your proposal),\nreserve '$$' as the precious prefix, and '$' followed by any other\nbyte reserved for future use, but there may be better ideas.\n\n> *Unfortunately*, users can't just add a local `.git/info/exclude` file with\n> `$.config` in it and expect `.config` to be considered precious as the pattern\n> search order will search this last as it's part of the exclude-globals.\n\nThat it nothing new and is the same for ignored files. The lower\nprecedence files do not override higher precedence files.\n\n> Thus, to make this work, projects that ship the `.gitignore` files would *have\n> to add patterns* that make certain files precious.\n\nNot really. They do not have to do anything if they are content\nwith the current Git ecosystem. And users who have precious stuff\ncan mark them in the.git/info/excludes no? The only case that is\nproblematic is when the project says 'foo' is ignored and expendable\nbut the user thinks otherwise. So to make this work, projects that\nship the \".gitignore\" files have to avoid adding patterns to ignore\nthings that it may reasonably be expected for its users to mark\nprecious.\n\n> Such opted-in projects would produce `.gitignore` files like these:\n>\n> .*\n> $.config\n\nI would understand if you ignored \"*~\" or \"*.o\", but why ignore \".*\"?\n\nTHanks.\n"},{"id":"483193","messageId":"9C4A2AFD-AAA2-4ABA-8A8B-2133FD870366@icloud.com","threadId":"60338","inReplyTo":"xmqqttqvg4lw.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Sebastian Thiel","fromEmail":"sebastian.thiel@icloud.com","sentAt":"2023-10-13T09:09:27Z","receivedAt":"2023-10-13T09:09:34Z","isPatch":false,"sender":{"key":"sebastian.thiel@icloud.com","avatar":"https://avatars.githubusercontent.com/u/63622?v=4"},"body":"On 12 Oct 2023, at 18:58, Junio C Hamano wrote:\n\n> I presume you picked '$' > exactly because of this reason?\n\nYes, and because I thought '$' seems a great fit to represent value.\n\n> I do not think it will be the end of the world if we don't do so,\n> but it would be really really nice if we at least explored a way (or\n> two) to make a big enough hole in the syntax to not just add\n> \"precious\", but leave room to later add other traits, without having\n> to worry about breaking the backward compatibility again. A\n> simplest and suboptimal way may be to declare that a path that\n> begins with '$' now needs '\\'-quoting (just like your proposal),\n> reserve '$$' as the precious prefix, and '$' followed by any other\n> byte reserved for future use, but there may be better ideas.\n\nEven though I'd love to go with the unextensible option assuming it would last\nanother 15 years, I can see the appeal of making it extensible from the start.\n\nIn a world where '$' is a prefix, I'd also think that it's now possible to\nspecify exclusion using '$!path' for completeness, if '$$path' marks 'path'\nprecious.\n\nBut if there is now a prefix, I feel that it might as well be chosen so that it\nis easier to remember and/or less likely to cause conflicts. I think it must\nhave been that reason for pathspecs to choose ':' as their prefix, and it seems\nto be an equally good choice here.\n\nThis would give us the following, taking the Linux kernel as example:\n\n .*\n !this-file-is-hidden-and-tracked\n :!new-syntax-for-negation-for-completeness\n \\!an-ignored-file-with-leading-!\n \\:an-ignored-file-with-leading-:-which-is-technically-breaking\n :$.config\n :x-invalid-as-:-needs-either-!-or-$-to-follow-it\n\nNow ':$path' would make any path precious, which is `:$.config` in the example\nabove.\n\nHow does that 'feel'? Is the similarity to pathspecs without being pathspecs\nan anti-feature maybe?\n\n>> Thus, to make this work, projects that ship the `.gitignore` files would *have\n>> to add patterns* that make certain files precious.\n>\n> Not really. They do not have to do anything if they are content\n> with the current Git ecosystem. And users who have precious stuff\n> can mark them in the.git/info/excludes no?\n\nYes, but only if they control all the ignore patterns in their global files. If\nthe repository decides to exclude a file they deem precious, now it won't be\nprecious anymore as their ':$make-this-precious' pattern is seen sequentially\nafter the pattern in the repository.\n\nFor instance, tooling-specific ignores are typically fully controlled by the\nuser, like '/.idea/', which could now easily be made precious with ':$/idea/'.\n\nBut as the Linux kernel repository ships with a '.gitignore' file that includes\nthe '.*' pattern, users won't be able to 'get ahead' of that pattern with their\n':$.config' specification.\n\n> The only case that is\n> problematic is when the project says 'foo' is ignored and expendable\n> but the user thinks otherwise. So to make this work, projects that\n> ship the \".gitignore\" files have to avoid adding patterns to ignore\n> things that it may reasonably be expected for its users to mark\n> precious.\n\nYes, I think my paragraph above is exactly that but with examples to practice\nthe new syntax-proposal.\n\n>\n>> Such opted-in projects would produce `.gitignore` files like these:\n>>\n>> .*\n>> $.config\n>\n> I would understand if you ignored \"*~\" or \"*.o\", but why ignore \".*\"?\n\nI don't have an answer, the example is from the Linux Kernel repository was\nadded in 1e65174a33784 [1].\n\nI am definitely getting excited about the progress the syntax is making :),\nthanks for proposing it!\n\n[ Reference ]\n\n1. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1e65174a33784\n\n"},{"id":"483194","messageId":"0deee2bc-1775-4459-906d-1d44b3103499@gmail.com","threadId":"60338","inReplyTo":"xmqqttqvg4lw.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-13T10:06:55Z","receivedAt":"2023-10-13T10:07:01Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 12/10/2023 17:58, Junio C Hamano wrote:\n> Sebastian Thiel writes:\n\nSebastian - thanks for raising this again, it would be really good to \nget a solution for handling \"ignored but not expendable\" files\n\n> I have been regretting that I did not make the quoting syntax not\n> obviously extensible in f87f9497 (git-ls-files: --exclude mechanism\n> updates., 2005-07-24), which technically was a breaking change (as a\n> relative pathname that began with '!' were not special, but after\n> the change, it became necessary to '\\'-quote it). A relative\n> pathname that begins with '$' would be now broken the same way, but\n> hopefully the fallout would be minor. I presume you picked '$'\n> exactly because of this reason?\n> \n> I do not think it will be the end of the world if we don't do so,\n> but it would be really really nice if we at least explored a way (or\n> two) to make a big enough hole in the syntax to not just add\n> \"precious\", but leave room to later add other traits, without having\n> to worry about breaking the backward compatibility again. A\n> simplest and suboptimal way may be to declare that a path that\n> begins with '$' now needs '\\'-quoting (just like your proposal),\n> reserve '$$' as the precious prefix, and '$' followed by any other\n> byte reserved for future use, but there may be better ideas.\n\nOne thought I had was that we could abuse the comment syntax to annotate \npaths something like\n\n#(keep)\n/my-precious-file\n\nwould prevent /my-precious-file from being deleted by git clean (and \nhopefully unpack-trees()[1]). It means that older versions of git would \ntreat the file as ignored. If we ever want more than one annotation per \npath we could separate them with commas\n\n#(keep,something-else)\n/my-file\n\nStrictly speaking it is a backward incompatible change but I doubt there \nare many people using comments like that. I also wondered about some \nkind of suffix on the file\n\n/my-precious-file #(keep)\n\nbut that means that older versions of git would not ignore the file.\n\nBest Wishes\n\nPhillip\n\n[1] Of the cases listed in [2] it is \"git checkout\" and friends \noverwriting ignored files that I worry about more. At least \"git clean\" \ndefaults to a dry-run and has in interactive mode to select what gets \ndeleted.\n\n[2] https://lore.kernel.org/git/xmqqttqytnqb.fsf@gitster.g\n\n"},{"id":"483195","messageId":"ZSkpOc/dcGcrFQNU@ugly","threadId":"60338","inReplyTo":"xmqqttqvg4lw.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-13T11:25:45Z","receivedAt":"2023-10-13T11:25:52Z","isPatch":false,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Thu, Oct 12, 2023 at 09:58:19AM -0700, Junio C Hamano wrote:\n>I do not think it will be the end of the world if we don't do so,\n>but it would be really really nice if we at least explored a way (or\n>two) to make a big enough hole in the syntax to not just add\n>\"precious\", but leave room to later add other traits, without having\n>to worry about breaking the backward compatibility again.\n>\nthat would invariably make the syntax more verbose, for dubious gain.\n\nthat the extension we're deliberating now (again) was coming (in some \nform) was clear for quite a while, while i'm not aware of anything else \nthat would semantically fit gitignore (*). \"other traits\" sounds awfully \nlike scope creep, and would most likely fit gitattributes better.\n\nanyway, such a hypothetical \"breaking\" change wouldn't have much impact, \nbecause versioned files aren't affected by gitignore. and for the \nmisclassification to be actually harmful, the user would have to be \nunable to notice or correct it.\n\n(*) this got me thinking about things that would fit, and i came up with \na modification of the proposal: one might want to specify just *how* \nprecious a file is (which i guess would translate to how many times the \nextra override option would have to be passed to git-clean). (**)\n\ni guess a suitable syntax for that would be\n\n 2>.config\n\nnote that even though using the dollar sign to denote \"precious\" is kind \nof intuitive, i'm not using it for two reasons: a) it's not \"crazy\" \nenough to use it at not quite the beginning of a file name (note that \ntraditionally it isn't even special on windows), and b) the visual \nseparation of the prefix isn't as good as with the \"arrow-like\" \ncharacter.\n\n(**) actually, one would probably want proper type tagging (e.g., config \nfiles vs. autotools-generated files (which do not belong into a repo, \nbut do into a tar-ball)). that really does sound a lot like \ngitattributes, only that the files aren't versioned.\n\nregards\n"},{"id":"483206","messageId":"xmqqfs2e3292.fsf@gitster.g","threadId":"60338","inReplyTo":"9C4A2AFD-AAA2-4ABA-8A8B-2133FD870366@icloud.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-13T16:39:53Z","receivedAt":"2023-10-13T16:39:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sebastian Thiel writes:\n\n> But if there is now a prefix, I feel that it might as well be chosen so that it\n> is easier to remember and/or less likely to cause conflicts.\n\nAnother criteria is that it is not very often used in real\npathnames, of course, and '!' and '$' are good ones.\n\nCome to think of it, we might be able to retrofit '!' without too\nmuch damage. Something like \"!unignored\" is now a deprecated but\nstill supported way to say \"!!unignored\", \"!*precious\" is new, and\n\"\\!anything\" is a pathname that begins with '!'.\n\n> Yes, I think my paragraph above is exactly that but with examples to practice\n> the new syntax-proposal.\n\nOK.\n"},{"id":"483240","messageId":"ZSouSI_zPusOefsv@localhost","threadId":"60338","inReplyTo":"xmqqttqytnqb.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2023-10-14T05:59:36Z","receivedAt":"2023-10-14T06:00:05Z","isPatch":false,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Tue, Oct 10, 2023 at 10:02:20AM -0700, Junio C Hamano wrote:\n> Sebastian Thiel writes:\n> \n> > I'd like to propose adding a new standard gitattribute \"precious\".\n> \n> ;-).\n> \n> Over the years, I've seen many times scenarios that would have been\n> helped if we had not just \"tracked? ignored? unignored?\" but also\n> the fourth kind [*]. The word \"ignored\" (or \"excluded\") has always\n> meant \"not tracked, not to be tracked, and expendable\" to Git, and\n> \"ignored but unexpendable\" class was missing. I even used the term\n> \"precious\" myself in those discussions. At the concept level, I\n> support the effort 100%, but as always, the devil will be in the\n> details.\n\n\"I've already wanted this for years\" is, honestly, the best response we\ncould *possibly* have hoped for.\n\n> Scenarios that people wished for \"precious\" traditionally have been\n> \n> * You are working on 'master'. You have in your .gitignore or\n> .git/info/exclude a line to ignore path A, and have random\n> scribbles in a throw-away file there. There is another branch\n> 'seen', where they added some tracked contents at path A/B. You\n> do \"git checkout seen\" and your file A that is an expendable file,\n> because it is listed as ignored in .git/info/exclude, is removed\n> to make room for creating A/B.\n\nOuch, I hadn't even thought about the issue of branch-switching\noverwriting a file like that, but that's another great reason to have\n\"precious\". (I've been thinking about \"precious\" as primarily to protect\nfiles like `.config`, where they'd be unlikely to be checked in on any\nbranch because they have an established purpose in the project. Though,\nof course, people *do* sometimes check in `.config` files in\nspecial-purpose branches that aren't meant for upstreaming.)\n\n> * Similar situation, but this time, 'seen' branch added a tracked\n> contents at path A. Again, \"git checkout seen\" will discard the\n> expendable file A and replace it with tracked contents.\n> \n> * Instead of \"git checkout\", you decide to merge the branch 'seen'\n> to the checkout of 'master', where you have an ignored path A.\n> Because merging 'seen' would need to bring the tracked contents\n> of either A/B (in the first scenario above) or A (in the second\n> scenario), your \"expendable\" A will be removed to make room.\n\n+1\n\n> In previous discussions, nobody was disturbed that \"git clean\" was\n> unaware of the \"precious\" class, but if we were to have the\n> \"precious\" class in addition to \"ignored\" aka \"expendable\", I would\n> not oppose to teach \"git clean\" about it, too.\n> \n> There was an early and rough design draft there in\n> \n> https://lore.kernel.org/git/7vipsnar23.fsf@alter.siamese.dyndns.org/\n> \n> which probably is worth a read, too.\n> \n> Even though I referred to the precious _attribute_ in some of these\n> discussions, between the attribute mechanism and the ignore\n> mechanism, I am actually leaning toward suggesting to extend the\n> exclude/ignore mechanism to introduce the \"precious\" class. That\n> way, we can avoid possible snafu arising from marking a path in\n> .gitignore as ignored, and in .gitattrbutes as precious, and have to\n> figure out how these two settings are to work together.\n\nSounds reasonable.\n\n> In any case, the \"precious\" paths are expected to be small minority\n> of what people never want to \"git add\" or \"git commit\", so coming up\n> with a special syntax to be used in .gitignore, even if that special\n> syntax is ugly and cumbersome to type, would be perfectly OK.\n\n[Following up both to this and to Sebastian's response.]\n\nOne potentially important question: should the behavior of old git be to\ntreat precious files as ignored, or as not-ignored? If the syntax were\nsomething like\n\n$.config\n\nthen old git would treat the file as not-ignored. If the syntax were\nsomething like\n\n$precious\n.config\n\nthen old git would treat the file as ignored.\n\nSeems like it would be obtrusive if `git status` in old git showed the\nfile, and `git add .` in old git added it.\n"},{"id":"483241","messageId":"9815705C-EF59-473F-A119-DE84C0E16A89@icloud.com","threadId":"60338","inReplyTo":"xmqqfs2e3292.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Sebastian Thiel","fromEmail":"sebastian.thiel@icloud.com","sentAt":"2023-10-14T07:30:53Z","receivedAt":"2023-10-14T07:30:59Z","isPatch":false,"sender":{"key":"sebastian.thiel@icloud.com","avatar":"https://avatars.githubusercontent.com/u/63622?v=4"},"body":"On 13 Oct 2023, at 18:39, Junio C Hamano wrote:\n\n> Come to think of it, we might be able to retrofit '!' without too\n> much damage. Something like \"!unignored\" is now a deprecated but\n> still supported way to say \"!!unignored\", \"!*precious\" is new, and\n> \"\\!anything\" is a pathname that begins with '!'.\n\nI don't know anything about statistics, and I don't which of the proposed\nsyntax thus far has the lowest probability of accidental breakage, possibly\nin combination with the best possible usability.\n\nHowever, I do like even more the idea to retro-fit `!` instead of having an\nentirely new prefix, it seems more intuitive to me.\n\nAn apparent disadvantage would be that using `!` prefix with\nbackwards-compatibility will make any additional future modifier more\nbreaking. For instance `!*` is potentially ignoring an additional file\nin old git, and another `!-` modifier is having the same effect.\n\nChances for this are probably low though, but if in doubt it would be possible\nto check certain patterns against all files of the top-3.5TB of\nGitHub repositories.\n\nUsing `!*` to signal precious files also seems like a less likely\npath prefix than `!$` would be, but then again, it's just a guess\nwhich most definitely doesn't have much bearing.\n\nI personally also like this more than using special comments as 'modifier',\neven though doing so would probably have the lowest probability for\naccidentally ignoring files in old git.\n\nMaybe it's time to choose one of the options with the possibility to validate\nit for accidental exclusion of files against the top 3.5TB of\nGitHub repositories to be more sure it?\n\n"},{"id":"483248","messageId":"xmqq5y39xjzx.fsf@gitster.g","threadId":"60338","inReplyTo":"0deee2bc-1775-4459-906d-1d44b3103499@gmail.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-14T16:10:42Z","receivedAt":"2023-10-14T16:10:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> One thought I had was that we could abuse the comment syntax to\n> annotate paths something like\n>\n> #(keep)\n> /my-precious-file\n>\n> would prevent /my-precious-file from being deleted by git clean (and\n> hopefully unpack-trees()[1]). It means that older versions of git\n> would treat the file as ignored. If we ever want more than one\n> annotation per path we could separate them with commas\n>\n> #(keep,something-else)\n> /my-file\n>\n> Strictly speaking it is a backward incompatible change but I doubt\n> there are many people using comments like that.\n\n;-)\n\nIf \"#(\" feels a bit too generic, that part can be bikeshed.\n\nI might find some example use cases why we shouldn't later, but\noffhand, the idea of (ab)using the comment is a very good idea.\n"},{"id":"483254","messageId":"xmqqil79t82q.fsf@gitster.g","threadId":"60338","inReplyTo":"ZSouSI_zPusOefsv@localhost","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-14T17:41:49Z","receivedAt":"2023-10-14T17:41:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Triplett writes:\n\n> On Tue, Oct 10, 2023 at 10:02:20AM -0700, Junio C Hamano wrote:\n>> Sebastian Thiel writes:\n>> \n>> > I'd like to propose adding a new standard gitattribute \"precious\".\n>> \n>> ;-).\n>> \n>> Over the years, I've seen many times scenarios that would have been\n>> helped if we had not just \"tracked? ignored? unignored?\" but also\n>> the fourth kind [*]. The word \"ignored\" (or \"excluded\") has always\n>> meant \"not tracked, not to be tracked, and expendable\" to Git, and\n>> \"ignored but unexpendable\" class was missing. I even used the term\n>> \"precious\" myself in those discussions. At the concept level, I\n>> support the effort 100%, but as always, the devil will be in the\n>> details.\n>\n> \"I've already wanted this for years\" is, honestly, the best response we\n> could *possibly* have hoped for.\n\nYeah, but that is not what I gave here.\n\nIt is something I saw people want from time to time over the years;\nI am not at all talking about my desire, or lack thereof, to add it\nto the system ;-)\n\n>> In previous discussions, nobody was disturbed that \"git clean\" was\n>> unaware of the \"precious\" class, but if we were to have the\n>> \"precious\" class in addition to \"ignored\" aka \"expendable\", I would\n>> not oppose to teach \"git clean\" about it, too.\n>> \n>> There was an early and rough design draft there in\n>> \n>> https://lore.kernel.org/git/7vipsnar23.fsf@alter.siamese.dyndns.org/\n>> \n>> which probably is worth a read, too.\n\nThe project can say something like\n\n # force older git to ignore\n .config\n # older git unignores \"$.config\" without touching \".config\"\n # but newer git applies the \"last one wins\" rule as usual\n # to mark \".config\" as precious.\n !$.config\n\nif our syntax were to retrofit '!' prefix, and even more simply\n\n #:(precious)\n .config\n\nif we adopt Phillip's \"comment abuse\" idea, where older Git will\ntreat it as saying \".config\" is not to be added but is expendable,\nwhile newer Git will treat it as saying \".config\" is not to be added\nand not to be clobbered.\n"},{"id":"483282","messageId":"CABPp-BEg6vxiUpcJAG_=KB_sTrVgCF19JZh-+ZGCTPXdbo9ekg@mail.gmail.com","threadId":"60338","inReplyTo":"ZSouSI_zPusOefsv@localhost","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-10-15T06:44:22Z","receivedAt":"2023-10-15T06:44:42Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Fri, Oct 13, 2023 at 11:00 PM Josh Triplett wrote:\n>\n> On Tue, Oct 10, 2023 at 10:02:20AM -0700, Junio C Hamano wrote:\n> > Sebastian Thiel writes:\n> >\n> > > I'd like to propose adding a new standard gitattribute \"precious\".\n> >\n> > ;-).\n> >\n> > Over the years, I've seen many times scenarios that would have been\n> > helped if we had not just \"tracked? ignored? unignored?\" but also\n> > the fourth kind [*]. The word \"ignored\" (or \"excluded\") has always\n> > meant \"not tracked, not to be tracked, and expendable\" to Git, and\n> > \"ignored but unexpendable\" class was missing. I even used the term\n> > \"precious\" myself in those discussions. At the concept level, I\n> > support the effort 100%, but as always, the devil will be in the\n> > details.\n>\n> \"I've already wanted this for years\" is, honestly, the best response we\n> could *possibly* have hoped for.\n>\n> > Scenarios that people wished for \"precious\" traditionally have been\n> >\n> > * You are working on 'master'. You have in your .gitignore or\n> > .git/info/exclude a line to ignore path A, and have random\n> > scribbles in a throw-away file there. There is another branch\n> > 'seen', where they added some tracked contents at path A/B. You\n> > do \"git checkout seen\" and your file A that is an expendable file,\n> > because it is listed as ignored in .git/info/exclude, is removed\n> > to make room for creating A/B.\n>\n> Ouch, I hadn't even thought about the issue of branch-switching\n> overwriting a file like that, but that's another great reason to have\n> \"precious\". (I've been thinking about \"precious\" as primarily to protect\n> files like `.config`, where they'd be unlikely to be checked in on any\n> branch because they have an established purpose in the project. Though,\n> of course, people *do* sometimes check in `.config` files in\n> special-purpose branches that aren't meant for upstreaming.)\n\nIf we're going to implement precious files, I think we should take a\nstep back and figure out what parts of the system are affected. It's\nway more than branch switching and `git clean`. Some notes (including\nsome useful implementation pointers):\n\nA) You will probably learn a lot and get a leg up on the\nimplementation by grepping for \"preserve_ignored\"; lots of the\nplumbing has already been created related to this.\n\nB) checkout has a --no-overwrite-ignore, which for checkout\noperations, essentially turns all ignored files into precious files.\nB1) The code behind --no-overwrite-ignore will probably be helpful in\nyour implementation of precious files\nB2) What happens to this --no-overwrite-ignore option?\nDeprecate/remove it after adding precious files, since precious files\nnow serve that purpose? Keep the flag anyway? What happens to the\ndocs around the flag?\nB3) If we keep --no-overwrite-ignore, do we also need to add a\n--overwrite-precious option to allow those to be similarly tweaked?\n\nC) merge has a --no-overwrite-ignore option, which for supported merge\noperations (which are sadly very few), essentially turns all ignored\nfiles into precious files.\nC1) Same comments as A1-A3 but for merges\nC2) Sadly, merge's --no-overwrite-ignore is almost garbage.\nbuiltin/merge.c will only pass this option to the \"fast-forward\" merge\nbackend, causing any other type of merge to overwrite ignored files\ndespite any such flag.\nC3) most merge backends don't have logic to handle\n--no-overwrite-ignore even if they were passed it, and would need\nexplicit support added.\nC4) merge-ort would only essentially need a one-liner; it basically\nhas the code in place and a comment but the flag was never plumbed\nthrough\nC5) it'd be a herculean effort to support this with merge-recursive,\nand a sisyphean effort to attempt to maintain. Deprecating and\nremoving merge-recursive is probably a better option\nC6) merge-resolve and merge-octopus could probably be handled\nautomatically by ensuring `git read-tree` gained support for it.\nC7) there'd be no way to ensure user-written merge algorithms would\nsupport it, but that's kind of a general problem with user-written\nanythings\n\nD) git ls-files would need to have a way to query for precious files,\nmuch as it can currently be used to query for ignored files (or\ntracked files, or conflicted file, or skip-worktree files, or...).\nBackward compatibility questions arise about whether precious files\nshould appear in `git ls-files -o` output.\n\nE) git status has an --ignored option, with multiple flags for\ncontrolling it. We'll likely need more flags to be able to pick out\nprecious files.\n\nF) We'd probably need to look through several other commands and look\nat what they need for special handling. e.g., am, stash, reset. I\nsuspect stash will be a particularly sore point, as its unfortunate\ndesign of implementing shell in C code and attempting to decompose the\ncommand in terms of other high-level commands is basically a leaky\nabstraction that is very likely to be susceptible to edge and corner\ncases here. (In fact, I think I may have left some of the issues for\nuntracked/ignored files in stash broken when I was fixing such\nproblems for other commands.)\n\nG) Documentation. Commands like `git reset --hard`, `git checkout\n-f`, and `read-tree --reset` are documented to nuke untracked files\nspecifically because we expect most commands to preserve untracked\nfiles. These would need to mention that \"precious\" files are also\nnuked (or, if we don't nuke them, why precious-and-ignored files are\nmore precious than untracked files).\n\nH) Design of `reset --hard`. As per\nhttps://lore.kernel.org/git/xmqqr1e2ejs9.fsf@gitster.g/, `git reset\n--hard` is a little funny and we have thought about changing it. Will\nthe addition of \"precious\" objects provide more impetus to do so, and\nshould a migration story be part of such a new feature?\n\nI) Although git's stated behavior was to nuke ignored files and\nprotect untracked files, just a couple years ago we found _many_ bugs\nwhere Git didn't do that. A series was pushed to fix most of those[1]\n(incidentally, the same series the introduced the preserve_ignored\nflag I pointed you to earlier), but it left a few things\ncommented/broken. The cover letter and some emails in the thread also\ndiscussed in more detail some of the ramifications around a \"precious\"\nsetting. It may be worth reading to catch other things to cover and\nthink about.\n\nJ) To implement this feature, you're going to have to touch dir.c.\nGood luck with that. (Seriously, good luck. The more people that\ntouch it that aren't me, the less I'll be pinged/queried about that\nmonstrosity.)\n\n\n> > In any case, the \"precious\" paths are expected to be small minority\n> > of what people never want to \"git add\" or \"git commit\", so coming up\n> > with a special syntax to be used in .gitignore, even if that special\n> > syntax is ugly and cumbersome to type, would be perfectly OK.\n>\n> [Following up both to this and to Sebastian's response.]\n>\n> One potentially important question: should the behavior of old git be to\n> treat precious files as ignored, or as not-ignored? If the syntax were\n> something like\n>\n> $.config\n>\n> then old git would treat the file as not-ignored. If the syntax were\n> something like\n>\n> $precious\n> .config\n>\n> then old git would treat the file as ignored.\n>\n> Seems like it would be obtrusive if `git status` in old git showed the\n> file, and `git add .` in old git added it.\n\nA very good set of questions, along similar lines as the question\nabout `git ls-files -o` handling.\n\n\nAnyway, I'm a bit worried after digging up and dumping all these\nconcerns on you, that it'll sound like I'm trying to bury the feature\nand discourage folks. In the past I have been against this at times,\nbut mostly because it looked like lots of work, I didn't want to touch\ndir.c anymore, and I was worried we'd add a bunch more edge & corner\ncases to the code (when we already had plenty with our more limited\nnumber of file types). In a way, the preserve_ignored stuff kind of\nmade this a lot more reasonable for us to switch to. But I do still\nthink it's a fair amount of work, and I am kind of worried about\npotential new edge and corner case.\n\n\nHope that helps,\nElijah\n"},{"id":"483283","messageId":"B088FC28-BE30-424D-9CDD-7A53EDFC1710@icloud.com","threadId":"60338","inReplyTo":"CABPp-BEg6vxiUpcJAG_=KB_sTrVgCF19JZh-+ZGCTPXdbo9ekg@mail.gmail.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Sebastian Thiel","fromEmail":"sebastian.thiel@icloud.com","sentAt":"2023-10-15T07:33:17Z","receivedAt":"2023-10-15T07:33:23Z","isPatch":false,"sender":{"key":"sebastian.thiel@icloud.com","avatar":"https://avatars.githubusercontent.com/u/63622?v=4"},"body":"Thanks so much Elijah for your eye-opening response. Thus far I was both\nnaive and ignorant about the complexity of the matter, and also never\nasked the question as to why it wasn't tackled earlier since it came up\nalready.\n\nSince so many areas of git are affected by precious files, it seems that\nrolling it out with everything working is unrealistic and I wonder if\nit even had to be behind a feature toggle at first.\n\nA particularly interesting question brought up here also was the question\nof what's more important: untracked files, or precious files? Are they\neffectively treated the same, or is there a difference?\n\nIn any case, it seems easiest to set the desired syntax for such a\nfeature and/or validate it, and then devise a plan for how it could all\ncome together.\n\n\nOn 15 Oct 2023, at 8:44, Elijah Newren wrote:\n\n> Hi,\n>\n> On Fri, Oct 13, 2023 at 11:00 PM Josh Triplett wrote:\n>>\n>> On Tue, Oct 10, 2023 at 10:02:20AM -0700, Junio C Hamano wrote:\n>>> Sebastian Thiel writes:\n>>>\n>>>> I'd like to propose adding a new standard gitattribute \"precious\".\n>>>\n>>> ;-).\n>>>\n>>> Over the years, I've seen many times scenarios that would have been\n>>> helped if we had not just \"tracked? ignored? unignored?\" but also\n>>> the fourth kind [*]. The word \"ignored\" (or \"excluded\") has always\n>>> meant \"not tracked, not to be tracked, and expendable\" to Git, and\n>>> \"ignored but unexpendable\" class was missing. I even used the term\n>>> \"precious\" myself in those discussions. At the concept level, I\n>>> support the effort 100%, but as always, the devil will be in the\n>>> details.\n>>\n>> \"I've already wanted this for years\" is, honestly, the best response we\n>> could *possibly* have hoped for.\n>>\n>>> Scenarios that people wished for \"precious\" traditionally have been\n>>>\n>>> * You are working on 'master'. You have in your .gitignore or\n>>> .git/info/exclude a line to ignore path A, and have random\n>>> scribbles in a throw-away file there. There is another branch\n>>> 'seen', where they added some tracked contents at path A/B. You\n>>> do \"git checkout seen\" and your file A that is an expendable file,\n>>> because it is listed as ignored in .git/info/exclude, is removed\n>>> to make room for creating A/B.\n>>\n>> Ouch, I hadn't even thought about the issue of branch-switching\n>> overwriting a file like that, but that's another great reason to have\n>> \"precious\". (I've been thinking about \"precious\" as primarily to protect\n>> files like `.config`, where they'd be unlikely to be checked in on any\n>> branch because they have an established purpose in the project. Though,\n>> of course, people *do* sometimes check in `.config` files in\n>> special-purpose branches that aren't meant for upstreaming.)\n>\n> If we're going to implement precious files, I think we should take a\n> step back and figure out what parts of the system are affected. It's\n> way more than branch switching and `git clean`. Some notes (including\n> some useful implementation pointers):\n>\n> A) You will probably learn a lot and get a leg up on the\n> implementation by grepping for \"preserve_ignored\"; lots of the\n> plumbing has already been created related to this.\n>\n> B) checkout has a --no-overwrite-ignore, which for checkout\n> operations, essentially turns all ignored files into precious files.\n> B1) The code behind --no-overwrite-ignore will probably be helpful in\n> your implementation of precious files\n> B2) What happens to this --no-overwrite-ignore option?\n> Deprecate/remove it after adding precious files, since precious files\n> now serve that purpose? Keep the flag anyway? What happens to the\n> docs around the flag?\n> B3) If we keep --no-overwrite-ignore, do we also need to add a\n> --overwrite-precious option to allow those to be similarly tweaked?\n>\n> C) merge has a --no-overwrite-ignore option, which for supported merge\n> operations (which are sadly very few), essentially turns all ignored\n> files into precious files.\n> C1) Same comments as A1-A3 but for merges\n> C2) Sadly, merge's --no-overwrite-ignore is almost garbage.\n> builtin/merge.c will only pass this option to the \"fast-forward\" merge\n> backend, causing any other type of merge to overwrite ignored files\n> despite any such flag.\n> C3) most merge backends don't have logic to handle\n> --no-overwrite-ignore even if they were passed it, and would need\n> explicit support added.\n> C4) merge-ort would only essentially need a one-liner; it basically\n> has the code in place and a comment but the flag was never plumbed\n> through\n> C5) it'd be a herculean effort to support this with merge-recursive,\n> and a sisyphean effort to attempt to maintain. Deprecating and\n> removing merge-recursive is probably a better option\n> C6) merge-resolve and merge-octopus could probably be handled\n> automatically by ensuring `git read-tree` gained support for it.\n> C7) there'd be no way to ensure user-written merge algorithms would\n> support it, but that's kind of a general problem with user-written\n> anythings\n>\n> D) git ls-files would need to have a way to query for precious files,\n> much as it can currently be used to query for ignored files (or\n> tracked files, or conflicted file, or skip-worktree files, or...).\n> Backward compatibility questions arise about whether precious files\n> should appear in `git ls-files -o` output.\n>\n> E) git status has an --ignored option, with multiple flags for\n> controlling it. We'll likely need more flags to be able to pick out\n> precious files.\n>\n> F) We'd probably need to look through several other commands and look\n> at what they need for special handling. e.g., am, stash, reset. I\n> suspect stash will be a particularly sore point, as its unfortunate\n> design of implementing shell in C code and attempting to decompose the\n> command in terms of other high-level commands is basically a leaky\n> abstraction that is very likely to be susceptible to edge and corner\n> cases here. (In fact, I think I may have left some of the issues for\n> untracked/ignored files in stash broken when I was fixing such\n> problems for other commands.)\n>\n> G) Documentation. Commands like `git reset --hard`, `git checkout\n> -f`, and `read-tree --reset` are documented to nuke untracked files\n> specifically because we expect most commands to preserve untracked\n> files. These would need to mention that \"precious\" files are also\n> nuked (or, if we don't nuke them, why precious-and-ignored files are\n> more precious than untracked files).\n>\n> H) Design of `reset --hard`. As per\n> https://lore.kernel.org/git/xmqqr1e2ejs9.fsf@gitster.g/, `git reset\n> --hard` is a little funny and we have thought about changing it. Will\n> the addition of \"precious\" objects provide more impetus to do so, and\n> should a migration story be part of such a new feature?\n>\n> I) Although git's stated behavior was to nuke ignored files and\n> protect untracked files, just a couple years ago we found _many_ bugs\n> where Git didn't do that. A series was pushed to fix most of those[1]\n> (incidentally, the same series the introduced the preserve_ignored\n> flag I pointed you to earlier), but it left a few things\n> commented/broken. The cover letter and some emails in the thread also\n> discussed in more detail some of the ramifications around a \"precious\"\n> setting. It may be worth reading to catch other things to cover and\n> think about.\n>\n> J) To implement this feature, you're going to have to touch dir.c.\n> Good luck with that. (Seriously, good luck. The more people that\n> touch it that aren't me, the less I'll be pinged/queried about that\n> monstrosity.)\n>\n>\n>>> In any case, the \"precious\" paths are expected to be small minority\n>>> of what people never want to \"git add\" or \"git commit\", so coming up\n>>> with a special syntax to be used in .gitignore, even if that special\n>>> syntax is ugly and cumbersome to type, would be perfectly OK.\n>>\n>> [Following up both to this and to Sebastian's response.]\n>>\n>> One potentially important question: should the behavior of old git be to\n>> treat precious files as ignored, or as not-ignored? If the syntax were\n>> something like\n>>\n>> $.config\n>>\n>> then old git would treat the file as not-ignored. If the syntax were\n>> something like\n>>\n>> $precious\n>> .config\n>>\n>> then old git would treat the file as ignored.\n>>\n>> Seems like it would be obtrusive if `git status` in old git showed the\n>> file, and `git add .` in old git added it.\n>\n> A very good set of questions, along similar lines as the question\n> about `git ls-files -o` handling.\n>\n>\n> Anyway, I'm a bit worried after digging up and dumping all these\n> concerns on you, that it'll sound like I'm trying to bury the feature\n> and discourage folks. In the past I have been against this at times,\n> but mostly because it looked like lots of work, I didn't want to touch\n> dir.c anymore, and I was worried we'd add a bunch more edge & corner\n> cases to the code (when we already had plenty with our more limited\n> number of file types). In a way, the preserve_ignored stuff kind of\n> made this a lot more reasonable for us to switch to. But I do still\n> think it's a fair amount of work, and I am kind of worried about\n> potential new edge and corner case.\n>\n>\n> Hope that helps,\n> Elijah\n"},{"id":"483286","messageId":"xmqqmswjsv8c.fsf@gitster.g","threadId":"60338","inReplyTo":"B088FC28-BE30-424D-9CDD-7A53EDFC1710@icloud.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-15T16:31:31Z","receivedAt":"2023-10-15T16:31:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sebastian Thiel writes:\n\n> A particularly interesting question brought up here also was the question\n> of what's more important: untracked files, or precious files? Are they\n> effectively treated the same, or is there a difference?\n\nThink of it this way. There are two orthogonal axes.\n\n (1) Are you a candidate to be tracked, even though you are not\n tracked right now?\n\n (2) Should you be kept and make an operation fail that wants to\n remove you to make room?\n\nFor untracked files, both are \"Yes\". As we already saw in the long\ndiscussion, precious files are \"not to be added and not to be\nclobbered\", so you'd answer \"No\" and \"Yes\" [*].\n\nIn other words, both are equally protected from getting cloberred.\n\n Side note: for completeness, for ignored files, the answers are\n \"No\", and \"No\". The introduction of \"precious\" class makes a\n combination \"No-Yes\" that hasn't been possible so far.\n\nElijah, thanks for doing a very good job of creating a catalog of\nkludges we accumulated over the years for the lack of proper support\nfor the precious paths. I think they should be kept for backward\ncompatibility, but for new users they should not have to learn any\nof them once we have the support for precious paths.\n\n"},{"id":"483294","messageId":"1EE716BB-C8D5-4543-A5BE-EB8518151077@icloud.com","threadId":"60338","inReplyTo":"xmqqmswjsv8c.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Sebastian Thiel","fromEmail":"sebastian.thiel@icloud.com","sentAt":"2023-10-16T06:02:25Z","receivedAt":"2023-10-16T06:11:53Z","isPatch":false,"sender":{"key":"sebastian.thiel@icloud.com","avatar":"https://avatars.githubusercontent.com/u/63622?v=4"},"body":"Thanks a lot, that makes perfect sense!\n\nThanks to Elijah we may also have discovered why the idea of precious files\ndidn't get implemented last time it came up: it's too much work to make\nall portions of the code aware.\n\nI don't know if this time will be different as I can only offer to implement\nthe syntax adjustment, whatever that might be (possibly after validating\nthe candidate against a corpus of repositories), along with the update\nto `git clean` so it leaves precious files alone by default and a new flag\nto also remove precious files.\n\nMaybe that already is something worth having, but I can also imagine\nthat ideally there is a plan for retrofitting other portions of git as\nwell along with the resources to actually do it.\n\nOn 15 Oct 2023, at 18:31, Junio C Hamano wrote:\n\n> Sebastian Thiel writes:\n>\n>> A particularly interesting question brought up here also was the question\n>> of what's more important: untracked files, or precious files? Are they\n>> effectively treated the same, or is there a difference?\n>\n> Think of it this way. There are two orthogonal axes.\n>\n> (1) Are you a candidate to be tracked, even though you are not\n> tracked right now?\n>\n> (2) Should you be kept and make an operation fail that wants to\n> remove you to make room?\n>\n> For untracked files, both are \"Yes\". As we already saw in the long\n> discussion, precious files are \"not to be added and not to be\n> clobbered\", so you'd answer \"No\" and \"Yes\" [*].\n>\n> In other words, both are equally protected from getting cloberred.\n>\n> Side note: for completeness, for ignored files, the answers are\n> \"No\", and \"No\". The introduction of \"precious\" class makes a\n> combination \"No-Yes\" that hasn't been possible so far.\n>\n> Elijah, thanks for doing a very good job of creating a catalog of\n> kludges we accumulated over the years for the lack of proper support\n> for the precious paths. I think they should be kept for backward\n> compatibility, but for new users they should not have to learn any\n> of them once we have the support for precious paths.\n"},{"id":"483650","messageId":"918D0772-CDEE-4892-828E-BD8A06C3F1F4@icloud.com","threadId":"60338","inReplyTo":"xmqqmswjsv8c.fsf@gitster.g","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Sebastian Thiel","fromEmail":"sebastian.thiel@icloud.com","sentAt":"2023-10-23T07:15:28Z","receivedAt":"2023-10-23T07:15:37Z","isPatch":false,"sender":{"key":"sebastian.thiel@icloud.com","avatar":"https://avatars.githubusercontent.com/u/63622?v=4"},"body":"On 16 Oct 2023, at 8:02, Sebastian Thiel wrote:\n\n> I don't know if this time will be different as I can only offer to implement\n> the syntax adjustment, whatever that might be (possibly after validating\n> the candidate against a corpus of repositories), along with the update\n> to `git clean` so it leaves precious files alone by default and a new flag\n> to also remove precious files.\n\nI am happy to announce this feature can now be contributed in full by me once\nyou give it a go. This would mean that the entirety of `git` would become\naware of precious files over time.\n\nTo my mind, and probably out of ignorance, it seems that once the syntax is\ndecided on it's possible for the implementation to start. From there I could\nuse Elijah's analysis to know which parts of git to make aware of precious files\nin addition to `git clean`.\n\nI am definitely looking forward to hearing from you :).\n\n\n"},{"id":"484045","messageId":"CABPp-BGZHUQz5Bnd1oUptKC_j680Vz0zykEGgXw+89W3Tv6hmw@mail.gmail.com","threadId":"60338","inReplyTo":"918D0772-CDEE-4892-828E-BD8A06C3F1F4@icloud.com","subject":"Re: [RFC] Define \"precious\" attribute and support it in `git clean`","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-10-29T06:44:02Z","receivedAt":"2023-10-29T06:44:18Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Sebastian,\n\nOn Mon, Oct 23, 2023 at 12:15 AM Sebastian Thiel\n wrote:\n>\n> On 16 Oct 2023, at 8:02, Sebastian Thiel wrote:\n>\n> > I don't know if this time will be different as I can only offer to implement\n> > the syntax adjustment, whatever that might be (possibly after validating\n> > the candidate against a corpus of repositories), along with the update\n> > to `git clean` so it leaves precious files alone by default and a new flag\n> > to also remove precious files.\n>\n> I am happy to announce this feature can now be contributed in full by me once\n> you give it a go. This would mean that the entirety of `git` would become\n> aware of precious files over time.\n>\n> To my mind, and probably out of ignorance, it seems that once the syntax is\n> decided on it's possible for the implementation to start. From there I could\n> use Elijah's analysis to know which parts of git to make aware of precious files\n> in addition to `git clean`.\n>\n> I am definitely looking forward to hearing from you :).\n\nSo, we typically don't pre-approve patches/features. Junio described\nthis recently at [1].\n\nHowever, starting things out with an RFC, as you've done, is certainly\na good first step to gauge whether folks think a feature is useful.\n\nOccasionally, when the feature is bigger or touches lots of areas of\nthe code, people will even write up a design document, and first get a\nreview on the document, which then streamlines later reviews since we\nhave some of the high-level aspects agreed to. Some examples:\n * Documentation/technical/hash-function-transition.txt\n * Documentation/technical/sparse-checkout.txt\n * Documentation/technical/sparse-index.txt\nEach of which are in various stages between \"these are ideas we think\nare good and our plans to get there\" to \"most of this document has\nsince been implemented\". There are others in that directory too,\nthough not everything in that directory is a planning document; some\nof the files are simply documentation of what already exists.\n\nAnyway, creating a similar planning document and covering the various\ncases I mentioned would likely be a very useful next step here. I did\nnote that multiple ideas have been presented in this thread about the\nsyntax for specifying precious files, and it'd be good to nail one\ndown. It would also be nice to see proposed answers to the several\ncases I brought up (some of which Junio answered, others of which I\nalso have potential answers for so I could potentially help you craft\nthis document, and a few others that someone else would need to fill\nin). Sometimes we also want to cover pros/cons of the approaches we\nhave decided upon, in part because others may come along later and if\nthey discover a new pro or con that we haven't thought of, then we may\nneed to rethink the plan.\n\nHope that helps,\nElijah\n\n[1] https://lore.kernel.org/git/xmqq8r9ommyt.fsf@gitster.g/\n"}]}