{"thread":{"id":"55840","subject":"Pro Git book: concerning data lost due to \".gitignore\"","startedAt":"2021-06-04T23:14:09Z","lastAt":"2021-07-10T08:32:43Z","messageCount":4,"participants":["grizlyk","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"426448","messageId":"3957861622848346@myt5-a5512e99e394.qloud-c.yandex.net","threadId":"55840","inReplyTo":null,"subject":"Pro Git book: concerning data lost due to \".gitignore\"","fromName":"grizlyk","fromEmail":"grizlyk1@yandex.ru","sentAt":"2021-06-04T23:12:26Z","receivedAt":"2021-06-04T23:14:09Z","isPatch":false,"sender":{"key":"grizlyk1@yandex.ru","avatar":null},"body":"Good day.\n\n1. Summary. \n\nIt should be explicitly warned in the Pro Git book https://git-scm.com/book/en/v2 (and in git man also) that the \".gitignore\" feature is very dangerous stuff and should be used with care. \n\nDue to \".gitignore\" usage, some data files in directory placed under git version control, can be lost for indexing and can be not placed into repo _unexpectedly_ for user. \n\nUser will call \"git add\" and git will answer about \"nothing to stage\". \n\nThe problem grows up if the project was obtained from network with unknown \".gitignore\" files or there are external tools (like Visual Studio IDE) that can in background setup and modify \"./.gitignore\" files by unknown for user way during project lifetime. \n\nIn follow text we list the problems, and git book editors will decide how to write about (esp. in english). \n\n2.\nIn ideal case \".gitignore\" should be disabled at all, user should provide clean directory to be controlled by any VCS, any all dummy files must be moved out by generic OS tools. \n\nBut for some reasons \".gitignore\" will exist. \n\nThere are some visible solutions to relax the data lost danger due to \".gitignore\": \n\n2.1\nplace files that no need to be staged into separate from repo subdirectories. \n\nIn ideal case for directory tree like \"a\\b\\c\": \n- all subdirectories placed under git control must be staged (\"git add src\"); \n\"\\src\\\" + \"a\\b\\c\"\n\n- all generated files must be outside of src source directory (possibly with the same subdir tree); \n\"\\var\\\" + \"a\\b\\c\"\n\nIn the case you will treat \".gitignore\" feature as workaround to projects that can not follow reliable repo rules (reliable here means files will not be omitted from stage by mistake). \n\n2.2\nprovide trivial \".gitignore\" file.\n\n- rename all unknown to you foreign or autogenerated \".gitignore\" into any kind of \".old-gitignore.old\"; \n- manually create \".gitignore\" file and place into it only the names, that you exactly know; \n- always call \"git status --ignored\". \n\n2.3\nuse \"git status --ignored\".\n\nalways use \"--ignored\" option of \"git status\" at least to complete daily commit but better for every commit. \n\nCheck ignored files list by \"git status --ignored\". You will see all 4 parts of status list (instead of 3 parts of the list without \"--ignored\" option): \n\nexample: \n$ git status --ignored\nOn branch master\nChanges to be committed:\n  (use \"git restore --staged <file>...\" to unstage)\n\nChanges not staged for commit:\n  (use \"git add <file>...\" to update what will be committed)\n  (use \"git restore <file>...\" to discard changes in working directory)\n\nUntracked files:\n  (use \"git add <file>...\" to include in what will be committed)\n\nIgnored files:\n  (use \"git add -f <file>...\" to include in what will be committed)\n\nIf you follow rule \"to separate output files\", the \"Ignored files\" list will be very short and you will see here only known to you names. \n\n2.4\nvisual tools to work with git should have separated tab to display \"Ignored files\" list. \n\nnot a user work\n\n2.5\nWe realize that \"Ignored files\" was invented exactly in intention to be skipped from repo.\n \nBut in status list, if option \"status --ignored\" was not used, it could be useful to warn users by message like this: \n\"There are ignored files (%u<number of files>) not placed into repo.\\n\\t(use git status --ignored to view)\"\n\ninstead of full list, if option \"status --ignored\" was used\nIgnored files:\n  (use \"git add -f <file>...\" to include in what will be committed)\n\nnot a user work\n\n3.\nMore explanations. \n\nIn the book one can read: \"2.2 Ignoring Files. Tips: The Linux kernel source repository has 206 .gitignore files\". \n\nBut in real the kernel source is not comprehensive list of \".gitignore\" usage. \n\nAny tool does not exist by itself, it works \"in terms of task the tool implements\". That means there are \"tasks\" the tool does. \n\nLike any VCS, git has a valuable \"task\" named \"every commit backup all files in the dir placed under version control\". The task means \"it is beter to include extra files into repo than to lose required files\". If even only one file will be lost from stage git will fail the task. \n\nIn the task git works in \"cp\" level and should be the same trivial and reliable as \"cp\", here we even should not try more tricks with git, just because \"enumerate files in directory without errors\" is very easy work that any software from CP/M times should implement. \n\n4. Example. \n\nThere is practical example of the git usage with errors due to \".gitignore\". \n\nI have a source \".\" directory. \n\nIn the directory i have some files (in my case 297 files, i was forced by git to count files in the directory, i have never tried to count files before git usage) and some levels of sub directories (in my case 9 or more levels of sub directories, i am not sure). \n\nI called \"git add .\" but i was not able to add all files in the directory in git repo without any error messages by answer about \"nothing to stage\". \n\nVS shows me that only 80 sacred files of 297 total files was blessed into repo by git in conjunction with VS without any error messages. \n\nThough first time (when repo was created) VS tried to stage more files (all 297 files) into git repo, but git did not accept the gift by answer about \"nothing to stage\". Later VS does not offer more than 80 files to add. \n\nI spent several hours to read git large mans, tens of flat git options, but i was still not able to add the directory in git repo. \n\nBecause \"gnu in windows\" has well known troubles with console and charsets (names can look like \"f??df???.???\"), the system was also under suspition and was also checked (as the system that is unable to enumerate files in directory). \n\n4.1\nlooking into \".gitignore\" file, created by several versions of VS, i found that some sub directories of the repo was marked to ignore. \n\nI removed the names from \".gitignore\" file and completed repo about 217 files.\n\nAt last the git repo was in work. Creepy. \n\n4.2\nAfter several days when the git repo was (as i believed) recovered i called \"git status --ignored\" by random event. \n\nI got \n>git status --ignored\n\nIgnored files:\n  (use \"git add -f <file>...\" to include in what will be committed)\n        .vs/\n        10\n        7\n        .../state/#bak/v02/\n        .../state/...reg_i.h\n\nI found yet 2 extra names ignored by VS and by git due to \".gitignore\" files. More creepy. \n\nAs you see by the real example: \".gitignore\" feature has real ability to produce repo with lost files. \n\nPS:\nIn Visual Studio related part it should be warned to do manual check \".gitattributes\" file often to adjust (for example to comment out) record \"* text=auto\". \n\n###############################################################################\n# Set default behavior to automatically normalize line endings.\n###############################################################################\n#* text=auto\n\nPPS:\nI also just have played in dice (no luck) to report the issue, not sure it can help to stage files. \n\nBest regards, \nMaksim.\n"},{"id":"426503","messageId":"87a6o459bh.fsf@evledraar.gmail.com","threadId":"55840","inReplyTo":"3957861622848346@myt5-a5512e99e394.qloud-c.yandex.net","subject":"Re: Pro Git book: concerning data lost due to \".gitignore\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-05T20:39:17Z","receivedAt":"2021-06-05T20:47:31Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Jun 05 2021, grizlyk wrote:\n\n> Good day.\n\nHi.\n\n> 1. Summary. \n>\n> It should be explicitly warned in the Pro Git book https://git-scm.com/book/en/v2 (and in git man also) that the \".gitignore\" feature is very dangerous stuff and should be used with care. \n\nThis mailing list does not maintain that book. For any issues with it\nsee the issues/PR's at https://github.com/progit/progit2.\n\n> Due to \".gitignore\" usage, some data files in directory placed under\n> git version control, can be lost for indexing and can be not placed\n> into repo _unexpectedly_ for user.\n\nI skimmed the rest of your mail, I think you might find the previous\ndiscussion(s) about a \"precious\" attribute at and adding something like\na backup log when we shred files due to gitignore[2] interesting and\nrelevant to much of what you point out.\n\nI don't think the notion of moving to some general workflow of compiled\nfiles being staged elsewhere than the source is something that's viable\nas a general constraint for a VCS like git.\n\nIt's way too common of a pattern to e.g. have a *.o file made from a\ncorresponding *.[ch] file(s) in the same directory.\n\nFor those for whom the solution you suggested works I believe git\nalready does a good job of supporting it. You'd e.g. compile all your\nassets outside of the repo via your build system, and just not have\nanything in .gitignore.\n\nI don't see how we could expect to smartly deal with having some\nparallel tool like VS that's (supposedly, I'm just taking your summary\nat face value) actively working against the wishes of its\nusers. Something like .git/info/exclude is useful, but only assuming\nthat your tooling isn't actively trying to subvert you.\n\n1. https://lore.kernel.org/git/20190216114938.18843-1-pclouds@gmail.com/\n2. https://lore.kernel.org/git/20181209104419.12639-1-pclouds@gmail.com/\n"},{"id":"429587","messageId":"4458411625892760@myt6-ab88f6e23eb0.qloud-c.yandex.net","threadId":"55840","inReplyTo":"87a6o459bh.fsf@evledraar.gmail.com","subject":"Re: Pro Git book: concerning data lost due to \".gitignore\"","fromName":"grizlyk","fromEmail":"grizlyk1@yandex.ru","sentAt":"2021-07-10T04:52:40Z","receivedAt":"2021-07-10T04:59:23Z","isPatch":false,"sender":{"key":"grizlyk1@yandex.ru","avatar":null},"body":"hi\n\n> On Sat, Jun 05 2021\n> It's way too common of a pattern to e.g. have a *.o file made from a\n> corresponding *.[ch] file(s) in the same directory.\n\nThe patterns were common for old times (before VCSes was involved). To deal with temporary files (like .o), generic OS tools like \"make remove_compiled\" can help to clean directory before stage. To keep derivative persistent files (like the same .o) separated directory can be used.\n\n> git already does a good job of supporting it. \n\nSure, the light message: \"There are ignored files (%u<number of files>) not placed into repo.\\n\\t(use git status --ignored to view)\"; will improve the activity. Otherwise some files somethimes will not be placed into repo unexpectedly for user. \n\n> You'd e.g. compile all your \n> assets outside of the repo via your build system, and just not have\n> anything in .gitignore.\n\nDo you suggest to copy desired src files into separated repo directory (the repo directory placed under VCS control) by generic OS tools (i.e. by cp command) and stage the separated directory? \n\nIf yes; for the first it is copies of all src files; for the second we could lost some src files due to the possible wrong copy patterns (the same reasons as by wrong .gitignore patterns). \nSo, to explicitly create output files into separated directory and to implicitly include all files in src directory into repo is only reliable way to keep commits without data lost (the way will sometimes include extra output files into repo). \n\nto explicitly create output files into separated directory is responsibility of translators and makefiles. \n\nBest regards,\nMaksim.\n"},{"id":"429596","messageId":"87zguu4li5.fsf@evledraar.gmail.com","threadId":"55840","inReplyTo":"4458411625892760@myt6-ab88f6e23eb0.qloud-c.yandex.net","subject":"Re: Pro Git book: concerning data lost due to \".gitignore\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-10T08:23:25Z","receivedAt":"2021-07-10T08:32:43Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Jul 10 2021, grizlyk wrote:\n\n> hi\n>\n>> On Sat, Jun 05 2021\n>> It's way too common of a pattern to e.g. have a *.o file made from a\n>> corresponding *.[ch] file(s) in the same directory.\n>\n> The patterns were common for old times (before VCSes was involved). To\n> deal with temporary files (like .o), generic OS tools like \"make\n> remove_compiled\" can help to clean directory before stage. To keep\n> derivative persistent files (like the same .o) separated directory can\n> be used.\n\nIt's still a very common pattern, e.g. the project whose ML you're\nposting to uses it, anecdotally most free software C or C-like projects\nI look at / work on use it.\n\nIn any case, git as a project can't say \"you should fix your code\". This\nVCS has to deal with the real world, people do use this pattern in the\nwild, and we can't willy-nilly eat their data.\n\nIt's not a good approach to advocate a change in git behavior to say\n\"people should do X, not Y, to avoid this problem\", when a cursory look\nat real-world use reveals that \"X\" is in wide use, and unless you did\n\"Y\" a proposed change in behavior would be detrimental to your use of\ngit.\n\nWhat is more productive is to either find out how we can support both\nwithout harming the other, or make new behavior opt-in, hence the\nthread(s) I linked to about \"precious\" etc.\n\n>> git already does a good job of supporting it. \n>\n> Sure, the light message: \"There are ignored files (%u<number of\n> files>) not placed into repo.\\n\\t(use git status --ignored to view)\";\n> will improve the activity. Otherwise some files somethimes will not be\n> placed into repo unexpectedly for user.\n>\n>> You'd e.g. compile all your \n>> assets outside of the repo via your build system, and just not have\n>> anything in .gitignore.\n>\n> Do you suggest to copy desired src files into separated repo directory\n> (the repo directory placed under VCS control) by generic OS tools\n> (i.e. by cp command) and stage the separated directory?\n\nI'm not really being serious here, as should be clear from the linked\nthreads I think the current behavior has sucky edge cases and does eat\npeople's data in some cases, that's bad, the problem is finding a way to\nchange it that doesn't cause badness for other use-cases.\n\nI am saying that if your proposed \"Y\" solution is effectively \"other\npeople should mostly/entirely rewrite their build systems to deal with a\nnew default I'm proposing\", then in this case you'd also approximately\nwhat you wou want if we keep the current behavior and you rewrite your\nbuild system(s), no?\n\nAnyway, maybe I misunderstood some of what you're saying...\n"}]}