{"thread":{"id":"59904","subject":"Clean up stale .gitignore and .gitattribute patterns","startedAt":"2023-06-23T15:30:13Z","lastAt":"2023-06-27T06:56:19Z","messageCount":9,"participants":["Sebastian Schuberth","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"478733","messageId":"CAHGBnuOR+MU50jhNBHw8buWS_Yr9D92mErvgoi=cK16a=4_YUA@mail.gmail.com","threadId":"59904","inReplyTo":null,"subject":"Clean up stale .gitignore and .gitattribute patterns","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2023-06-23T15:29:42Z","receivedAt":"2023-06-23T15:30:13Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"Hi,\n\nis there a command to easily check patterns in .gitignore and\n.gitattributes to still match something? I'd like to remove / correct\npatterns that don't match anything anymore due to (re)moved files.\n\n-- \nSebastian Schuberth\n"},{"id":"478737","messageId":"xmqqcz1may4g.fsf@gitster.g","threadId":"59904","inReplyTo":"CAHGBnuOR+MU50jhNBHw8buWS_Yr9D92mErvgoi=cK16a=4_YUA@mail.gmail.com","subject":"Re: Clean up stale .gitignore and .gitattribute patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-23T17:26:23Z","receivedAt":"2023-06-23T17:26:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sebastian Schuberth <sschuberth@gmail.com> writes:\n\n> is there a command to easily check patterns in .gitignore and\n> .gitattributes to still match something? I'd like to remove / correct\n> patterns that don't match anything anymore due to (re)moved files.\n\nI guess \"git check-attr --stdin\" and \"git check-ignore --stdin\" will\nbe part of the solution to your problem, but I do not know what the\nother parts would be.\n\nFeeding \"ls-files\" output to \"check-ignore --stdin\" feels sort-of\noxymoron because by definition the output from \"ls-files\" cannot\ncontain any ignored paths.\n"},{"id":"478758","messageId":"20230624011234.GA95358@coredump.intra.peff.net","threadId":"59904","inReplyTo":"CAHGBnuOR+MU50jhNBHw8buWS_Yr9D92mErvgoi=cK16a=4_YUA@mail.gmail.com","subject":"Re: Clean up stale .gitignore and .gitattribute patterns","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-24T01:12:34Z","receivedAt":"2023-06-24T01:12:50Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 23, 2023 at 05:29:42PM +0200, Sebastian Schuberth wrote:\n\n> is there a command to easily check patterns in .gitignore and\n> .gitattributes to still match something? I'd like to remove / correct\n> patterns that don't match anything anymore due to (re)moved files.\n\nI don't think there's a solution that matches \"easily\", but you can do a\nbit with some scripting. See below.\n\nFor checking .gitignore, I don't think you can ever say (at the git\nlevel) that a certain pattern is useless, because it is inherently about\nmatching things that not tracked, and hence generated elsewhere. So if\nyou have a \"*.foo\" pattern, you can check if it matches anything\n_currently_ in your working tree, but if it doesn't that may mean that\nyou simply did not trigger the build rule that makes the garbage \".foo\"\nfile.\n\nSo with that caveat, we can ask Git which rules _do_ have a match, and\nthen eliminate them as \"definitely useful\", and print the others. The\nlogic is sufficiently tricky that I turned to perl:\n\n-- >8 show-unmatched-ignore.pl 8< --\n#!/usr/bin/perl\n\n# The general idea here is to read \"filename:linenr ...\" output from\n# \"check-ignore -v\". For each filename we learn about, we'll load the\n# complete set of lines into an array and then \"cross them off\" as\n# check-ignore tells us they were used.\n#\n# Note that we'd fail to mention an ignore file which matches nothing.\n# Probably the list of filenames could be generated independently. I'll\n# that as an exercise for the reader.\nwhile (<>) {\n  /^(.*?):(\\d+):/\n    or die \"puzzling input: $_\";\n  if (!defined $files{$1}) {\n    $files{$1} = do {\n      open(my $fh, '<', $1)\n        or die \"unable to open $1: $!\";\n      [<$fh>]\n    };\n  }\n  $files{$1}->[$2] = undef;\n}\n\n# With that done, whatever is left is unmatched. Print them.\nfor my $fn (sort keys(%files)) {\n  my $lines = $files{$fn};\n  for my $nr (1..@$lines) {\n    my $line = $lines->[$nr-1];\n    print \"$fn:$nr $line\" if defined $line;\n  }\n}\n-- >8 --\n\nAnd you'd use it something like:\n\n  git ls-files -o |\n  git check-ignore --stdin -v |\n  perl show-unmatched-ignore.pl\n\nPretty clunky, but it works OK in git.git (and shows that there are many\n\"not matched but probably still useful\" entries; e.g., \"*.dll\" will\nnever match for me on Linux, but is probably something we still want to\nkeep). So I wouldn't use it as an automated tool, but it might give a\nstarting point for a human looking to clean things up manually.\n\nFor attributes, I think the situation is better; we only need them to\nmatch tracked files (though technically speaking, you may want to keep\nattributes around for historical files as we use the checked-out\nattributes during \"git log\", etc). Unfortunately we don't have an\nequivalent of \"-v\" for check-attr. It might be possible to add that ,but\nin the meantime, the best I could come up with is to munge each pattern\nto add a sentinel attribute, and see if it matches anything.\n\nSomething like:\n\n  # Maybe also pipe in .git/info/attributes and core.attributesFile\n  # if you want to check those.\n  git ls-files '.gitattributes' '**/.gitattributes' |\n  while read fn; do\n  \tlines=$(wc -l <\"$fn\")\n  \tmv \"$fn\" \"$fn.orig\"\n  \tnr=1\n  \twhile test $nr -le $lines; do\n  \t\tsed \"${nr}s/$/ is-matched/\" <\"$fn.orig\" >\"$fn\"\n  \t\tgit ls-files | git check-attr --stdin is-matched |\n  \t\tgrep -q \"is-matched: set\" ||\n  \t\techo \"$fn:$nr $(sed -n ${nr}p \"$fn.orig\")\"\n  \t\tnr=$((nr+1))\n  \tdone\n  \tmv \"$fn.orig\" \"$fn\"\n  done\n\nIt produces no output in git.git (we are using all of our attributes),\nbut you can add a useless one like:\n\n  echo '*.c -diff' >>Documentation/.gitattributes\n\nand then the loop yields:\n\n  Documentation/.gitattributes:2 *.c -diff\n\nSo I definitely wouldn't call any of that \"easy\", but it may help you.\n\n-Peff\n"},{"id":"478759","messageId":"20230624011651.GB95358@coredump.intra.peff.net","threadId":"59904","inReplyTo":"xmqqcz1may4g.fsf@gitster.g","subject":"Re: Clean up stale .gitignore and .gitattribute patterns","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-24T01:16:51Z","receivedAt":"2023-06-24T01:16:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 23, 2023 at 10:26:23AM -0700, Junio C Hamano wrote:\n\n> Sebastian Schuberth <sschuberth@gmail.com> writes:\n> \n> > is there a command to easily check patterns in .gitignore and\n> > .gitattributes to still match something? I'd like to remove / correct\n> > patterns that don't match anything anymore due to (re)moved files.\n> \n> I guess \"git check-attr --stdin\" and \"git check-ignore --stdin\" will\n> be part of the solution to your problem, but I do not know what the\n> other parts would be.\n> \n> Feeding \"ls-files\" output to \"check-ignore --stdin\" feels sort-of\n> oxymoron because by definition the output from \"ls-files\" cannot\n> contain any ignored paths.\n\nYou can feed \"ls-files -o\" (since without --exclude-standard it lists\nevery untracked file in the working tree), but note that this is\ninherently incomplete. Any solution like this can only tell you which\nones are unused by what's in your current working tree, not what might\nbe possible if you ran \"make foo\" or whatever.\n\nIt can be wrong the other way, too. You might have \"file.foo\" sitting\naround from a build last year (or even sightseeing an old commit), even\nthough support for building \".foo\" is long gone from the code base.\n\nSo you'd really want to start with a fresh clone, then run any build\ncommands that might possibly put cruft in the working tree (if that's\neven possible on a single platform), and then do your analysis (and see\nmy other mail in the thread for some hacky scripting there).\n\n-Peff\n"},{"id":"478788","messageId":"CAHGBnuPO63Hi8mfA+MkAGES-gs0eNCDPG2FcPZT=YsnVzKd30A@mail.gmail.com","threadId":"59904","inReplyTo":"20230624011234.GA95358@coredump.intra.peff.net","subject":"Re: Clean up stale .gitignore and .gitattribute patterns","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2023-06-26T07:51:27Z","receivedAt":"2023-06-26T07:51:44Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"Thanks Peff for the suggestion. I ended up scripting something via\nJGit [1], as we're anyway using it as part of our Gradle build system.\n\nPS: As a future idea, it might be good if \"git mv\" gives a hint about\nupdating .gitattributes if files matching .gitattributes pattern are\nmoved.\n\n[1]: https://github.com/oss-review-toolkit/ort/pull/7195/commits/e01945d41012db2d0bc2e53d7be4abd513888ba6\n\n-- \nSebastian Schuberth\n\nOn Sat, Jun 24, 2023 at 3:12 AM Jeff King <peff@peff.net> wrote:\n>\n> On Fri, Jun 23, 2023 at 05:29:42PM +0200, Sebastian Schuberth wrote:\n>\n> > is there a command to easily check patterns in .gitignore and\n> > .gitattributes to still match something? I'd like to remove / correct\n> > patterns that don't match anything anymore due to (re)moved files.\n>\n> I don't think there's a solution that matches \"easily\", but you can do a\n> bit with some scripting. See below.\n>\n> For checking .gitignore, I don't think you can ever say (at the git\n> level) that a certain pattern is useless, because it is inherently about\n> matching things that not tracked, and hence generated elsewhere. So if\n> you have a \"*.foo\" pattern, you can check if it matches anything\n> _currently_ in your working tree, but if it doesn't that may mean that\n> you simply did not trigger the build rule that makes the garbage \".foo\"\n> file.\n>\n> So with that caveat, we can ask Git which rules _do_ have a match, and\n> then eliminate them as \"definitely useful\", and print the others. The\n> logic is sufficiently tricky that I turned to perl:\n>\n> -- >8 show-unmatched-ignore.pl 8< --\n> #!/usr/bin/perl\n>\n> # The general idea here is to read \"filename:linenr ...\" output from\n> # \"check-ignore -v\". For each filename we learn about, we'll load the\n> # complete set of lines into an array and then \"cross them off\" as\n> # check-ignore tells us they were used.\n> #\n> # Note that we'd fail to mention an ignore file which matches nothing.\n> # Probably the list of filenames could be generated independently. I'll\n> # that as an exercise for the reader.\n> while (<>) {\n>   /^(.*?):(\\d+):/\n>     or die \"puzzling input: $_\";\n>   if (!defined $files{$1}) {\n>     $files{$1} = do {\n>       open(my $fh, '<', $1)\n>         or die \"unable to open $1: $!\";\n>       [<$fh>]\n>     };\n>   }\n>   $files{$1}->[$2] = undef;\n> }\n>\n> # With that done, whatever is left is unmatched. Print them.\n> for my $fn (sort keys(%files)) {\n>   my $lines = $files{$fn};\n>   for my $nr (1..@$lines) {\n>     my $line = $lines->[$nr-1];\n>     print \"$fn:$nr $line\" if defined $line;\n>   }\n> }\n> -- >8 --\n>\n> And you'd use it something like:\n>\n>   git ls-files -o |\n>   git check-ignore --stdin -v |\n>   perl show-unmatched-ignore.pl\n>\n> Pretty clunky, but it works OK in git.git (and shows that there are many\n> \"not matched but probably still useful\" entries; e.g., \"*.dll\" will\n> never match for me on Linux, but is probably something we still want to\n> keep). So I wouldn't use it as an automated tool, but it might give a\n> starting point for a human looking to clean things up manually.\n>\n> For attributes, I think the situation is better; we only need them to\n> match tracked files (though technically speaking, you may want to keep\n> attributes around for historical files as we use the checked-out\n> attributes during \"git log\", etc). Unfortunately we don't have an\n> equivalent of \"-v\" for check-attr. It might be possible to add that ,but\n> in the meantime, the best I could come up with is to munge each pattern\n> to add a sentinel attribute, and see if it matches anything.\n>\n> Something like:\n>\n>   # Maybe also pipe in .git/info/attributes and core.attributesFile\n>   # if you want to check those.\n>   git ls-files '.gitattributes' '**/.gitattributes' |\n>   while read fn; do\n>         lines=$(wc -l <\"$fn\")\n>         mv \"$fn\" \"$fn.orig\"\n>         nr=1\n>         while test $nr -le $lines; do\n>                 sed \"${nr}s/$/ is-matched/\" <\"$fn.orig\" >\"$fn\"\n>                 git ls-files | git check-attr --stdin is-matched |\n>                 grep -q \"is-matched: set\" ||\n>                 echo \"$fn:$nr $(sed -n ${nr}p \"$fn.orig\")\"\n>                 nr=$((nr+1))\n>         done\n>         mv \"$fn.orig\" \"$fn\"\n>   done\n>\n> It produces no output in git.git (we are using all of our attributes),\n> but you can add a useless one like:\n>\n>   echo '*.c -diff' >>Documentation/.gitattributes\n>\n> and then the loop yields:\n>\n>   Documentation/.gitattributes:2 *.c -diff\n>\n> So I definitely wouldn't call any of that \"easy\", but it may help you.\n>\n> -Peff\n"},{"id":"478797","messageId":"xmqqo7l25ibw.fsf@gitster.g","threadId":"59904","inReplyTo":"CAHGBnuPO63Hi8mfA+MkAGES-gs0eNCDPG2FcPZT=YsnVzKd30A@mail.gmail.com","subject":"Re: Clean up stale .gitignore and .gitattribute patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-26T15:55:31Z","receivedAt":"2023-06-26T15:55:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sebastian Schuberth <sschuberth@gmail.com> writes:\n\n> Thanks Peff for the suggestion. I ended up scripting something via\n> JGit [1], as we're anyway using it as part of our Gradle build system.\n>\n> PS: As a future idea, it might be good if \"git mv\" gives a hint about\n> updating .gitattributes if files matching .gitattributes pattern are\n> moved.\n\nInteresting.  \"git mv hello.jpg hello.jpeg\" would suggest updating\na \"*.jpg <list of attribute definitions>\" line in the .gitattributes\nto begin with \"*.jpeg\"?\n\n"},{"id":"478825","messageId":"CAHGBnuMjCsMetCJfhfDXb7aYttgUOc0WY+wJ_Q-tmoV4WES-pQ@mail.gmail.com","threadId":"59904","inReplyTo":"xmqqo7l25ibw.fsf@gitster.g","subject":"Re: Clean up stale .gitignore and .gitattribute patterns","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2023-06-26T16:42:17Z","receivedAt":"2023-06-26T16:42:33Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On Mon, Jun 26, 2023 at 5:55 PM Junio C Hamano <gitster@pobox.com> wrote:\n\n> > PS: As a future idea, it might be good if \"git mv\" gives a hint about\n> > updating .gitattributes if files matching .gitattributes pattern are\n> > moved.\n>\n> Interesting.  \"git mv hello.jpg hello.jpeg\" would suggest updating\n> a \"*.jpg <list of attribute definitions>\" line in the .gitattributes\n> to begin with \"*.jpeg\"?\n\nYes, right. Or as a simpler variant to start with (as patterns might\nmatch files in different directories, and not all of the matching\nfiles might be moved), just say that a specific .gitattributes line\nneeds updating (or needs to be duplicated / generalized in case files\nin both the old and new location match).\n\n-- \nSebastian Schuberth\n"},{"id":"478864","messageId":"20230627065103.GA1226768@coredump.intra.peff.net","threadId":"59904","inReplyTo":"CAHGBnuMjCsMetCJfhfDXb7aYttgUOc0WY+wJ_Q-tmoV4WES-pQ@mail.gmail.com","subject":"Re: Clean up stale .gitignore and .gitattribute patterns","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-27T06:51:03Z","receivedAt":"2023-06-27T06:51:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 26, 2023 at 06:42:17PM +0200, Sebastian Schuberth wrote:\n\n> On Mon, Jun 26, 2023 at 5:55 PM Junio C Hamano <gitster@pobox.com> wrote:\n> \n> > > PS: As a future idea, it might be good if \"git mv\" gives a hint about\n> > > updating .gitattributes if files matching .gitattributes pattern are\n> > > moved.\n> >\n> > Interesting.  \"git mv hello.jpg hello.jpeg\" would suggest updating\n> > a \"*.jpg <list of attribute definitions>\" line in the .gitattributes\n> > to begin with \"*.jpeg\"?\n> \n> Yes, right. Or as a simpler variant to start with (as patterns might\n> match files in different directories, and not all of the matching\n> files might be moved), just say that a specific .gitattributes line\n> needs updating (or needs to be duplicated / generalized in case files\n> in both the old and new location match).\n\nYeah, I don't think we could ever do anything automated here; a human\nneeds to judge the intent and how the patterns should be adapted.\n\nBut perhaps something like:\n\n  1. When git-commit makes a new commit that removes paths (whether they\n     were totally removed, or renamed), find all gitattribute lines\n     whose patterns match those paths.\n\n  2. For each such pattern, see if it still matches anything in the\n     resulting tree.\n\n  3. If not, print advise() lines showing the file/line of the pattern\n     which is no longer used.\n\nDoing so naively (by checking matches for each file in the tree) would\nbe a little expensive, but maybe OK in practice. It could perhaps be\ndone more efficiently with specialized code, but it might be tricky to\nright (and you still end up O(size of tree) in the worst case, because\nsomething like \"*.jpg\" needs to be compared against every entry).\n\nOf course on the way there you should end up with a decent tool for\n\"which patterns are not currently used?\". And you could just\nperiodically run that manually if you want to clean up (or even from a\npost-commit hook).\n\n\nRe-reading your email, though, I wonder if you meant something a little\nsimpler, like:\n\n  1. When a path is moved via git-mv, see if the attributes before/after\n     are the same.\n\n  2. If not, then mention which ones matched the old path via advise().\n\nThat is probably easier to write, though it does not help the \"git rm\"\ncase (where attributes may become obsolete).\n\n-Peff\n"},{"id":"478865","messageId":"CAHGBnuMKDE6nngaoajGfpViXy78toU4WCV_QNGvy-jqXuEaAZA@mail.gmail.com","threadId":"59904","inReplyTo":"20230627065103.GA1226768@coredump.intra.peff.net","subject":"Re: Clean up stale .gitignore and .gitattribute patterns","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2023-06-27T06:55:54Z","receivedAt":"2023-06-27T06:56:19Z","isPatch":false,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On Tue, Jun 27, 2023 at 8:51 AM Jeff King <peff@peff.net> wrote:\n\n> Re-reading your email, though, I wonder if you meant something a little\n> simpler, like:\n\nIndeed, I was only having the \"git mv\" case in mind and to advise() at\nthe time of that command being run, instead of advise()'ing at \"git\ncommit\" time.\n\n-- \nSebastian Schuberth\n"}]}