{"thread":{"id":"32761","subject":"[RFC] The design of new pathspec features","startedAt":"2013-01-29T04:35:17Z","lastAt":"2013-01-29T16:27:05Z","messageCount":10,"participants":["Duy Nguyen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"208148","messageId":"20130129043517.GA2878@duynguyen-vnpc.dek-tpc.internal","threadId":"32761","inReplyTo":null,"subject":"[RFC] The design of new pathspec features","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-01-29T04:35:17Z","receivedAt":"2013-01-29T04:35:17Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"For those who haven't followed closely, some coming changes allow us\nto extend current pathspec syntax. We should soon be able to do\ncase-insenstive matching for example, or introduce \"**\" wildcard that\nis currently used by gitignore. I just want to discuss about the new\nsyntax and behavior.\n\nMany of these are already implemented in [1]. But I don't want you to\nbother with buggy code yet. I'll resend it soon after 1.8.2.\n\n--literal-pathspecs\n===================\n\nThis feature is added by Jeff to disable globbing for all pathspecs. I\nwant to push it a bit further: disable all pathspec magic. This means\neven \":/\" is treated literally when --literal-pathspecs is set.\nBecause the intent behind this, as I understand, is for scripting, it\nmakes sense to keep it as literal as possible.\n\n:(literal) magic\n================\n\nThis magic is for people who want simple no-globbing pathspec (*). It\ncan be used in combination with other magic such as case-insensitive\nmatching. Incompatible with :(glob) magic below.\n\nGlobal option --noglob-pathspecs is added to add :(literal) to\nall. This is very similar to --literal-pathspecs. It just does not\ndisable pathspec magic. :(glob) magic overrides this global option.\n\n(*) you can always disable wildcards by quoting them using backslash,\nbut that's inconvenient\n\n:(glob) magic\n=============\n\nThis magic is for people who want globbing. However, it does _not_ use\nthe same matching mechanism the non-magic pathspec does today. It uses\nwildmatch(WM_PATHNAME), which basically means '*' does not match\nslashes and \"**\" does.\n\nGlobal option --glob-pathspecs is added to add :(glob) to all\npathspec. :(literal) magic overrides this global option.\n\nfnmatch without FNM_PATHNAME is deprecated\n==========================================\n\nWith the two magic above, people can switch between literal and new\nglobbing. There is no way to regain current matching behavior once\n--[no]glob-pathspecs is used. And I think that's a good thing. New\nglobbing is more powerful than the current one. At some point, I'd\nlike to switch the matching behavior when neither literal nor magic\npathspec is specified. Either:\n\n - make it literal by default\n - make it new globbing by default\n\nWhich is more often used should be come the default. The question is\nwhich.\n\nPathspec mnemonic\n=================\n\nAre :(literal) and :(glob) used often enough to deserve a short\nmnemonic (like :/ is equivalent to :(top))? Which symbols should be\nused? We can only use non-alphanumeric here, and '(', ')', ':' and '/'\nare taken. It should be friendly to UNIX shell, no quoting is\npreferred.\n\nAnother magic will come soon: case-insensitive matching. We may want\nto reserve a mnemonic symbol for it as well.\n\nWe may also want to reserve option shortcuts for --noglob-pathspecs\nand --glob-pathspecs. I suspect they'll be used more often.\n\nNew way to specify long pathspec magic\n======================================\n\nWhile testing the pathspec magic code, I grow tired of quoting :(glob)\nevery time because '(' is the start of a new shell. Which is one of\nthe reasons I introduce --[no]glob-pathspecs. Still I'd like a way to\nspecify long pathspec magic without quoting.\n\nHow about making \":q/xxx/\" an equivalence of \":(xxx)\"? Any character\n(except ',') following 'q' is used as separator (similar to s/// from\nsed). This violates the guidelines set in glossary-content.txt (only\nuse non-alphanumeric).\n\nAnother step futher is remove support for \":(xxx)\" in favor of\n\":q(xxx)\". We can do it today because I don't think anybody is using\n\":(top)\" (the only supported magic) yet.\n\n[1] https://github.com/pclouds/git/tree/parse-pathspec\n"},{"id":"208149","messageId":"7vpq0otw7y.fsf@alter.siamese.dyndns.org","threadId":"32761","inReplyTo":"20130129043517.GA2878@duynguyen-vnpc.dek-tpc.internal","subject":"Re: [RFC] The design of new pathspec features","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-29T05:05:37Z","receivedAt":"2013-01-29T05:05:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> Pathspec mnemonic\n> =================\n>\n> Are :(literal) and :(glob) used often enough to deserve a short\n> mnemonic (like :/ is equivalent to :(top))? Which symbols should be\n> used?\n\nI do not think we should discuss this before letting people gain\nexperience with various forms of magic; otherwise we would at best\nend up with a concensus guess among uninformed.\n\n> New way to specify long pathspec magic\n> ======================================\n>\n> While testing the pathspec magic code, I grow tired of quoting :(glob)\n> every time because '(' is the start of a new shell. Which is one of\n> the reasons I introduce --[no]glob-pathspecs. Still I'd like a way to\n> specify long pathspec magic without quoting.\n\nIs this a real issue, though? Often interesting pathspecs do have\nshell globs in them and we have to quote them anyway.\n\n> How about making \":q/xxx/\" an equivalence of \":(xxx)\"?\n\nA moderately strong no from here.\n"},{"id":"208150","messageId":"7vlibctw53.fsf@alter.siamese.dyndns.org","threadId":"32761","inReplyTo":"20130129043517.GA2878@duynguyen-vnpc.dek-tpc.internal","subject":"Re: [RFC] The design of new pathspec features","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-29T05:07:20Z","receivedAt":"2013-01-29T05:07:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> For those who haven't followed closely, some coming changes allow us\n> to extend current pathspec syntax. We should soon be able to do\n> case-insenstive matching for example, or introduce \"**\" wildcard that\n> is currently used by gitignore. I just want to discuss about the new\n> syntax and behavior.\n>\n> Many of these are already implemented in [1]. But I don't want you to\n> bother with buggy code yet. I'll resend it soon after 1.8.2.\n\nI think I agree with everything up to the \"fnmatch(~FNM_PATHNAME)\"\npart, from a quick read.\n\nThanks for writing this up.\n"},{"id":"208152","messageId":"CACsJy8BPhHJseDghWjCe1xK_KqSZOz1cfoOfPnTdex94qqrCzw@mail.gmail.com","threadId":"32761","inReplyTo":"7vpq0otw7y.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC] The design of new pathspec features","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-01-29T05:27:44Z","receivedAt":"2013-01-29T05:27:44Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jan 29, 2013 at 12:05 PM, Junio C Hamano wrote:\n>> New way to specify long pathspec magic\n>> ======================================\n>>\n>> While testing the pathspec magic code, I grow tired of quoting :(glob)\n>> every time because '(' is the start of a new shell. Which is one of\n>> the reasons I introduce --[no]glob-pathspecs. Still I'd like a way to\n>> specify long pathspec magic without quoting.\n>\n> Is this a real issue, though? Often interesting pathspecs do have\n> shell globs in them and we have to quote them anyway.\n\n:(icase) often won't (unless you combine with :(glob)). If we turn\ngrep's --max-depth feature into pathspec magic (feasible, just not\nsure if it's actually useful), it won't need quoting either.\n\nEven with :(glob), because of the complexity of the pathspec, it's\nless likely to match anything and be expanded by bash, so no quoting\nrequired.\n\n>> How about making \":q/xxx/\" an equivalence of \":(xxx)\"?\n>\n> A moderately strong no from here.\n-- \nDuy\n"},{"id":"208153","messageId":"7vd2wotuxm.fsf@alter.siamese.dyndns.org","threadId":"32761","inReplyTo":"20130129043517.GA2878@duynguyen-vnpc.dek-tpc.internal","subject":"Re: [RFC] The design of new pathspec features","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-29T05:33:25Z","receivedAt":"2013-01-29T05:33:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> :(literal) magic\n> ================\n>\n> This magic is for people who want simple no-globbing pathspec (*). It\n> can be used in combination with other magic such as case-insensitive\n> matching. Incompatible with :(glob) magic below.\n>\n> Global option --noglob-pathspecs is added to add :(literal) to\n> all. This is very similar to --literal-pathspecs. It just does not\n> disable pathspec magic. :(glob) magic overrides this global option.\n>\n> (*) you can always disable wildcards by quoting them using backslash,\n> but that's inconvenient\n\nHave you considered if it may be helpful to have a :(literal) magic\n(or any magic in general) that applies only to the first N\ncharacters of the pathspec pattern?\n\nWhen you are in subdirectory and do a pathspec limited operation,\ne.g.\n\n cd Documentation && git ls-files \"*.txt\"\n\nwe internaly do an equivalent of this:\n\n (1) first find out the \"prefix\", e.g. \"Documentation/\" in this\n case;\n (2) prepend the prefix to user-supplied pathspecs, e.g. yielding\n \"Documentation/*.txt\" in this case; and\n (3) use the resulting pathspecs to match against full pathnames\n relative to the root of the working tree.\n\nIf the prefix had globbing character in it (e.g. we started in a\ndirectory \"D*cumentati*n\" instead), we still should make sure that\nthat part matches literally, while allowing the globbing in\nuser-supplied part of the pathspec (e.g. \"*.txt\"). In the built in\ncode, you can work with the struct pathspec directly and mark the\nentire prefix part with nowildcard_len field to match literally, but\nif the above three-step logic needs to be implemented by a Porcelain\nscript like old days, they would need to quote glob specials in the\nprefix part before appending user-supplied part to form the full\npathspec string.\n\nI personally think we do not need to support something like this:\n\n\tprefix=$(git rev-parse --show-prefix)\n\tn=${#prefix}\n pathspec=\":(literal-$n)$prefix$1\"\n\nbut other aspiring Porcelain script writers may disagree and would\nwant to have it. We can always solve it by giving them an easy and\nuniform way to get the glob-quoted version of prefix to solve this\nparticular issue, i.e.\n\n\tprefixq=$(git rev-parse --show-prefix-glob-quoted)\n pathspec=\"$prefixq$1\"\n\nbut magic that applies only to a substring may have other uses.\n\nIf you do not immediately think of any, let's not overengineer this.\n\nThanks.\n"},{"id":"208155","messageId":"CACsJy8DsKFfGoPUt5b=3MuSmZ2z2Z5vJ20D_kqEowAUc8QTryA@mail.gmail.com","threadId":"32761","inReplyTo":"7vd2wotuxm.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC] The design of new pathspec features","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-01-29T06:13:37Z","receivedAt":"2013-01-29T06:13:37Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jan 29, 2013 at 12:33 PM, Junio C Hamano wrote:\n> Have you considered if it may be helpful to have a :(literal) magic\n> (or any magic in general) that applies only to the first N\n> characters of the pathspec pattern?\n\nNot user-driven. But the prefix part is :(literal)-applied. :(glob) is\ncurrently implemented this way, using nowildcard_len as you mentioned.\n:(icase) is more complicated and does not follow yet.\n\n> I personally think we do not need to support something like this:\n>\n> prefix=$(git rev-parse --show-prefix)\n> n=${#prefix}\n> pathspec=\":(literal-$n)$prefix$1\"\n>\n> but other aspiring Porcelain script writers may disagree and would\n> want to have it. We can always solve it by giving them an easy and\n> uniform way to get the glob-quoted version of prefix to solve this\n> particular issue, i.e.\n>\n> prefixq=$(git rev-parse --show-prefix-glob-quoted)\n> pathspec=\"$prefixq$1\"\n>\n> but magic that applies only to a substring may have other uses.\n\nYeah, that simplifies things. Supporting applying magic over just\nparts of the pathspec pattern sounds complex. Just a small\nmodification. That rev-parse needs to look at \"$1\" as well. If\n:(literal) is already specified, glob quoting will backfire. The user\nscript can deal with that, but it's harder (e.g. parsing magic from\nscripts and deal with magic combination) than letting rev-parse does\nit.\n\nI've done some form of this already, for supporting add--interactive.\ngit-add prefixes the pathspec but keeps all the magic in place, before\npassing pathspec to add--interactive. But I missed the quoting point\nyou mentioned above. I probably need de-quoting the prefix as well.\nMany optimizations stop short at the sign of any glob symbols,\nincluding backslash. This could be a new task for wildmatch.\n-- \nDuy\n"},{"id":"208156","messageId":"7v4ni0ts90.fsf@alter.siamese.dyndns.org","threadId":"32761","inReplyTo":"CACsJy8DsKFfGoPUt5b=3MuSmZ2z2Z5vJ20D_kqEowAUc8QTryA@mail.gmail.com","subject":"Re: [RFC] The design of new pathspec features","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-29T06:31:23Z","receivedAt":"2013-01-29T06:31:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n>> prefixq=$(git rev-parse --show-prefix-glob-quoted)\n>> pathspec=\"$prefixq$1\"\n>>\n>> but magic that applies only to a substring may have other uses.\n>\n> Yeah, that simplifies things. Supporting applying magic over just\n> parts of the pathspec pattern sounds complex. Just a small\n> modification. That rev-parse needs to look at \"$1\" as well.\n\nMakes sense. More like\n\n\tprefix=$(git rev-parse --show-prefix)\n\ttest -z \"$prefix\" || cd $(git rev-parse --show-cdup)\n\t...\n\tpathspec=$(git rev-parse --prefix-pathspec \"$prefix\" \"$1\")\n\tgit ls-files \"$pathspec\"\n\nwhich may take prefix=\"D*cumentati*n\" (from the previous example),\nand the user may have given \":(icase)hell*.txt\" to \"$1\". Ideally\nthe scriptors shouldn't have to worry about how to add the prefix to\nthe user supplied pathspec.\n"},{"id":"208183","messageId":"CACsJy8A5VJ4EchbeFEC5ikLYgHze47_xyLTtgH=kqVpP15Hogw@mail.gmail.com","threadId":"32761","inReplyTo":"20130129043517.GA2878@duynguyen-vnpc.dek-tpc.internal","subject":"Re: [RFC] The design of new pathspec features","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-01-29T12:17:10Z","receivedAt":"2013-01-29T12:17:10Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jan 29, 2013 at 11:35 AM, Duy Nguyen wrote:\n> :(glob) magic\n> =============\n>\n> This magic is for people who want globbing. However, it does _not_ use\n> the same matching mechanism the non-magic pathspec does today. It uses\n> wildmatch(WM_PATHNAME), which basically means '*' does not match\n> slashes and \"**\" does.\n>\n> Global option --glob-pathspecs is added to add :(glob) to all\n> pathspec. :(literal) magic overrides this global option.\n\nI forgot one thing. The current pathspec behavior, the pattern \"[a-z]\"\nwould match a file named \"[a-z]\" (iow, wildcards are also considered\nliteral characters). Do we want to keep this behavior in :(glob)? To\nme it's a surprise factor and therefore not good. But common sense may\nbe different.\n-- \nDuy\n"},{"id":"208194","messageId":"7vk3qwrmea.fsf@alter.siamese.dyndns.org","threadId":"32761","inReplyTo":"CACsJy8A5VJ4EchbeFEC5ikLYgHze47_xyLTtgH=kqVpP15Hogw@mail.gmail.com","subject":"Re: [RFC] The design of new pathspec features","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-29T16:20:45Z","receivedAt":"2013-01-29T16:20:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Tue, Jan 29, 2013 at 11:35 AM, Duy Nguyen wrote:\n>> :(glob) magic\n>> =============\n>>\n>> This magic is for people who want globbing. However, it does _not_ use\n>> the same matching mechanism the non-magic pathspec does today. It uses\n>> wildmatch(WM_PATHNAME), which basically means '*' does not match\n>> slashes and \"**\" does.\n>>\n>> Global option --glob-pathspecs is added to add :(glob) to all\n>> pathspec. :(literal) magic overrides this global option.\n>\n> I forgot one thing. The current pathspec behavior, the pattern \"[a-z]\"\n> would match a file named \"[a-z]\" (iow, wildcards are also considered\n> literal characters).\n\nThat sounds like a blatant bug to me (unless you are talking about\n\"literal\" case). We should fix it before we include it in any\nreleased version, I think.\n"},{"id":"208195","messageId":"7vd2worm3q.fsf@alter.siamese.dyndns.org","threadId":"32761","inReplyTo":"7vk3qwrmea.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC] The design of new pathspec features","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-29T16:27:05Z","receivedAt":"2013-01-29T16:27:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Duy Nguyen writes:\n>\n>> On Tue, Jan 29, 2013 at 11:35 AM, Duy Nguyen wrote:\n>>> :(glob) magic\n>>> =============\n>>>\n>>> This magic is for people who want globbing. However, it does _not_ use\n>>> the same matching mechanism the non-magic pathspec does today. It uses\n>>> wildmatch(WM_PATHNAME), which basically means '*' does not match\n>>> slashes and \"**\" does.\n>>>\n>>> Global option --glob-pathspecs is added to add :(glob) to all\n>>> pathspec. :(literal) magic overrides this global option.\n>>\n>> I forgot one thing. The current pathspec behavior, the pattern \"[a-z]\"\n>> would match a file named \"[a-z]\" (iow, wildcards are also considered\n>> literal characters).\n>\n> That sounds like a blatant bug to me (unless you are talking about\n> \"literal\" case). We should fix it before we include it in any\n> released version, I think.\n\nAh, no, I misread what you meant.\n\nYes, historically when matching a path with a pattern \"a/b[c-d]/e\",\nwe tried to first literally match the path with it (this allows the\npath \"a/b[c-d]/e\" to match with the pattern), and if it did not\nmatch, used fnmatch(3) to allow \"a/bc/e\" to also match. This was an\nugly hack to cope with the possiblity that such a funny path is\nactually used in projects.\n\nWith :(literal) magic and its friends you are working into the\nsystem, the hack should disappear at the same time when you stop\nrunning fnmatch(3) without FNM_PATHNAME. That should happen no\nlater than Git 2.0.\n\nBut until then, we should keep matching both paths \"a/bc/e\" and\n\"a/b[c-d]/e\" with pattern \"a/b[c-d]/e\", for backward compatibility.\n\nSorry about the earlier confusion.\n"}]}