Request feature: –no-submodule

Hello Git community.

As you already know, git ls-files command lists all of the tracked files, but submodule names are included.

My team would like a –no-submodule switch to exclude those.

Best rewards, Ilias Apostolou

Github: https://github.com/LiakosC

On Wed, Jun 02, 2021 at 01:31:11PM +0300, Ilias Apostolou wrote:

In all honesty, though this seems like a niche request for ls-files to fulfill, ls-files already has quite the collection of options, so I wouldn't be sad to see it learn how to do this, too.

The change boils down to having builtin/ls-files.c:show_ce() avoid printing out submoudles in when '--exclude-submoudles' is given). Something like this:

--- >8 ---

Taylor Blau <me@ttaylorr.com> writes:

I would be somewhat sad for two reasons.

 - If "I am not interested in any submodule" in a project with
   submodules is a common thing people would want, teaching a trick
   only to "ls-files" is an expensive and ineffective approach, and
   adding the option to everything would just be ugly.  "git diff
   --no-submodule"?  "git add --no-submodule ."?

 - Is "not interested in any submodule" so special and fundamental,
   or is it merely because the project the original requestor is
   looking at happens to have an optional submodule? If the project
   had that optional part as a subdirectory instead, would the
   request have been not --no-submodule but something else?  What
   happens when the project that led to the original request
   acquires another submodule that is more interesting, or what if
   the requestor's interest shifts and makes some submodules
   interesting but others not?  Would the --no-submodule option
   become totally useless in such a case?

I wonder if the "attr" magic of the pathspec, that allows you to choose paths based on the attributes you set on them, is what the original requestor missed.

On Thu, Jun 03, 2021 at 09:55:57AM +0900, Junio C Hamano wrote:

Mmm, that's a convincing set of reasons to think that this is a bad idea. (And I was sort of on the fence about it anyway by posting the whole thing as a short diff instead of a polished patch).

Maybe... but relying on the attr magic for this particular case would force the requester to set that attribute on all submodules in their project, and constantly keep that in-sync with their .gitmodules. So, it certainly make it easier to handle a request like "I don't care about ls-files telling me about any submodule(s) except these ones", but perhaps at the expense of some extra bookkeeping.

I might be missing something, though.

Thanks, Taylor

The reason we need to list all of the true files (except submodules) is for code refactoring using the sed unility, for example:

git ls-files | grep -Ev '(png$|ico$)' | xargs sed -i 's/\r\n/\n/'

All of the other alternatives we could think of are very ugly.

On 3/6/2021 5:33 π.μ., Taylor Blau wrote:

In my opinion, this should be a feature for "ls-files" only, since it would be nice to have a clean stream of true files.

Thank you for your replies, Ilias

Ilias Apostolou <ilias.apostolou.zero@gmail.com> writes:

Ah, then pathspec is still the right tool to use, I would think.

For example,

    git ls-files -s ':(exclude)*.png' ':(exclude)*.ico' |
    sed -n -e 's/^100[76][54][54] [0-9a-f]* 0       //p' |
    xargs sed -i 's/\r\n/\n/'

that is,

 (1) ls-files -s can be used to learn what kind of path it is.
     Regular files are either 100644 or 100755.  That way, you can
     also exclude symbolic links, which your example use case would
     probably not want to touch.  And you can filter the output to
     have 'a clean stream of true files' fairly easily Depending on
     the details of your needs, it can be tweaked into 'a clean
     stream of executable files' etc., too.

 (2) pathspec magic like ':(exclude)' can be used to lose your "we
     know png and ico are not text files".

Hope this helps.

Taylor Blau <me@ttaylorr.com> writes:

Well, it contradicts with the above "convincing" adjective, and shows that you are not convinced that "submodule"-ness is not all that essential and it merely is an artifact that the paths that the original requester happens to be uninterested in are all submodules.

But if we agree that focusing too narrowly on "submodule"-ness is a bad idea and open our mind to elsewhere, we'd realize that once we learn that we can "mark" any path with attributes and use that in magic pathspec, we can mark not just submodules but a subdirectory as "uninteresting", which will not become useless even when it turns out that "submodule"-ness wasn't really what the request was about.

Besides, you can iterate over the available submodules with "git submodule foreach" fairly mechanically, and maintaining the attribute per path shouldn't be all that hard, I would imagine.

On Fri, Jun 04, 2021 at 02:40:25AM +0900, Junio C Hamano wrote:

It doesn't seem outrageous to me for Git to automatically populate "pseudo-attributes" that connect properties of paths to the attribute system. I.e., could we just act as if every path that is a gitlink has the "gitlink" attribute set to true, and let people do:

  git ls-files ':(attr:-gitlink)'

That uses the existing generic mechanism, so it supports complex situations, but it also makes the "easy" case of "just ignore submodules" easy, with no attribute maintenance.

I didn't look at the code, though, so I'm not sure how awkward it would be to implement (usually we decide on attributes only from looking at the attribute files, not the trees/index themselves, but I think most code asking about a path would be iterating a list of files in the first place, and could feed the mode information).

An alternative view is allowing a pathspec that asks about the mode:

  git ls-files ':(mode=160000)'

That also lets you ask about other things, like:

  git ls-files ':(mode=100755)'

but it is probably unnecessarily arcane (even I had to look up the correct mode for a gitlink just now :) ).

-Peff

Jeff King <peff@peff.net> writes:

Sounds sensible. The attribute assignment was designed to purely depend on paths and not contents, so it might be a bit awkwarad to implement, but it should be doable.

The original request, as I understand the clarification posted upthread, is not "submodules are uninteresting", but is "we want regular files" (and we postprocess further the output), so such a "mode" (pseudo-)attribute that is automatically populated would be a better fit anyway. With the current system, they can already do:

    git ls-files -s ':(exclude)*.png' ':(exclude)*.ico)' |
    sed -n -e 's/^100[76][54][54] [0-9a-f]* 0       //p' |
    xargs dos2unix

(cf. <9cc98ca3-bdc5-61bf-450a-99bb47673d6c@gmail.com>)

and with such an auto-pseudo-attribute, presumably something along this line would work, removing the need for the intermediate filter:

    git ls-files \
	':(attr:mode=100755)' ':(attr:mode=100644)' \
	':(exclude)*.png' ':(exclude)*.ico' |
    xargs dos2unix

On Fri, Jun 04, 2021 at 06:54:10AM +0900, Junio C Hamano wrote:

Yeah, that makes sense.

By the way, another reason (beyond a simpler pipeline) that the "magic pathspec that understands modes" is nicer is that it can be applied to a more dynamic set of paths.

For instance, you could use a pipeline like the one you showed above to limit the ls-files output, and then you could feed that set of literal paths to a command like "git add" (perhaps with --literal-pathspecs). But you would not want to feed it to git-log like:

  git --literal-pathspecs log $(git ls-files -s | sed ...)

because you'd really want to expand the set of paths at each commit, not once based on the current index (i.e., it would fail to match paths that were removed or changed modes).

This is kind of a subtle and esoteric point, but it makes me more convinced that having powerful pathspec selectors is potentially useful. Or at least solving a problem that's hard to otherwise solve correctly. :)

-Peff

In the longer term, the project the original request wanted to invent the "--no-submodule" option for may want to lose the specific "we know that the only paths we do not want to run dos2unix happen to be png and ico files in the current codebase" from the above sample command line, and replace it with something like ':(attr:text)'

Alas, the "text" attribute does *not* work that way, though ;-) Just like any other attributes, what you assigned yourself counts, and our "is this file a text?" auto-detection logic only kicks in when there is no attribute that tells if a path is text or not.

It would be expensive at runtime; even if we were to introduce a "dynamic" pseudo attribute to tell text files and others apart, we probably shouldn't use the same "attr:*" magic but use something distinct (e.g. "dynamic-attr:*") in order to make sure that the users understand the performance implications. I think the above "mode=100755" matic (or anything that requires more than the basic "which entries in the .gitattributes files does this pathname match?") would fall into the same category.

If I understand correctly, we should learn to use git ls-files special parameters like 'attr' and 'exclude'. I'm impressed by the fact that these are not wide-spread known. We should learn to use these advanced features instead of extra piping. Your emails are precious to my team and will help us advance.

Please consider this request closed. Ilias