git-status and git-diff now very slow in project with a submodule

Hello,

I've just upgraded from 1.6.something to 1.7.1.  All very nice.

The new submodule reporting is nice too; but I'd like to be able to turn it off :-)

The problem is that I have a (relatively) small project as the supermodule, and a linux kernel clone as a submodule and an ffmpeg clone as a submodule. Now I used to be able to do git-status or git-diff and it would be instant, it now takes a number of seconds to report. I guess (but don't know), that it is the detection of "dirty" status in the submodule's that is slowing down the supermodule processing.

I wouldn't like to see the feature go, because in almost all circumstances it is exactly right; however, I'd like to be able to turn off dirty detection in submodules. Is this already possible, and I've just missed the configuration option?

One additional small point: why do untracked files in a submodule make the module dirty? I've often got a few "temp.ps" or "debug.log" or "backtrace.log" files lying around -- inappropriate to add to an ignore file, but they don't make my working directory dirty. "Dirty" in a working directory means uncommitted changes to tracked files, why does it mean something different in a submodule?

Andy

On 5/20/2010 12:01 PM, Andy Parkins wrote:

Maybe:

   git config status.submodulesummary false

That's IMHO a good point. I'd like to get an answer for that, too.

Regards,
  Stefan

Stefan Naewe wrote:

Hey! Thanks for the reply. Exactly the right option... except it doesn't work :-(

$ git --version git version 1.7.1 $ git config status.submodulesummary false $ git status -uno # On branch master # Changed but not updated: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # (commit or discard the untracked or modified content in submodules) # # modified: ffmpeg (modified content) # no changes added to commit (use "git add" and/or "git commit -a")

Andy

Andy Parkins venit, vidit, dixit 20.05.2010 13:37:

You see: No submodule summary here! Try setting the variable to true and see the difference. False is the default.

Git needs to check the submodule in order to produce the "modified" line even when no summary is required. Stopping Git from looking at the submodule at all is impossible, I think. One could only hope that it stops scanning after the first modification.

Michael

Michael J Gruber wrote:

Quite so; I hadn't understood submodulesummary -- I just tried it when it was suggested.

I realise that -- what I was after is a return to the old behaviour -- under the control of an option.

"Impossible" is a strong word for a behaviour that existed pre-1.7.

It's not that I want git not to look at the submodule at all; in fact it certainly should for those cases when the submodule commit has changed, and I guess that a check for a dirty index is pretty quick too; but scanning the whole submodule tree (which it has to do to find if anything was modified, even when nothing was modified) is a lot of extra time when the submodule is large. That extra time is inconvenient when you're working on a small project that makes use of a large project as a submodule. (Most of my personal use of submodule is embedding large projects that I want to be able to guarantee are at a particular version, but I don't really change them)

Andy

Am 20.05.2010 19:45, schrieb Andy Parkins:

But to guarantee they are at a particular version they have to be checked for local modifications (no matter if they happened accidentally or on purpose), no?

On Thursday 20 May 2010 18:49:41 Jens Lehmann wrote:

A valid point.

Surely though in my own .git/config I can be allowed to tell git that I don't care about that risk?

I've got a top level module that used to diff/status instantly; now git scans an entire Linux kernel checkout and an entire ffmpeg checkout. Painful. I fully accept that it was my own choice to arrange my repository in this way, but in my defence, it was fine last week :-)

Andy

On Thu, May 20, 2010 at 8:01 PM, Andy Parkins <andyparkins@gmail.com> wrote:

Sounds similar to the old "slow lstat" stuff. How about setting assume-unchanged bit on submodules? I don't know if it works. If it does not, maybe we could reuse the bit to ignore changes in submodules.

Andy Parkins <andyparkins@gmail.com> writes:

"They don't make my working directory dirty" is something only you can decide, until you tell git about that fact, isn't it?

The way to tell git about them is to use the ignore/exclude mechanism. Why are they "inappropriate to add to an ignore file"? At least you could have "*.log" in your personal exclude $GIT_DIR/info/exclude, no?

As to the not-working-configuration I don't remember the codepath well, so sorry but no answer from me right now.

On Thursday 20 May 2010 14:28:35 Junio C Hamano wrote:

Perhaps I've misunderstood then; I have always understood that "dirty" was the name we give to the state when tracked files have changes in the working directory. If not, then what word should be used to distinguish between tracked files unchecked in and untracked files?

Anyhoo; I don't mind.  Me starting a semantics debate isn't helpful is it?

I think you've taken me too literally; I was trying to get across the idea that they are files that are made on the fly, and when I notice them they just get deleted.

Also, I don't want *.log, or *.ps -- neither of them is guaranteed to be an ignore pattern. These throw away files have all sorts of names, made up on the spot as I'm working, adding them to an ignore file is overkill from my point of view.

Andy

Andy Parkins <andyparkins@gmail.com> writes:

I've learned to give them names that begin with an unusual letter (in my case, a colon or a plus sign), way before I started working on git, so the above is not a very convincing argument at least to me.

In any case, I am a bit torn about this whole issue.

On one hand, scanning for untracked files are not about these *.log cruft, but to catch mistakes that are caused by new paths that you forgot to add, and in that sense, uncommitted modifications to a path that happens to be tracked and a new path that you forgot to add have (semantically) similar chance of being a mistake that you might want to catch by running "git status".

On the other hand, adding new paths to an existing project is a rare event, compared to modifications to existing paths (e.g. even for a project as small and young as git.git, we have 10x as many revisions as we have paths), so by definition the chance that you might break others' builds by forgetting to commit a new file is much smaller than forgetting to commit necessary changes to existing files.

But ideally you would want your tool to catch mistakes that are rarer, as you would learn to avoid common mistakes on your own without help from your tool over time.

At least we should be able to let the users say, with "git status -uno", "I don't care about untracked and unignored paths; I don't make such a mistake to forget adding new paths", and optimize the scanning of submodule directories taking advantage of that statement. Is there a fundamental reason why things shouldn't work that way, or is it just a bug in the current code?

Am 21.05.2010 00:59, schrieb Junio C Hamano:

It works that way since 3bfc450. "git status" (and the diff family when comparing against the work tree) forks a "git status" for each submodule that is populated. If the status command in the superproject is started with "-uno" the same option is used for the "git status" forked in the populated submodules, so no checking for untracked files is done in that case.

But that doesn't speed up that process much, as the tracked files inside the submodules have to be checked for modifications anyway, no matter if "-uno" is used or not. Getting rid of the fork of a new "git status" by using an alternate odb is on my to do list. Apart from gaining some time by avoiding the fork (which is a per-submodule-constant) we could terminate early in case we find a modification (instead of continuing as the current approach does). But only dirty submodules would profit from that, a clean submodule won't be scanned much faster that way AFAICS.

On 2010-05-20, Junio C Hamano <gitster@pobox.com> wrote:

Please, correct me if I am wrong, but I always thought that repo's dirty designation has to do with changed files _in_ the repo. Untracked files are _not_ in the repo and, as such, are irrelevant for repo's clean/dirty status.

Speaking of .gitignore and untracked files. Explicitly mentioning all such untracked files in .gitignore is often unpractical. For example, during build process some large projects autogenerate many temporary *.c *.h *.cpp files. Hunting all of them down and adding to .gitignore is a waste of time and one cannot use globs *.c *.h for obvious reasons.

I think that it is consistent and logical to drop untracked files from repo's dirty/clean decision process.

I would be interested to know of any counter-example: that is, a use-case where it makes logical sense to declare a repo dirty when it gets an untracked file not mentioned in .gitignore.

--Leo--

Leo Razoumov <slonik.az@gmail.com> writes:

You can actually, since tracked files are never ignored.

Andreas.

Am 21.05.2010 19:54, schrieb Andreas Schwab:

Hm, but that would mean newly added files would never show up again (the same .gitignore is used by the developers of that submodule when they do work on it). So I wouldn't add *.c to .gitignore either ...

But I don't consider it a "waste of time" to get the .gitignore straight for pretty much the same reasons I want to see no warnings during compilation: So i can instantly see when something fishy might be going on. I consider that a "best practice" instead.

And for those projects where you can't or don't want to change the .gitignore: Just ignore when "git status" tells you a submodule has "untracked content" (it shows that information since 9297f7).

Am 21.05.2010 14:52, schrieb Leo Razoumov:

For submodules the presence of untracked files must be visible in the superproject. When new files are added to a submodule but the user forgot to commit them, the superproject might not even build at all when another developer clones the superproject. And yes, this is a real-life use case from my dayjob.

The alternative to these broader definitions about when to declare a submodule dirty (new commits, modified content and untracked content) would have been to handle all eight combinations of these states. In all relevant parts of the toolchain. Which seems pretty insane. So the submodule shows up as modified; and all but the short outputs of diff and status also tell you /why/ it is considered modified. So the user can decide what to do about that.