First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles

We have a diskimage/fileysystem that has a 50G Git repository + 900G of binary/build articles and untracked files. When we mount such a diskimage, The verify first “git status” command can take as long 40-50minutes. Subsequent “git status” finish in under 5-10 seconds.

If I had a diskimage of just the 50G source repository, and I mount and do a “git status” takes around 15 seconds.

How can we optimize this to be faster?

I suspect warming the filesystem caching is in play. But so is the fact that walk every tree to find untracked files as well.

My interest in git status is the 50G of sources/repository not the 900G of build generated articles in the workspace? I have tried adding .gitignore to whole directory trees that contain build artcles, 700G are excluded using git ignores, and that still drops the time for git status to only 30 minutes, which is high. time git status -uno --ignored=no

Any suggestions on how to root cause and optimize this case?

Thanks, Sarvi Occam’s Razor Rules

"Saravanan Shanmugham (sarvi)" <sarvi@cisco.com> writes:

Enable the untracked cache and "update-index --refresh", before freezing the repository + working tree state in the diskimage?

Thanks. That seems to help.

Just for context, we are using a copy-on-write/cloning solution to give developers a prebuilt workspace with all the platforms fully built, as well as sources all cloned, So they are ready for incremental development.

We create a ext4 diskimage, with a git clone workspace(50G) and 900G of fully built trees, then freeze that diskimage. Developers have a way to clone that diskimage(into a copy-on-write) copy of the diskimage(in 30 seconds) that they can mount and use for incremental build development.

I did the following.

Mounted the existing filesystem git update-index --untracked-cache git update-index --refresh

Not sure what you meant by " working tree state in the diskimage"

I then detached the diskimage, dropped all the caches. Remounted the diskimage.

git status  - The very first git status after dropping the cache now returns in in 1.54 seconds, which seems very much acceptable.

I obviously need to do more read up on what/how git caches.

Can you point to any documentation on what sort of information GIT caches, and how to understand debug that?
I would like to understand what the above git update-index commands actually do, that is making this faster.
 
Thanks,
Sarvi
Occam’s Razor Rules

On 8/22/19, 11:13 AM, "Junio C Hamano" <gitster@pobox.com> wrote:

    "Saravanan Shanmugham (sarvi)" <sarvi@cisco.com> writes:
    
    > I suspect warming the filesystem caching is in play.
    > But so is the fact that walk every tree to find untracked files as well. 
    
    Enable the untracked cache and "update-index --refresh", before
    freezing the repository + working tree state in the diskimage?
    

Am 22.08.19 um 20:02 schrieb Saravanan Shanmugham (sarvi):

Are you saying that you commonly mount and unmount the filesystem?

Git tracks a device number in the index. Could it happen that it is different every time you mount the filesystem? Because when it is, Git reads the data and checks whether it has changed. At this time, the device number is also fixed up in the index. Thereafter, "git status" is fast because it sees from the cached file properties that no change was made and does not have to read the data.

You may set "git config core.checkStat minimal" to avoid the problem. But it may come with its own problems (certain kinds of modifications would not be noticed, although these would be hard to trigger in practice).

-- Hannes

Johannes Sixt <j6t@kdbg.org> writes:

I read the above to mean that a diskimage file is treated as a virtual block device on which a filesystem image exists, and it is mounted via the lookback device mechanism. In such a case, I do not think stability of i-num would not be an issue (the filesystem image would record them all).

Yeah.

Am 22.08.19 um 21:32 schrieb Junio C Hamano:

Inode number would be stable, but st_dev may not be. But it looks like a default build does not use it anyway (I see that we do not define USE_STDEV), so my guess was most likely wrong.

-- Hannes

Johannes Sixt <j6t@kdbg.org> writes:

Ahh, thanks. I overlooked the device number, but I think the default settings excludes st_dev because it was unstable on nfs and friends.