threads / discuss / 43125

Re: cygwin, 44k files: how to commit only index?

Subject: Re: cygwin, 44k files: how to commit only index?

## tl;dr

18 messages between Dec 7, 2006 and Dec 9, 2006.

replies: 17people: 5as markdown or json

Alex Riesen· Dec 7, 2006, 14:27 UTC · lore

cygwin, 44k files: how to commit only index?

I have a kind of awkward project to work with (~44k files, many binaries).

The normal "git commit", which seem to be more than enough for anything and anyone else, is a really annoying procedure in my context. It spend too much time refreshing index and generating list of the files for the commit message.

At first I stopped using git commit -a (doing only update-index), now I'm about to start using write-tree/commit-tree/update-ref directly. It helps, but sometimes I really miss -F/-C. It's also ugly: I can (and almost did) commit an unchanged tree.

Is there any simple way to modify git commit for such a workflow? Failing that, any simple and _fast_ way to find out if the index

Junio C Hamano· Dec 7, 2006, 19:16 UTC · re: Alex Riesen · lore
"Alex Riesen" <raa.lkml@gmail.com> writes:
Show 8 quoted lines
> I have a kind of awkward project to work with (~44k files, many binaries).
>
> The normal "git commit", which seem to be more than enough
> for anything and anyone else, is a really annoying procedure
> in my context. It spend too much time refreshing index and
> generating list of the files for the commit message.
>
> At first I stopped using git commit -a (doing only update-index),

I am not sure what you are trying. Do you mean stat() is slow on your filesystem?

> Is there any simple way to modify git commit for such a workflow?
> Failing that, any simple and _fast_ way to find out if the index
> is any different from HEAD? (so that I don't produce empty commits).
Maybe you want "assume unchanged"?
Shawn Pearce· Dec 7, 2006, 19:26 UTC · re: Junio C Hamano · lore
Junio C Hamano <junkio@cox.net> wrote:
Show 13 quoted lines
> "Alex Riesen" <raa.lkml@gmail.com> writes:
> 
> > I have a kind of awkward project to work with (~44k files, many binaries).
> >
> > The normal "git commit", which seem to be more than enough
> > for anything and anyone else, is a really annoying procedure
> > in my context. It spend too much time refreshing index and
> > generating list of the files for the commit message.
> >
> > At first I stopped using git commit -a (doing only update-index),
> 
> I am not sure what you are trying.  Do you mean stat() is slow
> on your filesystem?
Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
same problem on my Cygwin systems.
 
Show 5 quoted lines
> > Is there any simple way to modify git commit for such a workflow?
> > Failing that, any simple and _fast_ way to find out if the index
> > is any different from HEAD? (so that I don't produce empty commits).
> 
> Maybe you want "assume unchanged"?

Yes, basically. The Cygwin/NTFS issues Alex is pointing out are exactly why git-gui has a "Trust File Modification Timestamp" option on both a per-repository and global level. My larger repositories (~10k files) are difficult to work with without that option enabled.

Shawn Pearce· Dec 7, 2006, 19:35 UTC · re: Shawn Pearce · lore
Shawn Pearce <spearce@spearce.org> wrote:
> Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
> same problem on my Cygwin systems.
Just to be clear, I'm not trying to blame Cygwin here.

Windows' dir command is slow. Windows Explorer is slow while browsing directories. Eclipse chugs hard while doing any directory scans (it normally runs very fast if its not rescanning the entire directory structure). The drive is just plain slow.

Yea, I know, get a faster disk... but some bean counters don't believe that a $50 more expensive disk could ever save enough time to warrant the extra $50 captial expenditure...

I spend at least an hour a week waiting for enough IO to finish so that the mouse pointer will move again. *sigh*

Christian MICHON· Dec 7, 2006, 21:26 UTC · re: Shawn Pearce · lore
On 12/7/06, Shawn Pearce <spearce@spearce.org> wrote:
Show 11 quoted lines
> Shawn Pearce <spearce@spearce.org> wrote:
> > Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
> > same problem on my Cygwin systems.
>
> Just to be clear, I'm not trying to blame Cygwin here.
>
> Windows' dir command is slow.  Windows Explorer is slow while
> browsing directories.  Eclipse chugs hard while doing any directory
> scans (it normally runs very fast if its not rescanning the entire
> directory structure).  The drive is just plain slow.
> (...)

before buying any new hardware, you could easily imagine the following scenario (I'm also "stuck" with windows, so it's an idea I've been toying around for a week or so).

There're virtualizers around, on which networking capabilities can be activated. And we could easily create a vm with linux+git inside, using ext2/ext3/ext4 fs virtual disks (you'd benefit from windows cache actually...)

example: YTech_Subversion_Appliance_v1.1 (ubuntu + subversion).
I've no prototype yet, but I've 2 scenario possible:
1) use vmplayer and a minimal uclibc initramfs with git onboard
2) use qemu+kqemu and a similar mini-distro (but right now networking
is an issue on windows hosts: I'm exploring tunneling)

The 1st scenario is "easy". And I start to prefer this idea over even mingw porting of git (I tried and it's hard, really).

But again, maybe jgit would be a better universal solution.
Torgil Svensson· Dec 9, 2006, 08:27 UTC · re: Christian MICHON · lore
On 12/7/06, Christian MICHON <christian.michon@gmail.com> wrote:
Show 9 quoted lines
> On 12/7/06, Shawn Pearce <spearce@spearce.org> wrote:
> > Shawn Pearce <spearce@spearce.org> wrote:
> > > Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
> > > same problem on my Cygwin systems.
> >
> > Just to be clear, I'm not trying to blame Cygwin here.
> >
> > Windows' dir command is slow.  Windows Explorer is slow while
> > browsing directories.

I think this is a very common scenario costing hideous amounts of money around the globe.

If you have lot's of files in a folder, don't even think of accidentally touching those folders in Windows Explorer, if you do - keep Process Explorer or similar ready. I've ended up using (even w/o Cygwin) scripts, automatic compressing and even a database functioning as directory cache - basically creating accessibility layers for a disabled file-system.

Show 25 quoted lines
>
> before buying any new hardware, you could easily imagine the
> following scenario (I'm also "stuck" with windows, so it's an idea
> I've been toying around for a week or so).
>
> There're virtualizers around, on which networking capabilities can
> be activated. And we could easily create a vm with linux+git
> inside, using ext2/ext3/ext4 fs virtual disks (you'd benefit from
> windows cache actually...)
>
> example: YTech_Subversion_Appliance_v1.1 (ubuntu + subversion).
>
> I've no prototype yet, but I've 2 scenario possible:
> 1) use vmplayer and a minimal uclibc initramfs with git onboard
> 2) use qemu+kqemu and a similar mini-distro (but right now networking
> is an issue on windows hosts: I'm exploring tunneling)
>
> The 1st scenario is "easy". And I start to prefer this idea over
> even mingw porting of git (I tried and it's hard, really).
>
> But again, maybe jgit would be a better universal solution.
>
> --
> Christian
> -
Very interesting!  Have you a time-frame for this?  Maybe even
Junio C Hamano· Dec 7, 2006, 19:57 UTC · re: Shawn Pearce · lore
Shawn Pearce <spearce@spearce.org> writes:
Show 13 quoted lines
>> I am not sure what you are trying.  Do you mean stat() is slow
>> on your filesystem?
>
> Its Cygwin/NTFS.  lstat() is slow.  readdir() is slow.  I have the
> same problem on my Cygwin systems.
>  
>> > Is there any simple way to modify git commit for such a workflow?
>> > Failing that, any simple and _fast_ way to find out if the index
>> > is any different from HEAD? (so that I don't produce empty commits).
>> 
>> Maybe you want "assume unchanged"?
>
> Yes, basically.
Then maybe "git grep assume.unchanged" would help?
Shawn Pearce· Dec 7, 2006, 20:29 UTC · re: Junio C Hamano · lore
Junio C Hamano <junkio@cox.net> wrote:
Show 10 quoted lines
> Shawn Pearce <spearce@spearce.org> writes:
> 
> >> I am not sure what you are trying.  Do you mean stat() is slow
> >> on your filesystem?
> >> 
> >> Maybe you want "assume unchanged"?
> >
> > Yes, basically.
> 
> Then maybe "git grep assume.unchanged" would help?

Hmm. OK, maybe I should have answered "No"" to your first question. I keep looking at the assume unchanaged feature of update-index, but refuse to use it because I'm a lazy guy who will forget to tell the index a file has been modified. Consequently I'm going to miss a change during a commit.

What may help (and without using assume unchanged) is:
 * skip the `update-index --refresh` part of git-status/git-commit
 * skip the status template in COMMIT_MSG when using the editor

As Git will still at least make sure a `commit -a` includes everything that is dirty.

Files whose modification dates may have been messed with (but whose content are unchanged) will just go through expensive SHA1 computation to arrive at the same value, which is fine.

Users skipping the first part are doing so under the assumption that their modification dates are usually always correct, and that then they aren't the SHA1 computation of a handful of files is cheap compared to stat'ing the entire set of files.

Users skipping the second part are doing so under the assumption that knowing the names of the files they are committing doesn't really improve their odds of writing a good commit message.

Junio C Hamano· Dec 7, 2006, 21:53 UTC · re: Shawn Pearce · lore
Shawn Pearce <spearce@spearce.org> writes:
Show 20 quoted lines
> What may help (and without using assume unchanged) is:
>
>  * skip the `update-index --refresh` part of git-status/git-commit
>  * skip the status template in COMMIT_MSG when using the editor
>
> As Git will still at least make sure a `commit -a` includes
> everything that is dirty.
>
> Files whose modification dates may have been messed with (but
> whose content are unchanged) will just go through expensive SHA1
> computation to arrive at the same value, which is fine.
>
> Users skipping the first part are doing so under the assumption that
> their modification dates are usually always correct, and that then
> they aren't the SHA1 computation of a handful of files is cheap
> compared to stat'ing the entire set of files.
>
> Users skipping the second part are doing so under the assumption
> that knowing the names of the files they are committing doesn't
> really improve their odds of writing a good commit message.

The second part is not about a good commit message but more about a path that should have been updated but forgotten (the same mistake you would be likely to make and that is the reason assume-unchanged is not good for you).

I do not mind too much if you added a new --quick option to "git commit" for this rather specialized need.

Alex Riesen· Dec 7, 2006, 22:15 UTC · re: Junio C Hamano · lore
Junio C Hamano, Thu, Dec 07, 2006 20:16:39 +0100:
Show 11 quoted lines
> > I have a kind of awkward project to work with (~44k files, many binaries).
> >
> > The normal "git commit", which seem to be more than enough
> > for anything and anyone else, is a really annoying procedure
> > in my context. It spend too much time refreshing index and
> > generating list of the files for the commit message.
> >
> > At first I stopped using git commit -a (doing only update-index),
> 
> I am not sure what you are trying.  Do you mean stat() is slow
> on your filesystem?

incredibly slow. That and the matter of having 44000 files to process with that slow stat().

Show 6 quoted lines
> > Is there any simple way to modify git commit for such a workflow?
> > Failing that, any simple and _fast_ way to find out if the index
> > is any different from HEAD? (so that I don't produce empty commits).
> 
> Maybe you want "assume unchanged"?
> 

If that is core.ignoreState you mean, than maybe this is what I mean. I haven't tried it yet (now I wonder myself why I haven't tried it). But (I'm repeating myself, in <81b0412b0612060235l5d5f93d0hd1aaf34924f7783@mail.gmail.com>) I do not really understand how it _can_ help: "I ask because it does not ignore stat info, as the name implies. Because if it would, there'd be no point of calling lstat at all, wouldn't it?" That last question was about refresh_cache_entry - it calls lstat unconditionally.

Still, I guess I'll have to try it.

But aside from me trying ignoreState, can anyone help me with that question regarding checking if the index is any different from HEAD? Because even on a very brocken filesystem and 40k files in a repo you sometimes do want to call git-update-index --refresh just to be sure you haven't missed anything. And than it'll quickly become annoying flicking ignoreState back and forth.

Junio C Hamano· Dec 7, 2006, 22:29 UTC · re: Alex Riesen · lore
fork0@t-online.de (Alex Riesen) writes:
> But aside from me trying ignoreState, can anyone help me with that
> question regarding checking if the index is any different from HEAD?

Comparing index and HEAD should be cheap on a system with slow lstat(), I think, as "git-diff-index --cached HEAD" should just ignore the working tree altogether. Is that what you want?

Alex Riesen· Dec 8, 2006, 05:27 UTC · re: Junio C Hamano · lore
Junio C Hamano, Thu, Dec 07, 2006 23:29:54 +0100:
Show 7 quoted lines
> > But aside from me trying ignoreState, can anyone help me with that
> > question regarding checking if the index is any different from HEAD?
> 
> Comparing index and HEAD should be cheap on a system with slow
> lstat(), I think, as "git-diff-index --cached HEAD" should just
> ignore the working tree altogether.  Is that what you want?
> 

yes, except that it'll compare the whole trees. Could I make it stop at first mismatch? "-q|--quiet" for git-diff-index perhaps? It's just not only stat, but also, open, read, mmap (yes, I try to use it for packs) and close are really slow here as well.

Junio C Hamano· Dec 8, 2006, 06:54 UTC · re: Alex Riesen · lore
fork0@t-online.de (Alex Riesen) writes:
> yes, except that it'll compare the whole trees. Could I make it stop
> at first mismatch? "-q|--quiet" for git-diff-index perhaps?
> It's just not only stat, but also, open, read, mmap (yes, I try to use
> it for packs) and close are really slow here as well.

That sounds like optimizing for a wrong case -- you expect the index to match HEAD and trying to catch mistakes by detecting a mismatch, right?

Having said that, I should point out that it is a low hanging fruit to optimize "diff-index --cached" for cases where index is expected to mostly match HEAD.

The current code for "diff-index --cached" reads the whole tree into the index as stage #1 entries (diff-lib.c::run_diff_index), and then compares stage #0 (from the original index contents) and stage #1 (the tree parameter from the command line). Even if you stop at the first mismatch, you would already have paid the overhead to open and read all tree objects before even starting the comparison.

However, this code is from the ancient time before cache-tree was introduced in the index. If the index is expected to mostly match HEAD, most of the cache-tree nodes are up-to-date, and whole subtree can be skipped with a single comparison between two tree SHA-1s at a shallower level of the directory tree.

In 'pu' (jc/diff topic), I have a very generic code to walk the index, working tree and zero or more trees in parallel, taking advantage of cache-tree. If somebody is interested to learn the internals of git, some of the code could be lifted from there and simplified to walk just the index and a single tree, and I think that would optimize "diff-index --cached" quite a bit.

A very unscientific test of running in the kernel repository I just pulled (hot cache) on my box is:

$ /usr/bin/time git diff-index -r --cached --abbrev v2.6.19 >/tmp/1 0.91user 0.20system 0:01.12elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+10949minor)pagefaults 0swaps

while the para-walk to produce the moral equivalent is:

$ /usr/bin/time test-para --no-work v2.6.19 >/tmp/2 0.11user 0.02system 0:00.13elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+4524minor)pagefaults 0swaps

Alex Riesen· Dec 8, 2006, 07:27 UTC · re: Junio C Hamano · lore
On 12/8/06, Junio C Hamano <junkio@cox.net> wrote:
Show 8 quoted lines
> > yes, except that it'll compare the whole trees. Could I make it stop
> > at first mismatch? "-q|--quiet" for git-diff-index perhaps?
> > It's just not only stat, but also, open, read, mmap (yes, I try to use
> > it for packs) and close are really slow here as well.
>
> That sounds like optimizing for a wrong case -- you expect the
> index to match HEAD and trying to catch mistakes by detecting
> a mismatch, right?

I expect the index to differ from HEAD. The test is to avoid the mistake of doing an empty commit.

Show 11 quoted lines
> Having said that, I should point out that it is a low hanging
> fruit to optimize "diff-index --cached" for cases where index
> is expected to mostly match HEAD.
>
> The current code for "diff-index --cached" reads the whole tree
> into the index as stage #1 entries (diff-lib.c::run_diff_index),
> and then compares stage #0 (from the original index contents)
> and stage #1 (the tree parameter from the command line).  Even
> if you stop at the first mismatch, you would already have paid
> the overhead to open and read all tree objects before even
> starting the comparison.

But I don't have to pay for the overhead of comparing all entries, if I can stop at first mismatch and exit with non-0. I think it'd make a difference (at least some difference). But, if we could avoid loading of the entries which will be never compared anyway, the speedup will be of course more substantial...

Show 6 quoted lines
> In 'pu' (jc/diff topic), I have a very generic code to walk the
> index, working tree and zero or more trees in parallel, taking
> advantage of cache-tree.  If somebody is interested to learn the
> internals of git, some of the code could be lifted from there
> and simplified to walk just the index and a single tree, and I
> think that would optimize "diff-index --cached" quite a bit.
Junio C Hamano· Dec 8, 2006, 07:36 UTC · re: Alex Riesen · lore
"Alex Riesen" <raa.lkml@gmail.com> writes:
Show 10 quoted lines
>> The current code for "diff-index --cached" reads the whole tree
>> into the index as stage #1 entries (diff-lib.c::run_diff_index),
>> and then compares stage #0 (from the original index contents)
>> and stage #1 (the tree parameter from the command line).  Even
>> if you stop at the first mismatch, you would already have paid
>> the overhead to open and read all tree objects before even
>> starting the comparison.
>
> But I don't have to pay for the overhead of comparing all
> entries, if I can stop at first mismatch and exit with non-0.
Bench it if you doubt me.

I'd bet that the time spent in comparison between stages inside index (and remember, you are not generating textual diff, only comparing the SHA-1) is dwarfed by the overhead of populating the stage #1 of the index with what is read from all the tree objects.

Alex Riesen· Dec 8, 2006, 07:48 UTC · re: Junio C Hamano · lore
On 12/8/06, Junio C Hamano <junkio@cox.net> wrote:
Show 12 quoted lines
> >> The current code for "diff-index --cached" reads the whole tree
> >> into the index as stage #1 entries (diff-lib.c::run_diff_index),
> >> and then compares stage #0 (from the original index contents)
> >> and stage #1 (the tree parameter from the command line).  Even
> >> if you stop at the first mismatch, you would already have paid
> >> the overhead to open and read all tree objects before even
> >> starting the comparison.
> >
> > But I don't have to pay for the overhead of comparing all
> > entries, if I can stop at first mismatch and exit with non-0.
>
> Bench it if you doubt me.

I don't question that the overhead of comparing is very much unnoticable. It just that it surely isn't zero, and it will grow with the size of repo (linearly, right?)... And I am sure that this repo will _only_ grow (typical corporate project).

Show 5 quoted lines
> I'd bet that the time spent in comparison between stages inside
> index (and remember, you are not generating textual diff, only
> comparing the SHA-1) is dwarfed by the overhead of populating
> the stage #1 of the index with what is read from all the tree
> objects.
I already understood that. I just haven't found yet what can I do
Alex Riesen· Dec 8, 2006, 08:43 UTC · re: Alex Riesen · lore
On 12/8/06, Alex Riesen <raa.lkml@gmail.com> wrote:
Show 10 quoted lines
> On 12/8/06, Junio C Hamano <junkio@cox.net> wrote:
> > In 'pu' (jc/diff topic), I have a very generic code to walk the
> > index, working tree and zero or more trees in parallel, taking
> > advantage of cache-tree.  If somebody is interested to learn the
> > internals of git, some of the code could be lifted from there
> > and simplified to walk just the index and a single tree, and I
> > think that would optimize "diff-index --cached" quite a bit.
>
> Will try to look at it.
>

And now I'm playing with that (against test-para.c from pu). I expect it to be broken by that webGmail, so it may not apply to anything, but you'll get the idea. More clearly than from me trying to explain.

commit 83642cdaca6dc1a2f94aa41923bc9e8f02d0e12f
Author: Alex Riesen <raa.lkml@gmail.com>
Date:   Fri Dec 8 09:38:18 2006 +0100
    add --quiet to test-para: stop at the first difference
diff --git a/test-para.c b/test-para.c
index bce5f0c..99d3792 100644
--- a/test-para.c
+++ b/test-para.c
@@ -21,6 +21,7 @@ int main(int ac, const char **av)
 	unsigned char trees[64][20];
 	int num_tree = 0, i, using_head, show_all = 0;
 	int index_wanted = 1, work_wanted = 1, tree_wanted = 0;
+	int quiet = 0;
 	const char *prefix;
 	const char **pathspec;

@@ -43,6 +44,8 @@ int main(int ac, const char **av)
 			work_wanted = 0;
 		else if (!strcmp(av[1] + 2, "no-index"))
 			index_wanted = 0;
+		else if (!strcmp(av[1] + 2, "quiet"))
+			quiet = 1;
 		else if (!av[1][2])
 			break;
 		else
@@ -118,7 +121,9 @@ int main(int ac, const char **av)
 				show_one(z, e->name, e->namelen,
 					 e->hash, e->mode);
 			}
-			else
+			else {
+				if (quiet)
+					exit(1);
 				for (i = 0; i < w.num_trees + 2; i++) {
 					char numbuf[10];

@@ -149,6 +154,7 @@ int main(int ac, const char **av)
 					show_one(z, e->name, e->namelen,
 						 e->hash, e->mode);
 				}
+			}
 		}
Alex Riesen· Dec 8, 2006, 14:16 UTC · re: Alex Riesen · lore
On 12/7/06, Alex Riesen <fork0@t-online.de> wrote:
Show 14 quoted lines
> > Maybe you want "assume unchanged"?
>
> If that is core.ignoreState you mean, than maybe this is what I mean.
> I haven't tried it yet (now I wonder myself why I haven't tried it).
> But (I'm repeating myself, in
> <81b0412b0612060235l5d5f93d0hd1aaf34924f7783@mail.gmail.com>)
> I do not really understand how it _can_ help: "I ask because it does
> not ignore stat info, as the name implies. Because if it would,
> there'd be no point of calling lstat at all, wouldn't it?" That last
> question was about refresh_cache_entry - it calls lstat
> unconditionally.
>
> Still, I guess I'll have to try it.
>
Tried. No noticeable difference:

$ git repo-config core.ignorestat true; time gup --refresh real 0m8.004s user 0m1.936s sys 0m5.702s $ git repo-config core.ignorestat false; time gup --refresh real 0m7.787s user 0m1.890s sys 0m5.703s $

← back to recent threads