Re: Switching from CVS to GIT

From: Eli Zaretskii <eliz@gnu.org>
Date: Oct 16, 2007, 04:30 UTC
Message-ID: <uodezisvg.fsf@gnu.org>
In-Reply-To: <Pine.LNX.4.64.0710151859590.7638@iabervon.org>

Show 8 quoted lines

> Date: Mon, 15 Oct 2007 20:45:02 -0400 (EDT)
> From: Daniel Barkalow <barkalow@iabervon.org>
> cc: Alex Riesen <raa.lkml@gmail.com>, Johannes.Schindelin@gmx.de, ae@op5.se, 
>     tsuna@lrde.epita.fr, git@vger.kernel.org, make-w32@gnu.org
> 
> I believe the hassle is that readdir doesn't necessarily report a README in 
> a directory which is supposed to have a README, when it has a readme 
> instead.

Sorry I'm asking potentially stupid questions out of ignorance: why would you want readdir to return `README' when you have `readme'?

> I think we want O(n) comparison of sorted lists, which doesn't 
> work if equivalent names don't sort the same.

You comparison function should be case-insensitive on Windows, or am I missing something?

Show 15 quoted lines

> > > - no acceptable level of performance in filesystem and VFS (readdir,
> > >   stat, open and read/write are annoyingly slow)
> > 
> > With what libraries?  Native `stat' and `readdir' are quite fast.
> > Perhaps you mean the ported glibc (libgw32c), where `readdir' is
> > indeed painfully slow, but then you don't need to use it.
> 
> We want getting stat info, using readdir to figure out what files exist, 
> for 106083 files in 1603 directories with a hot cache to take under 1s; 
> otherwise "git status" takes a noticeable amount of time with a medium-big 
> project, and we want people to be able to get info on what's changed 
> effectively instantly. My impression is that Windows' native stat and 
> readdir are plenty fast for what normal Windows programs want, but we 
> actually expect reasonable performance on an unreasonably-big 
> metadata-heavy input.

If that's the issue, then it's not a good idea to call `stat' and `readdir' on Windows at all. `stat' is a single system call on Posix systems, while on Windows it usually needs to go out of its way calling half a dozen system services to gather the `struct stat' info. You need to call something like FindFirstFile, which can do the job of `stat' and `readdir' together (and of `fnmatch', if you need to filter only some files) in one go. I don't know whether this will scan 100K files under one second (maybe I will try it one of these days), but it will definitely be faster than `readdir'+`stat' by maybe as much as an order of magnitude.

Show 6 quoted lines

> > > - no real "mmap" (which kills perfomance and complicates code)
> > 
> > You only need mmap because you are accustomed to use it on GNU/Linux.
> 
> I believe the need here is quick setup and fast access to sparse portions 
> of several 100M files. It's hard to beat a page fault for read speed.

If you need memory-mapped files, they are available on Windows. I thought the original comment about `mmap' was because it was used to allocate memory, not read files into memory.

> We also expect to be able to make a sequence of file system operations 
> such that programs starting at any time see the same database as the files 
> containing the database get restructured.

Sorry, I don't understand this; please tell more about the operations, ``the same database'' issue (what database?) and what do you mean by ``the files containing the database get restructured''.

> A unixy pipeline was convenient

Windows supports pipelines with almost 100% the same functionality as Posix. Again, perhaps I'm missing something.