MinGW port usable

I'd like to point interested parties to the MinGW port at

 git://repo.or.cz/git/mingw.git

which is now in a usable state, methinks. I'm using it with git-gui and gitk on a (almost) production repository.

The README.MinGW at http://repo.or.cz/w/git/mingw.git?a=blob_plain;f=README.MinGW;hb=master gives an overview on the state. The transfer via native git protocol does not work and cannot be made working without major surgery(*). Theoretically, using netcat (nc) as GIT_PROXY_COMMAND should work, but not in my setup for some reason that I still do not know.

(*) The reason is that on Windows read() and write() cannot operate on descriptors created by socket(). A work-around is to implement a (threaded) proxy, but that's almost the same as if netcat were used as GIT_PROXY_COMMAND.

Junio, you may like to cherry-pick these two non-critical commits from my repository:

8c8bb94f94f1d972c7ffadda4744cf343fac6f34 gitk: Use peek-remote instead of ls-remote. 46580d2192d79a469f8b40fc1081db9116ad5517 Add a missing fork() error check.

-- Hannes

Johannes Sixt <johannes.sixt@telecom.at> wrote:

Well, that's pretty sweet that git-gui is working in a MinGW case.

Last night I pushed out a number of changes to avoid cygpath when we don't have it in PATH (commit 20ddfcaa). Were you testing with that version, or something earlier?

On 1/29/07, Shawn O. Pearce <spearce@spearce.org> wrote:

my last compilation is less than 8 hours old, and I do not have cygwin installed. What is it you want to test actually ?

git-gui and gitk have been working for a bit more than a week, if I recall well. I'd like to see git-blame latest patch work with git-gui.

:)

Christian MICHON <christian.michon@gmail.com> wrote:

Try creating a desktop icon (Repository->Create Desktop Icon) and start git-gui from the resulting .bat file. It *should* come up right on MinGW, but I'm only guessing here.

The other one that I'm curious about is if fetch/push/merge work. merge requires git-merge right now, which is a shell script. fetch is the same, but push is pure C so it should work. But I don't think the Tcl environment will make it into the child, which means things like SSH_AUTH_SOCK don't get used.

But I really should just install the git-mingw port on one of
my windows systems and play with it.  If its faster than Cygwin
then it's worthwhile.
 

The blame feature is in git-gui (went in last night). It requires the --incremental patches from Linus&Junio, which are now in git.git master. It also needs a big display, as the interface is horrid. :)

On 1/30/07, Shawn O. Pearce <spearce@spearce.org> wrote:

ok noted for tomorrow's tests :)

I'll see what I can do.

one of the hassles with this mingw thingy is that you need to install it, and whatever you compile gets inside it, like libs, includes, etc... Whenever I come out with a stable product that can be sfx'ed inside a 7zip executable, I do it, to avoid installing/uninstalling or even modifying it.

it works like a charm with msys/mingw, and I guess git/gitk/git-gui will all work in this way too. Once this port is ready for mass testing, I guess we can either show how to compile it (actually some patches are still needed, like no symlinks exist on msys/mingw, so we should replace those links by bash scripts), or provide a 7z sfx binary that can help <<pushing>> git to the mass. On this last possibility I can help out...

your call, guys. :)

I'll find a big one then.

On 1/30/07, Shawn O. Pearce <spearce@spearce.org> wrote:

ok, yes it's working. :) but call gitk from inside it fail :(

"error in startup script" ( git-peek-remote is not working from git-gui, could be path related. It's hard to tell why yet)

On 1/30/07, Christian MICHON <christian.michon@gmail.com> wrote:

just to be more clear: when using the .bat shortcut, calling gitk from within git-gui fails.

But if I call git-gui from a msys rxvt+bash, I can invoke all tools, and calling gitk from within git-gui works.

I think this shortcut stuff may not be the best approach yet.

Hi,

On Tue, 30 Jan 2007, Christian MICHON wrote:

I guess you need to change the .bat shortcut, so that the tcl script is called via bash instead of directly from cmd.

Otherwise git-gui does not have a chance to find the MSYS environment.

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Hmmph. The [is_Cygwin] test within git-gui causes it to setup the .bat shortcut using:

  $(cygpath --windows /bin/sh) --login -c 'git git-gui'

for exactly that reason. I didn't think that with MSYS there was environment to hand down through a shell's init scripts. I thought it was all coming from Windows itself. Oops.

When starting gitk, git-gui assumes (rightly or wrongly) that gitk is a Tcl/Tk script and launches it with the same Tcl interpreter that is running git-gui. I did that to bypass the unnecessary fork/exec of a shell just to enter a wish process. I had assumed that would also work well on the MSYS case. Apparently not.

Patches to git-gui would be welcome. It will probably be a few weeks before I can get an MSYS+Git configuration working that I can test git-gui against. I'm just swamped with too many tasks right now, and this, although interesting enough to fix, is likely to take a back seat.

Christian MICHON wrote:

Calling gitk from git-gui works here, even if git-gui was invoked from the .bat file. But maybe just because I have all git tools in my $PATH. And I have this CMD script in my $PATH as well:

D:\>cat D:\MSYS\1.0\git\bin\gitk.cmd @echo off start wish84 D:/MSYS/1.0/git/bin/gitk %1 %2 %3 %4 %5 %6 %7 %8 %9

(and, btw, a similar script for git-gui).

-- Hannes

"Shawn O. Pearce" wrote:

yes, I tested that version. The new incremental blame works, too.

-- Hannes

On Mon, 29 Jan 2007, Johannes Sixt wrote:

Can you elaborate about any performance differences, especially with cygwin? Does this make git perform reasonably on Windows, or are the performance problems as bad as with cygwin?

gitk really shouldn't use either. It should probably use

	git-show-ref -h -d

instead, which has the same output format (modulo a space vs tab issue), and is entirely local, with no silly unnecessary remote connext.

Something like this patch.

Does that work on mingw too?

		Linus
----

On 1/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:

I do not know about cygwin recently (else than it was slow), but I cloned my packed git repository of all 117 2.6 linux kernels and the "git diff" rocks on it.

git-unpack and git-repack was slow as hell, though. Maybe this is the part to be improved. Mostly bad I/Os...

I'll try that tomorrow, but you'll have other reporting about it, for sure.

Hi,

On Tue, 30 Jan 2007, Christian MICHON wrote:

How do you want to improve bad I/O on Windows?

Ciao, Dscho

On 1/30/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

upgrade to Vista ? :lol:

seriously, no clue here. I want first to fairly compare mingw-git with git inside colinux inside a shared cofs directory.

Linus Torvalds wrote:

Performance is horrid, although better than I initially expected. I don't (want to) have cygwin installed to compare it (that's why I did the MinGW port in the first place ;) Maybe others can comment on this?

The test suite takes ~15min to complete on my box, which is an oooold 800MHz with a slow HD. (And it's still Win2K, if that matters.)

Quite frankly, the "performance boost" that I expect from this port is that it allows me the workflow that I want, instead of constantly banging my head against the walls of CVS/SVN/you-name-it.

Thanks, this works (I tested Junio's version). The problem with ls-remote was that it is a shell script, and for some reason it dumps its output into a cmd.exe that opens and closes right away instead of to the pipe.

-- Hannes

Hi,

On Mon, 29 Jan 2007, Linus Torvalds wrote:

On the git repository, using MinGW:

	$ time git-rev-list next > /dev/null

	real    0m0.320s
	user    0m0.031s
	sys     0m0.000s

	$ time git-rev-list next > /dev/null

	real    0m0.320s
	user    0m0.015s
	sys     0m0.015s

	$ time git-rev-list next > /dev/null

	real    0m0.352s
	user    0m0.015s
	sys     0m0.000s

Using cygwin:

	$ time git-rev-list next > /dev/null

	real    0m0.750s
	user    0m0.421s
	sys     0m0.140s

	$ time git-rev-list next > /dev/null

	real    0m0.750s
	user    0m0.374s
	sys     0m0.233s

	$ time git-rev-list next > /dev/null

	real    0m0.750s
	user    0m0.374s
	sys     0m0.218s

IOW MinGW wins hands down with an average 0.33s vs cygwin with an average 0.75s. (I am fairly certain that you must not trust sys times.)

I have no idea why, as the big show-stopper fork() is not even involved.

Now, for something different: git-log. On MinGW:

	$ time git-log next > /dev/null

	real    0m1.594s
	user    0m0.015s
	sys     0m0.015s

	$ time git-log next > /dev/null

	real    0m1.562s
	user    0m0.031s
	sys     0m0.000s

	$ time git-log next > /dev/null

	real    0m1.610s
	user    0m0.015s
	sys     0m0.015s

On Cygwin:

	$ time git-log next > /dev/null

	real    0m2.063s
	user    0m0.827s
	sys     0m1.031s

	$ time git-log next > /dev/null

	real    0m2.085s
	user    0m0.780s
	sys     0m1.061s

	$ time git-log next > /dev/null

	real    0m2.047s
	user    0m0.686s
	sys     0m1.171s

That's a bit less obvious: 1.589s for MinGW vs. 2.065s for Cygwin.

These are all hot-cache numbers, since the cold-cache performance of Windows is not easily triggered. (FWIW the first git-log on Cygwin -- which was cold-cache -- took over 10 seconds.)

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Just curious, but were these tests run with mmap(), or NO_MMAP?

Clearly, for a packed repository most of the overhead should be in getting to the pack data itself (as everything else is pure userspace).

Hi,

On Tue, 30 Jan 2007, Shawn O. Pearce wrote:

Both with NO_MMAP=YesPlease.

Right. That's why I did the test on the _same_ repository. (Not just both git repositories, but in the same path on the same machine.)

That said, I find the numbers quite convincing. On a (much weaker equipped) Linux bux, it takes 0.65s and 1.2s, respectively.

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

I wonder what the difference is between NO_MMAP=Yes and NO_MMAP=
on Windows.  This is something I've never tried, but probably should
do.  I run about ~25 developers on NO_MMAP= on Windows (all NTFS)
without issue, but now I'm curious if there's an actual performance
difference for some operations.  There's no real rationale behind
my NO_MMAP= setting, other than that's how we've always used it...
 

The time difference between the MinGW and Cygwin ports of Git is certainly an interesting one, and says good things about the MinGW work.

We're paying a price for POSIX compatibility on Windows, and its a pretty high one it seems. I can't blame the Cygwin folks, they've done a great job at trying to make the most of VMS. But sometimes you just need Linux (or *BSD, or Solaris). :-)

Hi,

On Tue, 30 Jan 2007, Shawn O. Pearce wrote:

Cygwin:

	$ time git-rev-list next >/dev/null

	real    0m0.812s
	user    0m0.358s
	sys     0m0.203s

	$ time git-rev-list next >/dev/null

	real    0m0.797s
	user    0m0.358s
	sys     0m0.234s

	$ time git-rev-list next >/dev/null

	real    0m0.845s
	user    0m0.390s
	sys     0m0.265s

and

	$ time git-log next >/dev/null

	real    0m2.094s
	user    0m0.718s
	sys     0m1.077s

	$ time git-log next >/dev/null

	real    0m2.078s
	user    0m0.718s
	sys     0m1.062s

	$ time git-log next >/dev/null

	real    0m2.124s
	user    0m0.796s
	sys     0m1.046s

IOW, the numbers are slightly worse (!) than with mmap().

MinGW does not even have mmap().

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Slightly?  That's double the time!
 

But Windows has something almost there. I just read a flame war thread about implementing mmap in libiberty for MinGW by stealing source from Cygwin, and how the FSF might feel about someone's dirty feet after playing rugby. Yea... Google is good. :)

Hi,

On Tue, 30 Jan 2007, Shawn O. Pearce wrote:

Nope. What I meant is this: rev-list takes 0.8s without NO_MMAP, while it takes 0.75s with NO_MMAP, and similarly log takes 2.09s without NO_MMAP, while it takes 2.065 with NO_MMAP.

Yeah, CreateFile(). Very intuitive name.

;-)

Well, I had _horrible_ results in the last two weeks, slowly getting better.

Ciao, Dscho

On Mon, 29 Jan 2007, Johannes Sixt wrote:

Can you do

#define read(fd, buffer, len) recv(fd, buffer, len, 0) #define write(fd, buffer, len) send(fd, buffer, len, 0)

in the appropriate file?

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow wrote:

I doubt that recv and send can operate on regular file descriptors, as opened by _pipe(), open(), can they?

-- Hannes

Johannes Sixt wrote:

Hence "in the appropriate file", I guess.

Hi,

On Tue, 30 Jan 2007, Johannes Sixt wrote:

They can't. Thus, you'd break at least inetd mode.

Of course, you really could start a thread which gets two pipe()d fds, and the proper socket, and just shuffles things back & forth. Not that hard.

Ciao, Dscho

On Tue, 30 Jan 2007, Johannes Sixt wrote:

I don't think so, but I think it should be possible to make packet_write/packet_read always get a socket, by calling receive-pack and upload-pack with a socket pair instead of a pair of pipes. Actually, IIRC, this would be beneficial for making programs not leak file descriptors or double-close them, because there's already the issue on *nix that it isn't consistant whether both directions are the same file descriptor.

	-Daniel
*This .sig left intentionally blank*

Hi,

On Tue, 30 Jan 2007, Daniel Barkalow wrote:

As I already mentioned in this thread, that would break inetd support.

Ciao, Dscho

On Tue, 30 Jan 2007, Johannes Schindelin wrote:

I was actually thinking of only using recv/send on mingw. So the rule could be: if git sets up the connection to a pkt_line-user itself, the connection is a socket; otherwise it might be a pair of pipes; if you're on mingw, pkt_line uses recv/send. Then everything should work except for inetd on mingw, and I don't think that's a plausible combination anyway.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow wrote:

Except that it won't work. Because in the same program, say git-fetch-pack, the packet_*() functions, where you intend to replace read/write by recv/send, are sometimes called with pipes, and sometimes with sockets, even on MinGW.

Call it object oriented programming with polymorphic behavior if you like ;)

-- Hannes

Johannes Sixt <johannes.sixt@telecom.at> writes:

Thanks; cherry-picked the fork() one.

Regarding gitk, I think the following patch would be better regardless of the platform.

-- >8 -- [PATCH] gitk: Use show-ref instead of ls-remote

It used to be ls-remote on self was the only easy way to grab the ref information. Now we have show-ref which does not involve fork and IPC, so use it.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

Johannes Sixt wrote:

Actually, I believe it can for the NT series kernels (at least 2000 or later, not sure about the earlier ones), but not for the DOS-based ones.

The trick is to use _open_osfhandle() to convert the file handle (a WinAPI construct) to a file descriptor (which in Windows is a construct of the C library.)

In 9x/ME, a socket isn't represented by a file handle, so _open_osfhandle() doesn't work on a socket object.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/html/_crt__open_osfhandle.asp

	-hpa

"H. Peter Anvin" wrote:

I tried this, but it doesn't seem to work. I get an EINVAL at the first write() to the socket. I conclude that the things returned by socket() are not WinAPI file handles that are valid for WriteFile(). :(

-- Hannes

Johannes Sixt wrote:

Except they are (for NT-based Windows), so you're doing something goofy. 
  This is a widely used construct, so it can't be that broken.

	-hpa

H. Peter Anvin wrote:

Erf... I dug through this, and it seems that WriteFile only works on a socket if it has an OVERLAPPED argument now, because the socket is opened for overlapping I/O. This must be new behaviour in XP-SP2, because this definitely wasn't the case when I last played with this stuff back in 2003. The Internet is full of people using this technique, but I haven't found a way to get a socket which is *not* opened for overlapping I/O.

How typical of Microsoft to break an incredibly powerful unified paradigm, sort-of repair it, and then break it again. There doesn't seem to be an obvious way to repair this, either, since MS DLLs won't let you override for example the write() function as called from inside the C runtime DLL.

"Some people are just a total waste of carbon..."

	-hpa

On Fri, 2 Feb 2007, H. Peter Anvin wrote:

Doesn't WSASocket() w/out WSA_FLAG_OVERLAPPED work for you? I use that and it works just fine with ReadFile/WriteFile (example, I use to to feed std handles to CreateProcess - and those are always accessed with ReadFile/WriteFile by childs).

- Davide

"H. Peter Anvin" wrote:

Guess, what? When I use WSASocket() instead of socket() to explicitly create a non-overlapped socket, plus _open_osfhandle(), it works!

This does not seem to be that new a behavior, since I observed all the failures on Windows 2000.

Thanks a lot for your tip!

-- Hannes

H. Peter Anvin wrote:

**HAH***

I found a workaround after all. The trick is to use WSASocket() with a last argument 0 instead of socket().

I feel dirty now...

#include <winsock2.h> #include <fcntl.h> #include <io.h>

#define socket(a,b,c) sane_socket(a,b,c)

int sane_socket(int domain, int type, int protocol)
{
	SOCKET s;
	int fd;

	s = WSASocket(domain, type, protocol, NULL, 0, 0);
	if (s == INVALID_SOCKET) {
		/*
		 * WSAGetLastError() values seem to be mostly
		 * errno+10000; verify this or replace this with
		 * a switch statement...
		 */
		errno = WSAGetLastError() - 10000;
		return -1;
	}
	
	fd = _open_osfhandle(s, 0);
	if (fd < 0)
		closesocket(s);

	return fd;
}