threads / discuss / 1925

git 0.99.7b doesn't build on Cygwin

Subject: git 0.99.7b doesn't build on Cygwin

## tl;dr

38 messages between Sep 23, 2005 and Sep 26, 2005.

replies: 37people: 11as markdown or json

Peter TB Brett· Sep 23, 2005, 13:33 UTC · lore
Hi folks,

I wanted to use git on a Windows-based project (yes, there are some out there still), so I fired up my Cygwin xterm, untarred the git sources and totally failed to succeed in building them:

                              ---- ----

$ make gcc -o apply.o -c -g -O2 -Wall '-DSHA1_HEADER=<openssl/sha.h>' apply.c gcc -o blob.o -c -g -O2 -Wall '-DSHA1_HEADER=<openssl/sha.h>' blob.c gcc -o commit.o -c -g -O2 -Wall '-DSHA1_HEADER=<openssl/sha.h>' commit.c gcc -o connect.o -c -g -O2 -Wall '-DSHA1_HEADER=<openssl/sha.h>' connect.c connect.c: In function `git_tcp_connect': connect.c:298: error: storage size of 'hints' isn't known connect.c:322: warning: implicit declaration of function `getaddrinfo' connect.c:324: warning: implicit declaration of function `gai_strerror' connect.c:324: warning: format argument is not a pointer (arg 3) connect.c:326: error: dereferencing pointer to incomplete type connect.c:327: error: dereferencing pointer to incomplete type connect.c:327: error: dereferencing pointer to incomplete type connect.c:327: error: dereferencing pointer to incomplete type connect.c:330: error: dereferencing pointer to incomplete type connect.c:330: error: dereferencing pointer to incomplete type connect.c:338: warning: implicit declaration of function `freeaddrinfo' connect.c:298: warning: unused variable `hints' make: *** [connect.o] Error 1

$ gcc --version gcc (GCC) 3.4.4 (cygming special) (gdc 0.12, using dmd 0.125) ...

                              ---- ----

It looks like the sort of problems I get when I'm missing header files, but all the headers #included by connect.c are present on my system, so I'm really not sure what's going on there...

Peter
P.S. Please Cc: me on any replies, I'm not subscribed to the list.
-- 
Quake II build tools:  http://peter-b.co.uk/
Latest QuArK:          http://quark.sourceforge.net/LatestVersion

v2sw6YShw7ln5pr6ck3ma8u7Lw3+2m0l7CFi6e4+8t4Eb8Aen4g6Pa2Xs5MSr5p4
  hackerkey.com
Johannes Schindelin· Sep 23, 2005, 13:44 UTC · re: Peter TB Brett · lore

Re: git 0.99.7b doesn't build on Cygwin

Hi,
On Fri, 23 Sep 2005, Peter TB Brett wrote:
> I wanted to use git on a Windows-based project (yes, there are some
> out there still), so I fired up my Cygwin xterm, untarred the git
> sources and totally failed to succeed in building them:
Also see my mail regarding cygwin:
http://www.gelato.unsw.edu.au/archives/git/0508/7956.html
Show 6 quoted lines
> gcc -o connect.o -c -g -O2 -Wall  '-DSHA1_HEADER=<openssl/sha.h>' connect.c
> connect.c: In function `git_tcp_connect':
> connect.c:298: error: storage size of 'hints' isn't known
> connect.c:322: warning: implicit declaration of function `getaddrinfo'
> connect.c:324: warning: implicit declaration of function `gai_strerror'
> [...]

This is the IPv6 stuff. There are patches to cygwin to support IPv6 somewhere, but they haven't made it into mainline.

As for the other problem I mentioned in my original mail:

It seems that the fixup of the mmap()ed regions after a fork() does not work properly in cygwin. Remember that cygwin just wraps the non-POSIX Win32API and tries to make it sort of POSIX compliant. The problem is that Win32API lacks a proper fork(). This is therefore emulated, and after that, all the mmap()ed regions have to be mapped again. That fails.

Sidenote: I ran it inside gdb, and it worked fine! So I tried to recompile 
the cygwin1.dll, but that wrecked my whole installation of cygwin and I 
spent 2 hours just to be able to "gcc -o" again.

Ciao, Dscho

Peter TB Brett· Sep 23, 2005, 13:50 UTC · re: Johannes Schindelin · lore

Re: git 0.99.7b doesn't build on Cygwin

Johannes Schindelin wrote:
Show 7 quoted lines
>> I wanted to use git on a Windows-based project (yes, there are some
>> out there still), so I fired up my Cygwin xterm, untarred the git
>> sources and totally failed to succeed in building them:
>
> Also see my mail regarding cygwin:
>
> http://www.gelato.unsw.edu.au/archives/git/0508/7956.html
Yes. I found that just after I'd spammed the list.
Show 10 quoted lines
> This is the IPv6 stuff. There are patches to cygwin to support IPv6
> somewhere, but they haven't made it into mainline.
>
> As for the other problem I mentioned in my original mail:
>
> It seems that the fixup of the mmap()ed regions after a fork() does not
> work properly in cygwin. Remember that cygwin just wraps the non-POSIX
> Win32API and tries to make it sort of POSIX compliant. The problem is that
> Win32API lacks a proper fork(). This is therefore emulated, and after
> that, all the mmap()ed regions have to be mapped again. That fails.
Hmph. Sounds like I'm stuffed.

Ah well, I'll just have to use something else -- I know Mercurial works on Windows. To be honest, I'd prefer to use git though; I've used it on other projects and it's really nice.

Ah well, can't have everything you want all of the time, neh?
Peter
-- 
Quake II build tools: http://peter-b.co.uk/
Latest QuArK:         http://quark.sourceforge.net/LatestVersion

v2sw6YShw7ln5pr6ck3ma8u7Lw3+2m0l7CFi6e4+8t4Eb8Aen4g6Pa2Xs5MSr5p4
  hackerkey.com
Martin Langhoff· Sep 23, 2005, 22:08 UTC · re: Peter TB Brett · lore

Re: git 0.99.7b doesn't build on Cygwin

On 9/24/05, Peter TB Brett <peter@peter-b.co.uk> wrote:
> Ah well, I'll just have to use something else -- I know Mercurial works on
> Windows.  To be honest, I'd prefer to use git though; I've used it on
> other projects and it's really nice.

Petr Baudis was working on a Mercurial-GIT gateway which could be useful, or perhaps you can use tailor.py. OTOH, if you have a unix machine in the network, you can probably make cretive use of samba...

cheers,
martin
Petr Baudis· Sep 23, 2005, 22:34 UTC · re: Martin Langhoff · lore

Re: git 0.99.7b doesn't build on Cygwin

Dear diary, on Sat, Sep 24, 2005 at 12:08:28AM CEST, I got a letter where Martin Langhoff <martin.langhoff@gmail.com> told me that...

Show 8 quoted lines
> On 9/24/05, Peter TB Brett <peter@peter-b.co.uk> wrote:
> > Ah well, I'll just have to use something else -- I know Mercurial works on
> > Windows.  To be honest, I'd prefer to use git though; I've used it on
> > other projects and it's really nice.
> 
> Petr Baudis was working on a Mercurial-GIT gateway which could be
> useful, or perhaps you can use tailor.py. OTOH, if you have a unix
> machine in the network, you can probably make cretive use of samba...
Clarification: that was a Monotone-GIT gateway.
-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Linus Torvalds· Sep 24, 2005, 00:09 UTC · re: Johannes Schindelin · lore

Re: git 0.99.7b doesn't build on Cygwin

On Fri, 23 Sep 2005, Johannes Schindelin wrote:
Show 6 quoted lines
> 
> It seems that the fixup of the mmap()ed regions after a fork() does not 
> work properly in cygwin. Remember that cygwin just wraps the non-POSIX 
> Win32API and tries to make it sort of POSIX compliant. The problem is that 
> Win32API lacks a proper fork(). This is therefore emulated, and after 
> that, all the mmap()ed regions have to be mapped again. That fails.

Now, I'm not a big fan of windows ("No, really? Tell us more!") but I'd actually like it if the _core_ git stuff worked in as wide a variety of situations as possible. Screw the shell scripts and the daemon or secondary things like that which windows users might as well generate their own stuff for, but I'd hope the really core stuff would work.

If I understood correctly, you said that "git-diff-tree" doesn't work due to the fork/mmap issue. Now, I assume that means that it's the builtin diff that has problems.

As far as I can tell, we can solve that two ways:
 - make Windows always use the external diff program. That may be the 
   right thing to do, since then the fork() just turns into a regular 
   fork+exec, which is how windows works anyway.
 - look at doing the diff internally.

I'm wondering if there is some stupid way to turn a diff generated by diff_delta() into a line-based one? If you have the original file and the xdiff, I think we should be able to just walk the original file and output a unified diff.

Davide, maybe I'm being stupid, but I'm thinking that it might be possible to generate a -u3 diff by basically walking the xdiff file in a linear fashion: if the edits are in strictly ascending order, we could walk the original file one line at a time, and keeping a buffer of the three last lines. Then, when the file offset hits the next "edit" in the xdiff, we start generating a line-based diff (and use the previous three lines as the context).

Does that sound possible? Maybe somebody has even done it? Is it a stupid idea?

I realize that it might not generate the same diff as GNU diff would do, and maybe it's really nasty, but it sounds like it _could_ be a "cheap" way of generating diffs, considering that we have something that already generates xdiffs..

		Linus
Linus Torvalds· Sep 24, 2005, 00:43 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

On Fri, 23 Sep 2005, Linus Torvalds wrote:
> 
> Davide, maybe I'm being stupid, but I'm thinking that it might be possible
> to generate a -u3 diff by basically walking the xdiff file in a linear
> fashion: if the edits are in strictly ascending order

Ahh, no. Looking at the xdiff format, the edits are strictly ascending in the destination, but they may take stuff from the source in any order, so it's not like you can generate a diff from it by just walking it.

Oh, well. I guess we're better off just using the external diff command, even if it is slower to execve an external diff.

The GNU diff sources are hard enough to read that I don't think we want to try to merge the unified diff generation from there.

		Linus
Junio C Hamano· Sep 25, 2005, 07:52 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

Linus Torvalds <torvalds@osdl.org> writes:
> The GNU diff sources are hard enough to read that I don't think we want to 
> try to merge the unified diff generation from there.

I was talking with GNU diff maintainer and his impression was that CVS folks may have done enough libification -- I'll find time to look at CVS code and see how much damage we are talking about.

Johannes Schindelin· Sep 25, 2005, 15:47 UTC · re: Junio C Hamano · lore

Implementing diff, was Re: git 0.99.7b doesn't build on Cygwin

Hi,
On Sun, 25 Sep 2005, Junio C Hamano wrote:
Show 9 quoted lines
> Linus Torvalds <torvalds@osdl.org> writes:
> 
> > The GNU diff sources are hard enough to read that I don't think we want to 
> > try to merge the unified diff generation from there.
> 
> I was talking with GNU diff maintainer and his impression was
> that CVS folks may have done enough libification -- I'll find
> time to look at CVS code and see how much damage we are talking
> about.

I am not sure if it would be wise to completely do away with the current method: Often, I call git-diff with my own wdiff-helper. Also, options like "-b" to diff are very useful, and would have to be implemented, too.

Ciao, Dscho

Davide Libenzi· Sep 25, 2005, 16:08 UTC · re: Johannes Schindelin · lore

Re: Implementing diff, was Re: git 0.99.7b doesn't build on Cygwin

On Sun, 25 Sep 2005, Johannes Schindelin wrote:
Show 13 quoted lines
>> Linus Torvalds <torvalds@osdl.org> writes:
>>
>>> The GNU diff sources are hard enough to read that I don't think we want to
>>> try to merge the unified diff generation from there.
>>
>> I was talking with GNU diff maintainer and his impression was
>> that CVS folks may have done enough libification -- I'll find
>> time to look at CVS code and see how much damage we are talking
>> about.
>
> I am not sure if it would be wise to completely do away with the current
> method: Often, I call git-diff with my own wdiff-helper. Also, options
> like "-b" to diff are very useful, and would have to be implemented, too.

What you'd have to do, if you chose to use diffutils stuff, is to transform the main() of diff in diff_main(), use setjmp/longjmp to capture its exit()s, and make it use a proper allocator (if you want to avoid leaks upon aborts). You can see an example inside the diff/libgdiff directory of this packages:

https://www.cvshome.org http://www.opencm.org

In that way, instead of executing "diff -u ...", you'd call diff_main() with the proper args array. The CVS one (the other project seems dead, and they lifted the thing from CVS anyway) should be readily usable.

- Davide
Linus Torvalds· Sep 25, 2005, 17:00 UTC · re: Davide Libenzi · lore

Re: Implementing diff, was Re: git 0.99.7b doesn't build on Cygwin

On Sun, 25 Sep 2005, Davide Libenzi wrote:
Show 5 quoted lines
>
> What you'd have to do, if you chose to use diffutils stuff, is to 
> transform the main() of diff in diff_main(), use setjmp/longjmp to capture 
> its exit()s, and make it use a proper allocator (if you want to avoid 
> leaks upon aborts).

I'd love to use libxdiff instead since you say it can do it, but quite frankly, the man-page didn't much help me. Do you have an example of how to generate a uni-diff with it? Something that mortal men can read and say "oh"?

		Linus
Davide Libenzi· Sep 25, 2005, 19:16 UTC · re: Linus Torvalds · lore

Re: Implementing diff, was Re: git 0.99.7b doesn't build on Cygwin

On Sun, 25 Sep 2005, Linus Torvalds wrote:
Show 11 quoted lines
> On Sun, 25 Sep 2005, Davide Libenzi wrote:
>>
>> What you'd have to do, if you chose to use diffutils stuff, is to
>> transform the main() of diff in diff_main(), use setjmp/longjmp to capture
>> its exit()s, and make it use a proper allocator (if you want to avoid
>> leaks upon aborts).
>
> I'd love to use libxdiff instead since you say it can do it, but quite
> frankly, the man-page didn't much help me. Do you have an example of how
> to generate a uni-diff with it? Something that mortal men can read and say
> "oh"?

Ahh, you looked at the docs. Don't do that ;) Take a look at the test/xdiff_test.c for an example on how to use its most important APIs. There is also a regression test (xregression) that create random files, renadomly changes them, and then try that (A-B)+B=A and (B-A)+A=B (for both text and binary). I'm dropping inline an example on how to text-diff ...

- Davide

#include "xmacros.h" #include "xdiff.h"

#define XDLT_STD_BLKSIZE (1024 * 8)
static int xdlt_load_mmfile(char const *fname, mmfile_t *mf, int binmode) {
         char cc;
         int fd;
         long size, bsize;
         char *blk;
         if (xdl_init_mmfile(mf, XDLT_STD_BLKSIZE, XDL_MMF_ATOMIC) < 0) {
                 return -1;
         }
         if ((fd = open(fname, O_RDONLY)) == -1) {
                 perror(fname);
                 xdl_free_mmfile(mf);
                 return -1;
         }
         if ((size = bsize = lseek(fd, 0, SEEK_END)) > 0 && !binmode) {
                 if (lseek(fd, -1, SEEK_END) != (off_t) -1 &&
                     read(fd, &cc, 1) && cc != '\n')
                         bsize++;
         }
         lseek(fd, 0, SEEK_SET);
         if (!(blk = (char *) xdl_mmfile_writeallocate(mf, bsize))) {
                 xdl_free_mmfile(mf);
                 close(fd);
                 return -1;
         }
         if (read(fd, blk, (size_t) size) != (size_t) size) {
                 perror(fname);
                 xdl_free_mmfile(mf);
                 close(fd);
                 return -1;
         }
         close(fd);
         if (bsize > size)
                 blk[size] = '\n';
         return 0;
}
static int xdlt_outf(void *priv, mmbuffer_t *mb, int nbuf) {
         int i;
         for (i = 0; i < nbuf; i++)
                 if (!fwrite(mb[i].ptr, mb[i].size, 1, (FILE *) priv))
                         return -1;
         return 0;
}
static void *wrap_malloc(void *priv, unsigned int size) {
         return malloc(size);
}
static void wrap_free(void *priv, void *ptr) {
         free(ptr);
}
static void *wrap_realloc(void *priv, void *ptr, unsigned int size) {
         return realloc(ptr, size);
}
int sample_textdiff(char const *pre, char const *post, FILE *outf) {
         int error;
         memallocator_t malt;
         mmfile_t mf1, mf2;
         xpparam_t xpp;
         xdemitconf_t xecfg;
         xdemitcb_t ecb;
         malt.priv = NULL;
         malt.malloc = wrap_malloc;
         malt.free = wrap_free;
         malt.realloc = wrap_realloc;
         xdl_set_allocator(&malt);
         if (xdlt_load_mmfile(pre, &mf1, 0) < 0)
                 return -1;
         if (xdlt_load_mmfile(post, &mf2, 0) < 0) {
                 xdl_free_mmfile(&mf1);
                 return -1;
         }
         xpp.flags = 0;
         xecfg.ctxlen = 3;
         ecb.priv = outf;
         ecb.outf = xdlt_outf;
         error = xdl_diff(&mf1, &mf2, &xpp, &xecfg, &ecb);
         xdl_free_mmfile(&mf2);
         xdl_free_mmfile(&mf1);
         return error;
}
Johannes Schindelin· Sep 24, 2005, 01:13 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

Hi,
On Fri, 23 Sep 2005, Linus Torvalds wrote:
Show 11 quoted lines
> On Fri, 23 Sep 2005, Johannes Schindelin wrote:
> > 
> > It seems that the fixup of the mmap()ed regions after a fork() does not 
> > work properly in cygwin. Remember that cygwin just wraps the non-POSIX 
> > Win32API and tries to make it sort of POSIX compliant. The problem is that 
> > Win32API lacks a proper fork(). This is therefore emulated, and after 
> > that, all the mmap()ed regions have to be mapped again. That fails.
> 
> Now, I'm not a big fan of windows ("No, really? Tell us more!") but I'd 
> actually like it if the _core_ git stuff worked in as wide a variety of 
> situations as possible.

It is sure worth to try to be as portable as possible. Just look at the bugs found by running git on x86_64 (for example by HPA), which were not apparent from x86 or PowerPC.

> Screw the shell scripts and the daemon or secondary things like that 
> which windows users might as well generate their own stuff for, but I'd 
> hope the really core stuff would work.

Whoa, slow! The shell scripts and the networking are important parts even of the core git suite. Without them, work is next to impossible.

> If I understood correctly, you said that "git-diff-tree" doesn't work due
> to the fork/mmap issue. Now, I assume that means that it's the builtin
> diff that has problems. 

No. It means that there is something weird going on inside cygwin1.dll. This library works perfectly when the program is run inside gdb. Which could well mean some timing issue. Unfortunately, I have problems rebuilding cygwin1.dll, and therefore cannot debug in detail.

BTW I am fairly convinced that the same issues would trouble a git-pull, once the networking is running, since the pack transfer relies on fork()ing.

> I'm wondering if there is some stupid way to turn a diff generated by 
> diff_delta() into a line-based one? If you have the original file and the 
> xdiff, I think we should be able to just walk the original file and output 
> a unified diff.

It sure would be nice to have a unified diff generator included, but I doubt that a reliable (=simple) one is easy to come by.

Ciao, Dscho

Linus Torvalds· Sep 24, 2005, 02:46 UTC · re: Johannes Schindelin · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Johannes Schindelin wrote:
> 
> BTW I am fairly convinced that the same issues would trouble a git-pull, 
> once the networking is running, since the pack transfer relies on 
> fork()ing.
I'm not sure.

Almost all other fork() users end up doing a more-or-less immediate execve() after the fork. Yes, they do some other minor setup, but not a whole lot.

The diff.c fork() is somewhat different. It actually ends up doing malloc and stdio IO before it actually gets to the exec(), so that one is more likely to hit any bugs in the fork() implementation.

Actually, looking a bit closer, the create_pack_file() thing also does malloc inside the child, but at least there it would be trivial to move that argument setup code into the parent.

But looking at send_pack() or fetch_pack(), for example, they are both _very_ traditional fork()+exec() calls, with just a few close() calls in between.

Looking a bit closer at the diff() usage, I actually think that we could move the fork() closer to the exec - we'd just have to move it _into_ all the different cases (ie you'd have two different fork() calls: one for the "builtin" case, one for the external pgm case, but then the child in both cases would be very simple).

Oh. Actually, I wonder if we could mke them "vfork()" calls. Does anybody know if cygwin has an easier time with vfork() + eventual exec? That _should_ map better to a non-UNIX process model, so maybe we could do it that way?

> It sure would be nice to have a unified diff generator included, but I 
> doubt that a reliable (=simple) one is easy to come by.

Yeah, I looked at GNU diffutils, and I had to rinse out my eyes with soap and water.

		Linus
Junio C Hamano· Sep 24, 2005, 03:04 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

Linus Torvalds <torvalds@osdl.org> writes:
Show 5 quoted lines
> Looking a bit closer at the diff() usage, I actually think that we could 
> move the fork() closer to the exec - we'd just have to move it _into_ all 
> the different cases (ie you'd have two different fork() calls: one for 
> the "builtin" case, one for the external pgm case, but then the child in 
> both cases would be very simple).

Looking back at what I did in the diff.c, I actually think the part near fork() is a total crap ;-).

Originally I intended to do more work in the child process (this is totally opposite of what is being proposed now), for example running prepare_temp_file() after child forked, so that the parent process does not have to worry about using memory for expanded blob to be written out to the temporary file and then later forgetting to free it ;-), but it seems the parent is doing more work than I intended to. I honestly think that the part of the code is ancient enough to deserve a major facelift.

Davide Libenzi· Sep 24, 2005, 05:26 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

On Fri, 23 Sep 2005, Linus Torvalds wrote:
Show 14 quoted lines
> But looking at send_pack() or fetch_pack(), for example, they are both
> _very_ traditional fork()+exec() calls, with just a few close() calls in
> between.
>
> Looking a bit closer at the diff() usage, I actually think that we could
> move the fork() closer to the exec - we'd just have to move it _into_ all
> the different cases (ie you'd have two different fork() calls: one for
> the "builtin" case, one for the external pgm case, but then the child in
> both cases would be very simple).
>
> Oh. Actually, I wonder if we could mke them "vfork()" calls. Does anybody
> know if cygwin has an easier time with vfork() + eventual exec? That
> _should_ map better to a non-UNIX process model, so maybe we could do it
> that way?

If you have only to run diff/patch, just use the native Win32 CreateProcess(). You abstract that on a git_exec(), and you use fork/exec on Unix and CreateProcess() on Winblows. If fork() is slow on Cygwin, fork+exec is pathetic. They do all that work to give you a fork(), and you throw it away with an exec().

- Davide
Linus Torvalds· Sep 24, 2005, 18:10 UTC · re: Davide Libenzi · lore

Re: git 0.99.7b doesn't build on Cygwin

On Fri, 23 Sep 2005, Davide Libenzi wrote:
Show 6 quoted lines
> 
> If you have only to run diff/patch, just use the native Win32 CreateProcess().
> You abstract that on a git_exec(), and you use fork/exec on Unix and 
> CreateProcess() on Winblows. If fork() is slow on Cygwin, fork+exec is 
> pathetic. They do all that work to give you a fork(), and you throw it 
> away with an exec().

CreateProcess doesn't work all that well, since we want to dup file descriptors around and close them in the child.

In general, CreateProcess() is a totally crap interface. I realize it's common (and especially in the VMS/Windows world it's how things are done), but hey, at that point it's better if somebody just waits until git is stable, and just makes a totally separate "git for windows" thing. The interfaces are certainly simple. There's no point in trying to maintain one tree.

However, vfork() really _is_ a nice interface. It's faster even on UNIX, and at least in theory it should be possible to do an efficient vfork() implementation on top of crap like windows. Does cygwin support that well?

Yes, git uses lots of filesystem stuff, and they suck under windows. Maybe cygwin adds its own overhead, but from everything I've ever been able to tell, filesystem access sucks under Windows regardless of any cygwin stuff. Add to an already slow FS interface the fact that virus checkers tend to hook into it and make it _even_slower_, and hey, you have a truly sucky OS.

But at least with pack-files, the filesystem access patterns are much less common. Opening one pack-file and mapping it gets the FS out of the way. So I don't think that's necessarily a huge problem.

		Linus
Davide Libenzi· Sep 24, 2005, 19:12 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Linus Torvalds wrote:
Show 12 quoted lines
>
>
> On Fri, 23 Sep 2005, Davide Libenzi wrote:
>>
>> If you have only to run diff/patch, just use the native Win32 CreateProcess().
>> You abstract that on a git_exec(), and you use fork/exec on Unix and
>> CreateProcess() on Winblows. If fork() is slow on Cygwin, fork+exec is
>> pathetic. They do all that work to give you a fork(), and you throw it
>> away with an exec().
>
> CreateProcess doesn't work all that well, since we want to dup file
> descriptors around and close them in the child.

You can do that (dup stuff and pass them around) even with CreateProcess(), if you want. Yes, the interface sucks (zillions of parameters/flags) ;)

Show 17 quoted lines
> In general, CreateProcess() is a totally crap interface. I realize it's
> common (and especially in the VMS/Windows world it's how things are done),
> but hey, at that point it's better if somebody just waits until git is
> stable, and just makes a totally separate "git for windows" thing. The
> interfaces are certainly simple. There's no point in trying to maintain
> one tree.
>
> However, vfork() really _is_ a nice interface. It's faster even on UNIX,
> and at least in theory it should be possible to do an efficient vfork()
> implementation on top of crap like windows. Does cygwin support that well?
>
> Yes, git uses lots of filesystem stuff, and they suck under windows. Maybe
> cygwin adds its own overhead, but from everything I've ever been able to
> tell, filesystem access sucks under Windows regardless of any cygwin
> stuff. Add to an already slow FS interface the fact that virus checkers
> tend to hook into it and make it _even_slower_, and hey, you have a truly
> sucky OS.

I also realized that git plays/handles with unix permissions too, and this might make the "interface layer" not so small. Dunno about vfork() on Cygwin, but if you really care about performance on Windows, I'd rather remove the external program execution and use an in-process diff library.

- Davide
Davide Libenzi· Sep 24, 2005, 21:28 UTC · re: Junio C Hamano · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Junio C Hamano wrote:
> Making really really core part usable on Windows would not need
> this, but there is another thing: .git/HEAD symlink.
Starting from Win2k, they *finally* added:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
- Davide
Junio C Hamano· Sep 24, 2005, 21:46 UTC · re: Davide Libenzi · lore

Re: git 0.99.7b doesn't build on Cygwin

Davide Libenzi <davidel@xmailserver.org> writes:
Show 6 quoted lines
> On Sat, 24 Sep 2005, Junio C Hamano wrote:
>
>> Making really really core part usable on Windows would not need
>> this, but there is another thing: .git/HEAD symlink.
>
> Starting from Win2k, they *finally* added:
Ah, that's good to know.
Junio C Hamano· Sep 24, 2005, 21:47 UTC · re: Davide Libenzi · lore

Re: git 0.99.7b doesn't build on Cygwin

Davide Libenzi <davidel@xmailserver.org> writes:
Show 8 quoted lines
> On Sat, 24 Sep 2005, Junio C Hamano wrote:
>
>> Making really really core part usable on Windows would not need
>> this, but there is another thing: .git/HEAD symlink.
>
> Starting from Win2k, they *finally* added:
>
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
It talks about "a hard link".  Can we readlink it?
Davide Libenzi· Sep 24, 2005, 21:52 UTC · re: Junio C Hamano · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Junio C Hamano wrote:
Show 12 quoted lines
> Davide Libenzi <davidel@xmailserver.org> writes:
>
>> On Sat, 24 Sep 2005, Junio C Hamano wrote:
>>
>>> Making really really core part usable on Windows would not need
>>> this, but there is another thing: .git/HEAD symlink.
>>
>> Starting from Win2k, they *finally* added:
>>
>> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
>
> It talks about "a hard link".  Can we readlink it?
Nope. It's an hardlink (ala link(2)).
- Davide
Linus Torvalds· Sep 24, 2005, 22:26 UTC · re: Davide Libenzi · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Davide Libenzi wrote:
> 
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
Don't you mean
	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createsymboliclink.asp
rather?
It mentions longhorn.
		Linus
Linus Torvalds· Sep 24, 2005, 22:27 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Linus Torvalds wrote:
> 
> It mentions longhorn.

Anyway, regardless, we could certainly make HEAD be a regular file containing the name of the head instead.

It probably wouldn't even require a whole lot of changes. HEAD already ends up getting some special attention, since most of the things that look for refs only look inside the .git/refs directory.

		Linus
Linus Torvalds· Sep 25, 2005, 16:59 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Linus Torvalds wrote:
Show 7 quoted lines
>
> Anyway, regardless, we could certainly make HEAD be a regular file 
> containing the name of the head instead.
> 
> It probably wouldn't even require a whole lot of changes. HEAD already 
> ends up getting some special attention, since most of the things that look 
> for refs only look inside the .git/refs directory.

This patch does some of it. I decided not to special-case HEAD, but to just improve "read_ref()" a bit.

Changing "read_ref()" was the trivial part - the bigger part that we had three different implementations of it, and this patch is thus bigger just because it collapses them all into "read_ref()" and makes the calling conventions acceptable to all.

NOTE! This makes "symbolic refs" usable in general, ie you can do
	echo "ref: refs/tags/v0.99.7" > .git/refs/tags/LATEST

and that essentially makes "LATEST" a symbolic ref that points to the v0.99.7 tag without using a filesystem symlink. But it does NOT mean that you can replace the HEAD symlink with a file containing "refs/tags/master" yet: there are _other_ parts of git that depend on it being a symlink.

(For the most core example, the "write new head" logic depends on just writing to HEAD, and that symlink will automatically change that write to the thing the HEAD _points_ to. That's the biggest one).

I'll change those too to accept a regular file, if people agree this is worthwhile. In theory there are even UNIXes out there that don't support symlinks, so maybe it's worth it. But maybe people dislike this.

In the meantime, you can test this out with
	echo "ref: HEAD" > .git/TEST_HEAD
	git-rev-parse HEAD TEST_HEAD master

which - if your HEAD points to master - should print out the same SHA1 three times ;)

		Linus
---
Subject: Allow reading "symbolic refs" that point to other refs

This extends the ref reading to understand a "symbolic ref": a ref file that starts with "ref: " and points to another ref file, and thus introduces the notion of ref aliases.

This is in preparation of allowing HEAD to eventually not be a symlink, but one of these symbolic refs instead.

Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---
diff --git a/cache.h b/cache.h
--- a/cache.h
+++ b/cache.h
@@ -227,6 +227,7 @@ extern int has_pack_index(const unsigned
 extern int get_sha1(const char *str, unsigned char *sha1);
 extern int get_sha1_hex(const char *hex, unsigned char *sha1);
 extern char *sha1_to_hex(const unsigned char *sha1);	/* static buffer result! */
+extern int read_ref(const char *filename, unsigned char *sha1);
 
 /* General helper functions */
 extern void usage(const char *err) NORETURN;
diff --git a/refs.c b/refs.c
--- a/refs.c
+++ b/refs.c
@@ -2,17 +2,38 @@
 #include "cache.h"
 
 #include <errno.h>
+#include <ctype.h>
 
-static int read_ref(const char *refname, unsigned char *sha1)
+/* We allow "recursive" symbolic refs. Only within reason, though */
+#define MAXDEPTH 5
+
+int read_ref(const char *filename, unsigned char *sha1)
 {
-	int ret = -1;
-	int fd = open(git_path("%s", refname), O_RDONLY);
+	int depth = 0;
+	int ret = -1, fd;
+
+	while ((fd = open(filename, O_RDONLY)) >= 0) {
+		char buffer[256];
+		int len = read(fd, buffer, sizeof(buffer)-1);
 
-	if (fd >= 0) {
-		char buffer[60];
-		if (read(fd, buffer, sizeof(buffer)) >= 40)
-			ret = get_sha1_hex(buffer, sha1);
 		close(fd);
+		if (len < 0)
+			break;
+
+		buffer[len] = 0;
+		while (len && isspace(buffer[len-1]))
+			buffer[--len] = 0;
+
+		if (!strncmp(buffer, "ref: ", 5)) {
+			if (depth > MAXDEPTH)
+				break;
+			depth++;
+			filename = git_path("%s", buffer+5);
+			continue;
+		}
+		if (len >= 40)
+			ret = get_sha1_hex(buffer, sha1);
+		break;
 	}
 	return ret;
 }
@@ -54,7 +75,7 @@ static int do_for_each_ref(const char *b
 					break;
 				continue;
 			}
-			if (read_ref(path, sha1) < 0)
+			if (read_ref(git_path("%s", path), sha1) < 0)
 				continue;
 			if (!has_sha1_file(sha1))
 				continue;
@@ -71,7 +92,7 @@ static int do_for_each_ref(const char *b
 int head_ref(int (*fn)(const char *path, const unsigned char *sha1))
 {
 	unsigned char sha1[20];
-	if (!read_ref("HEAD", sha1))
+	if (!read_ref(git_path("HEAD"), sha1))
 		return fn("HEAD", sha1);
 	return 0;
 }
@@ -101,33 +122,14 @@ static char *ref_lock_file_name(const ch
 	return ret;
 }
 
-static int read_ref_file(const char *filename, unsigned char *sha1) {
-	int fd = open(filename, O_RDONLY);
-	char hex[41];
-	if (fd < 0) {
-		return error("Couldn't open %s\n", filename);
-	}
-	if ((read(fd, hex, 41) < 41) ||
-	    (hex[40] != '\n') ||
-	    get_sha1_hex(hex, sha1)) {
-		error("Couldn't read a hash from %s\n", filename);
-		close(fd);
-		return -1;
-	}
-	close(fd);
-	return 0;
-}
-
 int get_ref_sha1(const char *ref, unsigned char *sha1)
 {
-	char *filename;
-	int retval;
+	const char *filename;
+
 	if (check_ref_format(ref))
 		return -1;
-	filename = ref_file_name(ref);
-	retval = read_ref_file(filename, sha1);
-	free(filename);
-	return retval;
+	filename = git_path("refs/%s", ref);
+	return read_ref(filename, sha1);
 }
 
 static int lock_ref_file(const char *filename, const char *lock_filename,
@@ -140,7 +142,7 @@ static int lock_ref_file(const char *fil
 		return error("Couldn't open lock file for %s: %s",
 			     filename, strerror(errno));
 	}
-	retval = read_ref_file(filename, current_sha1);
+	retval = read_ref(filename, current_sha1);
 	if (old_sha1) {
 		if (retval) {
 			close(fd);
diff --git a/sha1_name.c b/sha1_name.c
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -119,21 +119,6 @@ static int get_short_sha1(const char *na
 	return -1;
 }
 
-static int get_sha1_file(const char *path, unsigned char *result)
-{
-	char buffer[60];
-	int fd = open(path, O_RDONLY);
-	int len;
-
-	if (fd < 0)
-		return -1;
-	len = read(fd, buffer, sizeof(buffer));
-	close(fd);
-	if (len < 40)
-		return -1;
-	return get_sha1_hex(buffer, result);
-}
-
 static int get_sha1_basic(const char *str, int len, unsigned char *sha1)
 {
 	static const char *prefix[] = {
@@ -150,7 +135,7 @@ static int get_sha1_basic(const char *st
 
 	for (p = prefix; *p; p++) {
 		char *pathname = git_path("%s/%.*s", *p, len, str);
-		if (!get_sha1_file(pathname, sha1))
+		if (!read_ref(pathname, sha1))
 			return 0;
 	}
 
Jon Loeliger· Sep 26, 2005, 19:33 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 2005-09-24 at 17:27, Linus Torvalds wrote:
Show 6 quoted lines
> On Sat, 24 Sep 2005, Linus Torvalds wrote:
> > 
> > It mentions longhorn.
> 
> Anyway, regardless, we could certainly make HEAD be a regular file 
> containing the name of the head instead.

I cleverly removed a branch this morning (intentionally), but what I failed to realize was that it was also the current HEAD. The next series of command were oddly cryptic, only telling me the git-read-tree usage message. Not a clue in the world why, of course. After poking around, reading some scripts and all, I discovered that I had a dangling .git/HEAD symlink still point to my removed branch.

Would it be worthwhile for me to re-discover this Wonky Failure (Hi Linus! :-) and try to write it up somewhere or dream up a better error message patch?

Thanks, jdl

Junio C Hamano· Sep 26, 2005, 20:23 UTC · re: Jon Loeliger · lore

Re: git 0.99.7b doesn't build on Cygwin

Jon Loeliger <jdl@freescale.com> writes:
> Would it be worthwhile for me to re-discover this
> Wonky Failure (Hi Linus! :-) and try to write it up
> somewhere or dream up a better error message patch?
Thanks.
Davide Libenzi· Sep 24, 2005, 22:41 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Linus Torvalds wrote:
Show 13 quoted lines
>
>
> On Sat, 24 Sep 2005, Davide Libenzi wrote:
>>
>> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
>
> Don't you mean
>
> 	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createsymboliclink.asp
>
> rather?
>
> It mentions longhorn.
Hah, didn't know this one. Requiring LongHorn is pretty strict though ;)
- Davide
Giuseppe Bilotta· Sep 25, 2005, 19:59 UTC · re: Davide Libenzi · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005 15:41:27 -0700 (PDT), Davide Libenzi wrote:
Show 17 quoted lines
> On Sat, 24 Sep 2005, Linus Torvalds wrote:
> 
>>
>>
>> On Sat, 24 Sep 2005, Davide Libenzi wrote:
>>>
>>> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createhardlink.asp
>>
>> Don't you mean
>>
>> 	http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createsymboliclink.asp
>>
>> rather?
>>
>> It mentions longhorn.
> 
> Hah, didn't know this one. Requiring LongHorn is pretty strict though ;)

However, it might be possible to use .lnk files, which would work on both NTFS and FAT32, and even under Win9x.

-- 
Giuseppe "Oblomov" Bilotta

"I weep for our generation" -- Charlie Brown
Junio C Hamano· Sep 26, 2005, 04:57 UTC · re: Giuseppe Bilotta · lore

Re: git 0.99.7b doesn't build on Cygwin

Giuseppe Bilotta <bilotta78@hotpop.com> writes:
>> Hah, didn't know this one. Requiring LongHorn is pretty strict though ;)
>
> However, it might be possible to use .lnk files, which would work on
> both NTFS and FAT32, and even under Win9x.
Possibly, but it is a moot point now.

When textual "symbolic refs" support becomes mature, we will use it on boxes without symbolic links to express .git/HEAD.

Davide Libenzi· Sep 26, 2005, 05:05 UTC · re: Giuseppe Bilotta · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sun, 25 Sep 2005, Giuseppe Bilotta wrote:
> However, it might be possible to use .lnk files, which would work on
> both NTFS and FAT32, and even under Win9x.

The .lnk files are a shell thing, not an OS one. Try to open()+read() a .lnk file and look at what you get ...

- Davide
Giuseppe Bilotta· Sep 26, 2005, 11:00 UTC · re: Davide Libenzi · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sun, 25 Sep 2005 22:05:29 -0700 (PDT), Davide Libenzi wrote:
Show 7 quoted lines
> On Sun, 25 Sep 2005, Giuseppe Bilotta wrote:
> 
>> However, it might be possible to use .lnk files, which would work on
>> both NTFS and FAT32, and even under Win9x.
> 
> The .lnk files are a shell thing, not an OS one. Try to open()+read() a 
> .lnk file and look at what you get ...

Well, sure. I wasn't thinking about just substituting .lnk files for symlinks, but that's the closest thing you can get on Windows, currently, so maybe supporting this kind of thing would be the best approach.

-- 
Giuseppe "Oblomov" Bilotta

[W]hat country can preserve its liberties, if its rulers are not
warned from time to time that [the] people preserve the spirit of
resistance? Let them take arms...The tree of liberty must be
refreshed from time to time, with the blood of patriots and
tyrants.
	-- Thomas Jefferson, letter to Col. William S. Smith, 1787
H. Peter Anvin· Sep 26, 2005, 21:54 UTC · re: Davide Libenzi · lore

Re: git 0.99.7b doesn't build on Cygwin

Davide Libenzi wrote:
Show 9 quoted lines
> On Sun, 25 Sep 2005, Giuseppe Bilotta wrote:
> 
>> However, it might be possible to use .lnk files, which would work on
>> both NTFS and FAT32, and even under Win9x.
> 
> 
> The .lnk files are a shell thing, not an OS one. Try to open()+read() a 
> .lnk file and look at what you get ...
> 

Except that Cygwin uses them transparently, so if you do open() and read() under Cygwin they work as expected.

	-hpa
Davide Libenzi· Sep 26, 2005, 22:03 UTC · re: H. Peter Anvin · lore

Re: git 0.99.7b doesn't build on Cygwin

On Mon, 26 Sep 2005, H. Peter Anvin wrote:
Show 13 quoted lines
> Davide Libenzi wrote:
>> On Sun, 25 Sep 2005, Giuseppe Bilotta wrote:
>> 
>>> However, it might be possible to use .lnk files, which would work on
>>> both NTFS and FAT32, and even under Win9x.
>> 
>> 
>> The .lnk files are a shell thing, not an OS one. Try to open()+read() a 
>> .lnk file and look at what you get ...
>> 
>
> Except that Cygwin uses them transparently, so if you do open() and read() 
> under Cygwin they work as expected.

With Cygwin you don't even need .lnk files, since it already supports all the Unix symlinks APIs/cmds. The discussion born thinking about a native Win32 interface, w/out the Cygwin crud in it.

- Davide
H. Peter Anvin· Sep 26, 2005, 22:15 UTC · re: Davide Libenzi · lore

Re: git 0.99.7b doesn't build on Cygwin

Davide Libenzi wrote:
Show 8 quoted lines
>>
>> Except that Cygwin uses them transparently, so if you do open() and 
>> read() under Cygwin they work as expected.
> 
> With Cygwin you don't even need .lnk files, since it already supports 
> all the Unix symlinks APIs/cmds. The discussion born thinking about a 
> native Win32 interface, w/out the Cygwin crud in it.
> 

Cygwin symbolic links are implemented as .lnk files on the underlying filesystem was my point.

	-hpaa
Daniel Barkalow· Sep 25, 2005, 03:04 UTC · re: Junio C Hamano · lore

Re: git 0.99.7b doesn't build on Cygwin

On Sat, 24 Sep 2005, Junio C Hamano wrote:
> Making really really core part usable on Windows would not need
> this, but there is another thing: .git/HEAD symlink.

Cygwin supports symlinks without underlying filesystem support. It does basically the standard UNIX thing for symlinks, but inefficiently in userspace.

	-Daniel
*This .sig left intentionally blank*
Davide Libenzi· Sep 24, 2005, 05:11 UTC · re: Linus Torvalds · lore

Re: git 0.99.7b doesn't build on Cygwin

On Fri, 23 Sep 2005, Linus Torvalds wrote:
Show 19 quoted lines
>
>
> On Fri, 23 Sep 2005, Johannes Schindelin wrote:
>>
>> It seems that the fixup of the mmap()ed regions after a fork() does not
>> work properly in cygwin. Remember that cygwin just wraps the non-POSIX
>> Win32API and tries to make it sort of POSIX compliant. The problem is that
>> Win32API lacks a proper fork(). This is therefore emulated, and after
>> that, all the mmap()ed regions have to be mapped again. That fails.
>
> Now, I'm not a big fan of windows ("No, really? Tell us more!") but I'd
> actually like it if the _core_ git stuff worked in as wide a variety of
> situations as possible. Screw the shell scripts and the daemon or
> secondary things like that which windows users might as well generate
> their own stuff for, but I'd hope the really core stuff would work.
>
> If I understood correctly, you said that "git-diff-tree" doesn't work due
> to the fork/mmap issue. Now, I assume that means that it's the builtin
> diff that has problems.

Stay away from Cygwin if you use extensively fork(), since it becomes dog slow (the fork() implementation on Cygwin involves creating a new suspended task, *copy* - not COW - the *whole* VM space to the child, and resuming it). Actually, the whole Cygwin in general is dog slow, especially on FS operations (and git likes them). If you really like to have Windows support (uuu hoo WinTorvalds) you might be better using a small compat layer (should be fairly small for git).

Show 28 quoted lines
> As far as I can tell, we can solve that two ways:
>
> - make Windows always use the external diff program. That may be the
>   right thing to do, since then the fork() just turns into a regular
>   fork+exec, which is how windows works anyway.
>
> - look at doing the diff internally.
>
> I'm wondering if there is some stupid way to turn a diff generated by
> diff_delta() into a line-based one? If you have the original file and the
> xdiff, I think we should be able to just walk the original file and output
> a unified diff.
>
> Davide, maybe I'm being stupid, but I'm thinking that it might be possible
> to generate a -u3 diff by basically walking the xdiff file in a linear
> fashion: if the edits are in strictly ascending order, we could walk the
> original file one line at a time, and keeping a buffer of the three last
> lines. Then, when the file offset hits the next "edit" in the xdiff, we
> start generating a line-based diff (and use the previous three lines as
> the context).
>
> Does that sound possible? Maybe somebody has even done it? Is it a stupid
> idea?
>
> I realize that it might not generate the same diff as GNU diff would do,
> and maybe it's really nasty, but it sounds like it _could_ be a "cheap"
> way of generating diffs, considering that we have something that already
> generates xdiffs..

Hehe, the same library from where Nicolas lifted the code for the binary diff, has a totally portable diff/patch APIs (on top of xdiff/xpach):

http://www.xmailserver.org/xdiff.html

Generating text diffs, unfortunately is quite more complex than binary ones. Libxdiff uses the same algorithm of GNU diff (Eugene W. Myers). The library has zero dependency other than ANSI C. Another alternative, IIRC someone made a library by wrapping the diffutil stuff, but I do not remeber where it was.

- Davide

← back to recent threads