threads / discuss / 4173

Git 1.3.2 on Solaris

Subject: Git 1.3.2 on Solaris

## tl;dr

36 messages between May 16, 2006 and May 26, 2006.

replies: 35people: 7as markdown or json

Stefan Pfetzing· May 16, 2006, 23:52 UTC · lore
Hi,

I've been trying to get git to work on the latest Solaris Express release (with the help of NetBSD's pkgsrc).

It mostly miserabely fails because of common "shell commands" being used with GNU options. (like xargs, diff, tr and prob. some more) On my box (and thats AFAIK the default when you install gnu coreutils on Solaris) the commands do have a g prefix.

So there are 2 possible solutions to get git working on Solaris.
1.  fix every single shellscript automatically during the build phase
2.  setup a dir which contains symlinks to the "right" binaries and
put that dir into PATH.

No matter what solution is chosen to be the best, I'm volunteering to create a patch for it. :)

(although I personally prefer the second, because its easier...)
bye
Stefan
-- 
        http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.
Jason Riedy· May 17, 2006, 01:25 UTC · re: Stefan Pfetzing · lore

Re: Git 1.3.2 on Solaris

And "Stefan Pfetzing" writes:
 - I've been trying to get git to work on the latest Solaris Express
 - release (with the help of NetBSD's pkgsrc).

I've been using it on Solaris 8 and 9 with the GNU tools in pkgsrc for quite a while, as well as on AIX with the GNU tools available as modules (but I haven't compiled a new AIX version for a month or two).

 - It mostly miserabely fails because of common "shell commands" being
 - used with GNU options. (like xargs, diff, tr and prob. some more) On
 - my box (and thats AFAIK the default when you install gnu coreutils on
 - Solaris) the commands do have a g prefix.

In your pkgsrc mk.conf, use: GNU_PROGRAM_PREFIX= GTAR_PROGRAM_PREFIX=

I tried your first suggestion (patch all the commands) back in February. It's pretty fragile against future changes, and I wouldn't recommend it.

 - 2.  setup a dir which contains symlinks to the "right" binaries and
 - put that dir into PATH.
Setting a GIT_COMPAT_PATH in the Makefile and prepending
it to the path in git.c and git-sh-setup.sh might be more
sane.  A fragment like the following in git.c before adding
GIT_EXEC_PATH:
#ifdef GIT_COMPAT_PATH
	/* Search for sane external utilities */
	prepend_to_path(GIT_COMPAT_PATH, strlen(GIT_COMPAT_PATH));
#endif
And maybe in git-sh-setup.sh to help those of us who
use git-foo rather than git foo:
if [ ! -z "@GIT_COMPAT_PATH@" ] ; then
	PATH="@GIT_COMPAT_PATH@:${PATH}"
	export PATH
fi
Plus Makefile fun.
Jason
Linus Torvalds· May 17, 2006, 02:20 UTC · re: Stefan Pfetzing · lore

Re: Git 1.3.2 on Solaris

[ Junio - see the "grep" issue ]
On Wed, 17 May 2006, Stefan Pfetzing wrote:
Show 6 quoted lines
> 
> So there are 2 possible solutions to get git working on Solaris.
> 
> 1.  fix every single shellscript automatically during the build phase
> 2.  setup a dir which contains symlinks to the "right" binaries and
> put that dir into PATH.
If the biggest issue is git depending on some GNU extensions, I'd really 
suggest
 (a) install all the normal GNU binaries, and put them in the path before 
     git just to get it working (and don't try to change git at all)
 (b) help send in patches that just remove the dependency entirely.

I've been - on and off - trying to libify most of the core git sources, so that the shell scripts can be re-written to be just plain C. Most of the time it's not actually even a huge amount of work, it's just somewhat boring.

Writing them as C usually gets rid of any dependencies on any GNU tools, and hopefully even cygwin. For example, we got rid of one "xargs -0" in the development branch pretty recently, thanks to making "git grep" a built-in.

Of course, I don't think anybody tried the new "git grep" on Solaris, and I think the solaris "grep" lacks the "-H" flag, for example. But that should be easy to fix (for example, replace the use of "--" and "-H" with putting a "/dev/null" as the first filename).

I don't think it's worth it trying to add some compatibility layer for the shell-scripts. We really do want to get rid of them, and the more people that help, the merrier.

In many ways, the libification effort isn't even needed. It's perfectly ok to turn a stupid shell-script (and they really all _are_ pretty stupid) into a builtin-cmd.c C file that just does something really easy like a "fork + execve()" translation of the original shell script.

The complete libification will take some time, and in the meantime, a few silly C files that hard-code the shell logic is probably much preferable to using the shell and all the problems that involves (like the whole problem with quoting arguments - just _gone_ when you do it as a execve() in a simple C program).

So anybody can help with this. If you know shell (and the git shell-scripts aren't even _advanced_ shell), and know some basic C, you're all set to do a trivial conversion from one to the other. And when the libification gets further, your conversion will probably help that (ie maybe libificaiton isn't complete, but a _part_ of the thing can be written to use the library interfaces instead of spawning an external program).

There aren't _that_ many shell programs, and a lot of them are really really trivial (ie they parse the arguments, and then do just a couple of external git commands).

			Linus
Jason Riedy· May 17, 2006, 03:26 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

And Linus Torvalds writes:
 - 
 - The complete libification will take some time, and in the meantime, a few 
 - silly C files that hard-code the shell logic is probably much preferable 
 - to using the shell and all the problems that involves (like the whole 
 - problem with quoting arguments - just _gone_ when you do it as a execve() 
 - in a simple C program).

But for recommending and using git on these systems _now_... Simply translating the shell script into C with execs doesn't help if you're execing one of the known problems, or if the script has embedded, non-trivial Perl. git-clone is the major blocker; a trivial translation would be a great step but won't let people without GNU utilities clone repos.

Plus, alas, Perl modules and Python version drift can be a bit of a problem on the same semi-pristine (or unmaintained, or too-stable) systems, so shell isn't the only thing that needs to go. And that'll take a good deal of effort.

Note that my code snippets weren't a suggested patch. I wouldn't want the easy way out to impede progress on the right thing.

But some local installations may find it much easier to patch git than to instruct users to change their utilities to match what git expects, especially if users have old scripts that would break if they changed their path globally. Luckily, git makes it really easy to keep those patches locally...

Jason
Linus Torvalds· May 17, 2006, 03:49 UTC · re: Jason Riedy · lore

Re: Git 1.3.2 on Solaris

On Tue, 16 May 2006, Jason Riedy wrote:
> 
> But for recommending and using git on these systems _now_...

Yes. For that, I would literally suggest having people install the GNU tools (and/or a recent enough perl) somewhere early in the path.

If you use the git wrapper, for example, you can already depend on the fact that it will prepend the git installation directory to the path, so while the GNU tools might not _normally_ be on the path, if you put them in the same directory as your git install, you'll automatically get them as long as you use the "git cmd" format (rather than the "git-cmd" format).

		Linus
Stefan Pfetzing· May 17, 2006, 08:05 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

Hi Linus,
2006/5/17, Linus Torvalds <torvalds@osdl.org>:
Show 14 quoted lines
>
> On Tue, 16 May 2006, Jason Riedy wrote:
> >
> > But for recommending and using git on these systems _now_...
>
> Yes. For that, I would literally suggest having people install the GNU
> tools (and/or a recent enough perl) somewhere early in the path.
>
> If you use the git wrapper, for example, you can already depend on the
> fact that it will prepend the git installation directory to the path, so
> while the GNU tools might not _normally_ be on the path, if you put them
> in the same directory as your git install, you'll automatically get them
> as long as you use the "git cmd" format (rather than the "git-cmd"
> format).

Well I guess for my pkgsrc environment this won't work. I already (quite some time ago) tried to have gnu coreutils, findutils and diffutils installed without the g prefix. This broke several things on NetBSD and on Solaris.

So I'd prefer a solution where one could set one flag for the Makefile of git, and git would check for the g prefix, create somewhere a directory with symlinks to the "real" gnu binaries and put it into $PATH upon startup of every git c-program or shellscript.

I suggest having these gnu "tools" dependancies removed can only be a long term goal.

bye
dreamind
P.S.: I had to re-sent this mail, somehow gmail did put html crap into it.
--
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.
Linus Torvalds· May 17, 2006, 14:33 UTC · re: Stefan Pfetzing · lore

Re: Git 1.3.2 on Solaris

On Wed, 17 May 2006, Stefan Pfetzing wrote:
Show 5 quoted lines
> 
> So I'd prefer a solution where one could set one flag for the Makefile of git,
> and git would check for the g prefix, create somewhere a directory with
> symlinks to the "real" gnu binaries and put it into $PATH upon startup of
> every git c-program or shellscript.
So let me just quote the thing you quoted but apparently didn't read:
Show 6 quoted lines
> > If you use the git wrapper, for example, you can already depend on the
> > fact that it will prepend the git installation directory to the path, so
> > while the GNU tools might not _normally_ be on the path, if you put them
> > in the same directory as your git install, you'll automatically get them
> > as long as you use the "git cmd" format (rather than the "git-cmd"
> > format).

There already _is_ such a directory. It's your "prefix=" directory plus "bin".

So what you can do is make sure you compile with
	make prefix=/my/git/installation/prefix

and then install the GNU tools in /my/git/installation/prefix/bin, and you're all set.

At most you might have to make some of the tests use "git xyzzy" instead of "git-xyzzy", and run "make install" before "make test".

It wouldn't be wonderful, but hey, I've given alternatives (like using the GNU tools by default, or helping make git more portable in the first place). So it's a hack.

		Linus
Stefan Pfetzing· May 17, 2006, 15:08 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

Hi Linus,
2006/5/17, Linus Torvalds <torvalds@osdl.org>:
>
> So let me just quote the thing you quoted but apparently didn't read:
[snip]
I did read that.
Show 9 quoted lines
> There already _is_ such a directory. It's your "prefix=" directory plus
> "bin".
>
> So what you can do is make sure you compile with
>
>         make prefix=/my/git/installation/prefix
>
> and then install the GNU tools in /my/git/installation/prefix/bin, and
> you're all set.

Ok, if I would do so, my prefix would be /usr/pkg, and the bindir would be /usr/pkg/bin. So I would need to have an xargs and so on symlink in /usr/pkg/bin. But this is simply not acceptable, because it breaks other NetBSD pkgsrc scripts.

Besides that, installing git to a different location is not an option for me, because I want to have git packaged by pkgsrc.

I suggest Junio's solution will work (gitexecdir) but I have to try that later today.

Show 6 quoted lines
> At most you might have to make some of the tests use "git xyzzy" instead
> of "git-xyzzy", and run "make install" before "make test".
>
> It wouldn't be wonderful, but hey, I've given alternatives (like using the
> GNU tools by default, or helping make git more portable in the first
> place). So it's a hack.
Yes I know, as far as I can, I'm willing to help with this.
bye
dreamind
-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.
Linus Torvalds· May 17, 2006, 16:24 UTC · re: Stefan Pfetzing · lore

Re: Git 1.3.2 on Solaris

On Wed, 17 May 2006, Stefan Pfetzing wrote:
Show 6 quoted lines
> 
> Ok, if I would do so, my prefix would be /usr/pkg, and the bindir would be
> /usr/pkg/bin. So I would need to have an xargs and so on symlink in
> /usr/pkg/bin.
> But this is simply not acceptable, because it breaks other NetBSD
> pkgsrc scripts.
DON'T USE /usr/pkg then.

Use /usr/pkg/git-core/share/ or something that is normally not on your path.

And then install _just_ the "git" binary in /usr/pkg/bin.

That must be allowable by whatever solaris packaging rules: it's not like other projects don't have their own internal library files.

Then you install the GNU symlinks under that same
	/usr/pkg/git-core/share/bin

and you're all set. The only binary you can _see_ is "git", and when that executes any scripts or other git binaries, it will set up the path to include that magic hidden directory.

> Besides that, installing git to a different location is not an option
> for me, because I want to have git packaged by pkgsrc.

Now, I'm told pkgsrc is horrible, but it can't be so horrid as to not allow private directories?

		Linus
Jason Riedy· May 17, 2006, 16:35 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

And Linus Torvalds writes:
 - 
 - Now, I'm told pkgsrc is horrible, but it can't be so horrid as to not 
 - allow private directories?
Works fine.
  1) Depend on the GNU tools through the buildlink, um, stuff.
  2) Add a config.mak via a local patch that sets gitexecprefix.
  3) Add another local patch that sets up links within that
     gitexecprefix to the GNU tools.  Remember to check if the
     GNU tools were installed without the silly g prefix.

And pkgsrc itself works just fine without the silly g prefix, or at least does for me as a mere user (and as well as it does work). But if you intend on adding the package upstream, it'll need something to cope with the g. And pkgsrc handles local patches...

Jason
Stefan Pfetzing· May 23, 2006, 03:20 UTC · re: Jason Riedy · lore

Re: Git 1.3.2 on Solaris

Hi Jason,
2006/5/17, Jason Riedy <ejr@eecs.berkeley.edu>:
Show 5 quoted lines
> And pkgsrc itself works just fine without the silly g prefix,
> or at least does for me as a mere user (and as well as it does
> work).  But if you intend on adding the package upstream, it'll
> need something to cope with the g.  And pkgsrc handles local
> patches...

Well I had some problems on NetBSD without the g prefix for the gnu coreutils - since then I always used that prefix.

But now I have a completely different problem with the tests on solaris. It seems on solaris access() always returns 0 if a file is existant and the effective uid is 0.

so: --- snip --- #include <stdio.h> #include <unistd.h>

int
main (int argc, char **argv)
{
  printf ("access: %d\n", access("/etc/motd", X_OK));
  return 0;
}
--- snap ---

will return 0 on solaris - when run as root, even though /etc/motd is not executeable. This seems to break hooks on Solaris - but I'm not sure if this is only a Solaris Express bug. (I have no Solaris 10 system to verify it)

bye
Stefan
-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.
Jason Riedy· May 23, 2006, 04:51 UTC · re: Stefan Pfetzing · lore

Re: Git 1.3.2 on Solaris

And "Stefan Pfetzing" writes:
 -   printf ("access: %d\n", access("/etc/motd", X_OK));
[...]
 - will return 0 on solaris - when run as root, even though /etc/motd
 - is not executeable.
This is explicitly allowed by the SUS, even for non-root:
  http://www.opengroup.org/onlinepubs/000095399/functions/access.html
For non-root, some ACL systems could allow you to execute
the file even if there are no execute bits.  What a joy
ACLs are.  Or NFS uid mappings could play tricks on you,
or...  And as you've noticed, this kills [ -x ].  (Failing
to run the hooks in receive-pack.c is noisy but not fatal.
It's the shell scripts that stop.)

I think you're stuck. To disable the hooks for all possible users, OSes, file systems, etc., you need to remove them.

Or just don't run as root, and hope that the OS isn't completely insane.

BTW, ERR_RUN_COMMAND_EXEC is never returned. Any failure to exec will produce an exit code of 128 from die. This will be an issue when commit becomes a built-in, right?

Jason
Stefan Pfetzing· May 23, 2006, 12:04 UTC · re: Jason Riedy · lore

Re: Git 1.3.2 on Solaris

Hi Jason,
2006/5/23, Jason Riedy <ejr@eecs.berkeley.edu>:
Show 8 quoted lines
> This is explicitly allowed by the SUS, even for non-root:
>   http://www.opengroup.org/onlinepubs/000095399/functions/access.html
> For non-root, some ACL systems could allow you to execute
> the file even if there are no execute bits.  What a joy
> ACLs are.  Or NFS uid mappings could play tricks on you,
> or...  And as you've noticed, this kills [ -x ].  (Failing
> to run the hooks in receive-pack.c is noisy but not fatal.
> It's the shell scripts that stop.)

Yup, but this breaks test t5400 - I think because of the post-update hook is failing.

Show 5 quoted lines
> I think you're stuck.  To disable the hooks for all possible
> users, OSes, file systems, etc., you need to remove them.
>
> Or just don't run as root, and hope that the OS isn't
> completely insane.
As non-root it works fine.
> BTW, ERR_RUN_COMMAND_EXEC is never returned.  Any failure
> to exec will produce an exit code of 128 from die.  This
> will be an issue when commit becomes a built-in, right?
Think so.
bye
Stefan
-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.
Linus Torvalds· May 23, 2006, 14:53 UTC · re: Jason Riedy · lore

Re: Git 1.3.2 on Solaris

On Mon, 22 May 2006, Jason Riedy wrote:
Show 7 quoted lines
> And "Stefan Pfetzing" writes:
>  -   printf ("access: %d\n", access("/etc/motd", X_OK));
> [...]
>  - will return 0 on solaris - when run as root, even though /etc/motd
>  - is not executeable.
> 
> This is explicitly allowed by the SUS, even for non-root:
What kind of CRAP has Solaris become?

Yes, it's allowed. That doesn't mean that a quality implementation should do it.

SunOS used to be the best system around - it was, compared to others, _nice_ to work with. It wasn't about what was "allowed by the standards", that was the HP-SUX and AIX's excuses. It was the highest-quality implementation.

Now, Solaris had some serious problems early on (yes, I was there when they switched, and yes, I hated it), but I thought they had fixed their stuff long long ago.

Have Sun people forgotten the difference between "quality" and "crap that passes standards tests"?

Not doing a reasonable job on "access()" is a joke. It's like VMS being "posix". Sure, it's the letter of the law, but it's still not _unix_.

Btw, even SuS says:
    "The sentence concerning appropriate privileges and execute permission 
     bits reflects the two possibilities implemented by historical 
     implementations when checking superuser access for X_OK.
     New implementations are discouraged from returning X_OK unless at 
     least one execution permission bit is set."
which clearly says "Solaris is CRAP" to me.

What the heck is going on? First the totally broken stdio that doesn't retry on EINTR, now access(). And these people think they can compete?

Somebody hit some Solaris engineers with a 2x4 clue-stick, please.
		Linus
Edgar Toernig· May 23, 2006, 15:20 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

Linus Torvalds wrote:
Show 12 quoted lines
>
> >  -   printf ("access: %d\n", access("/etc/motd", X_OK));
> > [...]
> >  - will return 0 on solaris - when run as root, even though /etc/motd
> >  - is not executeable.
> > 
> > This is explicitly allowed by the SUS, even for non-root:
> 
>      New implementations are discouraged from returning X_OK unless at 
>      least one execution permission bit is set."
> 
> which clearly says "Solaris is CRAP" to me.

Just for the record: firefox's download manager performs exactly this test to decide whether you can 'open with' a file (pretty silly because the test is done on the freshly downloaded file in the temp dir which never has an x-bit set). But I was hit by this effect on my system which is - surprise surprise - Linux :-) Ok, it's a pretty old one with a 2.0 kernel and libc 5. But nevertheless, access(2) is not the right function to portably test the x-bit.

Ciao, ET.
Linus Torvalds· May 23, 2006, 15:31 UTC · re: Edgar Toernig · lore

Re: Git 1.3.2 on Solaris

On Tue, 23 May 2006, Edgar Toernig wrote:
> 
> But I was hit by this effect on my system which is - surprise surprise - 
> Linux :-)  Ok, it's a pretty old one with a 2.0 kernel and libc 5.

Yes, we've had that bug too, and yes, I was hit by a clue-stick, and still have the bruise. That's how you teach people.

[ And how the heck does anybody still run 2.0, btw? ]
		Linus
Edgar Toernig· May 23, 2006, 18:43 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

Linus Torvalds wrote:
>
> [ And how the heck does anybody still run 2.0, btw? ]

Why not? There was never the need for an upgrade. All connected hardware is well supported. Sure, there are no binary packages for it and you have to compile from sources but most apps can be made to run on it (on the way you learn to hate autoconf). And I wish the 2.6 system were as reliable as the 2.0 one ...

Ciao, ET.
Jason Riedy· May 23, 2006, 18:03 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

And Linus Torvalds writes:
 - 
 - What kind of CRAP has Solaris become?

Become? heh. Check mount's output; it's "mountpoint on device". Always has been. I think there might be a reason why certain other OSes have eaten their lunch, and it ain't the price.

 - It wasn't about what was "allowed by the standards", 
 - that was the HP-SUX and AIX's excuses.

No, AIX's excuse is that it's "dictated by these three standards over here and disallowed by those two, so clearly we have to support both behaviors depending on some footnote in our 1e9 page manual." wheee...

 - Have Sun people forgotten the difference between "quality" and "crap that 
 - passes standards tests"? 

As far as I've been told, Sun's more interested in near-perfect backwards compatibility than external standards tests. It worked for Intel, right? ;)

 - Btw, even SuS says:
[...]
 -      New implementations are discouraged from returning X_OK unless at 
 -      least one execution permission bit is set."

Now there is one possible, cross-OS problem that I haven't tested. You can chmod a-x and then use setfacl to grant one person execute access. I'm not sure if access works in that case, but that might possibly just say that current ACL systems are crap.

Hmm. Does access handle SELinux or the other systems? That might be interesting for a public git server, but I don't know enough about it.

 - Somebody hit some Solaris engineers with a 2x4 clue-stick, please.

I think you're targetting the wrong department... Their hands are tied.

Jason, wondering if you could resist the SUS bait...
Linus Torvalds· May 23, 2006, 18:24 UTC · re: Jason Riedy · lore

Re: Git 1.3.2 on Solaris

On Tue, 23 May 2006, Jason Riedy wrote:
Show 11 quoted lines
>
>  - Btw, even SuS says:
> [...]
>  -      New implementations are discouraged from returning X_OK unless at 
>  -      least one execution permission bit is set."
> 
> Now there is one possible, cross-OS problem that I
> haven't tested.  You can chmod a-x and then use
> setfacl to grant one person execute access.  I'm not
> sure if access works in that case, but that might
> possibly just say that current ACL systems are crap.

I absolutely agree. That is why the OS has a "access()" system call. It's there to ask the OS whether the file is executable (or readable/writable).

Otherwise, we'd just do
   static inline int executable(const char *path)
   {
	struct stat st;
	return  !stat(pathname, &st) &&
		S_ISREG(st.st_mode) &&
		(st.st_mode & 0111) != 0;
   }

and be done with it. But exactly because the OS knows what "executable" means, we ask it. We don't know about all the ACL's etc, the OS does.

(Similar issues are true for writability too - the file may be "writable" in the sense that the write permission bits are on, but if the filesystem is mounted read-only, it sure as hell ain't W_OK _anyway_).

> Hmm.  Does access handle SELinux or the other systems?
Yup. 

Modulo bugs, of course, but yes, access() on linux should check both POSIX ACL's and SELinux security extensions. It uses exactly the same code-paths that open()/execve() does: it uses the "vfs_permission()" function which is also what execve() uses.

Now, I think access() actually misses a no-exec mount (it doesn't seem to check MNT_NOEXEC for X_OK), and that looks like it might actually be a real bug.

		Linus
Linus Torvalds· May 23, 2006, 18:48 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

On Tue, 23 May 2006, Linus Torvalds wrote:
> 
> I absolutely agree. That is why the OS has a "access()" system call. It's 
> there to ask the OS whether the file is executable (or readable/writable).

Side note: I'm not claiming that "access()" is a wonderful thing. I do agree that we might want to replace it with something else inside of git, if only because of portability concerns.

So I'm really just ranting my normal "standards lawyerese doesn't mean much" rant..

(access() also has other isses: X_OK obviously means different things for directories and for regular files, so quite often you need to do a stat() on the thing _anyway_ just to determine whether it's "executable in the 'execve()' sense" or "executable in the 'path lookup' sense").

		Linus
Stefan Pfetzing· May 26, 2006, 03:30 UTC · re: Stefan Pfetzing · lore

Re: Git 1.3.2 on Solaris

Hi list,
2006/5/23, Stefan Pfetzing <stefan.pfetzing@gmail.com>:
Show 9 quoted lines
> 2006/5/17, Jason Riedy <ejr@eecs.berkeley.edu>:
> > And pkgsrc itself works just fine without the silly g prefix,
> > or at least does for me as a mere user (and as well as it does
> > work).  But if you intend on adding the package upstream, it'll
> > need something to cope with the g.  And pkgsrc handles local
> > patches...
>
> Well I had some problems on NetBSD without the g prefix for the
> gnu coreutils - since then I always used that prefix.
...

Well finally - after some patching around access() and after figuring out "merge" was broken in pkgsrc (and still is - I had to open a problem report) - I got all tests to complete successfully.

bye
Stefan

P.S.: merge from devel/rcs uses /bin/diff3 on solaris which somehow breaks merge.

-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.
Ryan Anderson· May 17, 2006, 05:15 UTC · re: Jason Riedy · lore

Re: Git 1.3.2 on Solaris

On Tue, May 16, 2006 at 08:26:24PM -0700, Jason Riedy wrote:
> Plus, alas, Perl modules and Python version drift can be a bit
> of a problem on the same semi-pristine (or unmaintained, or
> too-stable) systems, so shell isn't the only thing that needs to
> go.  And that'll take a good deal of effort.

The Perl used in core-git is pretty forgiving of older versions of Perl, back to at least 5.6. (Going back to 5.005.003 is rather painful, however, to be honest.)

The only major tool I can think of that has embedded Perl in the shell script is format-patch. That could probably be redone in pure Perl if it would help.

Junio C Hamano· May 17, 2006, 08:22 UTC · re: Ryan Anderson · lore

Re: Git 1.3.2 on Solaris

Ryan Anderson <ryan@michonline.com> writes:
> The only major tool I can think of that has embedded Perl in the shell
> script is format-patch.  That could probably be redone in pure Perl if
> it would help.
Actually, that one is in the process of migrating all C.
Junio C Hamano· May 17, 2006, 09:03 UTC · re: Linus Torvalds · lore

Re: Git 1.3.2 on Solaris

Linus Torvalds <torvalds@osdl.org> writes:
> [ Junio - see the "grep" issue ]
> ...
> Of course, I don't think anybody tried the new "git grep" on Solaris,...

I haven't tried the new grep on Solaris myself, as the Solaris box I have easy access is badly maintained (unmaintained is probably a better wording).

> ...and 
> I think the solaris "grep" lacks the "-H" flag, for example. But that 
> should be easy to fix (for example, replace the use of "--" and "-H" with 
> putting a "/dev/null" as the first filename).
You mean like this, I presume.

But I think this approach breaks -L; I do not think Solaris supports -L, so it does not matter there, but on platforms that knows how to do -L it does.

-- >8 -- [PATCH] builtin-grep: give /dev/null at the beginning instead of -H

---
diff --git a/builtin-grep.c b/builtin-grep.c
index 66111de..ff3c1f7 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -453,7 +453,6 @@ static int external_grep(struct grep_opt
 
 	len = nr = 0;
 	push_arg("grep");
-	push_arg("-H");
 	if (opt->fixed)
 		push_arg("-F");
 	if (opt->linenum)
@@ -503,7 +502,7 @@ static int external_grep(struct grep_opt
 		push_arg("-e");
 		push_arg(p->pattern);
 	}
-	push_arg("--");
+	push_arg("/dev/null");
 
 	hit = 0;
 	argc = nr;
Junio C Hamano· May 17, 2006, 09:54 UTC · re: Junio C Hamano · lore

[PATCH] builtin-grep: workaround for non GNU grep.

Some implementations do not know what to do with -H; define NO_H_OPTION_IN_GREP when you build git if your grep lacks -H.

Most of the time, it can be worked around by prepending /dev/null to the argument list, but that causes -L and -c to slightly misbehave (they both expose /dev/null is given), so when these options are given, do not run external grep that does not understand -H.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---
 Junio C Hamano <junkio@cox.net> writes:
 > But I think this approach breaks -L; I do not think Solaris
 > supports -L, so it does not matter there, but on platforms that
 > knows how to do -L it does.
 So this is an updated version.  I am not proud of the handling
 of the new Makefile variable, although I like the C code that
 does not need #ifdef thanks to it.
 Makefile       |   11 +++++++++++
 builtin-grep.c |   22 +++++++++++++++++++---
 2 files changed, 30 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile
index 9ba608c..c67108d 100644
--- a/Makefile
+++ b/Makefile
@@ -46,6 +46,8 @@ # Patrick Mauritz).
 #
 # Define NO_MMAP if you want to avoid mmap.
 #
+# Define NO_H_OPTION_IN_GREP if your grep does not understand -H.
+#
 # Define WITH_OWN_SUBPROCESS_PY if you want to use with python 2.3.
 #
 # Define NO_IPV6 if you lack IPv6 support and getaddrinfo().
@@ -444,6 +446,12 @@ ifdef NO_ACCURATE_DIFF
 	ALL_CFLAGS += -DNO_ACCURATE_DIFF
 endif
 
+ifdef NO_H_OPTION_IN_GREP
+	NO_H_OPTION_IN_GREP=1
+else
+	NO_H_OPTION_IN_GREP=0
+endif
+
 # Shell quote (do not use $(call) to accomodate ancient setups);
 
 SHA1_HEADER_SQ = $(subst ','\'',$(SHA1_HEADER))
@@ -526,6 +534,9 @@ git$X git.spec \
 %.o: %.S
 	$(CC) -o $*.o -c $(ALL_CFLAGS) $<
 
+builtin-grep.o: builtin-grep.c
+	$(CC) -o $*.o -c $(ALL_CFLAGS) -DNO_H_OPTION_IN_GREP=$(NO_H_OPTION_IN_GREP) $<
+
 exec_cmd.o: exec_cmd.c
 	$(CC) -o $*.o -c $(ALL_CFLAGS) '-DGIT_EXEC_PATH="$(gitexecdir_SQ)"' $<
 
diff --git a/builtin-grep.c b/builtin-grep.c
index 66111de..36512d8 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -453,7 +453,6 @@ static int external_grep(struct grep_opt
 
 	len = nr = 0;
 	push_arg("grep");
-	push_arg("-H");
 	if (opt->fixed)
 		push_arg("-F");
 	if (opt->linenum)
@@ -503,7 +502,13 @@ static int external_grep(struct grep_opt
 		push_arg("-e");
 		push_arg(p->pattern);
 	}
-	push_arg("--");
+
+	if (NO_H_OPTION_IN_GREP)
+		push_arg("/dev/null");
+	else {
+		push_arg("-H");
+		push_arg("--");
+	}
 
 	hit = 0;
 	argc = nr;
@@ -535,8 +540,19 @@ #ifdef __unix__
 	 * Use the external "grep" command for the case where
 	 * we grep through the checked-out files. It tends to
 	 * be a lot more optimized
+	 *
+	 * Some grep implementations do not understand -H nor --
+	 * but /dev/null can be used as a substitution in most
+	 * cases.
+	 *
+	 * However -L and -c would slightly misbehave (-L would
+	 * list /dev/null as a hit, and -c would report 0 hits
+	 * from /dev/null); so do not use the external one on
+	 * such platforms.
 	 */
-	if (!cached) {
+	if (!cached &&
+	    (!NO_H_OPTION_IN_GREP ||
+	     (!opt->count && !opt->unmatch_name_only))) {
 		hit = external_grep(opt, paths, cached);
 		if (hit >= 0)
 			return hit;
-- 
1.3.3.g8a24
Linus Torvalds· May 17, 2006, 14:24 UTC · re: Junio C Hamano · lore

Re: [PATCH] builtin-grep: workaround for non GNU grep.

No, please don't do it this way.
On Wed, 17 May 2006, Junio C Hamano wrote:
Show 17 quoted lines
>
> +	 * Some grep implementations do not understand -H nor --
> +	 * but /dev/null can be used as a substitution in most
> +	 * cases.
> +	 *
> +	 * However -L and -c would slightly misbehave (-L would
> +	 * list /dev/null as a hit, and -c would report 0 hits
> +	 * from /dev/null); so do not use the external one on
> +	 * such platforms.
>  	 */
> -	if (!cached) {
> +	if (!cached &&
> +	    (!NO_H_OPTION_IN_GREP ||
> +	     (!opt->count && !opt->unmatch_name_only))) {
>  		hit = external_grep(opt, paths, cached);
>  		if (hit >= 0)
>  			return hit;
That's the ugliest test ever, and at all the wrong levels.

Just make "external_grep()" test for the cases that it cannot handle, and return -1. That's how it's designed to work.

		Linus
Junio C Hamano· May 17, 2006, 17:41 UTC · re: Linus Torvalds · lore

Re: [PATCH] builtin-grep: workaround for non GNU grep.

Linus Torvalds <torvalds@osdl.org> writes:
Show 12 quoted lines
>> -	if (!cached) {
>> +	if (!cached &&
>> +	    (!NO_H_OPTION_IN_GREP ||
>> +	     (!opt->count && !opt->unmatch_name_only))) {
>>  		hit = external_grep(opt, paths, cached);
>>  		if (hit >= 0)
>>  			return hit;
>
> That's the ugliest test ever, and at all the wrong levels.
>
> Just make "external_grep()" test for the cases that it cannot handle, and 
> return -1. That's how it's designed to work.

Ah.... *BLUSH* I was not thinking when I saw that "if (hit >= 0)" stuff. Yes, you made it to fall back.

Bertrand Jacquin· May 17, 2006, 15:39 UTC · re: Junio C Hamano · lore

Re: [PATCH] builtin-grep: workaround for non GNU grep.

On 5/17/06, Junio C Hamano <junkio@cox.net> wrote:
Show 5 quoted lines
> +ifdef NO_H_OPTION_IN_GREP
> +       NO_H_OPTION_IN_GREP=1
> +else
> +       NO_H_OPTION_IN_GREP=0
> +endif
Show 6 quoted lines
> +       if (NO_H_OPTION_IN_GREP)
> +               push_arg("/dev/null");
> +       else {
> +               push_arg("-H");
> +               push_arg("--");
> +       }

Sorry, maybe a C code beginner question but while you define NO_H_OPTION_IN_GREP in Makefile, why don't use a build time ``if'' instead of a runtime one ?

Like :
#if NO_H_OPTION_IN_GREP
               push_arg("/dev/null");
#else
               push_arg("-H");
               push_arg("--");
#fi
-- 
Beber
#e.fr@freenode
Junio C Hamano· May 17, 2006, 17:42 UTC · re: Bertrand Jacquin · lore

Re: [PATCH] builtin-grep: workaround for non GNU grep.

"Bertrand Jacquin" <beber.mailing@gmail.com> writes:
Show 12 quoted lines
> Sorry, maybe a C code beginner question but while you define
> NO_H_OPTION_IN_GREP in Makefile, why don't use a build time ``if''
> instead of a runtime one ?
>
> Like :
>
> #if NO_H_OPTION_IN_GREP
>               push_arg("/dev/null");
> #else
>               push_arg("-H");
>               push_arg("--");
> #fi

Exactly because I wanted to avoid conditional compilation using C preprocessor directive (#if).

Linus Torvalds· May 17, 2006, 18:12 UTC · re: Junio C Hamano · lore

Re: [PATCH] builtin-grep: workaround for non GNU grep.

On Wed, 17 May 2006, Junio C Hamano wrote:
Show 12 quoted lines
>
> "Bertrand Jacquin" <beber.mailing@gmail.com> writes:
> >
> > #if NO_H_OPTION_IN_GREP
> >               push_arg("/dev/null");
> > #else
> >               push_arg("-H");
> >               push_arg("--");
> > #fi
> 
> Exactly because I wanted to avoid conditional compilation using
> C preprocessor directive (#if).
I think this is portable and correct.

Of course, it still ignores the fact that not all grep's support some of the flags like -F/-L/-A/-C etc, but for those cases, the external grep itself will happily just say "unrecognized option -F" or similar.

So with this change, "git grep" should handle all the flags the native grep handles, which is really quite fine. We don't _need_ to expose anything more, and if you do want our extensions, you can get them with "--uncached" and an up-to-date index.

No configuration necessary, and we automatically take advantage of any native grep we have, if possible.

		Linus
---
diff --git a/builtin-grep.c b/builtin-grep.c
index 66111de..d09ddf0 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -453,7 +453,6 @@ static int external_grep(struct grep_opt
 
 	len = nr = 0;
 	push_arg("grep");
-	push_arg("-H");
 	if (opt->fixed)
 		push_arg("-F");
 	if (opt->linenum)
@@ -503,17 +502,35 @@ static int external_grep(struct grep_opt
 		push_arg("-e");
 		push_arg(p->pattern);
 	}
-	push_arg("--");
+
+	/*
+	 * To make sure we get the header printed out when we want it,
+	 * add /dev/null to the paths to grep.  This is unnecessary
+	 * (and wrong) with "-l" or "-L", which always print out the
+	 * name anyway.
+	 *
+	 * GNU grep has "-H", but this is portable.
+	 */
+	if (!opt->name_only && !opt->unmatch_name_only)
+		push_arg("/dev/null");
 
 	hit = 0;
 	argc = nr;
 	for (i = 0; i < active_nr; i++) {
 		struct cache_entry *ce = active_cache[i];
+		const char *name;
 		if (ce_stage(ce) || !S_ISREG(ntohl(ce->ce_mode)))
 			continue;
 		if (!pathspec_matches(paths, ce->name))
 			continue;
-		argv[argc++] = ce->name;
+		name = ce->name;
+		if (name[0] == '-') {
+			int len = ce_namelen(ce);
+			name = xmalloc(len + 3);
+			memcpy(name, "./", 2);
+			memcpy(name + 2, ce->name, len + 1);
+		}
+		argv[argc++] = name;
 		if (argc < MAXARGS)
 			continue;
 		hit += exec_grep(argc, argv);
Junio C Hamano· May 17, 2006, 18:59 UTC · re: Linus Torvalds · lore

Re: [PATCH] builtin-grep: workaround for non GNU grep.

Linus Torvalds <torvalds@osdl.org> writes:
Show 13 quoted lines
> I think this is portable and correct.
>
> Of course, it still ignores the fact that not all grep's support some of 
> the flags like -F/-L/-A/-C etc, but for those cases, the external grep 
> itself will happily just say "unrecognized option -F" or similar.
>
> So with this change, "git grep" should handle all the flags the native 
> grep handles, which is really quite fine. We don't _need_ to expose 
> anything more, and if you do want our extensions, you can get them with 
> "--uncached" and an up-to-date index.
>
> No configuration necessary, and we automatically take advantage of any 
> native grep we have, if possible.
This makes -c misbehave in a subtle way.
	git grep -c -e no-such-string-anywhere | head -n 1
But I do not think we care.
Linus Torvalds· May 17, 2006, 19:42 UTC · re: Junio C Hamano · lore

Re: [PATCH] builtin-grep: workaround for non GNU grep.

On Wed, 17 May 2006, Junio C Hamano wrote:
Show 6 quoted lines
> 
> This makes -c misbehave in a subtle way.
> 
> 	git grep -c -e no-such-string-anywhere | head -n 1
> 
> But I do not think we care.

Ahh, yes. That appears to be unfixable without using special GNU extensions.

I agree that we probably don't care, though.
		Linus
Junio C Hamano· May 17, 2006, 08:28 UTC · re: Stefan Pfetzing · lore

Re: Git 1.3.2 on Solaris

"Stefan Pfetzing" <stefan.pfetzing@gmail.com> writes:
> 1.  fix every single shellscript automatically during the build phase
> 2.  setup a dir which contains symlinks to the "right" binaries and
> put that dir into PATH.
You forgot 3.
  3.  rewrite scripts so that they would require only POSIX;
      for ones that do need GNU extended coreutils to do in
      shell, find other ways, perhaps rewriting the stuff in C.

I am not looking forward to do the g- prefix in the main Makefile. The approach to have symlink forest under gitexecdir (<Pine.LNX.4.64.0605162047380.10823@g5.osdl.org> by Linus) is more palatable, and I am not opposed to host a script to do so under contrib/notgnu perhaps.

Stefan Pfetzing· May 17, 2006, 09:06 UTC · re: Junio C Hamano · lore

Re: Git 1.3.2 on Solaris

Hi Junio,
2006/5/17, Junio C Hamano <junkio@cox.net>:
Show 11 quoted lines
> "Stefan Pfetzing" <stefan.pfetzing@gmail.com> writes:
>
> > 1.  fix every single shellscript automatically during the build phase
> > 2.  setup a dir which contains symlinks to the "right" binaries and
> > put that dir into PATH.
>
> You forgot 3.
>
>   3.  rewrite scripts so that they would require only POSIX;
>       for ones that do need GNU extended coreutils to do in
>       shell, find other ways, perhaps rewriting the stuff in C.

Yes, thats right, but this can only be a long term goal, because I guess this will take significantly longer. - even "tr" and "diff" behave different on Solaris.

Show 5 quoted lines
> I am not looking forward to do the g- prefix in the main
> Makefile.  The approach to have symlink forest under gitexecdir
> (<Pine.LNX.4.64.0605162047380.10823@g5.osdl.org> by Linus) is
> more palatable, and I am not opposed to host a script to do so
> under contrib/notgnu perhaps.

Hm, gitexecdir is also the path where git is installed, right? So if I'd install git with pkgsrc it will be /usr/pkg/bin, right? - If so, putting symlinks there _will_ break pkgsrc.

bye
Stefan
-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.
Junio C Hamano· May 17, 2006, 09:22 UTC · re: Stefan Pfetzing · lore

Re: Git 1.3.2 on Solaris

"Stefan Pfetzing" <stefan.pfetzing@gmail.com> writes:
> Hm, gitexecdir is also the path where git is installed, right? So if I'd
> install git with pkgsrc it will be /usr/pkg/bin, right? - If so,
> putting symlinks
> there _will_ break pkgsrc.

If you look at our Makefile, you will see bindir does not have to be gitexecdir. The suggestion by Linus is that you set bindir to /usr/local/bin or whereever your distribution's packaging scheme wants the locally installed software to be that is on user's PATH, and gitexecdir to /usr/local/libexec/git (again, whereever), _and_ have:

	ln -s /usr/bin/gtr /usr/local/libexec/git/tr
	ln -s /usr/bin/gxargs /usr/local/libexec/git/xargs
        ...
Then:
	(1) git and gitk are available in /usr/local/bin;
        (2) while git and gitk runs, /usr/local/libexec/git will
            be prepended to the PATH, so when they want xargs,
            they will get gxargs;
        (3) but your users will _not_ have /usr/local/libexec/git
            on their PATH, so when they type xargs they will get
            the one that barfs on -0 option.

and train your users and user's scripts to use the officially sanctioned way to refer to git subprograms. From interactive sessions, say "git foo", not "git-foo". If your script _really_ cares about extra exec git wrapper does, use "git --exec-path" upfront in the script to obtain correct gitexecpath, export GIT_EXEC_PATH environment variable with that value, and prepend it to PATH so that it can find "git-foo" executable (you would probably need to do both, so that git-foo can find git-bar and its friends).

	
Stefan Pfetzing· May 17, 2006, 10:41 UTC · re: Junio C Hamano · lore

Re: Git 1.3.2 on Solaris

Hi Junio,
2006/5/17, Junio C Hamano <junkio@cox.net>:
Show 10 quoted lines
> If you look at our Makefile, you will see bindir does not have
> to be gitexecdir.  The suggestion by Linus is that you set
> bindir to /usr/local/bin or whereever your distribution's
> packaging scheme wants the locally installed software to be that
> is on user's PATH, and gitexecdir to /usr/local/libexec/git
> (again, whereever), _and_ have:
>
>         ln -s /usr/bin/gtr /usr/local/libexec/git/tr
>         ln -s /usr/bin/gxargs /usr/local/libexec/git/xargs
>         ...

Nice, that looks like a solution, but will this also "fix" the tr usage for the git tests? If so, I'll write a small shellscript to create the links and so on, and test it on Solaris later today.

bye
dreamind
-- 
       http://www.dreamind.de/
Oroborus and Debian GNU/Linux Developer.

← back to recent threads