"fatal: index-pack failed" on git-clone

I get the error "fatal: index-pack failed" when I attempt to clone a remote bare repository. The repository works well on other machines, including the repository's own.

The repository lives on a version of Mac OS X I'm not allowed to talk about (I repeat: It works well for working copies on other machines, and on its own). The client is RHEL5. Git is version 1.6.3 on both machines, and was built from the tarball.

Here's the debug transcript:

=========== $ sudo GIT_TRACE=1 git clone myusername@remote.example.com:/Users/ myusername/scientia/scientia.git trace: built-in: git 'clone' 'myusername@remote.example.com:/Users/ myusername/scientia/scientia.git' Initialized empty Git repository in /srv/scientia/.git/ trace: run_command: 'ssh' 'myusername@remote.example.com' 'git-upload- pack '\''/Users/myusername/scientia/scientia.git'\''' Password: trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '-- keep=fetch-pack 17580 on local.example.com' trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '-- keep=fetch-pack 17580 on local.example.com' trace: exec failed: No such file or directory trace: exec 'index-pack' failed: No such file or directory fatal: index-pack failed remote: Counting objects: 2797, done. remote: Compressing objects: 100% (2388/2388), done. $ ===========

/Users/myusername/scientia/scientia.git on remote is a symlink to a .git repository on another volume. I've verified that the path is valid.

One post I saw via Google said that attempting to clone large files can choke the process. I've done git-rm on my largest, reconstructable files, to no effect. (Though I suppose it does no good wrt the files that are still in the history.)

I've fooled around with git-verify-pack without result.

Prior to all this, my last push to the repository failed with "[rejected] ... non-fast forward". I got past that with a "git push -- force".

How can I get the clone done?

	— F

Fritz Anderson <fritza@uchicago.edu> writes:

Looking at the output of the trace, I do not think that you have to worry about people asking for a copy of your repository in order to diagnose this issue, as I suspect that even a much smaller toy repository will fail for you in the same way.

I have heard that pseudo resets the PATH so you are invoking "git" from one of those standard system PATH, perhaps /usr/bin.

This is saying that "git" on the local side (the one you are running "clone" on) couldn't find its "index-pack" subcommand. Why?

I think this is an issue with your RHEL5 box, not the MacOS box. A quick check that might be useful is to type:

	$ git index-pack
	$ sudo git index-pack

On Jul 8, 2009, at 11:42 AM, Junio C Hamano wrote:

...

Here is the result:

=== $ git index-pack usage: git index-pack [-v] [-o <index-file>] [{ ---keep | -- keep=<msg> }] [--strict] { <pack-file> | --stdin [--fix-thin] [<pack- file>] } $ sudo git index-pack usage: git index-pack [-v] [-o <index-file>] [{ ---keep | -- keep=<msg> }] [--strict] { <pack-file> | --stdin [--fix-thin] [<pack- file>] } ===

So git is apparently found. HOWEVER, if I do this, it's a different story:

=== $ which git /usr/local/bin/git $ sudo which git which: no git in (/usr/bin:/bin) ===

On that evidence, I reissued my problem command under sudo, specifying /usr/local/bin/git as the command. That worked. Thank you; I would not have found it without you.

I'm obviously ignorant on the path issue, but that's off-topic for this list.

Thanks again.

	— F

Fritz Anderson <fritza@uchicago.edu> writes:

I was told sudo does this path munging for security reasons (I do not use it personally) but it appears that it does _not_ do that for finding the top level command in "sudo $command $args".

Very interesting.

Which makes the initial "sudo git clone..." find git in _your_ path before sanitization (and that is why it even starts), but then the path is nuked for the git process it launches, and we cannot find git-index-pack on the PATH.

But this should be fine, as git is expected to find git-index-pack in its GIT_EXEC_PATH that is compiled in the binary of "git" itself.

Which makes me suspect that your "git" in /usr/local/bin may be misconfigured. You might want to check what these tell you.

	$ git --exec-path
	$ /usr/local/bin/git --exec-path

On Jul 8, 2009, at 12:34 PM, Junio C Hamano wrote:

Glad to oblige. These are the four possibilities:

$ git --exec-path /usr/local/libexec/git-core $ /usr/local/bin/git --exec-path /usr/local/libexec/git-core $ sudo git --exec-path /usr/local/libexec/git-core $ sudo /usr/local/bin/git --exec-path /usr/local/libexec/git-core $

Same path every time, sudo or not, full path to git or not.

I built git (after installing zlib) simply with "./configure" and "sudo make install".

	— F

Fritz Anderson <fritza@uchicago.edu> writes:

Hmm, there is something fishy going on, and I am a bit frustrated not being able to see what it is.

The callpath should look like this:

  git.c::main()
  -> setup_path()
  -> cmd_clone()
     -> transport_fetch_refs()
        -> fetch_refs_via_pack()
           -> fetch_pack()
              -> do_fetch_pack()
                 -> get_pack()
                    -> start_command(), running either
                       "index-pack" or "unpack-objects"
                       on the incoming stream

and start_command() forks and eventually does execv_git_cmd() which is a thin wrapper around execvp().

The PATH exported when this execvp() runs should have been adjusted to have the exec-path at the beginning by calling setup_path() and this is done way before cmd_clone() was called by git.c::main() function.

What am I not seeing? There should be something obvious that I am missing. I do not see how your original command can fail with "exec failed: No such file or directory".

Could you try your original (non-working) command with this debug patch?

 exec_cmd.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

On Wed, 8 Jul 2009, Junio C Hamano wrote:

Just to verify, /usr/local/libexec/git-core/git-index-pack exists, and is executable?

All I can think of is that this could happen if PATH already had git-index-pack, and the exec-path didn't have it.

	-Daniel
*This .sig left intentionally blank*

On Jul 8, 2009, at 2:05 PM, Daniel Barkalow wrote:

...

$ ls -l /usr/local/libexec/git-core/git-index-pack -rwxr-xr-x 1 root root 1700975 Jul 8 09:28 /usr/local/libexec/git- core/git-index-pack

Exists, and is executable.

	— F

On Jul 8, 2009, at 1:49 PM, Junio C Hamano wrote:

(I had to apply the patch by hand; RHEL patch complained about a malformatted patch file.)

===== $ sudo GIT_TRACE=1 git clone username@remote.example.com:/Users/ username/scientia/scientia.git trace: setup_path: the $PATH was: /usr/bin:/bin trace: setup_path: the $PATH is now: /usr/local/libexec/git-core:/usr/ bin:/bin trace: built-in: git 'clone' 'username@remote.example.com:/Users/ username/scientia/scientia.git' Initialized empty Git repository in /home/username/git-1.6.3/ scientia/.git/ trace: run_command: 'ssh' 'username@remote.example.com' 'git-upload- pack '\''/Users/username/scientia/scientia.git'\''' Password: trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '-- keep=fetch-pack 32503 on scientia.uchicago.edu' trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '-- keep=fetch-pack 32503 on scientia.uchicago.edu' trace: exec failed: No such file or directory trace: the $PATH was: /usr/local/libexec/git-core:/usr/bin:/bin trace: exec 'index-pack' failed: No such file or directory fatal: index-pack failed remote: Counting objects: 2802, done. remote: Compressing objects: 100% (2393/2393), done. =====

As a reminder: ===== $ sudo /usr/local/bin/git --exec-path /usr/local/libexec/git-core =====

	— F

On Mittwoch, 8. Juli 2009, Junio C Hamano wrote:

It failed because /usr/local/bin is not in the PATH when git is run with sudo. Look at the original trace output:

At this point PATH is "/bin:/usr/bin" and the invoked git is /usr/local/bin/git (appearently!).

At this point PATH is "/usr/local/libexec/git-core:/bin:/usr/bin". There is no /usr/local/bin.

The PATH doesn't have 'git'; this must fail.

However, if Fritz runs 'sudo /usr/local/bin/git clone ...', then the interim PATH is "/usr/local/bin:/usr/local/libexec/git-core:/bin:/usr/bin" because this time setup_path() finds a non-empty argv0_path, and the command works.

-- Hannes

On Wed, Jul 08, 2009 at 10:42:51PM +0200, Johannes Sixt wrote:

I think there are two possible improvements here:

  1. Hardlinking "git" into exec-path. That means we will always be able
     to find the wrapper, even if the PATH has been munged. Admittedly,
     it sounds far fetched to me that something would exec from the PATH
     and then munge the PATH afterwards, but that seems to be what sudo
     is doing (and it is pretty commonly used).

  2. Better error messages. This would have been much more obvious to
     diagnose if it had said:

        trace: exec("git") failed: No such file or directory

     Johannes, I saw you just posted some related improvements to
     run_command; do they improve this?

-Peff

On Jul 8, 2009, at 4:12 PM, Jeff King wrote:

Here's an interesting experiment (RHEL 5):

===== $ echo $PATH /usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/fritza/bin $ cat >tryme.sh echo $PATH $ chmod a+x tryme.sh $ sudo ./tryme.sh /usr/bin:/bin

$ sudo git --exec-path /usr/local/libexec/git-core $ cat >tryme.sh git --exec-path $ ./tryme.sh /usr/local/libexec/git-core $ sudo ./tryme.sh ./tryme.sh: line 1: git: command not found =====

That is to say, possibly there is some sudo magic that uses the invoker's PATH to find the command in the first argument. After that, however, PATH is a "safe" value. So if you invoke git via sudo, it will internally see a PATH different from the one at the time of invocation.

For what it's worth.

	— F

On Mittwoch, 8. Juli 2009, Jeff King wrote:

No.

-- Hannes

Johannes Sixt <j6t@kdbg.org> writes:

Ahh, that is what I was missing.

As I said elsewhere already, I personally do not think sudo is worth supporting compared to the cost of this kind of pain resulting from its misguided "safety" brokenness, but apparently it is widely used. I think what Peff suggests in this thread might be a reasonable workaround.

On Wed, Jul 08, 2009 at 03:48:15PM -0700, Junio C Hamano wrote:

Yes, I find sudo's restrictions silly, considering that most people use it to allow arbitrary code execution, which is why I wrote this some time ago:

  http://peff.net/tinysu/

However, sudo is pretty popular, and it should be easy enough for us to work around it in this case. Patch is below. It's longer than the one-liner necessary, because it now uses "git" as the magic "everything should link to this" file instead of "git-add", which I think is a bit more obvious.

-- >8 --
Subject: [PATCH] Makefile: install 'git' in execdir

When a git command executes a subcommand, it uses the "git foo" form, which relies on finding "git" in the PATH. Normally this should not be a problem, since the same "git" that was used to invoke git in the first place will be found. And if somebody invokes a "git" outside of the PATH (e.g., by giving its absolute path), this case is already covered: we put that absolute path onto the front of PATH.

However, if one is using "sudo", then sudo will execute the "git" from the PATH, but pass along a restricted PATH that may not contain the original "git" directory. In this case, executing a subcommand will fail.

To solve this, we put the "git" wrapper itself into the execdir; this directory is prepended to the PATH when git starts, so the wrapper will always be found.

Signed-off-by: Jeff King <peff@peff.net>
---
 Makefile |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

Jeff King venit, vidit, dixit 09.07.2009 08:37:

:)

I think writing "tinysu" is really the best statement one can make about "sudo"... although "sudoh" would have been the most appropriate name...

Your patch is welcome, of course, and also removes the somewhat surprising special role played by "git-add".

Michael

Jeff King wrote:

 >
 > Signed-off-by: Jeff King <peff@peff.net>
 > ---
 >  Makefile |   14 +++++++-------
 >  1 files changed, 7 insertions(+), 7 deletions(-)
 >
 > diff --git a/Makefile b/Makefile
 > index 78cc113..311ce7d 100644
 > --- a/Makefile
 > +++ b/Makefile
 > @@ -1641,15 +1641,15 @@ ifneq (,$X)
 >  endif
 >  	bindir=$$(cd '$(DESTDIR_SQ)$(bindir_SQ)' && pwd) && \
 >  	execdir=$$(cd '$(DESTDIR_SQ)$(gitexec_instdir_SQ)' && pwd) && \
 > -	{ $(RM) "$$execdir/git-add$X" && \
 > +	{ $(RM) "$$execdir/git$X" && \
 >  		test -z "$(NO_CROSS_DIRECTORY_HARDLINKS)" && \
 > -		ln "$$bindir/git$X" "$$execdir/git-add$X" 2>/dev/null || \
 > -		cp "$$bindir/git$X" "$$execdir/git-add$X"; } && \
 > -	{ for p in $(filter-out git-add$X,$(BUILT_INS)); do \
 > +		ln "$$bindir/git$X" "$$execdir/git$X" 2>/dev/null || \
 > +		cp "$$bindir/git$X" "$$execdir/git$X"; } && \
 > +	{ for p in $(BUILT_INS); do \
 >  		$(RM) "$$execdir/$$p" && \
 > -		ln "$$execdir/git-add$X" "$$execdir/$$p" 2>/dev/null || \
 > -		ln -s "git-add$X" "$$execdir/$$p" 2>/dev/null || \
 > -		cp "$$execdir/git-add$X" "$$execdir/$$p" || exit; \
 > +		ln "$$execdir/git$X" "$$execdir/$$p" 2>/dev/null || \
 > +		ln -s "git$X" "$$execdir/$$p" 2>/dev/null || \
 > +		cp "$$execdir/git$X" "$$execdir/$$p" || exit; \
 >  	  done; } && \
 >  	./check_bindir "z$$bindir" "z$$execdir" "$$bindir/git-add$X"
 >

This breaks the install if ${bindir} == ${execdir}. The following is needed on top Peff's patch.

On Thu, Jul 09, 2009 at 07:29:27PM -0400, A Large Angry SCM wrote:

Oops, I didn't even think to test that (and then I left town all weekend
leaving you to pick up the pieces...:) ). Thanks.
 
-Peff