Re: pull from stdin

From: Jeff King <peff@peff.net>
Date: Jul 7, 2011, 19:28 UTC
Message-ID: <20110707192813.GE12044@sigill.intra.peff.net>
In-Reply-To: <N1B-u7I8PJ0byN@Safe-mail.net>

On Thu, Jul 07, 2011 at 11:23:04AM -0400, miket99@Safe-mail.net wrote:

Show 10 quoted lines

> I am using a rather complicated distributed workflow to keep various
> git repositories in sync. Basically I am creating bundles, pushing
> them to a server, and pull them again using a special application
> which output the bundle content to stdin. Unfortunately git-fetch and
> friends do not like /dev/stdin as input. Is there any way to pass the
> bundle contents via stdin to git?
> [...]
> ~/test$ cat b | git pull -- /dev/stdin
> fatal: '/dev/stdin' does not appear to be a git repository
> fatal: The remote end hung up unexpectedly

I think the problem is that "git fetch" tests for a bundle by checking whether the argument is a regular file, which /dev/stdin is not. We could perhaps fix that, but I think it still wouldn't work. The bundle-reading code calls lseek, and your pipe will not be seekable.

So this works:

  $ git bundle unbundle /dev/stdin <b
  1c0138bbb920cdbf8e80bea38c4f77b0d01c3763 refs/heads/master
  1c0138bbb920cdbf8e80bea38c4f77b0d01c3763 HEAD

but this doesn't:

  $ cat b | git bundle unbundle /dev/stdin
  fatal: early EOF
  error: index-pack died

I don't think the problem is in the bundle format. I think it is simply an issue that we read the bundle header in one process via buffered I/O, and then use seek to pass the rest on to index-pack in another process. So you could solve it by either:

  1. Using unbuffered I/O and reading a character at a time from the
     bundle when the input is not seekable. This will involve many more
     syscalls, of course, but we would only have to do it for the list
     of refs, not for the actual pack data.

  2. Instead of passing the descriptor to index-pack, open a pipe to
     index-pack and pass the data through the original process. For
     the sake of efficiency, we could enable this only when the input
     isn't seekable.

-Peff