fast-import fails in read-only tree

I recently discovered that "git fast-import" signals an error if used in a tree to which we do not have write-access, because it tries to create a "objects/pack/tmp_pack_XXX" file even before starting to process the commands.

Usually this is not a problem (we'll create new commits and such, so write-access is indeed necessary), but in my case I was using fast-import only for its "reading" operations (in order to combine several inter-dependent "cat-file" operations into a single git session).

        Stefan

On Thu, Jan 28, 2016 at 05:17:36PM -0500, Stefan Monnier wrote:

The primary goal of fast-import is to write that packfile. It kind of sounds like you are using the wrong tool for the job.

Can you elaborate on what you are sending to fast-import (preferably with a concrete example)? There may be a way to accomplish the same thing with read-only tools like cat-file.

-Peff

Yes, I realize that. But in some cases it's the best tool available. `fast-import' is very close to being a "generic access API" which can be used instead of something like libgit. I think it'd be good to push it yet a bit closer.

My earlier "cat-blob applied to a tree" issue is another such case.

I'm sending a stream of "progress <foo>; cat-blob <foo>", basically.

The concrete example is in [BuGit](https://gitlab.com/monnier/bugit), see for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c

Yes, I switched to using "cat-file --batch" instead, but it's less convenient (I can't intersperse ad-hoc info in the output, the way I can with "progress" in fast-import) and there are cases where the list of files I need to extract cannot be determined without first looking at some of those extracted files (I currently have been able to avoid this in BuGit, luckily).

If I could use "cat-blob" on directories, there would be even more cases where I'd want to use fast-import for read-only operations to reduce the number of Git processes I fork.

        Stefan

On Fri, Jan 29, 2016 at 09:28:44AM -0500, Stefan Monnier wrote:

I'm not sure I agree. Git tries to make its innards available via flexible plumbing commands. If we're not succeeding, I think that should be fixed, rather than trying to shoe-horn an unrelated command to do the job, even if it would be less code.

You can use custom cat-file formatting to output your "name" strings as part of the same field. IOW, something like:

  git cat-file -p refs/heads/bugit-master:numbers |
  awk '{print $3 " " $4 }' |
  git cat-file --batch="%(rest)" |
  while read number; do
    read id; # assuming blob contents are single-line
    read _junk; # assumes blob ended in its own newline
    $fun "$id" "$number"
  done

That's from a fairly cursory reading of that bugit patch, though, so I might be missing some requirement.

I think the example above should handle the "intersperse" thing.

If you're really going to do a lot of interactive back-and-forth access of objects, though, I think you want to set up pipes to cat-file. It's a little tedious to allocate fifos, but something like:

  mkfifo in out
  (exec git cat-file --batch <in >out) &
  exec 8>in
  exec 9<out
  echo $sha >&8
  read mode type size <&9
  read content ;# or read $size, or read until newline
  echo $content >&8 ;# imagine content is another sha to look up
  ...read from &9, etc..

The fifos and numbered descriptors are annoying, but that's shell for you. I suspect using "fast-import" wouldn't be much different.

One feature I do think would be useful (and almost implemented when I added --batch-check=<format>) is a formatter for the object content, with a pretty modifier. I.e., it would be nice to do:

  echo $some_tree |
  git cat-file --batch-check="%(objectsize:pretty) %(contents:pretty)"

to work as the rough equivalent of "git cat-file -p" (but here you could feed multiple trees and get multiple answers).

-Peff

Jeff King <peff@peff.net> writes:

With bash's coproc it's a bit less tedious:

    coproc CAT_FILE git cat-file --batch
    echo $sha >&${CAT_FILE[1]}
    read mode type size <&${CAT_FILE[0]}

Andreas.

[...]

OMG, I didn't realize that cat-file doesn't buffer its output so it can be read&write to/from the same process. And the "%(rest)" thingy takes care of the rest of my needs, indeed.

Thanks!

That's not a problem.

Yes, that would be a good improvement,

        Stefan