# fast-import fails in read-only tree

6 messages from 2016-01-28 to 2016-01-30. Participants: Stefan Monnier, Jeff King, Andreas Schwab.
Thread: https://gitlist.dev/t/41280

## Stefan Monnier, 2016-01-28 22:17

Subject: fast-import fails in read-only tree
Message-ID: <jwvfuxhz72e.fsf-monnier+gmane.comp.version-control.git@gnu.org>
URL: https://gitlist.dev/e/jwvfuxhz72e.fsf-monnier%2Bgmane.comp.version-control.git%40gnu.org

```
I recently discovered that "git fast-import" signals an error if used in
a tree to which we do not have write-access, because it tries to create
a "objects/pack/tmp_pack_XXX" file even before starting to process
the commands.

Usually this is not a problem (we'll create new commits and such, so
write-access is indeed necessary), but in my case I was using
fast-import only for its "reading" operations (in order to combine
several inter-dependent "cat-file" operations into a single git
session).


        Stefan

```

## Jeff King, 2016-01-29 06:08

Subject: Re: fast-import fails in read-only tree
Message-ID: <20160129060802.GA23106@sigill.intra.peff.net>
URL: https://gitlist.dev/e/20160129060802.GA23106%40sigill.intra.peff.net
In-Reply-To: <jwvfuxhz72e.fsf-monnier+gmane.comp.version-control.git@gnu.org>

```
On Thu, Jan 28, 2016 at 05:17:36PM -0500, Stefan Monnier wrote:

> I recently discovered that "git fast-import" signals an error if used in
> a tree to which we do not have write-access, because it tries to create
> a "objects/pack/tmp_pack_XXX" file even before starting to process
> the commands.
> 
> Usually this is not a problem (we'll create new commits and such, so
> write-access is indeed necessary), but in my case I was using
> fast-import only for its "reading" operations (in order to combine
> several inter-dependent "cat-file" operations into a single git
> session).

The primary goal of fast-import is to write that packfile. It kind of
sounds like you are using the wrong tool for the job.

Can you elaborate on what you are sending to fast-import (preferably
with a concrete example)? There may be a way to accomplish the same
thing with read-only tools like cat-file.

-Peff

```

## Stefan Monnier, 2016-01-29 14:28

Subject: Re: fast-import fails in read-only tree
Message-ID: <jwv7fisxyhz.fsf-monnier+gmane.comp.version-control.git@gnu.org>
URL: https://gitlist.dev/e/jwv7fisxyhz.fsf-monnier%2Bgmane.comp.version-control.git%40gnu.org
In-Reply-To: <20160129060802.GA23106@sigill.intra.peff.net>

```
>> I recently discovered that "git fast-import" signals an error if used in
>> a tree to which we do not have write-access, because it tries to create
>> a "objects/pack/tmp_pack_XXX" file even before starting to process
>> the commands.
> The primary goal of fast-import is to write that packfile.  It kind of
> sounds like you are using the wrong tool for the job.

Yes, I realize that.  But in some cases it's the best tool available.
`fast-import' is very close to being a "generic access API" which can be
used instead of something like libgit.  I think it'd be good to push it
yet a bit closer.

My earlier "cat-blob applied to a tree" issue is another such case.

> Can you elaborate on what you are sending to fast-import (preferably
> with a concrete example)?

I'm sending a stream of "progress <foo>; cat-blob <foo>", basically.

The concrete example is in [BuGit](https://gitlab.com/monnier/bugit),
see for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c

> There may be a way to accomplish the same thing with read-only tools
> like cat-file.

Yes, I switched to using "cat-file --batch" instead, but it's less
convenient (I can't intersperse ad-hoc info in the output, the way I can
with "progress" in fast-import) and there are cases where the list of
files I need to extract cannot be determined without first looking at
some of those extracted files (I currently have been able to avoid
this in BuGit, luckily).

If I could use "cat-blob" on directories, there would be even more cases
where I'd want to use fast-import for read-only operations to reduce the
number of Git processes I fork.


        Stefan

```

## Jeff King, 2016-01-30 05:13

Subject: Re: fast-import fails in read-only tree
Message-ID: <20160130051340.GA1677@sigill.intra.peff.net>
URL: https://gitlist.dev/e/20160130051340.GA1677%40sigill.intra.peff.net
In-Reply-To: <jwv7fisxyhz.fsf-monnier+gmane.comp.version-control.git@gnu.org>

```
On Fri, Jan 29, 2016 at 09:28:44AM -0500, Stefan Monnier wrote:

> > The primary goal of fast-import is to write that packfile.  It kind of
> > sounds like you are using the wrong tool for the job.
> 
> Yes, I realize that.  But in some cases it's the best tool available.
> `fast-import' is very close to being a "generic access API" which can be
> used instead of something like libgit.  I think it'd be good to push it
> yet a bit closer.

I'm not sure I agree. Git tries to make its innards available via
flexible plumbing commands. If we're not succeeding, I think that should
be fixed, rather than trying to shoe-horn an unrelated command to do the
job, even if it would be less code.

> > Can you elaborate on what you are sending to fast-import (preferably
> > with a concrete example)?
> 
> I'm sending a stream of "progress <foo>; cat-blob <foo>", basically.
> 
> The concrete example is in [BuGit](https://gitlab.com/monnier/bugit),
> see for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c

You can use custom cat-file formatting to output your "name" strings as
part of the same field. IOW, something like:

  git cat-file -p refs/heads/bugit-master:numbers |
  awk '{print $3 " " $4 }' |
  git cat-file --batch="%(rest)" |
  while read number; do
    read id; # assuming blob contents are single-line
    read _junk; # assumes blob ended in its own newline
    $fun "$id" "$number"
  done

That's from a fairly cursory reading of that bugit patch, though, so I
might be missing some requirement.

> Yes, I switched to using "cat-file --batch" instead, but it's less
> convenient (I can't intersperse ad-hoc info in the output, the way I can
> with "progress" in fast-import) and there are cases where the list of
> files I need to extract cannot be determined without first looking at
> some of those extracted files (I currently have been able to avoid
> this in BuGit, luckily).

I think the example above should handle the "intersperse" thing.

If you're really going to do a lot of interactive back-and-forth access
of objects, though, I think you want to set up pipes to cat-file. It's a
little tedious to allocate fifos, but something like:

  mkfifo in out
  (exec git cat-file --batch <in >out) &
  exec 8>in
  exec 9<out
  echo $sha >&8
  read mode type size <&9
  read content ;# or read $size, or read until newline
  echo $content >&8 ;# imagine content is another sha to look up
  ...read from &9, etc..

The fifos and numbered descriptors are annoying, but that's shell for
you. I suspect using "fast-import" wouldn't be much different.

One feature I do think would be useful (and almost implemented when I
added --batch-check=<format>) is a formatter for the object content,
with a pretty modifier. I.e., it would be nice to do:

  echo $some_tree |
  git cat-file --batch-check="%(objectsize:pretty) %(contents:pretty)"

to work as the rough equivalent of "git cat-file -p" (but here you could
feed multiple trees and get multiple answers).

-Peff

```

## Andreas Schwab, 2016-01-30 09:05

Subject: Re: fast-import fails in read-only tree
Message-ID: <m2oac31lp2.fsf@linux-m68k.org>
URL: https://gitlist.dev/e/m2oac31lp2.fsf%40linux-m68k.org
In-Reply-To: <20160130051340.GA1677@sigill.intra.peff.net>

```
Jeff King <peff@peff.net> writes:

> If you're really going to do a lot of interactive back-and-forth access
> of objects, though, I think you want to set up pipes to cat-file. It's a
> little tedious to allocate fifos, but something like:

With bash's coproc it's a bit less tedious:

>   mkfifo in out
>   (exec git cat-file --batch <in >out) &
>   exec 8>in
>   exec 9<out
>   echo $sha >&8
>   read mode type size <&9

    coproc CAT_FILE git cat-file --batch
    echo $sha >&${CAT_FILE[1]}
    read mode type size <&${CAT_FILE[0]}

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

```

## Stefan Monnier, 2016-01-30 13:56

Subject: Re: fast-import fails in read-only tree
Message-ID: <jwvtwlv18jw.fsf-monnier+gmane.comp.version-control.git@gnu.org>
URL: https://gitlist.dev/e/jwvtwlv18jw.fsf-monnier%2Bgmane.comp.version-control.git%40gnu.org
In-Reply-To: <20160130051340.GA1677@sigill.intra.peff.net>

```
> You can use custom cat-file formatting to output your "name" strings as
> part of the same field. IOW, something like:
[...]
> If you're really going to do a lot of interactive back-and-forth access
> of objects, though, I think you want to set up pipes to cat-file.

OMG, I didn't realize that cat-file doesn't buffer its output so it can
be read&write to/from the same process.  And the "%(rest)" thingy takes
care of the rest of my needs, indeed.

Thanks!

> It's a little tedious to allocate fifos, but something like:

That's not a problem.

> One feature I do think would be useful (and almost implemented when I
> added --batch-check=<format>) is a formatter for the object content,
> with a pretty modifier. I.e., it would be nice to do:
>
>   echo $some_tree |
>   git cat-file --batch-check="%(objectsize:pretty) %(contents:pretty)"
>
> to work as the rough equivalent of "git cat-file -p" (but here you could
> feed multiple trees and get multiple answers).

Yes, that would be a good improvement,


        Stefan

```