Re: [PATCH 4/5] fast-export: Introduce --inline-blobs

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Jan 20, 2011, 05:41 UTC
Message-ID: <20110120054126.GA4770@burratino>
In-Reply-To: <7vtyh4smer.fsf@alter.siamese.dyndns.org>

Junio C Hamano wrote:

> Just thinking aloud, but is it possible to write a filter that converts an
> arbitrary G-F-I stream with referenced blobs into a G-F-I stream without
> referenced blobs by inlining all the blobs?

A few details to watch out for:

- A mark, as in

	M 100644 :1 path/to/file

  can refer to a blob from a previous import.  A mark can even refer
  to a manually prepared marks file.

- The syntax

	M 100644 0409ac9fd3f1ea36680189e07116e58b2630ccad path/to/file

  refers to a blob that might not have been mentioned elsewhere in the
  stream.  This is the variant used by "git fast-export --no-data" to
  avoid transferring blob data.  (In general, non-git backends would
  presumably use something other than git blob IDs if they use this
  feature.  A filter of the kind we are describing would probably pass
  them through.)

  These datarefs can be acquired out of band (probably not a big deal)
  or by using the "ls" command to copy from a previous revision:

	> ls :3 "path/to/other/file"
	100644 blob 0409ac9fd3f1ea36680189e07116e58b2630ccad	git.c
	> M 100644 0409ac9fd3f1ea36680189e07116e58b2630ccad path/to/file

- The cat-blob command ("cat-blob :1") allows frontends to request
  the content of a previously imported blob (presumably in order to
  apply a delta to it).

So while something like the filter you describe seems possible, it cannot be as simple as

	mkfifo replies &&
	fast-export-frontend 3<replies |
	inline-blobs |
	fast-import-backend --cat-blob-fd=3 <args> 3>replies

for general <frontend> and <backend>. The frontend might try to cat blobs by mark number or to pick off where it left off in a previous run using a marks file.