Filter-branch's "move tree to subdirectory" example fails with BSD sed?

The "Move tree to subdirectory" example in the 'git filter-branch' manpage fails on Mac OSX 10.7.3, but succeeds on Ubuntu 10.04. I'm using git version 1.7.0.4, which happened to be the version installed on the Ubuntu VM I had laying around. I think it's a difference between 'sed' on the two systems.

The example:
git filter-branch --index-filter \
        'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Demonstrating the problem is with sed: git init "test" cd "test" mkdir -p subdirA/subdirB echo content > subdirA/subdirB/file git add . git commit -m "initial commit" git ls-files -s | sed "s-\t\"*-&newsubdir/-"

On Mac 10.7.3 the final command outputs: 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0 subdirA/subdirB/file

On Ubuntu 10.04: 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0 newsubdir/subdirA/subdirB/file

I can solve my immediate problem using Ubuntu, but is there a way we could rewrite the example to work on both systems? I'm afraid differences between BSD and GNU 'sed' are a bit beyond me.

-- Christopher Tiwald

Am 14.04.2012 18:00, schrieb Christopher Tiwald:

Perhaps a literal TAB instead of \t makes the example work?

It would be difficult, though, to write this down in the manual in an unambiguous way.

-- Hannes

On Sat, Apr 14, 2012 at 09:25:05PM +0200, Johannes Sixt wrote:

Yes, I suspect that is the problem, too.

Maybe it would be simpler to just use perl:

 git filter-branch --index-filter '
   git ls-files -s |
     perl -pe "s{\t\"?}{$&newsubdir/}" |
     GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
   mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
 ' HEAD

I also think tweaking the indentation and line breaks as I did above makes it a little more readable. If we are using perl, we could also just use "-z" to get rid of the funny quote handling.

I wish we could also get rid of the temporary index. It exists solely for the side effect of removing the existing entries (otherwise, you get both "foo" and "newsubdir/foo" in the resulting history). It would be nice if there was some flag to update-index to say "before you do anything, clear the existing index" (or I guess, "don't bother reading the existing index"). And then result could look like:

  git filter-branch --index-filter '
    git ls-files -sz |
    perl -0pe "s{\t}{\tnewsubdir/}" |
    git update-index --from-scratch -z --index-info
  ' HEAD

which is IMHO much easier to read and understand.

-Peff

This just discards existing entries from the index, which can be useful if you are rewriting all entries with "--index-info" or similar.

Signed-off-by: Jeff King <peff@peff.net>
---
I tried to make something like:

  git update-index --from-scratch --index-info

work by avoiding reading all entries in the first place. However, update-index actually processes its arguments sequentially, so we _must_ read the index before we start processing arguments. But because it's sequential, a "clear" operation makes sense, since you clear, then add new entries.

 Documentation/git-update-index.txt |    3 +++
 builtin/update-index.c             |   10 ++++++++++
 2 files changed, 13 insertions(+)

On Mon, Apr 16, 2012 at 12:02:32PM -0400, Jeff King wrote:

At the very least I can give this patch series a go this evening. I'm in the process of combining 40 highly-intertwined, nested repositories into one, and should be able to test it under the conditions outlined in the updated example.

-- Christopher Tiwald

On Mon, Apr 16, 2012 at 5:48 PM, Christopher Tiwald <christiwald@gmail.com> wrote:

This worked great for me, at least in the case of adding trees to subdirectories. Fwiw, I found the newer, patched example a little easier to parse, but I also had spent a while figuring out what the original version did.

-- Christopher Tiwald

Over the years, this simple example has ended up quite hard to read because of the number of special cases that must be handled. Let's simplify it a bit:

  1. Use the new "index-info --clear" to avoid the need for
     a temporary index.

  2. Use "-z" and "perl -0" to avoid dealing with quoting
     issues. As a bonus, using perl means that "\t" will
     work consistently in regexps (the previous example
     using sed was reported to fail on OS X).

  3. Change the indentation to keep one logical unit per
     line and avoid extra backslash-escaping.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-filter-branch.txt |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

Jeff King <peff@peff.net> writes:

Wouldn't "git read-tree --prefix=newsubdir/" suffice without the pipeline? I.e.

        git filter-branch --index-filter '
		rm -f "$GIT_INDEX_FILE"
                git read-tree --prefix=newsubdir/ "$GIT_COMMIT"
	' HEAD

On Mon, Apr 16, 2012 at 10:03:03AM -0700, Junio C Hamano wrote:

Yeah, I think it does (at least it makes sense to me, and worked on a simple test case). I think I might use "git read-tree --empty" instead of "rm", as it is a little more obvious what is going on. But other than that, it is much more readable.

It is a slight shame not to show an "ls-files | update-index" example, because that is the most general form. But since this form is quite often cut-and-paste by people, I think simple is better here. If we want to show a more complex example, then we should add it separately.

-Peff