Using filter-branch to move repo contents in subdirectory

Hi

I'm trying to use git filter-branch to move all contents of a repo into a subdirectory of the repo root. The git filter-branch man page has the following example:

       To move the whole tree into a subdirectory, or remove it from there:

           git filter-branch --index-filter \
                   'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
                           GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                                   git update-index --index-info &&
                    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

So I've been trying to use this, the original structure of the repo is:

$ ls 00boot SFTdumpall.c debian Makefile.am SFTdumpheader.c ligodcc.cls README SFTvalidate.c pdfdraftcopy.sty SCCB_2004_09_09 SFTwrite.c run_validate.sh SFTReferenceLibrary.c T040164.tex sftreferencelibrary-2.2.tar.gz SFTReferenceLibrary.h configure.in sftreferencelibrary-2.3.tar.gz $

and I want all this in a subdirectory call "sftlib", so using the example from the manpage I ran:

$ git filter-branch --index-filter \
    'git ls-files -s | sed "s-\t\"*-&sftlib/-" |
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
            git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
Rewrite 223651f2ebd5d5d9341bcfc9e7cb6caaa3f4d171 (56/65)Ignoring path
00bootsftlib/
Rewrite c979ee835ebb89408e2f5f345ddefbb4d8cd75f5 (57/65)Ignoring path
00bootsftlib/
Rewrite 10037389e41fb4b4d04234ca2e302a7a491616d5 (58/65)Ignoring path
00bootsftlib/
Rewrite 21b4c31a02c6b7718145d471ba20e6fa70eb30d0 (59/65)Ignoring path
00bootsftlib/
Rewrite dc5cebbe14050f52f2effaa5eac4eb579dd440c3 (60/65)Ignoring path
00bootsftlib/
Rewrite 3b83fdd19b5d674445f5263df8cb2fc35c3c0092 (61/65)Ignoring path
00bootsftlib/
Rewrite 5246038ceeb2d28fba462076b62f3cf1b2b7e9ab (62/65)Ignoring path
00bootsftlib/
Rewrite 0fd3256ff401b405c677e83db350ad41ee74ef0d (63/65)Ignoring path
00bootsftlib/
Ignoring path debian/compatsftlib/
Ignoring path debian/copyrightsftlib/
Rewrite 9aca227c807658b951e81a6ca4460115b2fe8ccc (64/65)Ignoring path
00bootsftlib/
Ignoring path debian/compatsftlib/
Ignoring path debian/copyrightsftlib/
Rewrite 00274be10e10920d84db145268d85faf1b06539f (65/65)Ignoring path
00bootsftlib/
Ignoring path debian/compatsftlib/
Ignoring path debian/copyrightsftlib/

Ref 'refs/heads/to_merge' was rewritten $

now the directory structure is:

$ ls Makefile.am SCCB_2004_09_09 SFTReferenceLibrary.h SFTdumpheader.c SFTwritsftlib configure.in ligodcc.cls run_validatsftlib README SFTReferenceLibrary.c SFTdumpall.c SFTvalidatsftlib T040164.tsftlib debian pdfdraftsftlib sftsftlib $

which is clearly not what I'm wanting. What I am doing wrong here? As far as I can I'm using the example from the man-page?

Cheers

Adam

Adam Mercer wrote:

What OS is this? You seem to have a 'sed' version where \t does not match the horizontal tab character, or a shell that replaces the '\t' inside a double-quoted string with simply 't'.

As a short-term fix, you can try to replace it with $(printf '\t') which should always give a tab character. You need to quote the ' though, so that's

  $ git filter-branch --index-filter \
      'git ls-files -s | sed "s-$(printf '\''\t'\'')\"*-&sftlib/-" |
          GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
              git update-index --index-info &&
      mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

As for the actual issue: 'man 1p sed' on my system claims

        * The escape sequence '\n' shall match a <newline> embedded in the  pat-
          tern space. A literal <newline> shall not be used in the BRE of a con-
          text address or in the substitute function.

but does not mention \t at all, so I guess either that manpage is wrong or GNU sed is not POSIX compliant even with --posix (where it still treats \t as a tab).

On Thu, Jul 29, 2010 at 14:08, Thomas Rast <trast@student.ethz.ch> wrote:

This was on Mac OS X 10.6. Where sed is BSD sed.

Thanks, that did the trick!

Also using the original command, but specifying GNU sed also does the trick.

Cheers

Adam

Using \t to represent a tab character is not portable beyond GNU sed (see e.g. GNU sed's info pages). Use printf to generate the tab instead.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

 Adam Mercer wrote:

That's very funny however, since it means that nobody should ever have had a successful test run on OS X with the preinstalled tools. What gives?

 Documentation/git-filter-branch.txt |    3 ++-
 t/t7003-filter-branch.sh            |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

On 7/29/10 3:24 PM, Thomas Rast wrote:

v1.7.2-rc1 on 10.6:

fixed 3 success 6328 failed 0 broken 21 total 6355

*** t7003-filter-branch.sh ***
*   ok 1: setup
*   ok 2: rewrite identically
*   ok 3: result is really identical
*   ok 4: rewrite bare repository identically
*   ok 5: result is really identical
*   ok 6: correct GIT_DIR while using -d
*   ok 7: Fail if commit filter fails
*   ok 8: rewrite, renaming a specific file
*   ok 9: test that the file was renamed
*   ok 10: rewrite, renaming a specific directory
*   ok 11: test that the directory was renamed
*   ok 12: rewrite one branch, keeping a side branch
*   ok 13: common ancestor is still common (unchanged)
*   ok 14: filter subdirectory only
*   ok 15: subdirectory filter result looks okay
*   ok 16: more setup
*   ok 17: use index-filter to move into a subdirectory
*   ok 18: stops when msg filter fails
*   ok 19: author information is preserved
*   ok 20: remove a certain author's commits
*   ok 21: barf on invalid name
*   ok 22: "map" works in commit filter
*   ok 23: Name needing quotes
*   ok 24: Subdirectory filter with disappearing trees
*   ok 25: Tag name filtering retains tag message
*   ok 26: Tag name filtering strips gpg signature
*   ok 27: Tag name filtering allows slashes in tag names
*   ok 28: Prune empty commits
*   ok 29: --remap-to-ancestor with filename filters
*   ok 30: setup submodule
*   ok 31: rewrite submodule with another content
*   ok 32: replace submodule revision

Tomas Carnecky wrote:

[...]

[...]

Oh, I know. The test isn't correct, it should be something like the patch below. Otherwise a failure in git-diff (namely that directorymoved:newsubdir is invalid) is never detected, and since the output in this case is empty, also the test -z succeeds. Can you apply that and see if it makes the test fail for you?

On 7/29/10 4:52 PM, Thomas Rast wrote:

Now the test fails. And with the other patch applied on top of it, the test passes again.

tom

The test would not fail if the filtering failed to do anything, since in

  test -z "$(git diff HEAD directorymoved:newsubdir)"'

'directorymoved:newsubdir' is not valid, so git-diff fails without printing anything on stdout. But then the exit status of git-diff is lost, whereas test -z "" succeeds.

Use 'git diff --exit-code' instead, which does the right thing and has the added bonus of showing the differences if there are any.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

Tomas Carnecky wrote:

Ok. Here's a slightly nicer solution, let's add this on top of the earlier patch.

 t/t7003-filter-branch.sh |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

Thomas Rast wrote:

Bah, this subject is wrong on two counts.  It should be something like

  t7003: fix move-to-subdirectory test

instead.  Sorry for the noise.

On Thu, Jul 29, 2010 at 07:37, Adam Mercer <ramercer@gmail.com> wrote:

I'm trying to do this with another repo now and am getting a different error message:

$ git filter-branch --index-filter \
    'git ls-files -s | gsed "s-\t\"*-&sftlib/-" |
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
            git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
Rewrite b3100fb24a7eb2f349801f3b87cf995dc3d14bba (1/44)mv:
/Users/ram/git/eah2/.git-rewrite/t/../index.new: No such file or
directory
index filter failed: git ls-files -s | gsed "s-\t\"*-&sftlib/-" |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv
$GIT_INDEX_FILE.new $GIT_INDEX_FILE
$

Any ideas?

This is on Mac OS X 10.6.4, with git-1.7.2.1, and GNU sed 4.2.1

Cheers

Adam

On Mon, Aug 9, 2010 at 14:36, Adam Mercer <ramercer@gmail.com> wrote:

Anyone?

Cheers

Adam

On Wed, Aug 11, 2010 at 10:01:15AM -0500, Adam Mercer wrote:

It looks like git-update-index will not create $GIT_INDEX_FILE at all if you have no actual input lines to --index-info. So perhaps you have some commit in your repo that has no actual content in it. Either that or for some reason your "ls-files -s | gsed" invocation is producing no output.

Try adding a "touch $GIT_INDEX_FILE.new" at the beginning of your index filter and see if that helps. Then the file will always exist to be moved, even if we write nothing to it. If it works, then problem solved. If you get a repo full of empty commits, then something is wrong with your gsed pipeline.

-Peff

On Wed, Aug 11, 2010 at 14:32, Jeff King <peff@peff.net> wrote:

Yep, that was the problem. I'd been modifying the history with several git filter-branch calls and the initial commit in the repo was empty. Getting rid of this initial empty commit allowed the command to run.

Cheers

Adam